2015-03-20

How to use rsync over USB on Android with adb

This blog post explains how to copy files using rsync between the computer running Unix (typically Linux or Mac OS X) and the mobile device running Android, using an USB cable. It's not necessary to root the device. It's not necessary to install any app.

If you want to copy over wifi rather than USB, then please use the app rsync backup for Android (rsync4android) instead. The rest of this tutorial describes a method which needs the computer and the mobile device connected with a USB data cable.

Enable USB debugging on the device. If you don't know how to do it in Settings, then find a tutorial online.

Install adb (Android Debug Bridge) to the Unix system. For example, on Ubuntu: sudo apt-get install android-tools-adb

Connect the mobile device to the computer. Run: adb shell id on the computer. (All commands should be run on the computer unless asked otherwise.) It should display something like:

$ adb shell id
uid=2000(shell) gid=2000(shell) groups=1004(input),... context=u:r:shell:s0

If it doesn't work, you may have to enable Settings / Developer options / USB debugging on the device, then reconnect, then click OK on the dialog box in the device, then rerun adb shell id on the computer.

Please note that on Cyanogenmod rsync is installed by default and it is on the $PATH, so you can skip some of the steps below. (If you don't know what to skip, just do everything anyway.)

Download the rsync binary: wget -O rsync.bin http://github.com/pts/rsyncbin/raw/master/rsync.rsync4android

Copy the rsync binary to the device: adb push rsync.bin /data/local/tmp/rsync

Make the rsync binary on the device executable: adb shell chmod 755 /data/local/tmp/rsync

Make sure you have a backup copy of the binary in a more permanent directory: adb shell cp /data/local/tmp/rsync /sdcard/rsync.bin

Get the rsync version by running adb shell /data/local/tmp/rsync --version . Typical output:

$ adb shell /data/local/tmp/rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    no socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, no iconv, no symtimes, no prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Run rsync --version . If you get something like

rsync: command not found

, then rsync isn't installed to the computer. On Ubuntu you can install it with sudo apt-get install rsync

Create an rsyncd.conf config file and install it to the device by running: adb shell 'exec >/sdcard/rsyncd.conf && echo address = 127.0.0.1 && echo port = 1873 && echo "[root]" && echo path = / && echo use chroot = false && echo read only = false' . This must finish without displaying any (error) message.

Start the rsync daemon in the device by running: adb shell /data/local/tmp/rsync --daemon --no-detach --config=/sdcard/rsyncd.conf --log-file=/proc/self/fd/2 . It must start up with just a single message:

2015/03/19 15:58:14 [5814] rsyncd version 3.1.1 starting, listening on port 1873

Keep it running for the duration of the copies (below), and continue working in another terminal window. Or press Ctrl-C to exit right now, and restart it (so it will start running in the background on the device) like this: adb shell '/data/local/tmp/rsync --daemon --config=/sdcard/rsyncd.conf --log-file=/data/local/tmp/foo &'

Start port forwarding by running: adb forward tcp:6010 tcp:1873

Now you can start copying files with rsync (back and forth). An example command: rsync -av --progress --stats rsync://localhost:6010/root/sdcard/Ringtones .

You may find the --size-only flag useful if rsync is copying the same files over and over again.

You may want to copy from or to /storage/sdcard1 instead of /sdcard on the device.

Some newer storage devices have the exfat filesystem (older ones typically have fat or some emulation of it, and that's just fine). Writing to exfat drives rsync -av crazy: it reports steady progress with lots of Operation not permitted errors, but it actually doesn't create any files. This applies both rsync running on the Linux computer and rsync running on the device. A solution is to replace rsync -av with rsync -vrtlD , and restart the copy.

12 comments:

  1. I am trying to do this as root but I am getting this error:

    # rsync -va --progress rsync://localhost:6010/root/data/media ./
    @ERROR: invalid gid nogroup
    rsync error: error starting client-server protocol (code 5) at main.c(1653) [Receiver=3.1.0]

    ReplyDelete
  2. Awesome. Thanks so much for posting this. This worked a charm (with just minimal tweaking for my own situation). Now I can sync my Android music library to my iTunes library on my work computer! :)

    ReplyDelete
  3. There's a much easier way to do this if you use a binary-clean alternative to `adb shell`, such as fb-adb, if you use this trick involving a fake RSYNC_RSH and setting the remote hostname to "exec":

    rsync -av --progress --stats -e "fb-adb shell" exec:/sdcard/Ringtones .

    You can also set the remote hostname to an empty string:

    rsync -av --progress --stats -e "fb-adb shell" :/sdcard/Ringtones .

    ReplyDelete
  4. Useful tip: in Cyanogenmod and some AOSP ROMs, rsync can be found in /system/xbin.

    ReplyDelete
  5. Thank you so much for sharing.

    I backed-up all my photos/ebooks/etc from and old Samsung S4 to an external sdcard and now there are transferred onto my mac.
    When I try to run rsync the files to my new Nexus 6P (rooted, USB debugging on) it says:

    $ rsync -avP /Users/Simorgh/Desktop/card/Books/ rsync://localhost:6010/root/sdcard/Books/
    building file list ...
    6 files to consider
    created directory sdcard/Books
    rsync: failed to set times on "/sdcard/Books/." (in root): Operation not permitted (1)
    ./
    How to Talk to Anyone - 92 Little Tricks for Big Success in Relationships.PDF
    16189904 100% 2.80MB/s 0:00:05 (xfer#1, to-check=4/6)
    How to Win Every Argument.pdf
    2471871 100% 2.60MB/s 0:00:00 (xfer#2, to-check=3/6)
    aboosaeed-robayee[www.mybook.ir].pdf
    487536 100% 461.35kB/s 0:00:01 (xfer#3, to-check=2/6)
    assholism-smallscreen-2ndedition.pdf
    2237551 100% 2.88MB/s 0:00:00 (xfer#4, to-check=1/6)
    complete.book.of.intelligence.tests.pdf
    1325061 100% 1.09MB/s 0:00:01 (xfer#5, to-check=0/6)
    rsync: failed to set times on "/sdcard/Books/.How to Talk to Anyone - 92 Little Tricks for Big Success in Relationships.PDF.Me6634" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.How to Win Every Argument.pdf.vQ6634" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.aboosaeed-robayee[www.mybook.ir].pdf.MO6634" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.assholism-smallscreen-2ndedition.pdf.qv6634" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.complete.book.of.intelligence.tests.pdf.Jc6634" (in root): Operation not permitted (1)

    sent 22715258 bytes received 132 bytes 2672398.82 bytes/sec
    total size is 22711923 speedup is 1.00
    rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-45/rsync/main.c(992) [sender=2.6.9]

    I even tried the -vrtlD option, but no success.

    $ rsync -vrtlD /Users/Simorgh/Desktop/card/Books/ rsync://localhost:6010/root/sdcard/Books/
    building file list ... done
    rsync: failed to set times on "/sdcard/Books/." (in root): Operation not permitted (1)
    ./
    How to Talk to Anyone - 92 Little Tricks for Big Success in Relationships.PDF
    How to Win Every Argument.pdf
    aboosaeed-robayee[www.mybook.ir].pdf
    assholism-smallscreen-2ndedition.pdf
    complete.book.of.intelligence.tests.pdf
    rsync: failed to set times on "/sdcard/Books/.How to Talk to Anyone - 92 Little Tricks for Big Success in Relationships.PDF.Rr7314" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.How to Win Every Argument.pdf.cW7314" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.aboosaeed-robayee[www.mybook.ir].pdf.JU7314" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.assholism-smallscreen-2ndedition.pdf.va7314" (in root): Operation not permitted (1)
    rsync: failed to set times on "/sdcard/Books/.complete.book.of.intelligence.tests.pdf.Rq7314" (in root): Operation not permitted (1)

    sent 36389 bytes received 53946 bytes 36134.00 bytes/sec
    total size is 22711923 speedup is 251.42
    rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-45/rsync/main.c(992) [sender=2.6.9]

    The reason I would like to use rsync to preserve the timestamp on my family photos that I have taken for better sorting options and knowing when the photos were taken in the future. Even though the error messages, the files get copied, but the timestamp is the time of the rsync.
    Can anybody please help me to find out what I am doing wrong?


    Thank you in advance.

    ReplyDelete
  6. I had the same issue as @Pablo De Biase - "@ERROR: invalid gid nogroup"

    I was able to fix this by adding "&& echo uid = 0 && echo gid = 0" to the end of the of command that creates the rsyncd.conf

    ReplyDelete
  7. Wonderful, I've long been looking for a way to synchronize music and picts to my smartphone, and it's the first time I find something that works fine!!

    Thanks a lot!

    I just have one question: I cannot figure out why, after a sync, if I disconnect my phone and then later on reconnect it, sync does not work any more unless I reboot my computer (linux mint): there is probably some "ghost" process still running that can't talk with the new adapter to the phone, but I didn't find it yet...

    Any idea? Thanks in advance!

    ReplyDelete
  8. @Philippe Laroque: You may want to run `adb kill-server' instead of rebooting.

    ReplyDelete
  9. @pts: great, it works 100% fine now! ;)
    thank you so much!

    ReplyDelete
  10. Tried on Sony Z5, Android OS 7.1.1 and didn't work. Then I run into https://forum.xda-developers.com/android/software-hacking/guide-rsync-backup-usb-adb-lineage-14-x-t3620054 and adding double -t when starting daemon automagically worked. So instead of:

    adb shell /data/local/tmp/rsync --daemon --no-detach --config=/sdcard/rsyncd.conf --log-file=/proc/self/fd/2

    this worked for me:

    adb shell -t -t /data/local/tmp/rsync --daemon --no-detach --config=/sdcard/rsyncd.conf --log-file=/proc/self/fd/2

    Great post! Thank you!

    ReplyDelete
  11. I just upgraded my linux box. The rsync version now differs from that of the smartphone (3.1.2 computer, 3.1.1 phone), and the synchronisation fails:

    opening tcp connection to localhost port 6010
    rsync: safe_read failed to read 1 bytes [sender]: Connection reset by peer (104)
    rsync error: error in rsync protocol data stream (code 12) at io.c(285) [sender=3.1.2]

    Is there a workaround? I could not find a version 3.1.2 for my phone yet... :(

    ReplyDelete
  12. Hello,
    I recently upgraded my linux box and the synchronization between the box and the phone now fails. I guess it is due to the new version of rsync on the box, which is differetn from that of the phone:

    # rsync --version
    rsync version 3.1.2 protocol version 31
    Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    # adb shell /data/local/tmp/rsync --version
    rsync version 3.1.1 protocol version 31
    Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/

    I could not find a newer version for my phone...
    Any help appreciated! thanks in advance ;)

    ReplyDelete