google / adb-sync Goto Github PK

View Code? Open in Web Editor NEW

1.1K 46.0 165.0 35 KB

License: Apache License 2.0

Python 98.52% Shell 1.48%

adb-sync's Introduction

adb-sync

adb-sync is a tool to synchronize files between a PC and an Android device using the ADB (Android Debug Bridge).

DEPRECATED

This project is no longer being maintained; please use better-adb-sync instead. Thanks!

Related Projects

Before getting used to this, please review this list of projects that are somehow related to adb-sync and may fulfill your needs better:

better-adb-sync is an improved rewrite of this project.
rsync is a file synchronization tool for local (including FUSE) file systems or SSH connections. This can be used even with Android devices if rooted or using an app like SSHelper.
adbfs is a FUSE file system that uses adb to communicate to the device. Requires a rooted device, though.
adbfs-rootless is a fork of adbfs that requires no root on the device. Does not play very well with rsync.
go-mtpfs is a FUSE file system to connect to Android devices via MTP. Due to MTP's restrictions, only a certain set of file extensions is supported. To store unsupported files, just add .txt! Requires no USB debugging mode.

Setup

Android Side

First you need to enable USB debugging mode. This allows authorized computers (on Android before 4.4.3 all computers) to perform possibly dangerous operations on your device. If you do not accept this risk, do not proceed and try using go-mtpfs instead!

On your Android device:

Go to the Settings app.
If there is no "Developer Options" menu:
- Select "About".
- Tap "Build Number" seven times.
- Go back.
Go to "Developer Options".
Enable "USB Debugging".

PC Side

Install the Android SDK (the stand-alone Android SDK "for an existing IDE" is sufficient). Alternatively, some Linux distributions come with a package named like "android-tools-adb" that contains the required tool.
Make sure "adb" is in your PATH. If you use a package from your Linux distribution, this should already be the case; if you used the SDK, you probably will have to add an entry to PATH in your ~/.profile file, log out and log back in.
git clone https://github.com/google/adb-sync
cd adb-sync
Copy or symlink the adb-sync script somewhere in your PATH. For example: cp adb-sync /usr/local/bin/

Usage

To get a full help, type:

adb-sync --help

To synchronize your music files from ~/Music to your device, type one of:

adb-sync ~/Music /sdcard
adb-sync ~/Music/ /sdcard/Music

To synchronize your music files from ~/Music to your device, deleting files you removed from your PC, type one of:

adb-sync --delete ~/Music /sdcard
adb-sync --delete ~/Music/ /sdcard/Music

To copy all downloads from your device to your PC, type:

adb-sync --reverse /sdcard/Download/ ~/Downloads

ADB Channel

This package also contains a separate tool called adb-channel, which is a convenience wrapper to connect a networking socket on the Android device to file descriptors on the PC side. It can even launch and shut down the given application automatically!

It is best used as a ProxyCommand for SSH (install SSHelper first) using a configuration like:

Host sshelper
Port 2222
ProxyCommand adb-channel tcp:%p com.arachnoid.sshelper/.SSHelperActivity 1

After adding this to ~/.ssh/config, run ssh-copy-id sshelper.

Congratulations! You can now use rsync, sshfs etc. to the host name sshelper.

Contributing

Patches to this project are very welcome.

Before sending a patch or pull request, we ask you to fill out one of the Contributor License Agreements:

Disclaimer

This is not an official Google product.

adb-sync's People

Contributors

Stargazers

Watchers

Forkers

imclab guibog abdullah353 justincc anatol shanyou gubaojian hugofeng sungamagnus jalaziz getwingm ardumont alexconst cyvins baguspermanaarif qezt daoqiao3 thiblahute tonycuadra kleopatra999 iveskins markus-perl runt18 rjmcguire pantasio baohaojun liuyong240 pankajchunchun wyxy2005 dirkakrid stesen ruiwentang cng sivansong pramukesh jpathak rockyspade chatziko nicetester shirayu jontwo mxbees runer112 cryptogopher philr73 muruganandhan drakaz pengcy omunroe-com gbesjes emtee40 tuomassalo dstmath ifa6 ekystra-a1 jithumon dil3mm4 out0fmemory pashupati123 klozz tlvince lovinstar cherusk r0zbot jottr songtm exhorder yoesak theneutrinoraged primekun rtk4616 mydongistiny m13253 la-sora-cao msfjarvis mikegreat kennethso168 kaihongxia sts0mrg0 tastyblob sebhuet nuangua razerban dr-chaos c25x 5l1v3r1 er0k katyabeck3 linecode trainzje kunkun-tx krackers aviallon zhutony mkmark mounirerhili neotim gust4ng dlzr lokeshponnada

adb-sync's Issues

Scanning and diffing takes a long time for large directories

Hi,

I am trying to copy a large folder (several GiB with many thousands of files) and I'm stuck on INFO:root:Scanning and diffing... for a long time. Is there a way I can skip this and just delete/overwrite files by name?

Thanks!

full /sdcard/ download does not work

./adb-sync --reverse /sdcard/ /BACKUP/android/xperia/sdcard/
Sync: local /BACKUP/android/xperia/sdcard/, remote /sdcard
Scanning and diffing...
Note: unsupported file: /sdcard
Total: 0 KB/s (0 bytes in 0.317s)

I have to use /sdcard/* which seems weird.

adb-sync seems to close standard input?

foo (containing file 1.txt) and bar (containing file 2.txt) are two directories under /sdcard on my phone.

When I run the following script test.sh on my computer:

#!/bin/sh
while read -r dir; do
    ./adb-sync -R /sdcard/"$dir" .
done <<EOF
foo
bar
EOF

The result is that only the first directory foo is synced:

$ ./test.sh
INFO:root:Sync: local b'./foo', remote b'/sdcard/foo'
INFO:root:Scanning and diffing...
INFO:root:Pull: b'./foo'
INFO:root:Pull: b'./foo/1.txt'
[100%] /sdcard/foo/1.txt
INFO:root:Total: 1 KB/s (312 bytes in 0.225s)

The problem is read -r dir fails in the second iteration, perhaps implying stdin in closed.

adb-sync doesn't copy files properly if names are same on local and remote

I think this is due to self.both isn't processed for PerformCopies. I have a file with two lines that have the same length, where I comment out one line, and uncomment out another. Then I toggle this, and the file isn't copied. If I change the length of the lines, then src_only picks up the change.

This is with "adb-sync --delete --force ...". I also applied the "--times" fix in another issue, added --times, and fresh synced the content, but that still doesn't fix this issue on subsequent copies.

This self.both list seems to be created if the path on local matches the path on the remote, the stats aren't compared. But a copy is still needed if the modstamp differs. I only see self.both uses in overwrites and deletions, and not in the copy step. PerformCopies only walks the self_only list.

This means that the content on device is not correct, and doesn't reflect any mods that were made to the source directory. The modstamp, filesize, etc can all be different by adb-sync doesn't seem to check those. Please tell me I'm misreading something here, and that this important utility has been broken for all these years.

BuildFileLists
    this creates List[Tuple[path, stats]]

def DiffLists(...)
    if a_item[0] == b_item[0]:  <- seems to only compare the path portion of the tuple from BuildFileLists, should this be comparing a_item[1] == b_item[1], and same on two calls below
      both.append((a_item[0], a_item[1], b_item[1]))
      a_revlist.pop()
      b_revlist.pop()
    elif a_item[0] < b_item[0]:
      a_only.append(a_item)
      a_revlist.pop()
    elif a_item[0] > b_item[0]:
      b_only.append(b_item)
      b_revlist.pop()
    else:


def PerformCopies(self) -> None:
    """Perform all copying necessary for the file sync operation."""
    for i in [0, 1]:
      if self.src_to_dst[i]:
        for name, s in self.src_only[i]: <- this is only walking the src_only list build from DiffLists, self.both is ignored

The following code could handle it, but skips if the file sizes match. But the content may be different (modstamps differ) even if the file size is the same. I'm assuming file padding is ignored from st_size. My understanding of overwrites is that they were meant to handle folders <-> files.

def PerformOverwrites(self) -> None:

     for name, localstat, remotestat in self.both:  <- okay this walks the both list 
      if stat.S_ISDIR(localstat.st_mode) and stat.S_ISDIR(remotestat.st_mode):
        # A dir is a dir is a dir.
        continue
      elif stat.S_ISDIR(localstat.st_mode) or stat.S_ISDIR(remotestat.st_mode):
        # Dir vs file? Nothing to do here yet.
        pass
      else:
        # File vs file? Compare sizes.
        if localstat.st_size == remotestat.st_size:  <- this earlies out off file size
          continue

error running it. stacktrace.

./adb-sync
Traceback (most recent call last):
File "./adb-sync", line 61, in
class Stdout(object):
File "./adb-sync", line 81, in Stdout
def exit(self, exc_type: Optional[Type[BaseException]],
File "/usr/lib/python3.5/typing.py", line 649, in getitem
return Union[arg, type(None)]
File "/usr/lib/python3.5/typing.py", line 552, in getitem
dict(self.dict), parameters, _root=True)
File "/usr/lib/python3.5/typing.py", line 512, in new
for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
File "/usr/lib/python3.5/typing.py", line 512, in
for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
File "/usr/lib/python3.5/typing.py", line 1077, in subclasscheck
if super().subclasscheck(cls):
File "/usr/lib/python3.5/abc.py", line 225, in subclasscheck
for scls in cls.subclasses():
TypeError: descriptor 'subclasses' of 'type' object needs an argument

str/bytes/encoding problems on Windows

On Windows, subprocess.Popen process arguments as strings, and problem arises when bytes are passed in (a bytes-like object is required, not 'str').
Python 3.6 now uses UTF-8 for Windows filesystem and console encoding. Using locale.getdefaultlocale()[1] is not appropriate in this case.

I modified the script, changing all bytes to str and using UTF-8 only, and got a working script on my system (Win 7, Python 3.6), but it probably doesn't work on many other systems. Would love to see this tool improved in this regard.

Permission denied

Hi,
I've wrote a dummy app that creates files. With adb-sync I would try to keep track of change on the files creates. But the following command:

./adb-sync --dry-run --emulator --reverse /data/user/0/com.example.mex.dummyapp/files ./backup/
I got this permission denied error:
Sync: local ./backup//files, remote /data/user/0/com.example.mex.dummyapp/files
Scanning and diffing...
ls: /data/user/0/com.example.mex.dummyapp/files: Permission denied
No files seen. User error?
Total: 0 bytes

How can I fix this issue?

Thanks,
M

windows 10

Is there somewhere a version for Windows 10?

I use google translator

adb-sync doesn't work correctly with symlinks

I'd like use adb-sync to sync my pc with an external card of an android device (a Likebook).
The sync process stops when it comes at a symlink.
There is a way to say adbs-ync to ignore symlinks?
Or some another way to sync an extrenal card without eject it from device?

Hard-coded location of python3

The scripts assume that the python3 binary is located in /usr/bin/python3. This is true of most GNU/Linux distributions, but not other OSs (notably FreeBSD, which I use).

Problems with encoding: UnicodeDecodeError: 'utf-8' codec can't decode byte: invalid start byte

Hello,
when starting adb-sync I always get the following error message. I'm using Windows 10 1909, Python 3.8.1 and the latest version of the script.

python adb-sync --reverse /sdcard/test/ "C:\Backup\2019-12-29\sdcard"
INFO:root:Sync: local b'C:\\Backup\\2019-12-29\\sdcard', remote b'/sdcard/test/'
Traceback (most recent call last):
  File "adb-sync", line 883, in <module>
    main()
  File "adb-sync", line 870, in main
    if not syncer.IsWorking():
  File "adb-sync", line 507, in IsWorking
    return self.adb.IsWorking()
  File "adb-sync", line 219, in IsWorking
    with Stdout(self.adb +
  File "adb-sync", line 76, in __init__
    self.popen = subprocess.Popen(args, stdout=subprocess.PIPE)
  File "C:\Python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python38\lib\subprocess.py", line 1247, in _execute_child
    args = list2cmdline(args)
  File "C:\Python38\lib\subprocess.py", line 549, in list2cmdline
    for arg in map(os.fsdecode, seq):
  File "C:\Python38\lib\os.py", line 818, in fsdecode
    return filename.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 41: invalid start byte

As a workarround I tried different encodings for the script but without success:

UTF-8/UTF-8 with BOM/Windows 1252/ISO 8859-1: Same error message as above.
UTF-16 LE/BE:

SyntaxError: Non-UTF-8 code starting with '\xff' in file C:\Users\Kristian\adb-sync\adb-sync on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

Are there any other solutions ?
Thanks in advance!

Adb-sync fails whenever directory contains a very large file.

Whenever I attempt to sync a very large file to my android, the program will return an error. However, if I just adb push it, then the transfer will go through flawlessly. This problem does not arise with smaller files which are less than 1.5G.

OS: Fedora 26/ Linux.
Phone: Galaxy S4

Steps to replicate:

Have a very large file, in this case `~/Documents/ToPhone/Day2.mp4', 6.4G in size.
Attempt to sync the directory containing this file ./adb-sync --delete ~/Documents/ToPhone/ /mnt/extSdCard/fromComputer.

The file will first appear to be transferring correctly, with the progress indicator getting close to 100%.. Eventually, the transfer will result in an error message:

Sync: local /home/mikey/Documents/ToPhone, remote /mnt/extSdCard/fromComputer
Scanning and diffing...
Push: /mnt/extSdCard/fromComputer/Day2.mp4
adb: error: failed to copy '/home/mikey/Documents/ToPhone/Day2.mp4' to '/mnt/extSdCard/fromComputer/Day2.mp4': no response: Success
Interrupted-Push-Delete: /mnt/extSdCard/fromComputer/Day2.mp4
error: no devices/emulators found
Total: 0 KB/s (0 bytes in 1157.919s)
Traceback (most recent call last):
  File "/home/mikey/GoogleDrive/ScriptsThatIMightUse/requirements/adb-sync", line 774, in <module>
    main(*sys.argv)
  File "/home/mikey/GoogleDrive/ScriptsThatIMightUse/requirements/adb-sync", line 769, in main
    syncer.PerformCopies()
  File "/home/mikey/GoogleDrive/ScriptsThatIMightUse/requirements/adb-sync", line 585, in PerformCopies
    self.num_bytes += s.st_size
  File "/home/mikey/GoogleDrive/ScriptsThatIMightUse/requirements/adb-sync", line 474, in __exit__
    fs.unlink(name)
  File "/home/mikey/GoogleDrive/ScriptsThatIMightUse/requirements/adb-sync", line 254, in unlink
    raise OSError('unlink failed')
OSError: unlink failed`

Attempt to transfer the file by just using adb adb push Day1.mp4 /mnt/extSdCard/fromComputer. This operation will complete successfully.

Ignore unreadable files

When I execute the script, I get following output

INFO:root:Scanning and diffing...
ls: /storage/emulated/0/Android/data/ch.threema.app/files/tmp//.nomedia: Operation not permitted
INFO:root:Total: 0 bytes

Would it be possible to skip read errors? Maybe "only" the "Operation not permitted" errors?

Details:

Traceback (most recent call last):
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 883, in <module>
    main()
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 874, in main
    syncer.ScanAndDiff()
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 515, in ScanAndDiff
    self.local_only, self.both, self.remote_only = DiffLists(
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 403, in DiffLists
    b_revlist = sorted(b)
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 370, in BuildFileList
    for t in BuildFileList(fs, path + b'/' + n, follow_links,
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 370, in BuildFileList
    for t in BuildFileList(fs, path + b'/' + n, follow_links,
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 370, in BuildFileList
    for t in BuildFileList(fs, path + b'/' + n, follow_links,
  [Previous line repeated 2 more times]
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 367, in BuildFileList
    for n in files:
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 247, in listdir
    yield filename
  File "/mnt/c/git-repositories/adb-sync/adb-sync", line 86, in __exit__
    raise OSError('Subprocess exited with nonzero status.')
OSError: Subprocess exited with nonzero status.

ProxyCommand adb-connect failing (on Mac)

I followed the recommended setup for adb-connect using ProxyCommand, but it is not working. I am not sure which debug information is most relevant as I am unfamiliar with most of the commands run in the script. I suspect I might need the correct login name or put a key in the correct place, since the adb-connect script seems to work on its own, maybe.

It seemed adb-connect was failing in the last line, socat stdio unix:"${t}/sock", based on echos I placed on each line ("Running:" in output below) working except the final "Done".

ssh -v sshhelper
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/cgorichanaz/.ssh/config
debug1: /Users/cgorichanaz/.ssh/config line 1: Applying options for *
debug1: /Users/cgorichanaz/.ssh/config line 8: Applying options for sshhelper
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Executing proxy command: exec adb-channel tcp:2222 com.arachnoid.sshelper/.SSHelperActivity 1
debug1: permanently_drop_suid: 501
debug1: identity file /Users/cgorichanaz/.ssh/cng1408 type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/cgorichanaz/.ssh/cng1408-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.numi4L
debug1: ssh_exchange_identification: Running: adb shell am start -W com.arachnoid.sshelper/.SSHelperActivity
debug1: ssh_exchange_identification: Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.arachnoid.sshelper/.SSHelperActivity }
debug1: ssh_exchange_identification: Status: ok
debug1: ssh_exchange_identification: Activity: com.arachnoid.sshelper/.SSHelperActivity
debug1: ssh_exchange_identification: ThisTime: 1031
debug1: ssh_exchange_identification: TotalTime: 1031
debug1: ssh_exchange_identification: WaitTime: 1060
debug1: ssh_exchange_identification: Complete
debug1: ssh_exchange_identification: Running: adb forward localfilesystem:"/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.numi4L/sock" "tcp:2222"
debug1: ssh_exchange_identification: Running: socat stdio unix:"/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.numi4L/sock"
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.1
debug1: match: OpenSSH_7.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to sshhelper:2222 as 'cgorichanaz'
debug1: SSH2_MSG_KEXINIT sent
Bad packet length 1349676916.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: message authentication code incorrect

When I manually entered the commands from adb-connect, it seemed to open a connection based on SSHelper log showing sshd: Connection from 127.0.0.1 port 45008 on 127.0.0.1 port 2222.

➜  ~ t=`mktemp -d -t adb-channel.XXXXXX | tee /dev/tty`
/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.VvAB3N
➜  ~ adb shell am start -W com.arachnoid.sshelper/.SSHelperActivity
Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.arachnoid.sshelper/.SSHelperActivity }
Status: ok
Activity: com.arachnoid.sshelper/.SSHelperActivity
ThisTime: 1083
TotalTime: 1083
WaitTime: 1123
Complete
➜  ~ adb forward localfilesystem:"${t}/sock" "tcp:2222"
➜  ~ socat -d -d -d -d stdio unix:"${t}/sock"
2017/04/09 23:18:30 socat[42517] D getpid()
2017/04/09 23:18:30 socat[42517] D getpid() -> 42517
2017/04/09 23:18:30 socat[42517] D setenv("SOCAT_PID", "42517", 1)
2017/04/09 23:18:30 socat[42517] D setenv() -> 0
2017/04/09 23:18:30 socat[42517] D setenv("SOCAT_PPID", "42517", 1)
2017/04/09 23:18:30 socat[42517] D setenv() -> 0
2017/04/09 23:18:30 socat[42517] I socat by Gerhard Rieger and contributors - see www.dest-unreach.org
2017/04/09 23:18:30 socat[42517] I This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)
2017/04/09 23:18:30 socat[42517] I This product includes software written by Tim Hudson ([email protected])
2017/04/09 23:18:30 socat[42517] D socat version 1.7.3.2 on Jan 28 2017 18:25:05
2017/04/09 23:18:30 socat[42517] D setenv("SOCAT_VERSION", "1.7.3.2", 1)
2017/04/09 23:18:30 socat[42517] D setenv() -> 0
2017/04/09 23:18:30 socat[42517] D running on Darwin version Darwin Kernel Version 16.5.0: Fri Mar  3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64, release 16.5.0, machine x86_64
2017/04/09 23:18:30 socat[42517] D argv[0]: "socat"
2017/04/09 23:18:30 socat[42517] D argv[1]: "-d"
2017/04/09 23:18:30 socat[42517] D argv[2]: "-d"
2017/04/09 23:18:30 socat[42517] D argv[3]: "-d"
2017/04/09 23:18:30 socat[42517] D argv[4]: "-d"
2017/04/09 23:18:30 socat[42517] D argv[5]: "stdio"
2017/04/09 23:18:30 socat[42517] D argv[6]: "unix:/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.VvAB3N/sock"
2017/04/09 23:18:30 socat[42517] D sigaction(1, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(2, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(3, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(4, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(6, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(10, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(8, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(11, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D sigaction(15, 0x7fff56113ea8, 0x0)
2017/04/09 23:18:30 socat[42517] D sigaction() -> 0
2017/04/09 23:18:30 socat[42517] D signal(13, 0x1)
2017/04/09 23:18:30 socat[42517] D signal() -> 0x0
2017/04/09 23:18:30 socat[42517] D atexit(0x109aee3da)
2017/04/09 23:18:30 socat[42517] D atexit() -> 0
2017/04/09 23:18:30 socat[42517] D calloc(1, 808)
2017/04/09 23:18:30 socat[42517] D calloc() -> 0x7fad63500400
2017/04/09 23:18:30 socat[42517] D malloc(1024)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad64001000
2017/04/09 23:18:30 socat[42517] D calloc(1, 808)
2017/04/09 23:18:30 socat[42517] D calloc() -> 0x7fad635009a0
2017/04/09 23:18:30 socat[42517] D calloc(1, 808)
2017/04/09 23:18:30 socat[42517] D calloc() -> 0x7fad63500cd0
2017/04/09 23:18:30 socat[42517] D isatty(0)
2017/04/09 23:18:30 socat[42517] D isatty() -> 1
2017/04/09 23:18:30 socat[42517] D tcgetattr(0, 0x7fad63500af8)
2017/04/09 23:18:30 socat[42517] D tcgetattr(, {00006b02,00000003,00004b00,200005cf,04,ff,ff,7f,17,15,12,00,03,1c,1a,19,11,13,16,0f,01,00,14,00}) -> 0
2017/04/09 23:18:30 socat[42517] D isatty(1)
2017/04/09 23:18:30 socat[42517] D isatty() -> 1
2017/04/09 23:18:30 socat[42517] D tcgetattr(1, 0x7fad63500e28)
2017/04/09 23:18:30 socat[42517] D tcgetattr(, {00006b02,00000003,00004b00,200005cf,04,ff,ff,7f,17,15,12,00,03,1c,1a,19,11,13,16,0f,01,00,14,00}) -> 0
2017/04/09 23:18:30 socat[42517] D malloc(128)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad63500730
2017/04/09 23:18:30 socat[42517] D malloc(128)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad635007b0
2017/04/09 23:18:30 socat[42517] N reading from and writing to stdio
2017/04/09 23:18:30 socat[42517] D calloc(1, 808)
2017/04/09 23:18:30 socat[42517] D calloc() -> 0x7fad63501000
2017/04/09 23:18:30 socat[42517] D malloc(1024)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad64001400
2017/04/09 23:18:30 socat[42517] D malloc(128)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad63500890
2017/04/09 23:18:30 socat[42517] D malloc(128)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad63500910
2017/04/09 23:18:30 socat[42517] N opening connection to LEN=74 AF=1 "/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.VvAB3N/sock"
2017/04/09 23:18:30 socat[42517] D socket(1, 1, 0)
2017/04/09 23:18:30 socat[42517] I socket(1, 1, 0) -> 5
2017/04/09 23:18:30 socat[42517] D fcntl(5, 2, 1)
2017/04/09 23:18:30 socat[42517] D fcntl() -> 0
2017/04/09 23:18:30 socat[42517] D connect(5, {1,LEN=74 AF=1 "/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.VvAB3N/sock"}, 74)
2017/04/09 23:18:30 socat[42517] D connect() -> 0
2017/04/09 23:18:30 socat[42517] D getsockname(5, 0x7fff561138f0, 0x7fff5611387c{108})
2017/04/09 23:18:30 socat[42517] D getsockname(, {LEN=16 AF=1 ""}, {16}) -> 0
2017/04/09 23:18:30 socat[42517] N successfully connected from local address LEN=16 AF=1 ""
2017/04/09 23:18:30 socat[42517] I resolved and opened all sock addresses
2017/04/09 23:18:30 socat[42517] D malloc(16385)
2017/04/09 23:18:30 socat[42517] D malloc() -> 0x7fad64001800
2017/04/09 23:18:30 socat[42517] N starting data transfer loop with FDs [0,1] and [5,5]
2017/04/09 23:18:30 socat[42517] D data loop: sock1->eof=0, sock2->eof=0, closing=0, wasaction=1, total_to={0.000000}
2017/04/09 23:18:30 socat[42517] D select(6, &0x21, &0x22, &0x0, NULL/0.000000)
2017/04/09 23:18:30 socat[42517] D select -> (, 0x0, 0x22, 0x0, NULL/0.000000), 2
2017/04/09 23:18:30 socat[42517] D data loop: sock1->eof=0, sock2->eof=0, closing=0, wasaction=1, total_to={0.000000}
2017/04/09 23:18:30 socat[42517] D select(6, &0x21, &0x0, &0x0, NULL/0.000000)
2017/04/09 23:18:30 socat[42517] D select -> (, 0x20, 0x0, 0x0, NULL/0.000000), 1
2017/04/09 23:18:30 socat[42517] D read(5, 0x7fad64001800, 8192)
2017/04/09 23:18:30 socat[42517] D read -> 21
2017/04/09 23:18:30 socat[42517] D write(1, 0x7fad64001800, 21)
SSH-2.0-OpenSSH_7.1
2017/04/09 23:18:30 socat[42517] D write -> 21
2017/04/09 23:18:30 socat[42517] I transferred 21 bytes from 5 to 1
2017/04/09 23:18:30 socat[42517] D data loop: sock1->eof=0, sock2->eof=0, closing=0, wasaction=1, total_to={0.000000}
2017/04/09 23:18:30 socat[42517] D select(6, &0x21, &0x2, &0x0, NULL/0.000000)
2017/04/09 23:18:30 socat[42517] D select -> (, 0x0, 0x2, 0x0, NULL/0.000000), 1
2017/04/09 23:18:30 socat[42517] D data loop: sock1->eof=0, sock2->eof=0, closing=0, wasaction=1, total_to={0.000000}
2017/04/09 23:18:30 socat[42517] D select(6, &0x21, &0x0, &0x0, NULL/0.000000)
^C2017/04/09 23:21:23 socat[42517] N socat_signal(): handling signal 2
2017/04/09 23:21:23 socat[42517] N exiting on signal 2
2017/04/09 23:21:23 socat[42517] N socat_signal(): finishing signal 2
2017/04/09 23:21:23 socat[42517] N exit(130)
2017/04/09 23:21:23 socat[42517] D starting xioexit()
2017/04/09 23:21:23 socat[42517] D tcsetattr(0, 0, {00006b02,00000003,00004b00,200005cf,04,ff,ff,7f,17,15,12,00,03,1c,1a,19,11,13,16,0f,01,00,14,00})
2017/04/09 23:21:23 socat[42517] D tcsetattr() -> 0
2017/04/09 23:21:23 socat[42517] D tcsetattr(1, 0, {00006b02,00000003,00004b00,200005cf,04,ff,ff,7f,17,15,12,00,03,1c,1a,19,11,13,16,0f,01,00,14,00})
2017/04/09 23:21:23 socat[42517] D tcsetattr() -> 0
2017/04/09 23:21:23 socat[42517] I shutdown(5, 2)
2017/04/09 23:21:23 socat[42517] D shutdown()  -> 0
2017/04/09 23:21:23 socat[42517] D finished xioexit()
➜  ~ adb forward --remove-all
➜  ~ rm -rf "${t}"

Same error with ssh-copy-id:

➜  ~ ssh-copy-id sshhelper
/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.S1ejde
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/var/folders/jk/3fltf9vx0n15p2qhkpjx2zk80000gn/T/adb-channel.urVgpr

/usr/bin/ssh-copy-id: ERROR: Bad packet length 1349676916.
ERROR: ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: message authentication code incorrect

Warning: could not parse

whenever i type adb-sync --reverse /storage/emulated/0/ ./, to do a full sdcard backup, i get this:

Warning: could not parse 'drwxrwx--x 20 0 1015 4096 Sep 21 23:49 /storage/emulated/0'.

"bad date" from "touch" when using -t option

Hi, I'm trying to use adb-sync with the -t option to preserve file datestamps. Running on Ubuntu 20.04, connected to OnePlus 5 via USB. See code output below - seems like it's calling "touch" command which is taking issue with value (or format?) 20190103.103814. Is this an issue with adb-sync, or something on my end?

james@wagtail-fossa:~$ adb-sync -t ~/1p5\ backup\ 2021-06-18/DCIM/CameraTest /sdcard/DCIM/
INFO:root:Sync: local b'/home/james/1p5 backup 2021-06-18/DCIM/CameraTest', remote b'/sdcard/DCIM//CameraTest'
INFO:root:Scanning and diffing...
INFO:root:Push: b'/sdcard/DCIM//CameraTest/IMG_20190103_103812.jpg'
/home/james/1p5 backup 2021-06-18/DCIM/CameraTest/IMG_20190...812.jpg: 1 file pushed. 20.2 MB/s (4110909 bytes in 0.194s)
INFO:root:Push-Times: accessed Sat Jul 17 23:40:52 2021, modified Thu Jan 3 10:38:14 2019
touch: bad date 20190103.103814
INFO:root:Total: 7514 KB/s (4110909 bytes in 0.534s)
Traceback (most recent call last):
File "/home/james/.local/bin/adb-sync", line 883, in
main()
File "/home/james/.local/bin/adb-sync", line 877, in main
syncer.PerformCopies()
File "/home/james/.local/bin/adb-sync", line 646, in PerformCopies
self.dst_fs[i].utime(dst_name, (s.st_atime, s.st_mtime))
File "/home/james/.local/bin/adb-sync", line 312, in utime
raise OSError('touch failed')
OSError: touch failed

When the file length is the same, it will be considered as no difference

When push a file to the device, the file size does not change but the MD5 is different. At this time, adb-sync thinks that there is no difference, and I think an MD5 verification function should be added for this.

Add a flag to preserve time stamps similar to "adb pull -a"

The adb pull command has a -a flag which preserve time stamp. It would be nice if adb-sync had also such a flag.

This is very useful to keep time and date attributes when transferring photos. The gallery app uses these attributes to show the photos in the right order. If time and date attributes are lost during the transfer, the gallery app would show all photos as they were taken on the same day (day of transfer).

TypeError with latest rev

I get this error with rev fb7c549

# adb-sync/adb-sync -s b780df65 -R /storage/emulated/0/ backup/storage/emulated/0/
INFO:root:Sync: local b'backup/storage/emulated/0/', remote b'/storage/emulated/0/'
Traceback (most recent call last):
  File "adb-sync/adb-sync", line 883, in <module>
    main()
  File "adb-sync/adb-sync", line 870, in main
    if not syncer.IsWorking():
  File "adb-sync/adb-sync", line 507, in IsWorking
    return self.adb.IsWorking()
  File "adb-sync/adb-sync", line 221, in IsWorking
    b'date +%s' % (self.QuoteArgument(test_string),)]) as stdout:
TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple'

# python3 --version
Python 3.4.2

Does not work on MAC

I am trying to use this on mac:
I am getting this error:

adb-sync ../COPY_TEST/ mnt/sdcard/
Traceback (most recent call last):
File "/usr/local/bin/adb-sync", line 774, in
main(*sys.argv)
File "/usr/local/bin/adb-sync", line 693, in main
args_encoding = locale.getdefaultlocale()[1]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
return _parse_localename(localename)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8

can anyone help please

"Ordinal not in range"-error when syncing files with umlauts

Using the most recent version from this repos master branch:

$ adb-sync ü /sdcard/
Traceback (most recent call last):
  File "/Users/jonas/.bin/adb-sync", line 774, in <module>
    main(*sys.argv)
  File "/Users/jonas/.bin/adb-sync", line 695, in main
    localpatterns = [x.encode(args_encoding) for x in args.source]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

How to skip if file has already existed?

Copying from external storage

I have seen someone mention that they were able to get full functionality from their sd card through adb-sync but when I try to run the command with /storage/9C33-6BBD/ as the SRC I get "ls: /storage/9C33-6BBD///.android_secure: Permission denied"

What can I do to fix this issue?

Can't resume syncing folder in reverse mode after faliure

I tried to use adb-sync to sync the contents of /sdcard to a local directory. However, I had to terminate the command before it finished.

I expected to resume the operation later, just like rsync and other file syncing tools, but when I tried the following message appeared:

❯ adb-sync -R /sdcard .
Sync: local ./sdcard, remote /sdcard
Scanning and diffing...
Pull-Delete-Conflicting: ./sdcard
Would have to replace to do this. Use --force to allow this.
Total: 0 KB/s (0 bytes in 0.432s)

The error is too vague to understand what's going on. Is something like this not allowed?

Use my fork

Hi everyone. I recently rewrote this script from scratch (check out my fork). It was much easier to create anew instead of salvaging the awful code format and Python2-ish syntax from here, and the diffing function for excluded files required a nontrivial rewrite.

It has --exclude, --exclude-from, --delete-exluded options.

I have tried this extensively on Linux and it's great. Let me know if there's any path joining issues on Windows, though I think I've covered them all. I'm not too bothered about a pull request as long as people know about this. Enjoy!

Feature Request: Exclude option

it would be great if there would be an exclude options:
--Exclude

with which you can exclude certain directories, even better if you could also exclude certain file types (like rsync)

Colon not supported in folder name

It is not possible to create a folder with a colon : in its name on Android :

mkdir: '/sdcard/Music/Brian Tyler/Assassin's Creed 4: Black Flag (Original Game Soundtrack)': Invalid argument
INFO:root:Total: 0 KB/s (0 bytes in 58.417s)
Traceback (most recent call last):
  File "adb-sync", line 883, in <module>
    main()
  File "adb-sync", line 877, in main
    syncer.PerformCopies()
  File "adb-sync", line 634, in PerformCopies
    self.dst_fs[i].makedirs(dst_name)
  File "adb-sync", line 300, in makedirs
    raise OSError('mkdir failed')
OSError: mkdir failed

Maybe replace it with a _ ?

Is this project still active?

Hi, I was wondering if the script is still being developed. I'm asking this because last commit is dated Jan 2019.

Bye.

pull all /data directory

Hi,
by using adb-sync --reverse /data/ noAction/
I want to download all files from /data, but I've got an error:
adb: error: failed to copy '/data/var/run/netns/router' to 'noAction//var/run/netns/router': remote read failed: Invalid argument
Interrupted-Pull-Delete: noAction//var/run/netns/router
Total: 5 KB/s (81791 bytes in 14.811s)
Traceback (most recent call last):
File "../../adb-sync/adb-sync", line 774, in
main(*sys.argv)
File "../../adb-sync/adb-sync", line 769, in main
syncer.PerformCopies()
File "../../adb-sync/adb-sync", line 585, in PerformCopies
self.num_bytes += s.st_size
File "../../adb-sync/adb-sync", line 474, in exit
fs.unlink(name)
OSError: [Errno 2] No such file or directory: 'noAction//var/run/netns/router'

Two questions:

does this error interrupt the sync?
if yes, can I set adb-sync to go on anyway?
Thanks,
M

Problems with parameters: a bytes-like object is required, not 'str'

Hello,
when starting adb-sync I always get the following error message. I'm using Windows 10, Python 3.7.2 and the latest version of the script.

C:\Users\Marco>adb-sync --reverse /sdcard/DCIM/Camera ~/Downloads INFO:root:Sync: local b'~/Downloads/Camera', remote b'/sdcard/DCIM/Camera' Traceback (most recent call last): File "C:\Users\Marco\Desktop\adb-sync-master\adb-sync.py", line 883, in <module> main() File "C:\Users\Marco\Desktop\adb-sync-master\adb-sync.py", line 870, in main if not syncer.IsWorking(): File "C:\Users\Marco\Desktop\adb-sync-master\adb-sync.py", line 507, in IsWorking return self.adb.IsWorking() File "C:\Users\Marco\Desktop\adb-sync-master\adb-sync.py", line 221, in IsWorking b'date +%s' % (self.QuoteArgument(test_string),)]) as stdout: File "C:\Users\Marco\Desktop\adb-sync-master\adb-sync.py", line 76, in __init__ self.popen = subprocess.Popen(args, stdout=subprocess.PIPE) File "C:\Users\Marco\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 775, in __init__ restore_signals, start_new_session) File "C:\Users\Marco\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1119, in _execute_child args = list2cmdline(args) File "C:\Users\Marco\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 530, in list2cmdline needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: a bytes-like object is required, not 'str'

I saw under Issue #10 that chaning the script could solve this problem but I'm not familiar with Python at all. I also tried different Python versions but no success. Are there any other solutions or will this problem be fixed in the future? I love this tool and I would be happy to use it again.

Thanks in advance!

Warning: could not parse

With my new phone (Huawei P10) I can't use adb-sync anymore to sync my music (or any file). I always get this kind of error/warning (if the files are already on the phone):
Warning: could not parse u'-rw-rw---- 1 root sdcard_rw 329 2017-09-07 10:49 /sdcard/Download/testfile.txt'.
If the files are not on the phone I get this error:
Warning: could not parse u'ls: /sdcard/Download/testfile.txt: No such file or directory'.
It then continues to push the files, regardless if they are already on the phone.

The warning isn't always the same; sometimes the 1 infront of root and the number after sdcard_rw is a different number.
The same command worked fine on my previous phone (LG Nexus 5).

adb-sync fails with "encode() argument 1 must be string, not None" on Ubuntu 16.04

adb-sync -R /data/data/com.my.app .

outcome:

Traceback (most recent call last):
  File "/home/ciro/bin/adb-sync", line 774, in <module>
    main(*sys.argv)
  File "/home/ciro/bin/adb-sync", line 695, in main
    localpatterns = [x.encode(args_encoding) for x in args.source]
TypeError: encode() argument 1 must be string, not None

This happens because locale.getdefaultlocale()[1] is None.

Workaround: patch it with:

  args_encoding = locale.getdefaultlocale()[1] or 'utf-8'

but I'm not sure this is correct. But it worked for me.

Same also happens on a minimal test.py:

import locale
print locale.getdefaultlocale()

env is:

 'LANG': 'en_US:en',
 'LANGUAGE': 'en_US:en',
 'LC_ADDRESS': 'en_US.UTF-8',                                                        
 'LC_ALL': 'C',                                           
 'LC_COLLATE': 'C',          
 'LC_CTYPE': 'en_US.UTF-8',           
 'LC_IDENTIFICATION': 'en_US.UTF-8',
 'LC_MEASUREMENT': 'en_US.UTF-8',                         
 'LC_MESSAGES': 'en_US.UTF-8',   
 'LC_MONETARY': 'en_US.UTF-8',
 'LC_NAME': 'en_US.UTF-8',                                      
 'LC_NUMERIC': 'en_US.UTF-8',
 'LC_PAPER': 'en_US.UTF-8',
 'LC_TELEPHONE': 'en_US.UTF-8',
 'LC_TIME': 'en_US.UTF-8',

anyway to --reverse and --delete?

How to exclude a certain folder?

Is there a way to exclude certain folders from being transferred, in either a forward or reverse sync?

For e.g., my main conundrum is that I want to sync my phones /sdcard/ folder onto my backup drive, which would add all new pictures/videos and downloads. However, I want to exclude the /sdcard/Android/ folder which has unnecessary cache files.

Is there a way to do this, or would it be better for me to sync only the most needed folders (such as DCIM, Downloads, etc)?

Thank you

ssh for use with adb-channel uses wrong percent-expand

%d must be %p.

I won't file a pull-request as I find your requirement to fill out forms (https://github.com/google/adb-sync#contributing) rather ridiculous and won't do it. But I like this piece of software, so please make the fix yourself.

two way sync breaks when filename contains spaces

Ran under python 3.7 and with a local folder /tmp/sync-test and a remote folder (on device) /sdcare/sync-test.

It broken for the file with spaces in the name. The filename with full path is:
/tmp/sync-test/GitHub - google_adb-sync.mhtml

Here is the command and output.

$ python3.7 ./adb-sync -2 /tmp/sync-test /sdcard
INFO:root:Sync: local b'/tmp/sync-test', remote b'/sdcard/sync-test'
INFO:root:Scanning and diffing...
INFO:root:Push: b'/sdcard/sync-test/GitHub - google_adb-sync.mhtml'
4380 KB/s (1317055 bytes in 0.293s)
INFO:root:Pull: b'/tmp/sync-test/GitHub\ -\ google_adb-sync.mhtml'
remote object '/sdcard/sync-test/GitHub\ -\ google_adb-sync.mhtml' does not exist
INFO:root:Interrupted-Pull-Delete: b'/tmp/sync-test/GitHub\ -\ google_adb-sync.mhtml'
INFO:root:Total: 1653 KB/s (1317055 bytes in 0.778s)
Traceback (most recent call last):
File "./adb-sync", line 638, in PerformCopies
self.copy[i](src_name, dst_name)
File "./adb-sync", line 336, in Pull
raise OSError('pull failed')
OSError: pull failed

Not Issue. Possible Suggestion

Wasn't sure how else to share a suggestion.

Sometimes I have multiple devices I'd like to work with at a given moment. I have some simple bash scripts to handle some things, and the first part of the bash script runs "adb devices" and allows me to select a device that the rest of the operations in that script will be performed on.

Would it be possible to add this functionality to your adb-sync tool? I'm not up on my Python just yet.

Crashes when trying to copy files with question marks in the name.

I get the following crash and terminal output. For me, I have just renamed the file so that it doesn't include the funky Spanish upside-down question mark nor the closing question mark and now it works fine... Not sure if this is the scritp failing to escape the special character(s) or a limitation of adb...

adb: error: failed to copy '/home/jonathan/Personal/Music/Trumpet_Stuff/Books//Latin_Band/11 - Sway (¿Quién Será?) - Demo.mp3' to '/storage/3666-6130/Music/Trumpet//Latin_Band/11 - Sway (¿Quién Será?) - Demo.mp3': remote couldn't create file: Invalid argument
/home/jonathan/Personal/Music/Trumpet_Stuff/Books//Latin_Band/11 - Sway (¿Quién Será?) - Demo.mp3: 0 files pushed. 28.0 MB/s (65536 bytes in 0.002s)
INFO:root:Interrupted-Push-Delete: b'/storage/3666-6130/Music/Trumpet//Latin_Band/11 - Sway (\xc2\xbfQui\xc3\xa9n Ser\xc3\xa1?) - Demo.mp3'
rm: /storage/3666-6130/Music/Trumpet//Latin_Band/11 - Sway (¿Quién Será?) - Demo.mp3: No such file or directory
INFO:root:Total: 19375 KB/s (444203239 bytes in 22.388s)
Traceback (most recent call last):
  File "./adb-sync", line 638, in PerformCopies
    self.copy[i](src_name, dst_name)
  File "./adb-sync", line 331, in Push
    raise OSError('push failed')
OSError: push failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./adb-sync", line 883, in <module>
    main()
  File "./adb-sync", line 877, in main
    syncer.PerformCopies()
  File "./adb-sync", line 640, in PerformCopies
    self.num_bytes += s.st_size
  File "./adb-sync", line 465, in __exit__
    self.fs.unlink(self.name)
  File "./adb-sync", line 286, in unlink
    raise OSError('unlink failed')
OSError: unlink failed