Any idea what the problem could be?

__________       .__
\______   \ _____|__| ____   ____
|       _//  ___/  |/    \_/ ___\
|    |   \\___ \|  |   |  \  \___
|____|_  /____  >__|___|  /\___  >
\/     \/        \/     \/

Copyright 2019 C. J. Williams (CHURCHILL COLLEGE)
This is free software with ABSOLUTELY NO WARRANTY

Don't have: ".", entering first_sync mode
Ignore: []
Crawling: "." ⠹2020/02/26 21:17:21 Failed to lsjson: error in ListJSON: directory not found
Traceback (most recent call last):
File "/usr/bin/rsinc", line 11, in <module>
load_entry_point('rsinc==2.13.dev0', 'console_scripts', 'rsinc')()
File "/usr/lib/python3.8/site-packages/rsinc/rsinc.py", line 236, in main
rmt = lsl(path_rmt, HASH_NAME)
File "/usr/lib/python3.8/site-packages/rsinc/rclone.py", line 95, in lsl
list_of_dicts = ujson.load(result.stdout)
ValueError: Expected object or value

Here's what's in the config file:

{
    "BASE_L":"\/home\/dustin\/google-drive",
    "BASE_R":"Google:",
    "CASE_INSENSATIVE":false,
    "DEFAULT_DIRS":[

    ],
    "FAST_SAVE":false,
    "HASH_NAME":"MD5",
    "LOG_FOLDER":"\/home\/dustin\/.rsinc\/logs\/",
    "MASTER":"\/home\/dustin\/.rsinc\/master.json",
    "TEMP_FILE":"\/home\/dustin\/.rsinc\/rsinc.tmp"
}

AFAICT it is creating a new directory both locally and on my google drive named . then it fails. 😕

Hello,
wanted to try out your tool. Unfortunately the initial configuration failes. The rclone remote "hidrive" uses the webdav protocol to connect to the Strato HiDrive cloud. Rclone alone workes fine (listing files/directoryies and so on)

However runnig the initial rsinc --config command I get the following error error traceback

__________       .__               
\______   \ _____|__| ____   ____  
 |       _//  ___/  |/    \_/ ___\ 
 |    |   \\___ \|  |   |  \  \___ 
 |____|_  /____  >__|___|  /\___  >
        \/     \/        \/     \/ 

Copyright 2019 C. J. Williams (CHURCHILL COLLEGE)
This is free software with ABSOLUTELY NO WARRANTY

Starting configuration mode
Path to local root folder i.e "~/": /home/holzi/HiDrive/test/
Local root is: /home/holzi/HiDrive/test

Path to remote root folder i.e "onedrive:": hidrive:test
Remote root is: hidrive:test

Finding a matching hash function...
Searching: /home/holzi/HiDrive/test
Searching: hidrive:test
Traceback (most recent call last):
  File "/usr/bin/rsinc", line 11, in <module>
    load_entry_point('rsinc==2.13.dev0', 'console_scripts', 'rsinc')()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.8/site-packages/rsinc/rsinc.py", line 121, in <module>
    config_cli(config_path)
  File "/usr/lib/python3.8/site-packages/rsinc/config.py", line 63, in config_cli
    rmt_hashes = get_hashes(BASE_R)
  File "/usr/lib/python3.8/site-packages/rsinc/config.py", line 36, in get_hashes
    return set(files[0]["Hashes"].keys())
KeyError: 'Hashes'

Any suggestions? Many thanks for your help!

Hello I'm using a raspberry pi I already have installed rclone and its working fine,
but when i try to run rsinc after installing with pip3 install git+https://github.com/ConorWilliams/rsinc
I get the following error

rsinc

Traceback (most recent call last):
File "/usr/local/bin/rsinc", line 11, in
load_entry_point('rsinc==2.13.dev0', 'console_scripts', 'rsinc')()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2859, in load_entry_point
return ep.load()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2450, in load
return self.resolve()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2456, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/home/pi/.local/lib/python3.5/site-packages/rsinc/rsinc.py", line 100
"-v", "--version", action="version", version=f"rsinc version: {version}"
^
SyntaxError: invalid syntax

I also tried with rsinc --configure but i also get the same error

#rsinc --configure
Traceback (most recent call last):
File "/usr/local/bin/rsinc", line 11, in
load_entry_point('rsinc==2.13.dev0', 'console_scripts', 'rsinc')()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2859, in load_entry_point
return ep.load()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2450, in load
return self.resolve()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2456, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/home/pi/.local/lib/python3.5/site-packages/rsinc/rsinc.py", line 100
"-v", "--version", action="version", version=f"rsinc version: {version}"
^
SyntaxError: invalid syntax

Hi,

after installation and a first successful run I got the following error at the 2nd run:

pi@raspberrypi:~ $ rsinc OneDrive

    Copyright 2019 C. J. Williams (CHURCHILL COLLEGE)
    This is free software with ABSOLUTELY NO WARRANTY

Have: "OneDrive", entering sync & merge mode
Crawling: "OneDrive" ⠏Traceback (most recent call last):
  File "/home/pi/.local/bin/rsinc", line 10, in <module>
    sys.exit(main())
  File "/home/pi/.local/lib/python3.7/site-packages/rsinc/__main__.py", line 308, in main
    rmt = lsl(path_rmt, HASH_NAME, regexs)
  File "/home/pi/.local/lib/python3.7/site-packages/rsinc/rsinc.py", line 222, in lsl
    hashsize += d['Hashes'][hash_name]
KeyError: 'SHA-1'

Seems like SHA lib is not available on the pi. Could you please advise what to install or check to fix it?

Thanks,
Bodo

Rsinc requires an 'ID' for each file which is currently a concatenation of a files hash and its size. This must be unique for each file and independant of its name/path to allow file moves/renames to be tracked. Unfortuntly computing this hash for large files (e.g films) seems to take a huge amount of time (up to minutes per file). I welcome any suggestions to fix this problem. Ideally some way to cache the hashes and only recompute them if the file is modified?

Hi again, I was testing rsync stopping the transmission looking for possible errors.
As if the power had gone out or as if the system had stuck needing a force reboot.

I found that some files were not finished transferring.
Then when I run rsinc again, these files that are corrupt as they are new rsinc wants to delete the old ones that are fine and synchronize theese corrupted files.

One way that I thought to solve this problem is to have a record of the files that are being transfered and maybe use a flag to check if they were synchronized correctly.

And every time rsinc is run, check if there are files without that flag and ask if they want to be delete/clean

I just installed it and I get this error. This is the password I used when I set up rclone originally and works when I use it with rclone itself.

This is free software with ABSOLUTELY NO WARRANTY

Starting configuration mode
Path to local root folder i.e "~/": ~/   
Local root is: /root

Path to remote root folder i.e "onedrive:": "el_camino google-drive:"
Remote root is: "el_camino google-drive:"

Finding a matching hash function...
Searching: /root
Enter configuration password:
password:Enter configuration password:
password:
2020/01/12 01:23:02 ERROR : Couldn't decrypt configuration, most likely wrong password.

I've succesfully configured the rclone service with drive and I'm trying to set up the synchronization with rsinc.
I've configured rsinc with rsinc --config with the following parameters in the config.json:

{
"BASE_L":"/mnt/Ubuntu_data/Drive",
"BASE_R":"gdrive:",
"CASE_INSENSATIVE":true,
"DEFAULT_DIRS":[
"/mnt/Ubuntu_data/Drive"
],
"FAST_SAVE":false,
"HASH_NAME":"MD5",
"LOG_FOLDER":"/home/pablo/.rsinc/logs/",
"MASTER":"/home/pablo/.rsinc/master.json",
"TEMP_FILE":"/home/pablo/.rsinc/rsinc.tmp"
}

and trying to star the process with rsinc --default and I obtain the following error:

Don't have: ".", entering first_sync mode
Ignore: []
Crawling: "." ⠼2020/10/21 12:17:53 Failed to lsjson: error in ListJSON: directory not found
Traceback (most recent call last):
File "/usr/local/anaconda3/bin/rsinc", line 8, in
sys.exit(main())
File "/usr/local/anaconda3/lib/python3.8/site-packages/rsinc/rsinc.py", line 247, in main
rmt = lsl(path_rmt, HASH_NAME)
File "/usr/local/anaconda3/lib/python3.8/site-packages/rsinc/rclone.py", line 95, in lsl
list_of_dicts = ujson.load(result.stdout)
ValueError: Expected object or value

It's just too complex too setup. I want to sync ~/Desktop/lightdm to dropbox: and it throws errors after errors. Even though directory exists

rsinc ~/Desktop/lightdm Dropbox:Documents/

Don't have: "Desktop/lightdm", entering first_sync mode
Ignore: []
Error: unknown flag: --files-only

Hello

Thanks for the effort so far. Works like a charm. :)

Would it be possible to add a command line argument to send a rclone flag like '--fast-list' ?

Best regards.

rsinc only seems to work with ujson==1.35
It appears that more recent versions of ujson do not know how to read/write tuple as a json:
history, ignores, nest = read(MASTER)
write(MASTER, [[], [], empty()])
write(MASTER, (history, ignores, nest))

simple solution:
make history and ignores part of the master.json dict e.g. {'history': history, 'ignores': ignores}

Hello, there is an issue with google drive after letting it for a while i found these errors with rclone verbose
When one of the threads reaches 10/10 it resets the pulled file and it starts over
I belive this can be partially or fully corrected by reducing the multiprocess threads.

I see there is no command line arguments for this option in rsinc, so can you please point me where is the code that manages the number of multiprocess to change it and test with it, thanks

2020/08/14 11:06:57 INFO :
Transferred: 166.996M / 347.193 MBytes, 48%, 135.758 kBytes/s, ETA 22m39s
Transferred: 0 / 1, 0%
Elapsed time: 20m59.6s
Transferring:

•                             TT T01-08.mkv: 48% /347.193M, 59.334k/s, 51m49s
  

2020/08/14 11:06:58 INFO :
Transferred: 197.996M / 347.229 MBytes, 57%, 160.962 kBytes/s, ETA 15m49s
Transferred: 0 / 1, 0%
Elapsed time: 20m59.6s
Transferring:

•                             TT T02-01.mkv: 57% /347.229M, 148.126k/s, 17m11s
  

2020/08/14 11:06:58 DEBUG : TT T01-13.mkv: Reopening on read failure after 175771803 bytes: retry 2/10: read tcp 10.120.234.50:54694->172.217.6.42:443: read: connection reset by peer
2020/08/14 11:07:07 DEBUG : TT T01-11.mkv: Reopening on read failure after 198178641 bytes: retry 4/10: read tcp 10.120.234.50:34718->216.58.195.74:443: read: connection reset by peer
2020/08/14 11:07:38 DEBUG : TT T01-11.mkv: Reopening on read failure after 201404911 bytes: retry 5/10: read tcp 10.120.234.50:41028->172.217.6.74:443: read: connection reset by peer
2020/08/14 11:07:38 DEBUG : TT T01-10.mkv: Reopening on read failure after 181194896 bytes: retry 2/10: read tcp 10.120.234.50:55814->172.217.164.106:443: read: connection reset by peer
2020/08/14 11:07:43 DEBUG : TT T01-09.mkv: Reopening on read failure after 180146343 bytes: retry 2/10: read tcp 10.120.234.50:55796->172.217.164.106:443: read: connection reset by peer
2020/08/14 11:07:45 DEBUG : TT T01-13.mkv: Reopening on read failure after 182321597 bytes: retry 3/10: read tcp 10.120.234.50:41026->172.217.6.74:443: read: connection reset by peer
2020/08/14 11:07:53 DEBUG : TT T01-12.mkv: Reopening on read failure after 190032848 bytes: retry 2/10: read tcp 10.120.234.50:40984->172.217.6.74:443: read: connection reset by peer
2020/08/14 11:07:54 DEBUG : TT T01-11.mkv: Reopening on read failure after 206549487 bytes: retry 6/10: read tcp 10.120.234.50:34732->216.58.195.74:443: read: connection reset by peer

As in the title, initial configuration fails on Ubuntu 18.04 LTS.

$>rsinc --v
Copyright 2019 C. J. Williams (CHURCHILL COLLEGE)
This is free software with ABSOLUTELY NO WARRANTY
rsinc version: 2.12-DEV

$> rsinc

Copyright 2019 C. J. Williams (CHURCHILL COLLEGE)
This is free software with ABSOLUTELY NO WARRANTY

Starting configuration mode
Path to local root folder i.e "~/": ~/Dokumenty
Local root is: /home/afiergol/Dokumenty

Path to remote root folder i.e "onedrive:": Dokumenty:
Remote root is: Dokumenty:

Finding a matching hash function...
Searching: /home/afiergol/Dokumenty
Searching: Dokumenty:
Traceback (most recent call last):
  File "/home/afiergol/.local/bin/rsinc", line 11, in <module>
    load_entry_point('rsinc==2.12.dev0', 'console_scripts', 'rsinc')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/afiergol/.local/lib/python3.6/site-packages/rsinc/rsinc.py", line 121, in <module>
    config_cli(config_path)
  File "/home/afiergol/.local/lib/python3.6/site-packages/rsinc/config.py", line 64, in config_cli
    common = lcl_hashes.intersection(rmt_hashes)
AttributeError: 'NoneType' object has no attribute 'intersection'

During the interactive configuration, if the desired remote base is "mydropbox:rsinc", config.py adds a traling colon, making it "mydropbox:rsinc:"
I believe this may be in error. Or maybe I'm misusing BASE_R. Is it never intended that it should point to a directory on the remote?

Edit:nevermind - I found my user error...
Edit.next: OK, now I'm not convinced. I have reopened the issue.
If I put "mydropbox:rsinc:" in the config file during interactive config, then it creates an "rsync:" directory on the remote, which is not what I wanted. I can manually edit the config file and remove the trailing colon and it seems to behave as I desire.
So, I guess I leave this as an open question as to whether I'm using BASE_R incorrectly, or is it erroneous that config.py adds an undesired ":" on the end of "mydropbox:rsinc" if I pass that in during interactive config?

I really like your tool, seems to be working fine and is more efficient than the other rclone implementations. My only problem is that I cannot make it run through cron - I have tried multiple solutions to set the right environmental variables (because I think this is the problem) but I cannot make it work.
I was wondering if you were successful in using rsinc with cron.

I used rsinc before and it worked fine; I changed distro (debian now) and I am facing installations issues - cannot figure what the problem is!


any suggestions?
thanks, Dimitris

When folders are deleted from Google Drive and i sinc the changes, they are kept in local for no reason. Is this normal? There is a command to change this behavior?

Thanks