Giter Club home page Giter Club logo

Comments (7)

gilesknap avatar gilesknap commented on May 29, 2024

Sorry for late reply I missed this.

I think the cause is that writing to a sqlite db may not work on an nfs mount. there are some command line options to allow your to move the db directory to a seperate mount - I can't recall which right now but gphotos-sync --help will tell you.

from gphotos-sync.

JpMaster0 avatar JpMaster0 commented on May 29, 2024

Had this project laying on ice, now giving it a try again..

I think the cause is that writing to a sqlite db may not work on an nfs mount. there are some command line options to allow your to move the db directory to a seperate mount - I can't recall which right now but gphotos-sync --help will tell you.

I think the command "--dbpath" is specifying the location of the database. It's the one on the end of my command, or am I wrong?

$ sudo docker run --rm -v ~/.config/gphotos-sync:/config -v /nfs:/storage -v ~/db:/db -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync /storage --progress --db-path /db

any other ideas or ways to find informative logs?

from gphotos-sync.

gilesknap avatar gilesknap commented on May 29, 2024

Hi @JpMaster0 yes the db-path is specified in your command line. What filesystem is /db ??
If it is your container filesystem then I think that could also be an issue. If so try mounting /db to something local on your host running the container.

To get more logs use --log-level DEBUG and see the log file that should probably appear in /db if I recall.

from gphotos-sync.

JpMaster0 avatar JpMaster0 commented on May 29, 2024

Thanks for answering Giles!
/db is in the home folder of my host. it should be ext4. it is passed to the container via -v ~/db:/db

$ df -T ./db
Filesystem                        Type 1K-blocks    Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv ext4  11218472 6920408   3706400  66% /

the log level option is helping i guess. i noticed that /storage is detected as a tempfs and i'm guessing this is the fault...?
After a while the folder seems to fill up and the process comes to a halt but when i look into the folder /nfs/gphoto there is nothing new. (i've missed the ./gphoto in my commands before but the issue is the same.. so it has nothing to do with the nfs share?!)

I've cut the log a bit to reduce reoccurring lines

$ sudo docker run --rm -v ~/.config/gphotos-sync:/config -v /nfs/gphoto:/storage -v ~/db:/db -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync /storage --progress --db-path /db --log-level DEBUG
12-04 20:49:46 WARNING  gphotos-sync 3.1.3 2023-12-04 20:49:46.416610 
12-04 20:49:46 DEBUG    MINIMUM_DATE = 1800-01-01 00:00:00 
12-04 20:49:46 INFO     Target filesystem /storage is tmpfs 
12-04 20:49:46 DEBUG    Checking if is filesystem supports symbolic links... 
12-04 20:49:46 DEBUG    attempting to symlink /storage/test_src_434988099 to /storage/test_dst_1258840432 
12-04 20:49:46 DEBUG    Checking if File system supports unicode filenames... 
12-04 20:49:46 INFO     Filesystem supports Unicode filenames 
12-04 20:49:46 DEBUG    Checking if File system is case insensitive... 
12-04 20:49:46 INFO     Case sensitive file system found 
12-04 20:49:46 INFO     Max Path Length: 4096 
12-04 20:49:46 INFO     Max filename length: 255 
12-04 20:49:46 INFO     version: 3.1.3, database schema version 5.7 
12-04 20:49:46 DEBUG    no bad_ids file, bad ids list is empty 
12-04 20:49:46 WARNING  Indexing Google Photos Files ... 
12-04 20:49:46 INFO     searching for media start=2023-10-20 13:26:21, end=None, videos=True 
12-04 20:49:46 DEBUG    mediaItems.search with body:
{'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2023, 'month': 10, 'day': 20}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}} 
12-04 20:49:47 INFO     Indexed 1 photos/2023/12/IMG20231202125341.jpg 
...
12-04 20:49:48 INFO     Indexed 81 photos/2023/10/dji_fly_20231022_162132_325_1697985492562_video.mp4 
12-04 20:49:48 DEBUG    Skipped Index (already indexed) 1 photos/2023/10/IMG20231020152621.jpg 
...
12-04 20:49:48 DEBUG    Skipped Index (already indexed) 4 photos/2023/10/IMG20231020112523.jpg 
12-04 20:49:48 DEBUG    search_media parsed 85 media_items with 100 PAGE_SIZE 
12-04 20:49:48 WARNING  indexed 81 items 
12-04 20:49:48 WARNING  Downloading Photos ... 
12-04 20:49:49 INFO     downloading 1 photos/2023/09/IMG20230901173529.jpg 
...
12-04 20:49:49 INFO     downloading 20 photos/2023/08/IMG20230819140316.jpg 
12-04 20:49:51 DEBUG    COMPLETED 1 downloading photos/2023/08/IMG20230819141037.jpg 
...
12-04 20:51:14 DEBUG    COMPLETED 119 downloading photos/2023/08/dji_fly_20230813_190316_222_1691947175788_photo.jpg 
12-04 20:51:14 INFO     downloading 139 photos/2023/08/dji_fly_20230813_184232_205_1691945995921_photo.jpg 
12-04 20:51:15 ERROR    FAILURE 1 downloading photos/2023/08/dji_fly_20230813_185552_220_1691945935992_photo.jpg - [Errno 28] No space left on device 
12-04 20:51:26 ERROR    FAILURE 2 downloading photos/2023/08/dji_fly_20230816_174052_268_1692201255998_video.mp4 - [Errno 28] No space left on device 
12-04 20:51:26 INFO     Saving Database ... 
12-04 20:51:26 INFO     Database Saved. 
12-04 20:51:26 ERROR    
Process failed. 
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 174, in download_photo_media
    self.download_batch(batch)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 215, in download_batch
    self.download_file(media_item, media_item_json)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 246, in download_file
    self.do_download_complete(done_list)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete
    raise e
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 279, in do_download_file
    shutil.copyfileobj(response.raw, temp_file)
  File "/usr/local/lib/python3.10/shutil.py", line 198, in copyfileobj
    fdst_write(buf)
  File "/usr/local/lib/python3.10/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 507, in main
    self.start(args)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 447, in start
    self.do_sync(args)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 415, in do_sync
    files_downloaded = self.google_photos_down.download_photo_media()
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 178, in download_photo_media
    self.do_download_complete(futures_left)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete
    raise e
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 279, in do_download_file
    shutil.copyfileobj(response.raw, temp_file)
  File "/usr/local/lib/python3.10/shutil.py", line 198, in copyfileobj
    fdst_write(buf)
  File "/usr/local/lib/python3.10/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
12-04 20:51:26 WARNING  Done.

edit:
I've tried to download to a folder also in the home directory and it straight up works.
so the tempfs is the fault here..

$ df -T /nfs/gphoto/
Filesystem                           Type  1K-blocks  Used  Available Use% Mounted on
192.168.2.143:/mnt/t31B/gphotobackup nfs4 4647862016   128 4647861888   1% /nfs/gphoto

from gphotos-sync.

gilesknap avatar gilesknap commented on May 29, 2024

OK I might have an answer for you here. The code downloads to a temporary file first and then moves it to your storage folder when done. It looks like the line that tries to write the stream from the server to the temporary file is failing.

Can you also mount /tmp to your host or at least make sure that there is a reasonable sized and writeable temp file system inside your container?

from gphotos-sync.

JpMaster0 avatar JpMaster0 commented on May 29, 2024

Figured it out eventually but still not sure what really fixed my error. I am using a docker compose setup now.

compose.yaml:

services:
  gsync:
    image: ghcr.io/gilesknap/gphotos-sync
    command: /storage1 --progress --db-path /db
    ports:
      - 8080:8080
    volumes:
      - /home/USER/db:/db
      - /home/USER/.config/gphotos-sync:/config
      - type: volume
        source: nfsshare
        target: /storage1
        volume:
          nocopy: true
volumes:
  nfsshare:
    driver_opts:
      type: "nfs"
      o: "addr=192.168.2.143,nolock,soft,rw"
      device: ":/mnt/t31B/gphotobackup"

had to remove a old docker volume to make this work, as my first attempt had bad parameters and docker compose wasn't replacing it with new parameters :)
Also I am using docker compose here, not docker-compose
(first remove old container: docker ps -a, docker rm X
than remove old volume: docker volume ls, docker volume rm X)

now it just works with a docker compose up

from gphotos-sync.

gilesknap avatar gilesknap commented on May 29, 2024

Thanks for letting me know. It looks like you did not need to do anything with /tmp - but then that would normally be writeable inside the container.

from gphotos-sync.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.