cloudbox / autoscan Goto Github PK

I have 3 Plex servers and not all got the same libraries (1080p only, 1080p with 4k and 1080P with 4K and 4K Remux)
Now when autoscan processes a remux it will have a fatal failure.

ERR Fatal error occurred while processing targets, processor stopped, triggers will continue... error="no target libraries found:

While I understand the thought behind it it would be better to (have the option) to not stop the processor and just continue with autoscan.

PS: This wasn't a problem with plex_autoscan, it would just gracefully skip it and continue so a regression in function in that regard.

Would it be possible to add Readarr as one of the triggers in the config.yml? I currently put the path to point to my audiobooks folder location and autoscan usually grabs those files, but it sometimes misses them and I have to manually refresh my audiobooks library in Plex.

Currently, autoscan binds to all interfaces by default.

Would be nice if it could bind to specified interfaces:

host: 127.0.0.1
host:
  - 127.0.0.1
  - 10.0.0.2

hey guys. don't know if you will be implanting this. but will there be any chance of adding a web-ui. so we can set of a manual scan. and would it be possible for us to make changes on the web-ui. e.g adding other Tdrive etc. and it auto generates and add's it to the config file. sorry if i am asking for too much.

The code uses v1.10.0 of the modernc.org/sqlite module.

v1.10.6 fixed a memory leak, and v1.13 introduced FreeBSD/amd64 support.
As a FreeBSD user, it would be nice if you could upgrade the module so I can do a go build on FreeBSD.
At the moment I'm running autoscan under linuxemu, which works fine. But it would be nice to build it native on FreeBSD.

Thanks for autoscan!

I am using autoscan on Synology Docker and my Plex install is on a Windows machine. My libraries are all mapped using UNC paths. I am able to get autoscan to work if I create a new library and use mapped network drives in Windows instead of UNC paths. Is it possible for autoscan to recognize, or be updated to recognize, UNC paths in a Plex target rewrite?

Hi all, first off thanks for this new rewrite, is super simple and slick!

I've got everything running smoothly so far but have noticed that if a file is upgraded, the old file remains until I manually 'Empty trash' for the associated library.

I can't see anything regarding trash emptying in the config or code, is this to be expected?

plex_autoscan includes a script for fetching the plex auth token (below). It would be cool if there was a cli option that did a similar method

https://github.com/l3uddz/plex_autoscan/blob/master/scripts/plex_token.sh

Is it possible to get Plex to reanalyze a file?

I use Tdarr to create Stereo AAC streams from 5.1 audio tracks, which keeps transcodes nice and light on my server.

Tdarr copies a file to a cache location, does its transcoding, then copies the file back to where it came from. The problem is that Plex doesn't know about these new audio tracks, so playback fails because the expected media components are all out of order. I've found that having autoscan notify Plex about the file isn't enough to get it to notice the new audio tracks and I have to manually Analyze the files that Tdarr touches.

This is what happens in Plex during Analyzing: https://support.plex.tv/articles/200289336-analyze-media/

Is it possible to get autoscan to tell Plex to Analyze the media it scans too?

Thank you!

Right now if autoscan fails to connect to plex it shuts down with the error "Failed initialising target". When using autoscan and plex with docker, it can sometimes take plex a couple minutes to start up, whereas autoscan is much faster. As a result even if starting plex first, autoscan often fails to start and therefore has to be manually restarted later. This makes starting or updating the docker automatically difficult.

Might it be possible to add either a configurable delay to the startup script, or possibly even a method for retrying the connection every few minutes if it initially fails?

Thanks!

Hi,

is bernard able to use an rclone encrypted drive ?

not seeing anything in the https://github.com/m-rots/bernard repo about this

Can you confirm ?

Thanks

Hey I tried to use autoscan with radarr and have an issue.

As you can see from the screenshot the test succeeds.
But saving it fails

And I get theese errors in the logs (I obviously added the trigger in my config file)

As you can see in the logs I get a missing SQL column for OnDelete (the On delete option is not checked in radarr).
Is it normal ?
Is it a configuration issue ?

I don't know if this is the way inotify is supposed to work or not but the behavior I'm seeing is that inotify will detect a change for newly create files (ie. imports of new movies or upgrades (ie. movie-bluray720p.mkv changing to movie-bluray1080p.mkv)) but it will not detect file changes when the file remains the same (ie. file being converted where the file size gets reduced but the name remains the same).

I've been getting the following error in the last 24 hours and I'm not sure why it isn't able to reach the plex server. I've had no problem on my end being able to view movies/tv from the plex server.

PMS version: Version 1.20.4.3517

4:27AM INF Initialised processor anchors=["/mnt/unionfs/mounted-movies-4k.bin","/mnt/unionfs/mounted-movies.bin","/mnt/unionfs/mounted-tv.bin"] min_age=10m0s,
4:27AM INF Initialised triggers bernard=0 inotify=0 lidarr=0 radarr=2 sonarr=1,
4:27AM INF Starting server on port 3030,
4:27AM FTL Failed initialising target error="version: Get \"https://plex.domain.tld/\": dial tcp 192.99.x.x:443: i/o timeout: target unavailable" target=plex target_url=https://plex.domain.tld,
4:27AM INF Initialised processor anchors=["/mnt/unionfs/mounted-movies-4k.bin","/mnt/unionfs/mounted-movies.bin","/mnt/unionfs/mounted-tv.bin"] min_age=10m0s,
4:27AM INF Initialised triggers bernard=0 inotify=0 lidarr=0 radarr=2 sonarr=1,
4:27AM INF Starting server on port 3030,`

Hi I'm getting this error below. Can anyone help me with this?

Jul 4 00:38:16 INF Initialised processor anchors=["/mnt/unionfs/.anchors/media.anchor"] min_age=2m30s,
panic: runtime error: invalid memory address or nil pointer dereference,
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x9ae32c],
,
goroutine 1 [running]:,
github.com/m-rots/stubbs.ParseKey(0x0, 0x0, 0xc000214080, 0x0, 0x0),
github.com/m-rots/[email protected]/stubbs.go:73 +0x6c,
github.com/m-rots/stubbs.FromFile(0xc0000ab290, 0xf, 0xc0002181a0, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc00009f3c0, 0x1, ...),
github.com/m-rots/[email protected]/stubbs.go:105 +0x10a,
github.com/cloudbox/autoscan/triggers/bernard.New(0xc0000ab290, 0xf, 0xc0000ab2a4, 0xb, 0x0, 0x0, 0x0, 0x0, 0xc00014df40, 0x1, ...),
github.com/cloudbox/autoscan/triggers/bernard/bernard.go:48 +0x445,
main.main(),
github.com/cloudbox/autoscan/cmd/autoscan/main.go:220 +0xfc8

i just set up autoscan using "bernard".

im having just one issue. in some cases autoscan triggers multiple libraries to scan. in this case plex won't find any new file.

of course the path is not set to both of these libraries.

for example:

Mar 11 10:07:32 INF Scan moved to target library="Serien (GER)" path="/mnt/fclone/fusion/tvshows-4k/Snowpiercer (2020) [tvdbid-364928]/Season 02" target=plex url=http://plex-0:32400

Mar 11 10:07:32 INF Scan moved to target library="Serien 4K (GER)" path="/mnt/fclone/fusion/tvshows-4k/Snowpiercer (2020) [tvdbid-364928]/Season 02" target=plex url=http://plex-0:32400

or

Mar 11 13:04:14 INF Scan moved to target library="Serien (GER)" path="/mnt/fclone/fusion/tvshows-trash/The Bachelor (DE) (2003) [tvdbid-265925]/Season 11" target=plex url=http://plex-0:32400

Mar 11 13:04:14 INF Scan moved to target library="Serien Reality-TV (GER)" path="/mnt/fclone/fusion/tvshows-trash/The Bachelor (DE) (2003) [tvdbid-265925]/Season 11" target=plex url=http://plex-0:32400

I have the Empty trash automatically after every scan on my Plex install disabled because my mount often crashes and starts again automatically and if everything is detected as missing then it could wipe my entire library after it's all deleted.

Could a function be added to autoscan for managing the Plex emptyTrash function? Like it could run only if the threshold of missing/unavailable files is below a certain number?

Thanks again for making another awesome media server utility.

I had a few questions about the bernard google drive monitorig.

I see it now relies on json files. Does this mean it will not work with "My Drive" since service accounts, as far as I know, can't be granted rights on my drive?

Second, does Drive ID with Bernard mean the actual drive name or the drive ID string, e.g. the team drive ID from rclone?

While Sonarr and Radarr triggers work perfectly, Lidarr triggers fail from time to time. The message I get is failed determining libraries. Doesn't seem to be a configuration issue, as most of the time it works.

As you can see, I am conforming to the default Plex naming scheme. Library path is /mnt/google/Musik in Plex, as well as in Lidarr, with no rewriting configured for the Lidarr trigger.

Because of the very basic logging with unclear messages I cannot be sure. I suspect, however, that the error turns up whenever Plex doesn't know the artist yet. Whenever a new album for a known artist is added, it seems to work fine. Might this have to do with directory structure being a bit different for music libraries compared to movie and tv libraries (additional sub-directory in artist/album/file.mp3 as opposed to movie/file.mp4 or series/file.mp4)?

Log:

Jan  6 21:00:45 INF Scan moved to processor event=Download method=POST path="/mnt/google/Musik/Eric Clapton/There’s One in Every Crowd (1975)" request_id=c7bkkrff2ncs6vpuqoj0 url=/triggers/lidarr
Jan  6 21:01:45 WRN No target libraries found error="/mnt/google/Musik/Eric Clapton/There’s One in Every Crowd (1975): failed determining libraries" target=plex url=http://plex.local

Config:

# <- triggers ->
port: 80
triggers:
  sonarr:
    - name: lidarr # /triggers/sonarr
      priority: 1
  radarr:
    - name: lidarr # /triggers/radarr
      priority: 1
  lidarr:
    - name: lidarr # /triggers/lidarr
      priority: 1
# <- processor ->
minimum-age: 5m
scan-delay: 5s
scan-stats: 1m
anchors:
  - /mnt/google/.mntchk
authentication:
  username: [*****]
  password: [*****]
# <- targets ->
targets:
  plex:
    - url: http://plex.local
      token: [*****]

Hello,
Love autoscan and it's been working great on my NAS.

I've noticed however that it is scanning the .Recycle.Bin hidden folder on my share, which is causing erroneous files to show up in my plex library.

Is it possible to exclude hidden files/folders from the inotify scanner?

Thanks!

Any plans on creating a default config.yml file if the file isn't found at start? Would make this a bit more intuitive (and reduces the work I need for my Helm chart).

I'm getting errors in my logs and it's not working... Sorry I can't be more descriptive but i'm not technical. I'm trying to get it working with Plex and to the best of my knowledge everything else is setup correctly and I'm told these errors are a docker problem. Please see the attached file.
_autoscan_logs.txt

Retryable error occurred while syncing drive error="XXXXXXXXXXXXX: performing partial sync: ID does not exist from hook: datastore: data anomaly" attempts=2 drive_id=XXXXXXXXXXXXX drive_name="Shared Drive" trigger=bernard

I'm getting this error for bernard. What does that mean?

config:

  bernard:
    - account: /config/app/sa.json
      cron: "*/5 * * * *" # every five minutes (the "" are important)
      drives:
        - id: XXXXXXXXXXXXX

      # rewrite drive to the local filesystem
      rewrite:
        - from: ^/unencrypted/
          to: /data/m/

      # filter with regular expressions
      include:
        - ^/data/m/tv-shows/
        - ^/data/m/movies/
      exclude:
        - '\.srt$'

It would be really cool to have an interface listing jobs, completed, and pending. Could also have a force button to send it right away, as well as a remove button

Can we get an ARM (32-bit + 64-bit) supported docker image so we can work with it on Raspberry Pi's?

I run autoscan with systemd with the following targets: 2 plex, 1 jellyfin, 1 emby.

More often than I'd like, one of these will be down, sometimes resulting in a script exit.

If missing upon startup:

FTL Failed initialising target error="libraries: Get \"http://REDACTED:8097/Library/VirtualFolders\": dial tcp REDACTED:8097: connect: connection refused: target unavailable" target=jellyfin target_url=http://REDACTED:8097

If missing after it's recieved a task to send to targets:

ERR Not all targets are available, retrying in 15 seconds... error="availability: Get \"http://REDACTED:8097/System/Info\": dial tcp REDACTED:8097: connect: no route to host: target unavailable"

What I'd suggest is handling the error with:

Target {target_info} is unavailable, will try to initialize again in 3 minutes.

And

Target {target_info} is unavailable, will try to send task again in 3 minutes. (attempt 1 of 3)

This way, tasks can get sent to the Available targets, and not get hung up on a single downed target.

Has the auto empty trash for the scanned section been removed? I'm running autoscan on a test Plex server and I'm seeing a lot of "Unavailable" duplicates or trash icons, my other server running PAS has no issues.

The addition of "/" to all library paths from PLEX if they do not end with a "/" (#102) does not work for PLEX servers running on Windows machines,

This is currently an issue in my setup. I have my "arrs" installed in Docker containers with say "/tv/" as the path. My PLEX server is installed on a Windows machine with the TV folder as "T:\tv".

When Autoscan returns the libraries it is showing TV as having a path of "T:\tv/".

In the triggers I have setup a rewrite of:

from: /tv
to: T:\tv

In the Targets I have:

from: /
to: \

Say a request comes in for /tv/Show/Season 1/Show - s01e01.mkv the triggers will rename it to T:\\tv/Show/Season 1/Show - s01e01.mkv This would work if this was the path used for PLEX but it is not. The path used is the one generated from the targets. which is changed to T:\\tv\\Show\\Season 1\\Show - s01e01.mkv which does not match the section path of T:\\tv/ If I do not change the / for \ then the path is wrong for plex.

a proposed solution would be to add a few extra checks and a ELSE IF.

//Add trailing forwardslash if there is none for Unix file system.
if len(libPath ) > 0 && !strings.Contains(libPath , ":\\\\") && libPath[len(libPath)-1] != '/' {
	libPath += "/"
}
			
//Add trailing backslash if there is none for Windows file system.	
else if len(libPath ) > 0 && strings.Contains(libPath , ":\\\\") && libPath[len(libPath)-1] != '\\\\' {
	libPath += "\\\\"
}

Error when renaming a file via Radarr.

2:35PM TRC Received JSON body event={"eventType":"Rename","isUpgrade":false,"movie":{"FolderPath":"/home/plex/media/Movies4K/Asterix The Secret of the Magic Potion (2018)"},"movieFile":{"RelativePath":""}} method=POST request_id=bt5kh5qn153b54c7101g url=/triggers/4k
2:35PM ERR Required fields are missing method=POST request_id=bt5kh5qn153b54c7101g url=/triggers/4k
2:35PM DBG Request processed duration=0.261941 method=POST request_id=bt5kh5qn153b54c7101g status=400 url=/triggers/4k

Plex, Sonarr and radarr : Docker linuxserver.io
Autoscan : docker hotio

My config 👍

port: 3030

triggers:
  sonarr:
    - name: sonarr
      - from: /tv/
      to: /Plex/

  manual:
    rewrite:
    - from: /Google/
      to: /Plex/Google/
      
targets:
  plex:
    - url: http://192.168.0.28:32400
      token: *mytoken*

My log 👍

When i stop the container autoscan, in plex log 👍

Anyone have a idea ? :(

Would be nice if we could get binaries for the Raspberry Pi, which is based on ARM in both 32-bit and 64-bit.

The old plex_autoscan can be triggered with this tdarr plugin:

https://github.com/HaveAGitGat/Tdarr_Plugins/blob/master/Community/Tdarr_Plugin_MC93_MigzPlex_Autoscan.js

I'd imagine that something similar could be made for Autoscan to do the same thing. I assume something would have to be coded for each repo for it to work?

I started getting this error to a TD. I deleted the autoscan.db and let it rebuild, but it still happens after a full rebuild:
ERR Consecutive errors occurred while syncing drive, drive has been stopped... error=["REMOVED: performing partial sync: ID does not exist from hook: datastore: data anomaly","REMOVED: performing partial sync: ID does not exist from hook: datastore: data anomaly","REMOVED: performing partial sync: ID does not exist from hook:

Any ideas on what could be causing this?

Hi,

I'm not sure what I'm doing wrong but I'm getting errors about an anchor file being unavailable even though as its displaying the error I can login to the docker container and ls the file. It happens constantly even with the network share and docker volume are mounted.

I have two anchor files specified as follows:

anchors::
  - /data/movies/autoscan-movies-anchor
  - /data/tv/autoscan-tv-anchor

They are empty files, the tv anchor never has issues.

I want to write to the config file dynamically to add and remove plex servers listed in the config. What is the best way to do this, without doing it manually. Just curious if anyone else is doing this dynamically, through cURL or similar, or using something to dynamically edit the config file in another lang, like PHP

Would it be possible to support pre/post hooks/task that run with new scan requests? I have to run Plexarr often to fix matches that Plex couldn't find and doing it on a schedule (cron) is possible but I was hoping to run it as soon as the autoscan task finished.

In Sonarr I have path/AnimeTV, path/KidsTV and path/TV, because I like to keep these things seperate, with Autoscan I don't see anything in the documentation about whether or not we can add multiple paths, we can add multiple triggers, but not paths. Also can we add an anchor to more than one mount if we're not using UnionFS or MergerFS? If I have AnimeTV in one mount and TV/KidsTV in another mount?

Here's an example of how i think I can write this, but would greatly appreciate any help.

anchors:
  - /mnt/fusemount/Gdrive-Series/anchor.txt
  - /mnt/fusemount/Gdrive-AnimeTV/anchor.txt
  - /mnt/fusemount/Gdrive-Movies/anchor.txt

triggers:
  sonarr:
    - name: sonarr-docker # /triggers/sonarr-docker
      priority: 2

      # Rewrite the path from within the container
      # to your local filesystem.
      rewrite:
        - from: /mnt/fusemount/Gdrive-Series/TV/
          to: /mnt/fusemount/Gdrive-Series/TV/

        - from: /mnt/fusemount/Gdrive-Series/KidsTV/
          to: /mnt/fusemount/Gdrive-Series/KidsTV/

        - from: /mnt/fusemount/Gdrive-AnimeTV/
          to: /mnt/fusemount/Gdrive-AnimeTV/

Firstly, thank you so much for the awesome app. It works so well as it uses less processing by scanning only the updated folder.

To stop plex from adding a movie to recently added movies when upgraded, I added Radarr's recycle bin folder as a media source for plex. So that when Radarr upgrades a movie, the existing movie file will be moved to the recycle bin and plex will consider the file was moved to a different location.

With this setup, plex will retain the metadata and will not show as a newly added movie on the home page and once radarr clears the recycle bin, only the latest version of the movie will be on plex.

Is there a way to add a default/manual directory (RecycleBin folder) that will be scanned as well every time the plex was triggered to initiate a scan? or to send the recycle bin directory along with the movie directory to plex?

Thanks in advance

Set up yesterday for the first time and it's started throwing this error:
Mar 9 12:06:33 ERR Failed watching new directory error="watch directory: /mnt/unionfs/movies/The Perfect Teacher (2010): no space left on device" path="/mnt/unionfs/movies/The Perfect Teacher (2010)" trigger=inotify

Have it setup to notify from radarr for the /mnt/unionfs/movies directory

When attempting to access the webpage for /triggers/manual via a external IP the following is logged

WRN Invalid authentication method=GET request_id=c31km77bsjdkuckstr1g url=/triggers/manual

and no authentication popup is displayed on the browser

however accessing the same on localhost works

For my usecase, I have my storage server mounted via SMB/CIFS which does not create inotify events.

I'm not sure if watching inotify being enabled and not used causes any performance hit, but being able to disable it seems like a good idea.

Alternatively, some sort of watchdog for the directory with a configurable scan interval might be useful as well.

I am stumped on this one. Im getting an expected " : ". I can change the line removing and adding the colon no change. Any ideas? Line 23 the rewrite in the plex target

triggers:
sonarr:
- name: sonarr
- rewrite:
- from: /TV
to: /media/TV/
radarr:
- name: radarr
- rewrite:
- from: /movies
to: /media/Movies

targets:
plex:
url: http://10.0.0.6:32400# URL of your Plex server
token:7LVznDdGScLTi6VP3V37 # Plex API Token
- rewrite:
- from: /media/
to: /data/

Hi,

I have no clue if its possible or not. I run manual trigger most of the time and that means I can only scan one TV Show or Movie at a time. But would be great if I can select the date from calendar which will trigger the scan based on last number of days. This way I have lot of files scanned but not the whole library.

I hope this explains little bit.

Thanks

No scans are available, retrying in 15 seconds...

Is this normal? How can I make this longer?

Setting scan-delay: 90s doesn't change it

My understanding is inotify won't work with CIFS shares. Is there a way to do a ls -R type function on a cron schedule to parse out deltas for autoscan? Thanks in advance!

Hi,

thanks for the tool ! coming from the old plex_autoscan,

what should be the format and keys of the file service-account.json ?

Thanks

i have this running on synology but im getting this in the log:

ERR Required fields are missing method=PUT request_id=c0k38u24qlig1jq0gc40 url=/triggers/sonarr

do i have to install somthing or add to the config ?

# <- processor ->

# override the minimum age to 30 minutes:
minimum-age: 30m

# set multiple anchor files

# <- triggers ->

# Optionally, protect your webhooks with authentication
authentication:
  username: 
  password: 

# port for Autoscan webhooks to listen on
port: 3030

triggers:
  sonarr:
    - name: sonarr # /triggers/sonarr
      priority: 2

  radarr:
    - name: radarr   # /triggers/radarr
      priority: 2
  lidarr:
    - name: lidarr   # /triggers/lidarr
      priority: 1

# <- targets ->

targets:
  plex:

I'm wondering if it's possible to implement a file check in the processor before it initiates the scan? My media files are downloaded on a separate machine than my Plex server, and due to the nature of the way the scan is triggered, it can sometimes take quite a while before the file is available to my Plex server. I realize that is what the minimum age setting is for, but it's annoying to have to set a really high minimum age just to handle the occasional slow transfer, when most files are available relatively quickly otherwise.

PAS has a retry/delay system to check for the existence of the file every x number of seconds, y number times (after an initial delay), before starting the scan, which is immensely useful. It would be great if this were available in Autoscan as well.

Thanks!!

I have a Plex library for 4K content and below (that I use for my personal use), and a separate library for 1080p and below (minus the 4k stuff that I share with a few family members).

The 4K > library contains the folders /Media/Movies4k and /Media/Movies
The 1080 > library contains only the folder /Media/Movies

When Radarr adds a movie and requests Autoscan to scan that Movie, only the 4K > library gets updated the vast majority of the time.

I have an identical setup for my TV Shows and Sonarr. Since I only notice the issue when a family member notices something missing, I can’t for sure say if there’s any pattern to when it does or doesn’t happen. This is only a guess, but it seems like perhaps the issue happens more with TV Shows than it does Movies.

Is this due to an error on my part with how I configured Autoscan, or is it a limitation of the script somehow?

Autoscan looks good, testing it now and noticed that when a target is not available it will restart the autoscan process until it's available. This is giving issues with autoscan not working when multiple targets are configured and one of those is down (for whatever reason)

An x retry and then ignore the missing target would be ideal.