Subscriptions dissappear after page refresh. about youtubedl-material HOT 9 CLOSED

Hm, that is definitely weird. Is this happening for all channels/playlists?

Also, can you check the logs for any errors? It's likely a server-side problem, so if an error was logged it would be found in appdata/logs/combined.txt.

I'm guessing that the playlist info was retrieved correctly (the name pops up and all), but the downloads for the videos within it failed for some reason. Still doesn't explain it disappearing, though...

from youtubedl-material.

@Tzahi12345 thanks for the quick response!

I can confirm it is happening for all Playlists and Channels I've tried, I commented on the UnRaid forums and another user confirmed the same: https://forums.unraid.net/topic/87798-support-selfhostersnets-template-repository/?do=findComment&comment=850354

Unfortunately the logs don't show much at all.
appdata/logs/error.log is completely empty.
appdata/logs/combined.log doesn't show much at all, just this:

2020-04-28T02:35:48.440Z INFO: Beginning migration: 3.5->3.6+
2020-04-28T02:35:48.452Z INFO: 3.5->3.6+ migration complete!
2020-04-28T02:35:48.453Z INFO: YoutubeDL-Material v3.6 started on PORT 17442
2020-04-28T02:51:37.329Z INFO: Beginning migration: 3.5->3.6+
2020-04-28T02:51:37.345Z INFO: 3.5->3.6+ migration complete!
2020-04-28T02:51:37.346Z INFO: YoutubeDL-Material v3.6 started on PORT 17442

However in UnRaid you can click the container and view container logs, and in that window I am getting a bit more information:

at Object.mkdirSync (fs.js:823:3)
at /app/subscriptions.js:90:32
at done (/app/node_modules/youtube-dl/lib/youtube-dl.js:25:12)
at /app/node_modules/youtube-dl/node_modules/universalify/index.js:23:46
at processTicksAndRejections (internal/process/task_queues.js:94:5)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)

from youtubedl-material.

The container logs are quite helpful! It seems like it's failing at line 90 in the subscriptions.js file, at this line of code:

fs.ensureFileSync(archive_path);

Basically, it's making sure that the archive text file exists, which gets used if the archiving functionality is turned on. When the server realizes the parent directory doesn't exist (subscriptions/archives/{subscription_name}) and it can't create it with mkdir, it fails.

I assume you have a volume mapped to the subscriptions folder, can you make sure you have write access without sudo? Maybe you can run chmod -R 777 subscriptions it and see if that changes anything (just so we can track down what's wrong, it's not a permanent solution).

If you see any other errors in the container logs, let me know. It might give us another hint as to what's going awry.

from youtubedl-material.

@Tzahi12345 permissions I believe are okay:

I did figure out I needed to map app/subscriptions/ so I did that like so:

It appears to be creating the directory in subscriptions correctly (using playlist name as the folder name), then an archive.txt file inside it (although archive.txt is blank).

Before testing this all again I restarted the container, logs aren't showing any errors but same behavior is occurring. Add the channel or playlist, refresh, and it's gone.

Let me know if you need anything else!

from youtubedl-material.

Hm this is getting interesting. Maybe the archive stuff was a red herring? With the subscription disappearing still, this might be a database issue. Hard to tell at this point, but can you share your appdata/db.json file? The subscriptions would appear here if they still existed. If they're not, maybe something is deleting or overriding it.

from youtubedl-material.

@Tzahi12345 Ooo the appdata/db.json file is definitely doing something funny!

When I download a single video, I can see the db.json file populating. However as soon as I save a playlist or channel, db.json is wiped. If I go back and try and add another single video it is restored, but "subscriptions": [] is empty, and trying to add any always clears it.

Recording below:

from youtubedl-material.

Awesome detective work! I thought about it a bit more, and I think it's a bug that's quite easy to fix. Basically, I import the database twice (once in app.js, and another in subscriptions.js), which I don't think is the correct way to do it. I should instead import it once and pass variable wherever I need to use a database. It seems like the two imports are interfering with each other (maybe trying to write/read at the same time), and that's why it's behaving this way.

Not sure why it only happens on some platforms. I've dealt with inconsistencies in the past with the double imports, but I assumed because they dealt with entirely different parts of the database, it would be fine.

Having all the database calls be under one umbrella should resolve this issue. I'll leave this issue open until that gets done.

EDIT:

I've implemented what I discussed above, you can see the commit here. Hopefully this fixes the issue, if you want to test it out you can use the nightly tag on Docker.

from youtubedl-material.

I had the same issue as @Mattyfaz. Indeed fixed in the latest nightly.

Thank you very much for your amazing work @Tzahi12345 by the way :D

from youtubedl-material.

Great to hear! It's a pleasure, thank you for the confirmation :)

@Mattyfaz @saponace Closing issue now, thanks for all the help!

from youtubedl-material.