tubearchivist / browser-extension Goto Github PK

View Code? Open in Web Editor NEW

115.0 115.0 16.0 3.51 MB

Tube Archivist Companion for your Browser

License: GNU General Public License v3.0

JavaScript 87.72% HTML 6.28% CSS 3.55% Shell 2.46%

browser-extension's Introduction

more screenshots and video

Core functionality

Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. This includes:

Subscribe to your favorite YouTube channels
Download Videos using yt-dlp
Index and make videos searchable
Play videos
Keep track of viewed and unviewed videos

Resources

Discord: Connect with us on our Discord server.
r/TubeArchivist: Join our Subreddit.
Browser Extension Tube Archivist Companion, for Firefox and Chrome
Jellyfin Plugin: Add your videos to Jellyfin
Plex Plugin: Add your videos to Plex

Installing

For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. This project requires docker. Ensure it is installed and running on your system.

The documentation has additional user provided instructions for Unraid, Synology, Podman and True NAS.

The instructions here should get you up and running quickly, for Docker beginners and full explanation about each environment variable, see the docs.

Take a look at the example docker-compose.yml and configure the required environment variables.

TubeArchivist:

Environment Var	Value
TA_HOST	Server IP or hostname	Required
TA_USERNAME	Initial username when logging into TA	Required
TA_PASSWORD	Initial password when logging into TA	Required
ELASTIC_PASSWORD	Password for ElasticSearch	Required
REDIS_HOST	Hostname for Redis	Required
TZ	Set your timezone for the scheduler	Required
TA_PORT	Overwrite Nginx port	Optional
TA_UWSGI_PORT	Overwrite container internal uwsgi port	Optional
TA_ENABLE_AUTH_PROXY	Enables support for forwarding auth in reverse proxies	Read more
TA_AUTH_PROXY_USERNAME_HEADER	Header containing username to log in	Optional
TA_AUTH_PROXY_LOGOUT_URL	Logout URL for forwarded auth	Optional
ES_URL	URL That ElasticSearch runs on	Optional
ES_DISABLE_VERIFY_SSL	Disable ElasticSearch SSL certificate verification	Optional
ES_SNAPSHOT_DIR	Custom path where elastic search stores snapshots for master/data nodes	Optional
HOST_GID	Allow TA to own the video files instead of container user	Optional
HOST_UID	Allow TA to own the video files instead of container user	Optional
ELASTIC_USER	Change the default ElasticSearch user	Optional
REDIS_PORT	Port that Redis runs on	Optional
TA_LDAP	Configure TA to use LDAP Authentication	Read more
ENABLE_CAST	Enable casting support	Read more
DJANGO_DEBUG	Return additional error messages, for debug only

ElasticSearch

Environment Var	Value	State
ELASTIC_PASSWORD	Matching password `ELASTIC_PASSWORD` from TubeArchivist	Required
http.port	Change the port ElasticSearch runs on	Optional

Update

Always use the latest (the default) or a named semantic version tag for the docker images. The unstable tags are only for your testing environment, there might not be an update path for these testing builds.

You will see the current version number of Tube Archivist in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To update, you need to update the docker images, the method for which will depend on your platform. For example, if you're using docker-compose, run docker-compose pull and then restart with docker-compose up -d. After updating, check the footer to verify you are running the expected version.

This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported and you might have to reset your index and configurations to update. Ideally apply new updates at least once per month.
There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. Always check the release notes: Any breaking changes will be marked there.
All testing and development is done with the Elasticsearch version number as mentioned in the provided docker-compose.yml file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use bbilly1/tubearchivist-es to automatically get the recommended version.

Getting Started

Go through the settings page and look at the available options. Particularly set Download Format to your desired video quality before downloading. Tube Archivist downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:

bestvideo[vcodec*=avc1]+bestaudio[acodec*=mp4a]/mp4

Subscribe to some of your favorite YouTube channels on the channels page.
On the downloads page, click on Rescan subscriptions to add videos from the subscribed channels to your Download queue or click on Add to download queue to manually add Video IDs, links, channels or playlists.
Click on Start download and let Tube Archivist to it's thing.
Enjoy your archived collection!

Port Collisions

If you have a collision on port 8000, best solution is to use dockers HOST_PORT and CONTAINER_PORT distinction: To for example change the interface to port 9000 use 9000:8000 in your docker-compose file.

For more information on port collisions, check the docs.

Common Errors

Here is a list of common errors and their solutions.

`vm.max_map_count`

Elastic Search in Docker requires the kernel setting of the host machine vm.max_map_count to be set to at least 262144.

To temporary set the value run:

sudo sysctl -w vm.max_map_count=262144

To apply the change permanently depends on your host operating system:

For example on Ubuntu Server add vm.max_map_count = 262144 to the file /etc/sysctl.conf.
On Arch based systems create a file /etc/sysctl.d/max_map_count.conf with the content vm.max_map_count = 262144.
On any other platform look up in the documentation on how to pass kernel parameters.

Permissions for elasticsearch

If you see a message similar to Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot) or failed to obtain node locks, tried [/usr/share/elasticsearch/data] and maybe these locations are not writable when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.
To fix that issue, shutdown the container and on your host machine run:

chown 1000:0 -R /path/to/mount/point

This will match the permissions with the UID and GID of elasticsearch process within the container and should fix the issue.

Disk usage

The Elasticsearch index will turn to read only if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like disk usage exceeded flood-stage watermark.

Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.

`error setting rlimit`

If you are seeing errors like failed to create shim: OCI runtime create failed and error during container init: error setting rlimits, this means docker can't set these limits, usually because they are set at another place or are incompatible because of other reasons. Solution is to remove the ulimits key from the ES container in your docker compose and start again.

This can happen if you have nested virtualizations, e.g. LXC running Docker in Proxmox.

Known limitations

Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection.
Every limitation of yt-dlp will also be present in Tube Archivist. If yt-dlp can't download or extract a video for any reason, Tube Archivist won't be able to either.
There is no flexibility in naming of the media files.

Roadmap

We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:

Implemented:

User Scripts

This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. Make sure to check the respective repository links for detailed license information.

This is your time to shine, read this then open a PR to add your script here.

danieljue/ta_dl_page_script: Helper browser script to prioritize a channels' videos in download queue.
dot-mike/ta-scripts: A collection of personal scripts for managing TubeArchivist.
DarkFighterLuke/ta_base_url_nginx: Set base URL with Nginx when you can't use subdomains.
lamusmaser/ta_migration_helper: Advanced helper script for migration issues to TubeArchivist v0.4.4 or later.
lamusmaser/create_info_json: Script to generate .info.json files using ffmpeg collecting information from downloaded videos.
lamusmaser/ta_fix_for_video_redirection: Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response.
RoninTech/ta-helper: Helper script to provide a symlink association to reference TubeArchivist videos with their original titles.

Donate

The best donation to Tube Archivist is your time, take a look at the contribution page to get started.
Second best way to support the development is to provide for caffeinated beverages:

GitHub Sponsor become a sponsor here on GitHub
Paypal.me for a one time coffee
Paypal Subscription for a monthly coffee
ko-fi.com for an alternative platform

Notable mentions

This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media, newest on top.

ycombinator: Tube Archivist on Hackernews front page, [2023-07-16][link]
linux-community.de: Tube Archivist bringt Ordnung in die Youtube-Sammlung, [German][2023-05-01][link]
noted.lol: Dev Debrief, An Interview With the Developer of Tube Archivist, [2023-03-30] [link]
console.substack.com: Interview With Simon of Tube Archivist, [2023-01-29] [link]
reddit.com: Tube Archivist v0.3.0 - Now Archiving Comments, [2022-12-02] [link]
reddit.com: Tube Archivist v0.2 - Now with Full Text Search, [2022-07-24] [link]
noted.lol: How I Control What Media My Kids Watch Using Tube Archivist, [2022-03-27] [link]
thehomelab.wiki: Tube Archivist - A Youtube-DL Alternative on Steroids, [2022-01-27] [link]
reddit.com: Celebrating TubeArchivist v0.1, [2022-01-09] [link]
linuxunplugged.com: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [link] and [2021-10-05] [link]
reddit.com: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [link]

Sponsor

Big thank you to Digitalocean for generously donating credit for the tubearchivist.com VPS and buildserver.

browser-extension's People

Contributors

Stargazers

Watchers

Forkers

insuusvenerati pairofcrocs priestd09 nietschy bakkot puc9 zyzto strogo merlinscheurer gautamkrishnar corentinb forksbot ajayyy ritiek farcaller

browser-extension's Issues

no button on short videos

Hi,

First issue :

Sorry if I'm wrong, but I would like to know if I have to do something from my side to have the button to download directly from youtube short ?

The button appears with links "https://www.youtube.com/watch?v=xxxxxxxxxxx" but not with links like "https://www.youtube.com/shorts/xxxxxxxxxxx".

The second issue :

I need also sometimes to refresh (F5) the page with link like "https://www.youtube.com/watch?v=xxxxxxxxxxx" because the button doesn't appear.

Extensions :

TubeArchivist Companion 0.2.0 with chromium (last version)

Thanks

Thumbnail download button with AR

I think this an issue with only arabic language since it reverse the ui, where the download button covers the watch later button

Fails without `http://` in the TA IP settings field

Little improvement suggestion, as the option in settings says "Tube Archivist IP", but if you indeed enter just the IP, it is going to fail (because it does not default to http protocol apparently?). I think the code should attempt to prepend http:// if no protocol is specified there already to avoid this, as it's not so obvious to figure out why it fails (if you check the logs and attempt to open the URL it tries to GET in a new tab, it will work...)

Observer to inject buttons into youtube search page

Documenting a snipped provided by @bakkot on how to inject a download button into youtube search results.

function ensureTALinks() {
  let titles = document.querySelectorAll('#title-wrapper');
  for (let title of titles) {
    if (title.hasTA) continue;
    let titleLink = title.querySelector('#video-title')?.href;
    if (titleLink == null) continue;
    let link = document.createElement('a');
    link.href = '#';
    link.innerHTML = 'Send to TA';
    link.addEventListener('click', e => {
      e.preventDefault();
      alert(`do something with ${titleLink}`);
    });
    title.append(link);
    title.hasTA = true;
  }
}
ensureTALinks();

let observer = new MutationObserver(list => {
  if (list.some(i => i.type === 'childList' && i.addedNodes.length > 0)) {
    ensureTALinks();
  }
});

observer.observe(document.body, { attributes: false, childList: true, subtree: true });

Just keeping this here for future reference.

Cookie import fails with "validated: undefined"

Heya,

I just tried importing my cookies and I'm getting the following error inside the browser extension:

In the Inspect tab I get the message {"message":{"cookie_import":"fail","cookie_validated":false}}, the network tab tells me a 400 Bad Request occured.

Not sure if I'm doing something wrong (again) tho :)

Thanks already!

Extension will not open no matter what.

I am running Firefox 125.0.1 (64-bit) on Windows 11 pro.
Whenever I click the extension to enter my API token and where my server is loaded, it does not pop up. link to see what i mean
I tried extracting the xpi file and manually loading the .js file and i was able to get it to show me what is supposed to load but it will not allow me to enter the credentials and save them.
Any help would be appreciated, and I can provide more information as necessary.

Support for Invidious

Hi. Invidious is a privacy respecting frontend developed for youtube.
I'd like to request extending this extensions compatibility to Invidious. URL's scheme is identical to youtube which should be helpful if you decide to implement. Instances are listed here
Thank you for your time.

Login failed when HTTPS-Only Mode is enabled in Firefox.

Hello!
I encountered a small issue.

In firefox, if the "HTTPS-Only Mode" parameter is set to "Enable HTTPS-Only Mode in all windows", then even when using URL within a manage exceptions (i.e. http://tubearchivist.local:8000).
Authentication will fail with the error "NetworkError when attempting to fetch resource."
If you disable this option, everything is fine.

Addon's console log (version 0.2.1)

ping TA server popup.js:138:11
sync cookies not set popup.js:203:15
Object {  } popup.js:211:13
autostart not set popup.js:213:15
message background.js listener got message Object { type: "verify" } background.js:249:11
GET: http://tubearchivist.local:8000/api/ping/ background.js:29:11
HTTPS-Only Mode: Upgrading insecure request “http://tubearchivist.local:8000/api/ping/” to use “https”. ping
XHROPTIONS https://tubearchivist.local:8000/api/ping/ CORS Failed
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://tubearchivist.local:8000/api/ping/. (Reason: CORS request did not succeed). Status code: (null).
HTTPS-Only Mode: Upgrading insecure request “https://tubearchivist.local:8000/api/ping/” failed. (M14-C1012) ping
XHRGET https://tubearchivist.local:8000/api/ping/
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://tubearchivist.local:8000/api/ping/. (Reason: CORS request did not succeed). Status code: (null).
TypeError: NetworkError when attempting to fetch resource. background.js:290:15
Verify got error: NetworkError when attempting to fetch resource.

thanks in advance

Extension fails to show pop-up (most of the time) when clicking toolbar icon

I've tried both chromium & firefox with the same result.

Clicking the toolbar icon should present settings etc. but it hasn't done so since the recent update.

UPDATE: if you repeatedly click the toolbar icon it sometimes will show the proper popup instead of a tiny spec in the bottom right corner of the toolbar icon. (also refining the title of the bug as it may not just be new installs)

I've installed FF extension and I would to help

Hi, @bbilly1. I am running a docker stack on localhost and have installed and configured the firefox extension.

It connects to the instance and displays a link on YT pages in its popup window. When clicked, it adds it to the download queue.

Ping me if you want me to test something 😄

"Open in TA" button opens wrong TA video URL (missing port and assumed HTTPS)

When I click "Open in TA" on a previously downloaded video, it opens https://10.0.0.2/video/7xYRqY3T4tw/ instead of the correct http://10.0.0.2:8000/video/7xYRqY3T4tw/. Note the missing port number and the fact that it leads me to https:// instead of http://.

I have the "Tube Archivist URL" in the extension is filled out as http://10.0.0.2:8000, so this should be used to generate the correct video URL.

Subscribe and join buttons overlap

The subscribe and join buttons overlap when there is not enough space horizontally. It also cuts off the channel info as well.

Example without join button -

Example with join button -

Cookie sync fails

Hi,
trying to auto import cookie with extension yields validated: undefined

screenshot:

logs from tubearchivist host:

cookie preview:

# Netscape HTTP Cookie File
# https://curl.haxx.se/rfc/cookie_spec.html
# This is a generated file! Do not edit.

.youtube.com    TRUE    /       TRUE    [some numbers]      NID     [long-ass string]
cookie: activated and stored in Redis
validating cookie
cookie file is invalid
cookie validation failed, exiting...
cookie: revoked
cookie: {'cookie_import': 'fail', 'cookie_validated': False}
[pid: 46|app: 0|req: 81/81] 192.168.1.163 () {50 vars in 864 bytes} [Mon Jul 25 18:12:05 2022] PUT /api/cookie/ => generated 61 bytes in 23 msecs (HTTP/1.1 400) 9 headers in 357 bytes (1 switches on core 0)

I've no idea why TA says the cookie is invalid. I could try the manual import if you think that would make a difference, although I do not actually need the cookie functionality, I just wanted to test if the automatic cookie import works.

Failing to login while exposing behind https proxy

Hi, I've setup my selfhosted my TA instance behind a reverse-proxy which redirects everything to https, that is how I access everything from within the browser, however while trying to login with the extension it fails. After looking at the inspector I see the following error:

Access to fetch at 'https://<redacted>/api/ping/' from origin 'chrome-extension://jjnkmicfnfojkkgobdfeieblocadmcie' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

My installed version is 0.2.1 and the reported error in the interface is Could not connect to server

FR: Open the instance page

Hi
It would be great if the extension has a link to the local TA server page. That way the user can open the server link easily to check the server status.

thanks

No Subscribe or Download button when logged into Youtube

When signed into a Youtube/Google account, the subscribe and download button isnt injected.

This is how it appears when logged into a Google/Youtube Account.

This is how the same page appears after logging out of Google/Youtube Account.

When used in combination with DeArrow (crowdsourced titles), many titles just show up blank.

It's always a little awkward when two separate projects conflict like this, but I figured I'd raise an issue here. The relatively new extension DeArrow replaces clickbait video titles with crowdsourced improvements, and it has really improved the ergonomics of the site (maybe worth integrating with TA like Sponsorblock, but I digress...). Unfortunately, when both DeArrow and the TA extension are used at the same time and I visit a list of videos (subscriptions, channel), any video without a crowdsourced title improvement shows up with a blank title, i.e.

Perhaps one way to avoid needing close coordination between these two projects would be to make certain DOM modifications able to be disabled in the TA extension. I mostly use it to sync the cookie and on individual video pages, both of which work great today.

Wrong hoverover text

Button works as implied, but the hover over on subscribe shows video URL instead of the channel URL

Tapping the download icon shouldn't open up video uploader's channel page in new YT redesign

Hi. I noticed this problem in the new YouTube redesign, where tapping the TA download button also seems to open up the video uploader's channel page (besides correctly downloading the video).

Download button on search page disappears on hover.

Since YouTube introduced playback directly on the search page, our download buttons disappears when hovering over the thumbnail.

Easy workaround would be to disable Inline playback in your account settings at https://www.youtube.com/account_playback to revert that change.

At least until we can come up with a better selector there.

"Unexpected token '<', " <!doctype "... is not valid JSON"

Heya,

when I'm trying to set up the companion I simply get the following error after entering the URL:

Unexpected token '<', "
<!doctype "... is not valid JSON

I don't even have to enter a API key for it to display that error.
I'm hosting Tube Archivist on my Synology as a Docker container with the default port of 8000. If it's also important, I'm using Brave (which is Chromium based).

Extension buttons cut off content

Download button cuts off video titles

With addon:

Without:

It's quite annoying, because creators create their titles to be displayed under regular YT apps to be comprehensible, and when they're cut off more, it doesn't make sense as you often don't even know what the video is about anymore.

The issue is there's a download button from TA there, and it sits there on the right with opacity: 0. So the solution is to either have it do display: none instead of opacity: 0, or to do something like position: absolute; right: 2rem;, which would then look like this:

Or even position: aboslute; right: 0; top: 3rem; (in this case the button should be shown on hover over whole card and not just title though):

In general though I find these buttons pretty questionable. I think each of them should be configurable, on whether they should be shown at all. The reason is that a lot of people would have used this extension in the first place to just sync their cookies, and anything beyond that was just an extra (personally I had to disable cookie syncing as it was breaking yt-dlp lately, which now makes me question if I even need this extension enabled at all currently xd)

Subscribe/download combo cuts off channel name

Almost equally bad. What channel am I even watching?

This happens because YouTube sets their responsive layout with the assumption that only the buttons they put there would be there, and even they themselves would have often gotten it wrong and caused things to get cut off, hilariously. But with an extra wide button, any issues with this layout are only amplified sadly (under certain screen widths...).

I think in this case it'd be better to make the button much tinier, preferably just a tiny button with TA logo, that'd show dropdown with these options. And to "Subscribe" and "Download", there could be one named something like "Open in TA instance" added, to show the video if it's already there.

Extension fails to show pop-up

Duplicate of #17 but posting again as no real solution was given.

On the same computer, not the computer hosting TA btw, both Chrome and Firefox refuse to show the login pop-up. I've tried through the official store install and the manifest.json. No matter what I do, it doesn't seem to show the pop-up. I'm at a loss as I've tried just about everything. I did however try on another PC and it showed up just fine so it must be something within my PC that is causing the issue but as to what it is, I have no idea.

Any ideas?

Main PC is on Windows 10 64-bit. Both Chrome and Firefox fully up to date. Chrome has no other extensions installed also.
Server with TA is on Unraid, fully up to date.

Running in private windows on either browser has no effect on its functionality.