Giter Club home page Giter Club logo

tripper's Introduction

TRipper - a Tatort download utility

The software downloads Tatort episodes from the mediathek. The tool is aimed for people that want to create a local collection of all episodes.

TRipper comes with two data-wrappers:

  1. Mediathekview - to retrieve the available episodes and their download urls

  2. Wikipedia - to retrieve the metadata

Future releases will therefore also come with a scheduler such that the tool can run peridodicly.

The implementation heavily relies on ffmpeg and youtube-dl (as a python dependency)

Special thanks to the folkes at mediathekview for providing an easy to use api.

Installation

apt install ffmpeg # (or similar commands)
pip install git+https://github.com/stheid/tripper.git

to install current development version (or any other branch)

pip install git+https://github.com/stheid/tripper.git@develop

Usage

Adapt the config.yaml to your liking and run the program:

tripper

Setting persistent systemd.service

/etc/systemd/system/tripper.service doc

[Unit]
Description=Downloading Tatort collection

[Service]
Type=oneshot
User=<user> # if this line is omited the service will be run as root
WorkingDirectory=<path where your collection resides>
ExecStart=/home/<user>/.local/bin/tripper # or whatever "which tripper" returns


/etc/systemd/system/tripper.timer doc

[Unit]
Description=Downloading Tatort collection

[Timer]
OnCalendar=Sun *-*-* 20:15:00
Persistent=true

[Install]
WantedBy=timers.target

Execute the following lines as root

su
systemctl daemon-reload
systemctl enable --now tripper.timer
# verify using:
systemctl list-timers | grep "tripper\|UNIT"

Manually run systemd.service or debug service

Run tripper service

sudo systemctl start --no-block tripper

Inspect complete service logs

journalctl -b -u tripper

Roadmap

  1. Add docker image

tripper's People

Contributors

stheid avatar

Watchers

avatar

tripper's Issues

Better prediction of file names

Right now there are many files being put into the "check" folder that should be filtered alltogether.

Maybe the following simple classifier can be used:
Each possible tatort name is a class, than we extract the following features:
Name similarity; IsTeamMemberInDescription, IsCityInDescription...

Namesimilarity is probably passed to a transformation function such that only very high values are considered. Than i simply calculate the covex combination of the features with weights. Maybe the current function can be used to generate a dataset such that we can tune the weights or i will simply tune them by hand

prioritize downloding zürich tatort from SRF

basically, in line 44 we should no longer only look at the size, but implement a more complex comparable function. Maybe the mediathekwrapper should no longer be a pandas array, but a list of objects that implement le magic function. than they could be compared.
Currently the tatorts from zürich are "playlist downloads" meaning its not possible to determine the download filesize beforehand. therefore they will not be considered larger than the version from ARD.

tripper/tripper/exec/runner.py

Lines 42 to 46 in 9a609bd

tid = ids[0]
new_size = model.get_size_if_missing_or_smaller(tid, tatort.url)
if new_size is not None and new_size > processed.get(tid, 0):
processed[tid] = new_size
downloads[tid] = (tatort, model.filename(tid))

Refactoring: allow more download queries

Right now there is a Mediathek wrapper and a wikipedia wrapper.
Gernerally for each kind of collection there should be:

  • a Query class, that takes care of doing the initial filtering and is very close to mediathek
  • a Transformer class that takes care of more advanced deduplication, file renaming and so on. In the Tatort case Closely related to the Wiki data.

The plan is to create Query and Transformer as a general interface and make the current classes implement the interface. this allows for easy extension to collect other data from the mediathek.

ffprobe does not work on odroid xu4

ffprobe fails with an encoding error:
image

some ideas how to work around that issue:

  • get a more recent docker-arm image and try the code there, with hopefully a newer ffmpeg version
  • use console version of youtube-dl with --get-url to check if the final video url contains the string "geoblock", because the cdn that we get redirected to is called similarly for the orf. Thats however a very fragile fix.

The main problem is, that ffprobe is currently used to detect the filesize and duration of the video file. This in turn is used to determine if a the filesize is realistic. This can detect download issues such as geoprotection where we would only download a small video that informs about the geoprotection.

All in all this issue only appears on the ubuntu 18.04 on my XU4. The code works properly on a normal machine.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.