voc / voctopublish Goto Github PK

Not all thumbnails could be regenerated during #10 - it is a minor problem, since the old images were not deleted.

It should be investigated and fixed what the problem was.

In one case it is clear, the api has no correct poster image set, so the check-script could not find it.

In another case, there is no video file and therefore no thumbnail can be generated.

2016/12/21 16:17:34 image 'https://static.media.ccc.de/media/_new/conferences/froscon/2011/hs1_2_-_2011-08-20_12:45_-_software_freedom_means_business_value_-_simon_phipps_-_830.jpg' not found for 'Software Freedom Means Business Value' (https://api.media.ccc.de/public/events/323)
2016/12/21 16:17:34 image 'https://static.media.ccc.de/media/_new/conferences/froscon/2011/hs1_2_-_2011-08-20_12:45_-_software_freedom_means_business_value_-_simon_phipps_-_830_preview.jpg' not found for 'Software Freedom Means Business Value' (https://api.media.ccc.de/public/events/323)
2016/12/21 16:24:40 image 'https://static.media.ccc.de/media/_new/congress/2004/173 Closing Event.jpg' not found for 'Closing Event' (https://api.media.ccc.de/public/events/874)
2016/12/21 16:24:40 image 'https://static.media.ccc.de/media/_new/congress/2004/173 Closing Event_preview.jpg' not found for 'Closing Event' (https://api.media.ccc.de/public/events/874)
2016/12/21 16:29:13 image 'https://static.media.ccc.de/media/_new/conferences/mrmcd/mrmcd15/7080-sd.jpg' not found for 'RFC Reading' (https://api.media.ccc.de/public/events/2652)
2016/12/21 16:29:13 image 'https://static.media.ccc.de/media/_new/conferences/mrmcd/mrmcd15/7080-sd_preview.jpg' not found for 'RFC Reading' (https://api.media.ccc.de/public/events/2652)

This makes the logic easyer but need to be checked in the whole code

We should either place all muxes in one dir and and the master in another or create a dir per language.

probably in https://github.com/voc/publishing/blob/dev-andi/postprocessing/generate_thumb_autoselect_compatible.sh#L37

Verschoben aus voc/voctoweb#40

Variante A:

• beim ersten Release eines Encoding-Tickets die media.ccc.de recording_id die beim releasen zurück kommt in den tracker ins entsprechende Encoding-Ticket zurückschreiben.
• bei einem Rerelease dann anstatt POST recording ein PUT recording/ machen
  Offene Frage:
• Es gibt Fälle in denen ein Encoding Ticket mehrere Recordings auf media erzeugt: Bei mehreren Audio-Tracks wird aktuell im releasing Skript gesplittet und nicht im Tracker...
  @a-tze Soll das so bleiben oder wolltet ihr das ändern?

Variante B:

• Release eines Encoding-Tickets wie gehabt.
• wenn beim Rerelease via POST nen Fehler von der Validation zurück kommt weil es für diesen Mime-Type + Sprache + Quality-Flag (TODO ist das der Fall?) oder den Filename statt dessen ein PUT recording/ machen

Variante C:

• Beim Release eines Encoding-Tickets wird im Ticket ein Flag released_to_media=yes gesetzt
• Bei einem Rerelease wird auf Basis dieses Flags entschieden ob POST oder PUT das richtige ist.
• Bei PUT muss man sich vorher über die UUID des Events die Recordings hohlen und schauen welche Recordings man konkret überschreiben muss. Lässt sich über den Filename relativ einfach rausfinden...

Currently its hard to recover from an youtube error as we dont want to also talk again to media

ERROR - Exception: Incompatible ssh peer (no acceptable kex algorithm)

kex algos:['[email protected]', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256'] client encrypt:['[email protected]', 'aes128-ctr', 'aes192-ct
r', 'aes256-ctr', '[email protected]', '[email protected]'] server encrypt:['[email protected]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', '[email protected]', '[email protected]'] client mac:['[email protected]', '[email protected]', '[email protected]', '[email protected]', 'hmac-sha1-etm@op
enssh.com', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] c
lient compress:['none', '[email protected]'] server compress:['none', '[email protected]'] client lang:[''] server lang:[''] kex follows?False

Voctoweb guesses the events url as follows:
https://github.com/voc/voctoweb/blob/master/app/models/concerns/fahrplan_updater.rb#L18

Does publishing set the link directly via the API? If it does, https://github.com/voc/voctoweb/blob/master/app/controllers/api/events_controller.rb#L87 should never overwrite params from the same request.

As of now we only put conference acronym, local frab id and track into the voctoweb tags attribute list.
We should discuss how we get more tags for in individual talk into voctoweb, part 2:
(part 1, see #61)

• add tag for Foundation talks
• use frab's event_classifieres when they reach some specified level (not yet in schedule xml, c.f. frab/frab#443 and the orange/blue buttons on https://halfnarp.events.ccc.de
• search title, abstract and description for keywords, c.f. https://github.com/Thorsten-Sick/tags_for_media_ccc_de/blob/master/dropdata.py#L22

Did not happen yet but was suggested in the irc. A publishing process should stop if a file has 0byte size as this is most likely not a video

• master release contains 2. video track. Nothing to do here
  • figure out what needs to be done on vocotoweb code
• remux a slide only video from master that contains only the slide video track an the native audio track
• do the remuxing for h264 HD / webm HD, also for SD?
• figure out what needs to be done on vocotweb to show these files

add the track if present in the schedule to the tags

A 503 from youtube results in

uploading video failed with error-code 503: {
"error": {
"errors": [
{
"domain": "global",
"reason": "backendError",
"message": "Backend Error"
}
],
"code": 503,
"message": "Backend Error"
}
}

this needs more infos to be usable

this can either be used to prevent the intro from showing up in the thumbnail or to generate thumbs always conatining the intros. The behavior should be controlled by an optional tracker propertie:
Publishing.Voctoweb.IntroToThumb yes / no

An optional hash value filed in the xml should be added to verify downloaded files.

möglicher Vorschlag:
Conference: Titel (Vortragener 1, Vortragender 2)

Alternativ: mit Suffix und Prefix arbeitern, oder gleich mit nem Templates, sprich ein String in dem man über $variable den entsprechenden Wert einfügen kann.

fix youtube switch in twitter code

Change the runtime behavior of the publisher to loop until not new tickets are available.
This will align its behavior with the other CRS scripts and will speed up publishing

The tracker and thereby the publishing worker knows which language is used in recoding. Publish this information also via YouTube's API. See https://developers.google.com/youtube/v3/docs/videos/update – defaultLanguage is probably the field we are looking for.

It looks like there is also a localisation option for title and description, cf. https://developers.google.com/youtube/v3/docs/videos#localizations .

2016-10-13 13:49:55,326 - requests.packages.urllib3.connectionpool - DEBUG - "POST /api/events HTTP/1.1" 422 None
2016-10-13 13:49:55,327 - root - INFO - event already exists. => publishing
2016-10-13 13:49:55,328 - root - INFO - server said: {"guid":["has already been taken"]}
2016-10-13 13:49:55,328 - root - INFO - ## uploading /srv/releases/datengarten/datengarten-70-deu-xwing_Drohnenprojekt_hd.mp4 ##
2016-10-13 13:49:55,328 - root - INFO - ## Establishing SSH connection ##

Aktuell sind zwei Sprachen hartcodiert im Code. Da wir beim 33C3 auch offiziell mehr als eine Übersetzerspur unterstützen, würde ich vorschlagen den folgenden Codeblock von extra Variablen pro Sprache auf eine Schleife umzustellen:
https://github.com/voc/publishing/blob/dev-andi/script_H_publishing.py#L189-L219

Prevent broken paths by trimming the input

Aus dem Tracker fällt ein MP4 mit mehren Audiospuren. Für Youtube und für den voctoweb-Workaround brauchen wir aber auch MP4's mit nur einer Audiospur. Aus historischen Gründen wird das aktuell zwei mal gemacht: einmal für YouTube und einmal für Media. Aus meiner Sicht sollten wir das vereinheitlichen und die selben Dateien dafür benutzen.

Currently we maintain language strings in the voctoweb and the youtube client.
Also each new language requires changes in the code.
We need a central and more generic approach on this

Currently the importing worker places the files in the fuse folder. If we get files that can be used as master encoding already there should be an option to place the files in the tmp folder so they get only postencoded instead of encoded.

Instead of requering e.g. Publishing.Mastodon.Enable to be set even if we dont want it we should handle this as false. This would reduce the amount of properties needed to be set.
This should be changed for:

• Twitter
• Mastodon
• Voctoweb
• Youtube

thumb path composition is not save for missing trailing /

currently we have many string compares as the tracker uses yes / no. To clean this up the ticket module should convert yes / no to True / False

If the tracker is not reachable we still try to set the ticket failed which results in an error.

C3TT.setTicketFailed[b'C3TTException: A OS error occurred\nError message: [Errno -2] Name or service not known',

It would be nice to have a boolean property to enable or disable the YouTube comment section for all videos of an event.

Feb 08 16:12:08 releasing.c3voc.de sh[32709]: 2019-02-08 16:12:08,918 - root - ERROR {voctopublish.py:325} Could not create format subdir /cdn.media.ccc.de/contributors/cccac/vortraege/h264-hd : [Errno 13] Permission denied
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: Traceback (most recent call last):
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/home/voc/publishing/voctopublish/api_client/voctoweb_client.py", line 233, in upload_file
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self.sftp.stat(format_folder)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 493, in stat
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: t, msg = self._request(CMD_STAT, path)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 813, in _request
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: return self._read_response(num)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 865, in _read_response
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self._convert_status(msg)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 894, in _convert_status
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: raise IOError(errno.ENOENT, text)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: FileNotFoundError: [Errno 2] No such file
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: During handling of the above exception, another exception occurred:
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: Traceback (most recent call last):
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/home/voc/publishing/voctopublish/api_client/voctoweb_client.py", line 237, in upload_file
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self.sftp.mkdir(format_folder)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 460, in mkdir
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self._request(CMD_MKDIR, path, attr)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 813, in _request
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: return self._read_response(num)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 865, in _read_response
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self._convert_status(msg)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 896, in _convert_status
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: raise IOError(errno.EACCES, text)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: PermissionError: [Errno 13] Permission denied
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: The above exception was the direct cause of the following exception:
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: Traceback (most recent call last):
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "voctopublish/voctopublish.py", line 321, in
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: publisher.publish()
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "voctopublish/voctopublish.py", line 118, in publish
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: self._publish_to_voctoweb()
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "voctopublish/voctopublish.py", line 228, in _publish_to_voctoweb
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: vw.upload_file(self.ticket.local_filename, filename, self.ticket.folder)
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: File "/home/voc/publishing/voctopublish/api_client/voctoweb_client.py", line 239, in upload_file
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: raise VoctowebException('Could not create format subdir ' + format_folder + ' : ' + str(e)) from e
Feb 08 16:12:08 releasing.c3voc.de sh[32709]: api_client.voctoweb_client.VoctowebException: Could not create format subdir /cdn.media.ccc.de/contributors/cccac/vortraege/h264-hd : [Errno 13] Permission denied
Feb 08 16:12:08 releasing.c3voc.de systemd[1]: publishing.service: Main process exited, code=exited, status=255/n/a
Feb 08 16:12:08 releasing.c3voc.de systemd[1]: Failed to start voctopublish.
Feb 08 16:12:08 releasing.c3voc.de systemd[1]: publishing.service: Unit entered failed state.
Feb 08 16:12:08 releasing.c3voc.de systemd[1]: publishing.service: Failed with result 'exit-code'

this is usually done by the cut worker, as we skip this we need to do it in the recording step

See https://www.youtube.com/watch?v=oV_xKDB-mdA and Talk 373 in 33c3-sz.

As of now we only put conference acronym, local frab id and track into the voctoweb tags attribute list.
We should discuss how we get more tags for in individual talk into voctoweb, part 1:

• add year as tag voc/voctoweb#14
• add new property for static conference tags named Publishing.Tags?
• merge YouTube tags from tracker properties Publishing.Tags and Publishing.YouTube.Tags
• use static conference tags from tracker property Publishing.Tags also for Voctoweb
  • maybe recycle some? code from

    voctopublish/voctopublish/api_client/youtube_client.py

    Lines 246 to 286 in a2f798b

    	def _select_tags(self, lang=None):
    	"""
    	Build the tag list
    	:param lang: if present the language will be added to the tags
    	:return: Returns an array of tag strings
    	"""
    	tags = []
    	# if tags are set - copy them into the metadata dict
    	if self.t.youtube_tags:
    	tags.extend(map(str.strip, self.t.youtube_tags.split(',')))
    	if self.t.track:
    	tags.append(self.t.track)
    	if self.t.day:
    	tags.append('Day %s' % self.t.day)
    	if self.t.room:
    	tags.append(self.t.room)
    	if lang:
    	if lang in self.lang_map.keys():
    	if self.t.languages[0] == lang:
    	tags.append(self.t.acronym + self.lang_map[lang])
    	tags.append(self.t.acronym + ' ov')
    	else:
    	tags.append(self.lang_map[lang] + ' (' + self.translation_strings[lang] + ')')
    	tags.append(self.t.acronym + ' ' + lang)
    	else:
    	raise YouTubeException('language not in lang map')
    	else:
    	tags.append(self.t.acronym + ' ov')
    	tags.append(self.t.acronym + ' ' + self.t.languages[0])
    	tags.extend(self.t.people)
    	tags.append(self.t.acronym)
    	logging.debug('YouTube Tags: ' + str(tags))
    	return tags

(extracted from #57)

As we always fail to clean up when an event is done we need a worker that services the "removing" state of the tracker. This should be a second worker running on the same host as the publishing worker and and also share some code.

• Service removing state
• Make sure to only remove master encoding files when all subformats are released
  • We may need / want tracker support for this
• Clean up encoded / tmp / capture / on the releasing host
• Clean up on the encoder machines / storage is out of scope for now

@a-tze @jjeising RFC

It can happen that events and links to recordings are listed on media.ccc.de but no recording files are present. The player is not working and files will 404.
People will notice this as a bug and lose confidence in the app.

According to mazdermind and atze we could use the state field to control visibility of events until the files are uploaded.

It would be nice for bigger conferences like the congress with multiple translations and a lot of talks to have playlists for each language. As a viewer on a mobile device it becomes really hard to find the desired video especially since we append the language at the end of the video title.

I think there should be a playlist for all languages and one with the original recordings.

33c3 Playlists (youtube)

For 33c3 I sorted them using a script afterwards but there should be an easier way, we could create the playlists in advance and auto-add the video according to the tag for the conference in combination with a tag for the language. e.g "34c3 German" and "34c3 OV". If youtube can't search for multiple tags in this feature we might use s.th. like "34c3_OV" I guess. youtube auto-add-playlist screenshot

It seems like the is already code that can do something similar but the uploaded videos do not have the tags.

One Example:

Additionally the "OV" feature is missing and the tag of translated versions is a bit verbose. Youtube Client Tag code: youtube_client.py#L377

There are only the tracker configured tags present:

At 33C3 we talked about the following solution for rerelease and deletion:

• Support deletion of talks via ready to remove. This is an explicit command.

• Always delete recordings and replace on rerelease. Update metadata when master ticket is rereleased.

A missing propertie currently results in error that is not reportet back to tracker as script assumes it has no valid ticket. This is not optimal because there are many cases where a ticket is "valid enough" to report back the problem but noch still misses a propertie.

This will be fixed when the current "http upload" branch is merged back as some groudnwork to handle this kind of errors is already done there

The OpenChaos in Cologne has been using paths of the form {conference root}/{year}/{month}/{filename}, i.e. lump all file formats together, but split them by year and month. The recording path for this case is {year}/{month}.
For 2017 we have decided to switch to {conference root}/{year}/{format}/{filename} to gain some consistency with other conferences. The recording path for this case is {year}/{format}.

To enable either workflow two additions to publishing would be needed:

1. Allow specifying a particle that is added to the recording path in front of the format. This should ideally be appended to the CDN path (Publishing.Media.Path) and prepended to the recording path (currently EncodingProfile.MirrorFolder).
2. Add the option to not add the format to the recording path, i.e. ignore EncodingProfile.MirrorFolder.

currently we just add here a default link. This should be the feedback link to frab (if frab is used) or if not e.g. the conference webpage

Seit dem EMF Camp und bei der Subscribe8 hab ich die Youtube Klasse um ein paar Funktionen erweitert: Man kann jetzt auch einfach aus dem Code Playlists anlegen oder Thumbnails updaten.
Vorschlag alle youtube-* Dateien in den Master-Branch übernehmen und den restlichen Code so anzupassen das er mit den neuen Youtube-Klasse (youtube-client.py) funktioniert.

As voctoweb supports it we should also support it in the publishing.

• pdf should be placed the same folder as the encoded videos
• pdf should be named .pdf
• we need an "encoding profile" to create the jobs for this.