bavc / signalserver Goto Github PK

View Code? Open in Web Editor NEW

19.0 19.0 6.0 1.7 MB

License: MIT License

Python 41.47% HTML 21.42% Shell 0.29% JavaScript 2.63% CSS 34.19%

signalserver's People

Contributors

Stargazers

Watchers

Forkers

yayoiukai metacynicv2 lbjohnson01 iamdamosuzuki fr33ky igamemedia

signalserver's Issues

Rename policy button leads to error

The rename option raises a FieldError at /policy/rename with Cannot resolve keyword 'policy_name' into field. Choices are: filename, id, process, process_id, row, status, task_id

LICENSE.md

Repo needs an open source license declaration!

show info and edit ability per rule in policies

On the "Add/Delete Filter on Policies" as rules are edited some value doesn't show. For instance if saying that psnr.Y exceeds a cutoff number of 1. The '1' doesn't appear after the rule is supplied. There should be a way to view and edit existing rules of a policy (so far they can only be deleted).

Also Filter2 should only be shown contextually if average-difference is selected, else it has no meaning.

in quickstart add a check for dependency

could be more elegant

FILES_VOLUME_PATH=/files ./quickstart.sh
./quickstart.sh: line 2: docker-machine: command not found
./quickstart.sh: line 3: docker-machine: command not found
./quickstart.sh: line 4: docker-machine: command not found
./quickstart.sh: line 5: docker-machine: command not found
./quickstart.sh: line 6: docker-compose: command not found
./quickstart.sh: line 7: docker-compose: command not found
./quickstart.sh: line 11: docker-compose: command not found
./quickstart.sh: line 11: docker-compose: command not found
./quickstart.sh: line 11: docker-compose: command not found
./quickstart.sh: line 11: docker-compose: command not found

store upload timestamp

I suggest storing the upload timestamp of new files supplied to the database and showing that timestamp on the file report.

create mockup for file view

Show file's record, along with what summarizations have been made and the resulting output of each summarization

create group at point of upload

creating a group is a multi-step process: upload individual files, search for individual files that you just uploaded, create group

suggesting a more simplified approach:

choose to upload either a directory or individual files (as in MediaConch)
at the prompt, choose if you want to create a new group or add to an existing group

possibly the name of the directory can default as the new group name

supply method to edit/reapply evaluation

provide a ui so that the user may create, edit, delete, import, export evaluation criteria by selecting from a list of unique metadata keys (BRNG, TOUT, etc) and an operator such as average, count of frames greater than X (or equal to, less than, etc),

localhost doesn't seem to be updating correctly

When I run the localhost version, I cannot click the file button in the navigation bar. Because of this, I'm unable to do any to do any testing on the most recent version of signalserver.

When I make updates in my local repo, the updates push correctly to the remote repo, but the updates are not seen in localhost:8000.

Is there a step missing in the docker instructions? Or maybe an update?

Kelly

add format and stream metadata

QCTools will soon be able to push <streams> and <format> data to signalserver.

Here is an example:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Created by QCTools 0.6.0 -->
<ffprobe:ffprobe xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ffprobe='http://www.ffmpeg.org/schema/ffprobe' xsi:schemaLocation='http://www.ffmpeg.org/schema/ffprobe ffprobe.xsd'>
    <program_version version="git-2017-02-05-e57fd92" copyright="Copyright (c) 2007-2017 the FFmpeg developers" build_date="Feb  8 2017" build_time="20:52:25" compiler_ident="not available" configuration="not available"/>
    <library_versions>
        <library_version name="libavutil" major="55" minor="46" micro="100" version="3616356" ident="Lavu55.46.100"/>
        <library_version name="libavcodec" major="57" minor="75" micro="100" version="3754852" ident="Lavc57.75.100"/>
        <library_version name="libavformat" major="57" minor="66" micro="101" version="3752549" ident="Lavf57.66.101"/>
        <library_version name="libavfilter" major="6" minor="73" micro="100" version="412004" ident="Lavfi6.73.100"/>
        <library_version name="libswscale" major="4" minor="3" micro="101" version="263013" ident="SwS4.3.101"/>
    </library_versions>
    <frames>
        <frame media_type="video" stream_index="0" key_frame="1" pkt_pts="0" pkt_pts_time="0.0000000" pkt_duration_time="0.0333667" pkt_pos="36" pkt_size="337252" width="720" height="486" pix_fmt="yuv422p10le" pict_type="I">
            <tag key="lavfi.signalstats.YMIN" value="4"/>
            <tag key="lavfi.signalstats.YLOW" value="119"/>
            <tag key="lavfi.signalstats.YAVG" value="235.387"/>
            <tag key="lavfi.signalstats.YHIGH" value="460"/>
            <tag key="lavfi.signalstats.YMAX" value="861"/>
            <tag key="lavfi.signalstats.UMIN" value="368"/>
            <tag key="lavfi.signalstats.ULOW" value="471"/>
            <tag key="lavfi.signalstats.UAVG" value="500.972"/>
            <tag key="lavfi.signalstats.UHIGH" value="515"/>
            <tag key="lavfi.signalstats.UMAX" value="578"/>
            <tag key="lavfi.signalstats.VMIN" value="356"/>
            <tag key="lavfi.signalstats.VLOW" value="415"/>
            <tag key="lavfi.signalstats.VAVG" value="485.547"/>
            <tag key="lavfi.signalstats.VHIGH" value="513"/>
            <tag key="lavfi.signalstats.VMAX" value="530"/>
            <tag key="lavfi.signalstats.VDIF" value="0"/>
            <tag key="lavfi.signalstats.UDIF" value="0"/>
            <tag key="lavfi.signalstats.YDIF" value="0"/>
            <tag key="lavfi.signalstats.SATMIN" value="0"/>
            <tag key="lavfi.signalstats.SATLOW" value="2"/>
            <tag key="lavfi.signalstats.SATAVG" value="30.3061"/>
            <tag key="lavfi.signalstats.SATHIGH" value="108"/>
            <tag key="lavfi.signalstats.SATMAX" value="182"/>
            <tag key="lavfi.signalstats.HUEMED" value="44"/>
            <tag key="lavfi.signalstats.HUEAVG" value="136.591"/>
            <tag key="lavfi.signalstats.TOUT" value="0.00519547"/>
            <tag key="lavfi.signalstats.VREP" value="0"/>
            <tag key="lavfi.signalstats.BRNG" value="0.0266918"/>
            <tag key="lavfi.cropdetect.x1" value="0"/>
            <tag key="lavfi.cropdetect.x2" value="720"/>
            <tag key="lavfi.cropdetect.y1" value="0"/>
            <tag key="lavfi.cropdetect.y2" value="486"/>
            <tag key="lavfi.cropdetect.w" value="720"/>
            <tag key="lavfi.cropdetect.h" value="486"/>
            <tag key="lavfi.psnr.mse.v" value="0"/>
            <tag key="lavfi.psnr.mse.u" value="0"/>
            <tag key="lavfi.psnr.mse.y" value="0"/>
            <tag key="lavfi.psnr.psnr.v" value="0"/>
            <tag key="lavfi.psnr.psnr.u" value="0"/>
            <tag key="lavfi.psnr.psnr.y" value="0"/>
            <tag key="lavfi.ssim.All" value="0"/>
            <tag key="lavfi.ssim.V" value="0"/>
            <tag key="lavfi.ssim.U" value="0"/>
            <tag key="lavfi.ssim.Y" value="0"/>
            <tag key="lavfi.idet.single.bff" value="0"/>
            <tag key="lavfi.idet.single.tff" value="0"/>
            <tag key="lavfi.idet.single.progressive" value="0"/>
            <tag key="lavfi.idet.single.undetermined" value="0"/>
            <tag key="lavfi.idet.multiple.bff" value="0"/>
            <tag key="lavfi.idet.multiple.tff" value="0"/>
            <tag key="lavfi.idet.multiple.progressive" value="0"/>
            <tag key="lavfi.idet.multiple.undetermined" value="0"/>
            <tag key="lavfi.idet.repeated.bottom" value="0"/>
            <tag key="lavfi.idet.repeated.top" value="0"/>
            <tag key="lavfi.idet.repeated.neither" value="0"/>
            <tag key="pkt_duration_time" value="0.0333667"/>
            <tag key="pkt_size" value="337252"/>
        </frame>
        <frame media_type="video" stream_index="0" key_frame="0" pkt_pts="33.3667" pkt_pts_time="0.0333667" pkt_duration_time="0.0333667" pkt_pos="349576" pkt_size="336151" width="720" height="486" pix_fmt="yuv422p10le" pict_type="I">
            <tag key="lavfi.signalstats.YMIN" value="4"/>
            <tag key="lavfi.signalstats.YLOW" value="119"/>
            <tag key="lavfi.signalstats.YAVG" value="235.289"/>
            <tag key="lavfi.signalstats.YHIGH" value="459"/>
            <tag key="lavfi.signalstats.YMAX" value="874"/>
            <tag key="lavfi.signalstats.UMIN" value="376"/>
            <tag key="lavfi.signalstats.ULOW" value="471"/>
            <tag key="lavfi.signalstats.UAVG" value="500.745"/>
            <tag key="lavfi.signalstats.UHIGH" value="515"/>
            <tag key="lavfi.signalstats.UMAX" value="579"/>
            <tag key="lavfi.signalstats.VMIN" value="358"/>
            <tag key="lavfi.signalstats.VLOW" value="416"/>
            <tag key="lavfi.signalstats.VAVG" value="485.555"/>
            <tag key="lavfi.signalstats.VHIGH" value="513"/>
            <tag key="lavfi.signalstats.VMAX" value="539"/>
            <tag key="lavfi.signalstats.VDIF" value="2.93949"/>
            <tag key="lavfi.signalstats.UDIF" value="3.89837"/>
            <tag key="lavfi.signalstats.YDIF" value="7.6502"/>
            <tag key="lavfi.signalstats.SATMIN" value="0"/>
            <tag key="lavfi.signalstats.SATLOW" value="2"/>
            <tag key="lavfi.signalstats.SATAVG" value="30.3544"/>
            <tag key="lavfi.signalstats.SATHIGH" value="108"/>
            <tag key="lavfi.signalstats.SATMAX" value="175"/>
            <tag key="lavfi.signalstats.HUEMED" value="44"/>
            <tag key="lavfi.signalstats.HUEAVG" value="132.945"/>
            <tag key="lavfi.signalstats.TOUT" value="0.0051126"/>
            <tag key="lavfi.signalstats.VREP" value="0"/>
            <tag key="lavfi.signalstats.BRNG" value="0.0265947"/>
            <tag key="lavfi.cropdetect.x1" value="0"/>
            <tag key="lavfi.cropdetect.x2" value="720"/>
            <tag key="lavfi.cropdetect.y1" value="0"/>
            <tag key="lavfi.cropdetect.y2" value="486"/>
            <tag key="lavfi.cropdetect.w" value="720"/>
            <tag key="lavfi.cropdetect.h" value="486"/>
            <tag key="lavfi.psnr.mse.v" value="0"/>
            <tag key="lavfi.psnr.mse.u" value="0"/>
            <tag key="lavfi.psnr.mse.y" value="0"/>
            <tag key="lavfi.psnr.psnr.v" value="0"/>
            <tag key="lavfi.psnr.psnr.u" value="0"/>
            <tag key="lavfi.psnr.psnr.y" value="0"/>
            <tag key="lavfi.ssim.All" value="0"/>
            <tag key="lavfi.ssim.V" value="0"/>
            <tag key="lavfi.ssim.U" value="0"/>
            <tag key="lavfi.ssim.Y" value="0"/>
            <tag key="lavfi.idet.single.bff" value="0"/>
            <tag key="lavfi.idet.single.tff" value="0"/>
            <tag key="lavfi.idet.single.progressive" value="0"/>
            <tag key="lavfi.idet.single.undetermined" value="0"/>
            <tag key="lavfi.idet.multiple.bff" value="0"/>
            <tag key="lavfi.idet.multiple.tff" value="0"/>
            <tag key="lavfi.idet.multiple.progressive" value="0"/>
            <tag key="lavfi.idet.multiple.undetermined" value="0"/>
            <tag key="lavfi.idet.repeated.bottom" value="0"/>
            <tag key="lavfi.idet.repeated.top" value="0"/>
            <tag key="lavfi.idet.repeated.neither" value="0"/>
            <tag key="pkt_duration_time" value="0.0333667"/>
            <tag key="pkt_size" value="336151"/>
        </frame>
    </frames>
    <streams>
        <stream index="0" codec_name="ffv1" codec_long_name="FFmpeg video codec #1" codec_type="video" codec_time_base="100/2997" codec_tag_string="FFV1" codec_tag="0x31564646" width="720" height="486" coded_width="720" coded_height="486" has_b_frames="0" sample_aspect_ratio="10:11" display_aspect_ratio="400:297" pix_fmt="yuv422p10le" level="-99" field_order="bt" refs="1" r_frame_rate="2997/100" avg_frame_rate="2997/100" time_base="1/11988" start_pts="0" start_time="0.000000">
            <disposition default="1" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0" timed_thumbnails="0"/>
            <tag key="language" value="eng"/>
            <tag key="handler_name" value="DataHandler"/>
            <tag key="encoder" value="ffv1"/>
        </stream>
        <stream index="1" codec_name="pcm_s24le" codec_long_name="PCM signed 24-bit little-endian" codec_type="audio" codec_time_base="1/48000" codec_tag_string="in24" codec_tag="0x34326e69" sample_fmt="s32" sample_rate="48000" channels="2" channel_layout="stereo" bits_per_sample="24" r_frame_rate="0/0" avg_frame_rate="0/0" time_base="1/48000" start_pts="0" start_time="0.000000" bits_per_raw_sample="24">
            <disposition default="1" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0" timed_thumbnails="0"/>
            <tag key="language" value="eng"/>
            <tag key="handler_name" value="DataHandler"/>
        </stream>
    </streams>
    <format filename="/Users/davidrice/Desktop/Dave/qctools_dhc_samples/ffv1/DHC0386_Ghosting.mov_ffv1.mov" nb_streams="2" nb_programs="0" format_name="mov,mp4,m4a,3gp,3g2,mj2" format_long_name="QuickTime / MOV" start_time="0.000000" duration="10.678000" size="111637099" bit_rate="83638957" probe_score="100">
        <tag key="major_brand" value="qt  "/>
        <tag key="minor_version" value="512"/>
        <tag key="compatible_brands" value="qt  "/>
        <tag key="encoder" value="Lavf55.13.102"/>
    </format>
</ffprobe:ffprobe>

This provides much more info about the file as a whole than we had before (where it had been mostly about frames).

I think we need a design of what data to use and how to present it and then to integrate it into a file view. I suggest these values are relevant to include in display:

stream/@codec_name
stream/@codec_type
stream/@width (video)
stream/@height (video)
stream/@sample_aspect_ratio (video)
stream/@display_aspect_ratio (video)
stream/@pix_fmt (video)
stream/@field_order (video)
stream/@avg_frame_rate (video)
stream/@sample_rate (audio)
stream/@bits_per_raw_sample (audio)
stream/@channels (audio)
format/@filename
format/@format_log_name
format/@duration
format/@SiZe
format/@bit_rate

line breaks in reports

With this sample there are line breaks in the resulting report:
6_DHC36.mkv.qctools.xml.gz

But with this sample there are no line breaks:
1A_seattle_parade_transfer_a.qctools.xml.gz

edit button might be unnecessary

On http://localhost:8000/policy/show/1, I note that each rule has an edit form underneath it but then also includes an 'edit' button, but these 'edit' buttons all just lead back to http://localhost:8000/policy/show/1. So are they needed here?

dynamically add filters to policy test config based on incoming qctools data

Presently the dropdowns list frame data from lavfi.cropdetect, lavfi,rebur128, etc, but when new qctools is uploading with new values, the list doesn't update, so it is not possible to make a configuration for metadata from qctools with new values. The filter dropdown on "Add/Delete Filter on Policies" should either update according to supplied data or have some place where the lavfi terms may be registered.

Running locally with ./quickstart throws repeating error

Here is the warning and error:

WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
Creating volume "signalserver_dbdata" with default driver
Creating volume "signalserver_files" with default driver
ERROR: create .: volume name invalid: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed

policy buttons

The buttons on the Policy page are a little wonky. There are two rename buttons, and they surround the rename field in an odd way. But I'm thinking it's better to just list the policies and their descriptions with an edit button. When they click through to edit, then they can rename, reset, or delete. Kinda like:

And maybe also include a dropdown menu of all the groups so

Dancing Banana | Flags files with high BRNG metrics | edit | group to process

Thoughts? Bueller?

Rewrite language for policy percentages or remove them all together (for now)

This is totally up for debate, especially since you guys know better than me about the functionalities of signalserver.

Try as we might the two features of policies below are near impossible for us (me) to understand, let alone use or explain.

Percentage for the Group Process reporting**
Percentage for the File Process reporting***

I am wondering if Ashley or Dave or a savvy third party can re-write the description as if they will be read by people who haven't thought about math since the 5th grade (me).

But I'm also wondering if we can take these features out for now and revisit them at a later date, perhaps after the tester workshop.

page to show/edit policy fails if policy name includes space

For instance: http://52.34.188.9:8000/operations/show/Policy%20with%20Spaces gives an error page, but http://52.34.188.9:8000/operations/show/Digibeta_Tests is ok.

policy rule "edit" button redirects to top of page

When clicking "edit" on any rule, it refreshes the whole page and sets it at the top, it'd be nicer if it just popped up instead!

More than one upload of same XML throws error

It returns...

"MultipleObjectsReturned at /videostorage/list/show/test.mkv.qctools
get() returned more than one Video -- it returned 4!"

(etc etc etc)

cannot add # when editing policy

When editing a rule in a policy, if the user had picked 'average' but then changes it to a something more specific requiring a number, that box does not appear so the user has to edit, change, save, edit again, and then add the number. It should be streamlined so it appears as needed.

create mockup for configuration editor

Allow create, edit, delete of summarization configurations.

For each summarization allow create, edit, delete of rules of the summarization (such as average of YDIF, or count of frames with BRNG over 77). I'm reminded of the MediaConch policy editor for this.

use a default value for

If running quickstart.sh without declaring FILES_VOLUME_PATH, then the output is like:

WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
WARNING: The FILES_VOLUME_PATH variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service web: create .: "." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed

Could ./quickstart.sh just use /files unless the global var is already defined. That way we'd only have to define it if we want to overwrite the default value.

Wishlist: window message with status bar while files are uploading

It took several minutes to upload 16 QCtools files. Maybe we could have a window with a fun status bar display while files are uploading? Like this:

Add logo styled in white, sized and exported for web

For dark backgrounds!

add section on docker maintenance

Suggest adding a section to readme on cleanup and maintenace of docker:

from debugging earlier we used.

docker ps -a -q | xargs docker rm -f
docker images -a -q | wc -l
docker images -a -q | xargs docker rmi -f

update readme to document setup and installation

Wishlist: Public Policy Sharing?

Any thoughts about a public policy sharing feature, ala Media Conch?

https://mediaarea.net/MediaConchOnline/policyEditor

HTTPS support

Nowadays, communication between client and server is sensitive, and a server should accept HTTPS communications, especially if some sensitive information is transported (e.g. a password, users use to use the same password everywhere so scanning the HTTP signalserver communication would permit e.g. an attacker to access the email account of the user).

Related to bavc/qctools#242 (comment)

some imagery gets cut off

create mockup for table view

Show file records in table to allow comparative evaluation.
Sortable per column.
Columns are defined by the applied summarization.

error on password reset

I went to http://localhost:8000/register/ to register an account with an email. Then clicked on http://localhost:8000/password_reset/ to reset the password but get an error:

SMTPSenderRefused at /password_reset/

(530, b'5.5.1 Authentication Required. Learn more at\n5.5.1  https://support.google.com/mail/?p=WantAuthError q3sm8571800qte.0 - gsmtp', '[email protected]')

Request Method: 	POST
Request URL: 	http://localhost:8000/password_reset/
Django Version: 	1.10.5
Exception Type: 	SMTPSenderRefused
Exception Value: 	

(530, b'5.5.1 Authentication Required. Learn more at\n5.5.1  https://support.google.com/mail/?p=WantAuthError q3sm8571800qte.0 - gsmtp', '[email protected]')

Exception Location: 	/usr/local/lib/python3.4/smtplib.py in sendmail, line 783
Python Executable: 	/usr/local/bin/python
Python Version: 	3.4.5
Python Path: 	

['/usr/src/app',
 '/usr/local/lib/python34.zip',
 '/usr/local/lib/python3.4',
 '/usr/local/lib/python3.4/plat-linux',
 '/usr/local/lib/python3.4/lib-dynload',
 '/usr/local/lib/python3.4/site-packages',
 '/usr/src/app']

Server time: 	Thu, 9 Feb 2017 04:05:55 -0500

support/document programmatic method to upload qctools reports

I'm not sure if this is currently possible, if so let's document it, if not let's implement it.

click through file names on group report and go to individual file report

extra base.html file

Hey @yayoiukai

I see fileuploads/templates/base.html and it doesn't seem to be called anywhere -- seems to be the same as the higher-level base.html. Can I remove it?

404 Page Error opens on file deletion

When I try to delete a file in the File Uploads list page (http://52.34.188.9:8000/fileuploads/list/) I get a "page not found" error message

timestamps should be adjusted to local time

presently if seems that timestamps are UTC

Either local time of server or of client is fine.

sort filter names in policy editor

On the policy editor page ("Add/Delete Filter on Policies" ), please alphabetically sort the filter names.

extra 'Rename' button on policy page

On http://localhost:8000/policy/, each policy provides a text box to rename the policy and then there is a "Rename" button above that box which doesn't seem to do anything, and one rename button to the right of the text box which applies the rename.

Support of tokens instead of password only

Currently the API supports only a password for credentials, and clients have to store the password on their side if they don't want to ask the user to enter the password each time they access the server.
This may be an issue if the computer of the user is compromised, as the password would be retrieved.
A best practice nowadays is to store a token instead of the password on the client side, so the password is transmitted only once and not stored on the client machine (only the token is stored, and can be revoked if the machine is compromised, without providing the password)