alfredosequeida / fvid Goto Github PK

Okay, we need to address the elephant in the room - the intended use of the fvid module - you encode some file and put a video on youtube. In theory, it's okay, but.

If you are to upload a video on youtube it'll change the framerate to 60. So, say, you download the video by using, let say, youtube-dl and try to decode it - if you've used another framerate, you get a different number of images and an error of unzipping.

You can check it yourself - https://www.youtube.com/watch?v=JFL2vSxVzsU (if you don't have youtube-dl or something like this, just change youtube to youtubepp and you will be redirected to some site on which you can download 1080p video)

 py -m fvid -i pap_encoded_youtube-1-5.mp4 -d -o pap_decoded.mobi` 
<...>
frame=  240 fps=1.5 q=-0.0 Lsize=N/A time=00:00:40.00 bitrate=N/A speed=0.243x
<...>

in the original was 8 frames.

But, if I change a framerate to default 1/5:
ffmpeg -i pap_encoded_youtube.mp4 -filter:v fps=fps=1/5 pap_encoded_youtube-1-5.mp4
and use video instead:
py -m fvid -i pap_encoded_youtube-1-5.mp4 -d -o pap_decoded.mobi
I get a decoded file.

What I propose. Since -f 60 doesn't work, can we change the framerate from 1/5 to 1 in order to somewhat reduce the filesize (I assume 1 will work for any file and system, something like 5 - I don't really now - something like 6 and higher works for some files but not for all) and hardcode it in code and thus getting rid of -f flag.
After that, we add
ffmpeg -i from_youtube.mp4 -filter:v fps=fps=some_frame_rate_we_will_agree_upon temp_file_to_decode.mp4
in code and all will be working as intended.

Dear dev,

the fvid command is not recognized after installing via pip3
command not found: fvid

Any advice?
Thank you

Okay, folks, I manage to make a very rudimentary GUI for this #21 version of fvid

import PySimpleGUI as sg
import os
import sys

class MissingArgument(Exception):
    pass


layout = [
    [sg.Button('Password', key='popup_pass', size=(20, 1))],
    [sg.Button('File to de/encode', key='popup_file', size=(20, 1))],
    [sg.Radio('Encoding', 1, enable_events=True, default=True, key='R1'), sg.Radio('Decoding',1, enable_events=True, key='R2')],
    [sg.Button('Start', key='start', size=(20, 1))],
]
window = sg.Window('Simple fvid GUI', layout, font='a 16')

password = None
line = None 

while True:  # Event Loop
    event, values = window.Read()

    if event in [None ,'Exit']:
            break

    if event == 'popup_pass':
        password = sg.popup_get_text('Password', password_char='*') 

    if event == 'popup_file':
        fname = sg.popup_get_file('File to open')
        try:
            fname2 = os.path.basename(fname)
        except:
            pass

    if event == 'start':
        try:
            fname = fname + ' '
        except:
            raise MissingArgument('You need a file to en/decrypt.')
        if password:
            pwd = '-p ' + password
        else:
            pwd = ''
            
        part1 = 'python -m fvid -i ' + fname

        if values['R1'] == True:
            part2 = '-e ' + pwd + ' -o ' + fname2 + '_encoded.mp4'
        else:
            part2 = '-d ' + pwd 


        line = part1 + part2

        break

if line:
    # if sys.platform == 'win32':
        # os.system('cls')
    # else:
        # os.system('clear')
    print('Command:', line)
    os.system(line)


window.Close()

This tool works great for small files, but trying to convert a relatively large file (1GB) will cause a MemoryError, even with 32 GB of RAM

@dobrosketchkun If you have time, do you have any ideas for how to fix the issues shown in #34? I'm not that good with pickle or such.

I'm trying to run fvid and after installing all the modules im getting this error.

C:\Users\Yakir Oren\Desktop\fvid>py fvid.py -i ./image.png -e

Traceback (most recent call last):
  File "fvid.py", line 275, in <module>
    make_video(video_file_path, image_sequence)
  File "fvid.py", line 200, in make_video
    ffmpeg.input(frames[0], loop=1, t=1).output(
  File "C:\Users\Yakir Oren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\ffmpeg\_run.py", line 313, in run
    process = run_async(
  File "C:\Users\Yakir Oren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\ffmpeg\_run.py", line 284, in run_async
    return subprocess.Popen(
  File "C:\Users\Yakir Oren\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\Yakir Oren\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

After running the command, it just goes to a new line. My code setup is the folder "fvid" extracted into downloads and the file I want to test for now is in the same folder.

C:\Users\windows\Downloads\fvid-master\fvid>py -m fvid -i Ppsspp.zip -e

C:\Users\windows\Downloads\fvid-master\fvid>

No new files are created.

@AlfredoSequeida There have been a lot of changes since the last release (nearly all within 3 weeks of 0.02), and I was wondering if you could publish a new release to PyPi? It would have to be 1.0.0, as this new release would have passwords, and therefore break backwards compatibility.

Hi,
I'm getting a syntax error when trying execute. Doesn't seem to be related to file formats... Can you help.

python fvid.py -i test.mp4 -e
File "fvid.py", line 90
f"{FRAMES_DIR}decoded_frames%03d.png"
^
SyntaxError: invalid syntax

python fvid.py -i image.jpeg -e
File "fvid.py", line 90
f"{FRAMES_DIR}decoded_frames%03d.png"
^
SyntaxError: invalid syntax

python fvid.py -i image2.png -e
File "fvid.py", line 90
f"{FRAMES_DIR}decoded_frames%03d.png"
^
SyntaxError: invalid syntax

python fvid.py -i test.txt -e
File "fvid.py", line 90
f"{FRAMES_DIR}decoded_frames%03d.png"
^
SyntaxError: invalid syntax

In the fvid.py code there is a Syntax Error in line 121,
for filename in sorted(glob.glob(f"{FRAMES_DIR}decoded_frames*.png"), key=os.path.getmtime) :
^
SyntaxError: invalid syntax

I don't know how to fix this, can you please advise?

I see you have no LICENSE file for this project. The default is copyright.

I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

fvid installed through pip3 to bring along dependencies.

Test file: 9.1Mb Photoshop document. Successfully encoded with fvid, uploaded to YouTube, downloaded with ytdl, then run through fvid's decoder. After seeing Bits are in place, fvid unpacks the frames 100%, but:

Unziping...

Traceback (most recent call last):

  File "/usr/local/bin/fvid", line 8, in <module>
   sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/fvid/fvid.py", line 274, in main
   save_bits_to_file(file_path, bits, key)
  File "/usr/local/lib/python3.9/site-packages/fvid/fvid.py", line 144, in save_bits_to_file
   bitstring = fo.read()
  File "/usr/local/Cellar/[email protected]/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/gzip.py", line 300, in read
    return self._buffer.read(size)
  File "/usr/local/Cellar/[email protected]/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/gzip.py", line 478, in read
    self._read_eof()
  File "/usr/local/Cellar/[email protected]/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/gzip.py", line 524, in _read_eof
    raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32),
gzip.BadGzipFile: CRC check failed 0xebd7fa1e != 0xc3ae18db

I notice that regardless of the command syntax, fvid run multiple times on the same input file generates MP4s of consistently varying byte counts suggesting the algorithm or one of the modules it depends on is flaky. This is consistent across the following syntaxes:

fvid -ei filename.ext
fvid --encode -i filename.ext
fvid -i filename.ext -e

file.mp4 came out as lengths of 159,351,426; 159,351,436; 159,351,435; 159,351,433 and the resulting downloaded YT videos had similar variances.

Whenever I run this script to decode a video it generated, the output file is file.bin. I tried renaming the extension to match the original file I encoded but it didn't work. Morevover, this file.bin is lighter than the original file.

I've just started to use fvid again, but:

E:\>fvid -i Documents.rar -e -o Documents.mp4
Traceback (most recent call last):
  File "c:\users\tran bich dung\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\tran bich dung\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Tran Bich Dung\AppData\Local\Programs\Python\Python39\Scripts\fvid.exe\__main__.py", line 4, in <module>
  File "c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages\fvid\fvid.py", line 16, in <module>
    import magic
ModuleNotFoundError: No module named 'magic'

I don't know what happened. I tried to install 'magic':

E:\>py -m pip install magic
ERROR: Could not find a version that satisfies the requirement magic
ERROR: No matching distribution found for magic

And upgrade fvid to the latest master branch:

E:\>py -m pip install --upgrade git+https://github.com/AlfredoSequeida/fvid
Collecting git+https://github.com/AlfredoSequeida/fvid
  Cloning https://github.com/AlfredoSequeida/fvid to c:\users\tran bich dung\appdata\local\temp\pip-req-build-zyhpx15q
  Running command git clone -q https://github.com/AlfredoSequeida/fvid 'C:\Users\Tran Bich Dung\AppData\Local\Temp\pip-req-build-zyhpx15q'
Requirement already satisfied: bitstring>=3.1.6 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from fvid==1.0.0) (3.1.7)
Requirement already satisfied: pillow>=7.2.0 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from fvid==1.0.0) (8.0.1)
Requirement already satisfied: tqdm>=4.49.0 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from fvid==1.0.0) (4.54.1)
Requirement already satisfied: cryptography>=3.1.1 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from fvid==1.0.0) (3.2.1)
Requirement already satisfied: pycryptodome>=3.9.8 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from fvid==1.0.0) (3.9.9)
Requirement already satisfied: six>=1.4.1 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from cryptography>=3.1.1->fvid==1.0.0) (1.15.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from cryptography>=3.1.1->fvid==1.0.0) (1.14.4)
Requirement already satisfied: pycparser in c:\users\tran bich dung\appdata\local\programs\python\python39\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=3.1.1->fvid==1.0.0) (2.20)
Building wheels for collected packages: fvid
  Building wheel for fvid (setup.py) ... done
  Created wheel for fvid: filename=fvid-1.0.0-py3-none-any.whl size=9930 sha256=23a8357e046bb26ea95a09bc2dd433ab5a1e4872a064e09f19ed726fba392229
  Stored in directory: C:\Users\Tran Bich Dung\AppData\Local\Temp\pip-ephem-wheel-cache-xf5q5eny\wheels\7d\29\2f\3abea42ac756b48077987d878a53a93f6ec3b1cee1c8560187
Successfully built fvid
Installing collected packages: fvid
  Attempting uninstall: fvid
    Found existing installation: fvid 1.0.1
    Uninstalling fvid-1.0.1:
      Successfully uninstalled fvid-1.0.1
Successfully installed fvid-1.0.0

But it still doesn't work...
Any help appreciated. Thanks!

I see that currently only 1 frame is changed per 5 seconds. I was wondering the reason behind this choice. Also is it possible to encode it to higher frames per second ?

Can someone help me with download? Where i need put FFmpeg folder and where i can download libmagic. Where are i need put my file to encode, and where i can find output?

Here is an issue I came across while testing today. If the input file name to decode has a space like this for example: file 1.mp4, I get this error:

Reading video...
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-shared --enable-version3
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
file_1.mp4: No such file or directory
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-shared --enable-version3
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
_temp.mp4: No such file or directory
Traceback (most recent call last):
  File "/home/alfredo/.local/bin/fvid", line 33, in <module>
    sys.exit(load_entry_point('fvid==1.0.0', 'console_scripts', 'fvid')())
  File "/home/alfredo/.local/lib/python3.9/site-packages/fvid/fvid.py", line 498, in main
    bits = get_bits_from_video(args.input)
  File "/home/alfredo/.local/lib/python3.9/site-packages/fvid/fvid.py", line 218, in get_bits_from_video
    os.remove(TEMPVIDEO)
FileNotFoundError: [Errno 2] No such file or directory: '_temp.mp4'

It looks like the error is happening because the file file 1.mp4 is being read in as file_1.mp4 and therefore fvid can't find the input file.

There should be a requirements.txt file as well as a "Dependendies" section in the Readme (#4) to make usage easier (rather than guessing from the code which packages/libraries to install).

I had ffmpeg in PATH and fvid still doesn't work somehow...

PS C:\Users\Tran Bich Dung> py -m fvid -i "[hSa] Daichan, Daisuki. [????, ????.] MOVIE.mp4" -e
Traceback (most recent call last):
  File "C:\Users\Tran Bich Dung\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Tran Bich Dung\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Tran Bich Dung\AppData\Local\Programs\Python\Python38\lib\site-packages\fvid\__main__.py", line 2, in <module>
    from fvid import main
ImportError: cannot import name 'main' from 'fvid' (C:\Users\Tran Bich Dung\AppData\Local\Programs\Python\Python38\lib\site-packages\fvid\__init__.py)
PS C:\Users\Tran Bich Dung> py -m pip install fvid
Requirement already satisfied: fvid in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (0.0.2)
Requirement already satisfied: bitstring in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (3.1.7)
Requirement already satisfied: python-magic in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (0.4.18)
Requirement already satisfied: pillow in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (8.0.1)
Requirement already satisfied: numpy in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (1.19.2)
Requirement already satisfied: tqdm in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (4.51.0)
Requirement already satisfied: ffmpeg-python in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from fvid) (0.2.0)
Requirement already satisfied: future in c:\users\tran bich dung\appdata\local\programs\python\python38\lib\site-packages (from ffmpeg-python->fvid) (0.18.2)
PS C:\Users\Tran Bich Dung>

• add an -h265 flag allowing users to encode/decode with that
• make the code try h265 first and default back to libx264rbg if that doesn't work
• or, bump fvid to 2.0.0 when/if this is done to signify that h264 encoded videos won't be decodable anymore

Thoughts? I'm learning towards the second option.

This project is definitely missing a Readme file to explain people what it does and how cool it is.
Here are some templates :

• https://github.com/dbader/readme-template
• https://github.com/othneildrew/Best-README-Template
• https://github.com/scottydocs/README-template.md

@dtaivpp @dobrosketchkun
Currently, we together have 4 different PRs open. Do you guys/gals think we should combine them into one PR to make it easier and faster for Alfredo to review? Plus, if we all have different versions of the code, there will be redundancies and missing pieces when it's all merged.

(Also, dobro, I'm actually still having an issue with your cryptography stuff, so if other people have the same issue yours may need to be left as a separate PR).

Hi everyone, I've been busy, which is why I haven't been able to check-in as often. So first of all I want to thank everyone for their contributions and a special thanks to @Theelgirl and @dobrosketchkun since I know you two have been putting in a lot of work into the program. Your work does not go unnoticed.

I have some ideas I want to share and get some feedback.

1. Increasing storage capacity.
   Originally, the program was made with the simple thought that 1-bit (black and white) pixels would allow the program to be more efficient at data retrieval against compression algorithms. But of course, using a single pixel to represent a bit, leaves a lot to be desired as far as maximizing storage goes. I think we can still achieve the same logic by adding the option for another type of encoding (in addition to 1-bit color) by using colors to represent a set of 2 bits per pixel thus doubling the storage capacity while still keeping the colors simple enough to guard against compression. Here is the math/logic:

Using the RGB color spectrum, the simplest colors are red, green, and blue. This means that they are easy to distinguish from one another even if compression changes them a bit.

With that in mind, using binary numbers we can double storage by storing 2 bits in a single color, thus giving us a possibility of 2^2 or 4 different combinations. Then we can assign a color to each combination and use black or white for the remaining color:

00: Black
10: Red
01: Green
11: Blue

Then as far as decoding goes, the logic would be the same as for black and white. We would check which color the values is closest to and assume that color:

For example, if the pixel is (255,12,30), then the color must be Red (bin: 10), since the pixel contains more red than anything else.

I haven't done the research yet, but we might also be able to take advantage of an alpha channel using the RGBA color spectrum, but I would assume that A might not be as easy to guard against compression.

2. Adding some sort of error checking /correction algorithm
   Even though, the current implementation of fvid seems to work the times I have tried it. It's not perfect and there have been some reports of it not working with certain files. Because of this, I think we should look into adding some error checking or correction. A simple implementation might be adding a parity check where we add an extra bit or set of bits for every byte (8 bits) if the number is odd or even. However, that doesn't fix the data, it only tells us that the data is wrong and that assumes that the data was decoded correctly, which are probably too many assumptions for it to be a good solution. So I am open to hearing if anyone has any suggestions on this.

3. GUI
   Recently, the program has been getting more attention (because of a TikTok I made lol) and I have seen more requests for a GUI. I know @dobrosketchkun made a GUI for the program, but I have not seen that implemented yet. I have not made a GUI for a Python program before, so I was doing some research and was thinking of building one using PyQT, or Kivy, but since @dobrosketchkun already did some work using Tkinter, I rather their work not go to waste. In addition, I think it would be cool to make this optional, so maybe during the install process or with a different package.

4. Changing the License
   MIT was the original option since that is the most open license I know and I like the idea of sharing the source code and allowing people to do whatever they want with the program, but I don't want the work contributed by others to be "taken advantage of" for lack of a better phrase. With the amount of time contributors have put into their work, I would like the program and any copies to remain open source. So I propose, we change the license to GPL V3, but of course I am open to suggestions.

Everything here is just a suggestion, I want to hear what others have to say.

I believe the possible optimizations (without a refactoring allowing for processing multiple pixels at a time) for the lengthy decode process are severely diminished now, and it's only twice the speed that it originally was. So, I am proposing that I make an optional Cython extension. Cython is a superset of Python that compiles to C, which compiles to machine code. If you want to take a look at what it looks like and how fast it is, check out the pybase122.pyx file in my pybase122 decoder in my starred repos. I included benchmarks for the best Python version I could write, and the best Cython version I could write (with some help). You'll notice that the Cython version I wrote ended up being at least 10x faster than the python version, and the version that I got help with is over 20x faster. This is a huge speedup. While this won't have as major a speedup because it uses less math/bit operations, I bet you that I can get at least a 4x speedup for the section inside the two for loops in the get_file_from_image (I think that's the name) function.

The main obstacle to including Cython is that it needs to be compiled with every change, and you need to pip install Cython to compile. However, luckily, it is possible to fall back to a Python version of the code should a user not want to install Cython or compile it. This is why I'm proposing that I only Cythonize the slow part inside the loop, and add a check for if the user's system supports Cython before it actually tries running the Cython code.

For an example of a Cython library, google "pomegranate Cython". Pomegranate is a machine learning library that is blazingly fast because it uses Cython to compile its algorithms. It's also open-source, on GitHub, so you can check the internals.

I'm marking this as a proposal because I don't currently want to do the work if you're not comfortable with it.