pozitronik / sinner Goto Github PK

1. BatchProcessingCore.suggest_output_path() doesn't applicable when path to a directory passed in --output-path.
2. If no --output-path is provided, the output path is generated without source path reference.

Check if it is possible to use multiple sources for swapping. Every source keeps a face with a certain position, and if the target face position is near to the face position,this face source should be used.

Implement inpainting to do better handling of items over the faces.

I definitely need an option to select which video handler backend should be used.

Show, at least, all applied parameters and all characteristics of target and source.

In GUI mode do a realtime player: get frames and the sound from the target, do in-memory frames swap (may be with some buffering) or any other possible operation, and play the result. Also add quality presets: to improve processing speed, frames can be shrunken before processing, and restored in size after.
Also it may be possible to enable frame drop: render as many frames as possible, and just drop any other.

• when wrong path is passed as source, gui produces an unhandled error
• GUI won't start without a provided target
• add quality control to change processing image scale on the fly. Also show last/median frame processing time
• use pygame.display to show rendered frame as fast, as possible
• implement pygame display resize stuff
• Do not resize display on input frame change, resize the frame instead
• combine with WebCam module, there's no need to multiply entities
• fix/check behavior with image target
• add a pause button to stop player only (continue background buffering) there's no need due the current design
• add frame rotation controls
• fix issue with long processing restart on position change
• move bootstrap_frames() to a background thread, to not to block program. Get frames right from target, if they are not bootstrapped.
• Remember and restore previous windows positions on the screen
• Add "stay on top" control/option
• Move frames extracting to the separate thread (make another queue). The experiment was unsuccessful: there's no profit to buffer all frames, because the next required frame index is unknown, while playing. The feature is postponed.
• fix the issue with broken frame navigation while playing
• need to try to implement new control to edit frame processors set and order
• show gui progressbar on long operations (buffering, extracting, etc)
• sound
• documentation

Known issues:

• The PygameFramePlayer may not refresh when showing frame while playing (e.g. shows two frames at same time)

Add a button, that will add current preview parameters to a certain batch list.

Currently, if there is some gap in processed frames, sinner will not handle it: final state check will report about some inconsistency, but lost frames are ignored on restart anyway.

It is about the case, when a task was stopped, and already processed frames deleted by user (as example, extracted frames when they are already swapped, and now enhancing in progress). Sinner will reprocess those frames again on the next task run, although they are not required anymore.
There is required some lookahead procedure to determine, what processor is running now. It is simple: find last state folder with files, look, what frames are still unprocessed, and prepare only them (if they're not exists).

Add something like --intervals=(start1, stop2) (start1, stop2) , where start and stop are integer markers of frame intervals to process.

As example, run FaceSwapper on the 16 CPU threads, and FaceEnhancer on the 2 CUDA threads.

Allow to pass multiple sources for processing, which should result to process target for the every source repeatedly.
In the GUI mode multiple sources may display as a list of images, and preview result should render for everyone of them, resulting to a list of previews.

If all benchmark runs has a small delta, the benchmarking process may newer stop. It will be better to measure delta between the fastest and the slowest runs.

It seems that when source FPS is fractionary (e.g. like 29.97004) and target FPS is the same, any h264/h265 encoded videos have the same problem with broken rewind in the end part of the target video.
The solution is to round up FPS values by default.

Every module should be able to handle it own set of command-line parameters. Those parameters should be gathered before program start. Every module also should be able to validate parameters by itself.

In the current processing logic all frame processors runs in turns, one after one. The advantages of this logic are obvious: it is simple, there is a possibility to stop and resume processing at any time, e.t.c.
But there are disadvantages too, and they seems to be critical at this moment. There is no possibility to implement any realtime preview with this logic, and there is no place for speed enhancements, because we can't parallelize processing task.
So, there is a big (really big) task to implement multithreading there.

There will be named buffers, each buffer named to a linked processor. Each processor processing will start in the separate thread, it will read a frame from the input buffer, process it, and write to the output buffer (which is input buffer for the next processor).

There are subtasks for this task:

• Implement the buffered processing chain.
• Make *FrameBuffer classes for the different buffers types (memory, file, e.t.c).
• Implement threads manager. Some processors may work faster than others, so their buffers will fill quickly. So it make a sense to pause those processors threads to redistribute computing powers to the other threads.
• Implement states in this approach. Saving and loading buffers as a files can be useful: this logic partially coincides with the current state logic.
• #68
• Implement a memory control code. It is important, because several loaded models may require a lot of memory, and frame buffers are too.
• #70
• Show the general progress bar and the every processor progress bar and statistics (memory usage, buffers occupancy, e.t.c).
• Properly handle in-threads exceptions.

tkinter sucks so everybody is migrating to gradio (original roop & roop unleashed)

your fork is the most OOP so migration should be better

When processing directory with --extract-frames parameter, there is no processing, because DirectoryHandler tries to process self.state.in_dir which is not applicable for this type of input.

I want to store all previously selected target/source directories in the config file to use them in the file dialogs.

I definitely want to add more control flags and ini options for ffmpeg, because now ffmpeg resulting can produce unoptimal and, sometimes, broken, videos.

Let user choose what faces should be processed.

subj

There is a collision in the case, when two different videos have the same source file. It produces the same temporary directory name for the FaceEnhancer processor. The solution: the target (not the source) file name should be used in that case somehow.

Originally posted by @pozitronik in #28 (comment)

Implement support to read module setting from the current config file, like

[FaceSwapper]
execution-provider=cpu
threads=16

[FaceEnhancer]
execution-provider=cuda
threads=2

as addition to current global configurations (config file and command line)

Add a control to the preview picture, that allow to compare original and processed frame.

Just to make sure, that all is ok.

Instead using frame handler, when frames extraction is required, it is possible to create FrameExtractor processor and use it in the processing chain.
It allow to make code more clean.

When trying to use 'cara.jpg' as a source.
The thing is in the image itself, not in the format (it can be any).

It is possible to use different set of processors to make the quick preview (as example: halved image and no enhancing). The full processing can be done by request.

It seems, that sometimes the frame count of some videos still incorrect, leads to unpredictable errors.
In one case, frame count of the video is 24750 (according both to ffprobe or cv2), but only 24745 were extracted for swapping. FaceEnhancer, somehow, sees 27844 frames, which is wrong in any way. Here can be a bug in the state module.
Also, for this video resulting frame count determined as 360.0, which also isn't right.

Add support to config, that will store default parameters for every run, if those parameters are not overlapped in command line.

Due to changed logic, there is no source_path in Core anymore.
Also, the source (and maybe the target) can be replaced in the current processors without full reloading.

There are different usage scenarios, and most of them are stateless. In fact, states are required only within batch processing chain, but state object is heavily integrated into BaseFrameProcessor class. The same is applicable for data entries (source/input/output files/directories/etc).
So, i have to:

1. Move out states from BaseFrameProcessor to some kind of top-level handler (I see it in rewritten Core).
2. Make more general FrameProcessors to work only with frames, not files.

Preview control should be properly sized, and support windows resize as well

It is possible to run some processing instances on CUDAExecutionProvider and some on CPUExecutionProvider. In theory, it can utilize GPU and CPU and increase overall speed.

Add a frame processor, which will receive frames from camera (via cv2.VideoCapture).

When face in the target is rotated, FaceSwapper result on it is bad. So, it is possible to counter rotate the frame before swapping, and rotate it back after, to achieve better swapping.

Currently, all processed frames are renamed by their numerical positions. It is okay for video frames, but in other cases processed files should be saved with their original names.

Before create resulting video and deleting processed frames, it is need to check frames validity: if are there all frames in the sequence and all frames are not zero-sized.

It seems that current implementation is really counting all frames, one by one (maybe, only when no appropriate value is in video header). It may lasts for minutes.
I need to find a way to get frames count in faster manner (may be use a binary search approach, as in C2VideoHandler).
Or i just can implement a workaround in VideoHandler: use detect_fc() from C2VideoHandler and result() from FFmpegVideoHandler.

It will be nice to have the some graphical interface.

It turns out that variables, initialized with values right in the class declaration will be initialized on the class module import, not on the class creation. It may cause issues, so it is better to move initialization to constructors everywhere.

In some cases processing must be stopped, to prevent waste of power. As example, if no space left on device.

State always creates path directory inside the attribute getter, even if path will be changed later. It is not a big deal, but i want to find a clever solution.

Every time DirectoryHandler rescan the whole files list, which is slow.

Add the benchmark mode, which runs the every frame processor with every allowed execution provider, varying execution thread count. This mode should provide the user with the best variation of provider and threads for future usage.

• State class needs tests
• FaceAnalyser class needs tests

stdout redirection sometimes causes problems (it seems that previous stdout is not restoring somehow)

Originally posted by @pozitronik in #71 (comment)

Add a sources list with previews of images from the certain directory to quick switch between.

• Add help strings to the every loadable attribute.
• Add -h help to show help output. It requires to collect all available help data somehow. As the first approach, it requires to find all descendants of the AttributeLoader class.
• Fix spacing in errors output.
• Do not show attributes without help, and modules without attributes in help output.
• Fix doubling help strings in the help output