bzip2 about avideo-storage HOT 18 CLOSED

what do you think will be the best format?

from avideo-storage.

@DanielnetoDotCom Why aren't you using the same as HLS ? the php-zip ?

from avideo-storage.

I think that you don't realize the issue with your configuration !

1- The video does encode
2- The video is zipped and sent to the streamer site .
3- The video does unzip on the streamer server .
4- The video is zipped once again with bzip2 compression ( on the streamer server )
5- The video is starting to be sent to YPT-Storage
6- The video is has started to be unzipped on YPT-Storage

Is this all necessary ? Huge wast of time and resources .

from avideo-storage.

@DanielnetoDotCom

I'm reopening this ,because this is madness . I mean rly .
Long streams takes longer to transfer than to encode ...

Can you change this type of compression ?? Why the HLS is transfered to streamer site with php-zip and from streamer to storage with gzip2 ?

from avideo-storage.

Agree, as long as I finish the project I am engaged in now, I will review this process.

from avideo-storage.

btw pigz is the best I've found , but I don't know how it manages the permissions . Can't confirm ( but uses multi-threading compression )

from avideo-storage.

pigz looks a good option, I will research about it.

from avideo-storage.

Hello after almost year. Since I got tired of this compression ratio and wait time, I decided to take a look at the code for once. I though that is was hardcoded on the plugin, but it seems it's just a letter that was killing my patience.

I'm personally using pigz , but I'm not pushing since users must first install it ( you'll need to add it on description of the plugin, and error log ). sudo apt-get install pigz . For now only gzip ( 1 core usage , but ~8 times faster than bzip2

from avideo-storage.

Question, does pigz also create a .tgz file?

for example can I use pigz to compress and tar to decompress?

from avideo-storage.

I found this

what do you think?

from avideo-storage.

@DanielnetoDotCom The one that I use is much simple; this:

$cmd = "tar -c --use-compress-program=pigz -f {$backupFile} -C {$backupDir} .";

Question, does pigz also create a .tgz file?

for example can I use pigz to compress and tar to decompress?

pigz = tar.gz = tgz ; just in parallele . In other words, tar uses only 1 core-cpu to compress , pigz uses all your core-cpu to compress. The more cores you have, the fastest it will compress the data.

But again, if we gonna make a change, we better not compress at all. We just use tar , without compression. Like this ;

tar -cf test.tar -C my_folder ; This will result in pure Disk IO ( in my case 550mb/s speed sata ssd )

from avideo-storage.

that is my dilema.

maybe if we compress a 10 GB file and the storages are in a different network a compression worth. otherwise just pack without compression.

from avideo-storage.

@DanielnetoDotCom

maybe if we compress a 10 GB file and the storages are in a different network a compression worth.

I'm not sure Daniel if you ever tested it. There's no compression. .ts files doesn't support compression
Example a folder of hls res720 = 1gb ; when you compress it it's = 1gb

Live example;

$ du -h res720/
767M	res720/

$ tar -c --use-compress-program=pigz -f res720.tgz -C res720/ .

du -h res720.tgz 
767M	res720.tgz

As you can see, we saved literally 0 mbs

Give it a try , and you will understand what I mean. We're wasting a lot of resources for nothing.

from avideo-storage.

Ok, you are right
so replace all for -cf shoud do the job correct?
this should reflect on Streamer, Encoder and YPTStorage

from avideo-storage.

Yes. That's why I've asked if you're using a specific line to decompress the tar, so I could test it and debug.

this should reflect on Streamer, Encoder and YPTStorage

Yes, this will speed up everything.

from avideo-storage.

That is all we have

I will remove the compress options from all commands, I guess it will improve a lot

from avideo-storage.

I sent updates to YPTStorage server and plugin

from avideo-storage.

That should do it.
So , I'll start testing once I can change the code locally

I will remove the compress options from all commands, I guess it will improve a lot

In my case improvement from bzip to no compression was over x80-100 times. Was compression at 5-10mb/s , now 550mb/s

from avideo-storage.