CPU is completely utilized about freyr-js HOT 12 CLOSED

Hi @MoroseCorpse, this is probably the fault of ffmpeg reencoding the audio stream gotten from YouTube into an AAC container for M4A audio

You should probably turn down the number of concurrent tasks for the encoder.

By default, (as at 6d8a90b see conf.json), freyr uses 6 concurrent ffmpeg instances to parallelize encoding multiple tracks at once.

You can override this with the -z, --concurrency <SPEC> flag, like this;

$ freyr -z encoder=1 <QUERY>

Or modify the conf.json file by changing the default 6 to 1 or whatever you are comfortable with.

{
  ...
  "concurrency": {
    "queries": 1,
    "tracks": 1,
    "trackStage": 6,
    "downloader": 4,
-   "encoder": 6,
+   "encoder": 1,
    "embedder": 10
  },
  ...
}

from freyr-js.

I have changed the encoder variable to 1 in the config but all 4 cores are still up to 100% utilization. Even when I write this -z encoder=1 in my command

from freyr-js.

Can you inspect this with htop or something similar to see which processes are hogging up the CPU?

from freyr-js.

from freyr-js.

Clearly, the culprit seems to be youtube-dl, which understandably does some work trying to extract audio feeds for the youtube sources

You can turn concurrency down on that with the trackStage=1 spec in the command or trackStage entry in the conf file.

from freyr-js.

If you want to combine these specs for the concurrency in the cli flags, you can merge them like this

$ freyr -z "encoder=1,trackStage=1" <QUERY>

or like this

$ freyr -z encoder=1 -z trackStage=1 <QUERY>

(as documented here)

$ freyr get --help | grep '\--concurrency' -A 4

-z, --concurrency <SPEC>     key-value concurrency pairs (repeatable and optionally `,`-separated)
                               (format: <[key=]value>) (key omission implies track concurrency)
                               (valid(key): queries,tracks,trackStage,downloader,encoder,embedder)
                               (example: `queries=2,downloader=4` processes 2 CLI queries,
                               downloads at most 4 tracks concurrently)

from freyr-js.

Okay thanks, that helped a lot. Are now "only" 50% utilization. But individual cores still pull up to 100%. If I give my VM only one core, it is completely overloaded. For my server this may not be so important, but I would like to intriegieren your tool in a project of me that should also work on worse servers.

I have now also everything already set to 1 and are still 100% utilized at a core:

freyr -z "encoder=1,trackStage=1,downloader=1,embedder=1" spotify:playlist:37i9dQZF1DXcBWIGoYBM5M

from freyr-js.

That seems quite problematic, please include a screenshot of htop on this run.

from freyr-js.

from freyr-js.

Well from that, it's clear that the brunt of the work is solely coming from youtube-dl and ffmpeg where one is sourcing feeds and the other is reencoding audio streams.

That seems justified, although youtube-dl does come as a shock seeing its computational load, surely there could be a better way.

Maybe #21 can help in that, by using youtube-dl as a library as opposed to individual processes.

from freyr-js.

Closing this for now, seeing as the issue isn't from freyr itself.

from freyr-js.

So I profiled the source for youtube-dl using scalene and it shows that most of the CPU work was spent simply reading from the HTTP response

• HEADER
  
• youtube_dl/utils.py (% of time = 89.23% out of 12.68s)
  ••• (source)
  
  ••• (source)
  

from freyr-js.