High CPU usage about chromic_pdf HOT 29 CLOSED

Thanks for pointing me to this issue! Hope it'll work for you.

And @andreasknoepfle is going to get sooo many emails 😄

from chromic_pdf.

Hey,
Thanks again of the detailed answer!

The load is consistently on that level, and I am pretty sure we are not generating that many pdfs.
The templates have no js in them.
I will make some test, it could be something else as well, just this was my first suspect.
I will update the issue with my findings!

from chromic_pdf.

👍

Just found out (slight facepalm here):

div(System.schedulers_online(), 2)

If you only have 1 scheduler online (usually this means = 1 core), this will return 0.

from chromic_pdf.

@maltoe Good catch! I think then it is that, we have a tiny vm.

from chromic_pdf.

Waiting for the deployment at the moment

from chromic_pdf.

Yeah, that is why I was not too keen to use that option on prod if I didn't have to, plus the potential performance penalty...

from chromic_pdf.

I will give an update once the release(not up to me) is out, but I am pretty confident it will work.
And then down the line just update the lib once your fix is merged, and rely on that.

Thanks for all the help, very cool library!

from chromic_pdf.

So there was high CPU load because it was stuck in this recompile loop, and that manifested itself in the high cpu usage.
Somehow even with the pool size it was stuck on 0, but that could have been something else with it being stuck.
The on_demand mode did go through.
I will give the update a go at some point and let you know!

from chromic_pdf.

Hey @benonymus,
I suspect that the high CPU usage that you are seeing is due to Chromic PDF starting a Chrome instance with some tabs ready for PDF generation. This is great for a production server, so the PDF generation can happen very fast while keeping the same memory footprint, since the chrome tabs that are used to print are recycled.
You can also turn off this behavior by using on_demand mode. In this case there won't be any chrome instances started and kept around when you start the application, but only on demand when you generate a PDF (similar to what wkhtmltopdf will do).

I hope this already helps to bring some light into the case.

Cheers,
Andi

from chromic_pdf.

Hey @andreasknoepfle,

Thanks for the quick reply!
Is this also something that could be tweaked by these settings:

defp chromic_pdf_opts do
  [
    session_pool: [size: 3]
    ghostscript_pool: [size: 10]
  ]
end

?
Or is that for different things?

from chromic_pdf.

Yes, the session_pool flag is the right one to tweak how many browser tabs will be opened. However, I think the main CPU load comes form starting chrome itself, which only happens once. However this explains only a short term raise in CPU load when the system starts. A chrome process that is just open and waiting for PDF generation to happen should not eat up much CPU. In case you experience this load continuously there is probably something else going on.

Could you run htop or something while you generate your PDFs, so we can see which process consumes the CPU (if it is the Erlang process or Chrome itself). Also could you give some context on what exactly you are printing? If it is a page with JavaScript running on page load, this might also be an explanation...

The ghostscript_pool configuration however will not affect any of this, as long as you do not convert you PDF to PDF-A since there Chromic PDF does not have open ghostscript sessions or anything like that. This config is just for having an upper limit of maximum ghostscrpt executions that will run in parallel.

from chromic_pdf.

Hey @andreasknoepfle is there no default session_pool?
I just checked the logs, I have this error:

** (Mix) Could not start application xxxx: XXXX.Application.start(:normal, []) returned an error: shutdown: failed to start child: ChromicPDF
2021-02-05T15:48:34.558+01:00 ** (EXIT) shutdown: failed to start child: ChromicPDF.Browser
2021-02-05T15:48:34.558+01:00 ** (EXIT) shutdown: failed to start child: ChromicPDF.Browser.SessionPool
2021-02-05T15:48:34.558+01:00 ** (EXIT) an exception was raised:
2021-02-05T15:48:34.558+01:00 ** (ArgumentError) pool_size must be more than 0, got: 0
2021-02-05T15:48:34.558+01:00 (nimble_pool 0.2.3) lib/nimble_pool.ex:219: NimblePool.start_link/1
2021-02-05T15:48:34.558+01:00 (stdlib 3.12.1) supervisor.erl:379: :supervisor.do_start_child_i/3

But locally I am not getting any errors

from chromic_pdf.

hey @benonymus

The error reads indicates you have session_pool: [size: 0] in your options which is not allowed. As @andreasknoepfle said, if you want to disable the session pool, please consult the docs for the "on demand" mode. This will effectively disable the background Chrome process and spawn it when needed.

However, I don't think the session pool, or the background Chrome process is the reason for your high CPU load - or if it is, I'd really like to understand why, as so far I've never heard of any CPU load issues caused by an idling Chrome process. As long as you don't use it (as in, print PDFs) this process is just sitting there and doing nothing.

To be able to debug your issue further, it would be really helpful if you provided us with more information. htop as @andreasknoepfle suggested to see which OS process is actually eating your CPU, then environment information (esp. Chrome version, but also OS, deployment environment, etc.)

from chromic_pdf.

Hey @maltoe

I think the high cpu usage is just a symptom, the compilation keeps failing and that cause the high cpu.

But I am not setting session_pool in my settings, this 0 I think somehow comes from the default calculation which would mean there are 0 schedulers online.

I will try to set my session_pool and report back.

from chromic_pdf.

But I am not setting session_pool in my settings, this 0 I think somehow comes from the default calculation which would mean there are 0 schedulers online.

uh? interesting that during compilation you have 0 schedulers, how does it even compile anything? But that's actually a good point, it should get the number of schedulers at runtime. Will add a fix for that.

from chromic_pdf.

Is the compilation failing due to ChromicPDF though?

from chromic_pdf.

@maltoe Yes it looks like everything goes fine and we fail with the error I sent earlier.

from chromic_pdf.

You can get the same error locally if you hard code 0 as size for the session_pool

from chromic_pdf.

#102

from chromic_pdf.

@maltoe Also so the compilation passes, we fail when the application tries to start as the error shows, but not important anymore.

from chromic_pdf.

awesome!

So and when you set the option to some value > 0 it works? Or did you say you were going to try that out another day?

[session_pool: [size: 1]]

from chromic_pdf.

You can get the same error locally if you hard code 0 as size for the session_pool

makes sense, I'll see if I can add a validation so people don't set that accidentally and run into the issues you were seeing.

[edit] 🤔 hmm actually the exception from NimblePool might be enough

from chromic_pdf.

I was also wondering when you use on_demand mode, shouldn't that ignore the pool_size, or how does that interaction work?
If you set 0 and do on_demand: true you still get the same error

from chromic_pdf.

🤔

shouldn't that ignore the pool_size

It should and it does, and it works for me.. ?

iex(1)> ChromicPDF.start_link(on_demand: true, session_pool: [size: 0])
{:ok, #PID<0.248.0>}
iex(2)> ChromicPDF.print_to_pdf({:html, "test"})
{:ok,

from chromic_pdf.

Oh right, I guess I had it mixed up with false on_demand, sorry 🥇

So if I use on_demand the pool size is always one?

Edit yes it is I see

from chromic_pdf.

yes, it is because the pool size becomes kind of irrelevant as the "on demand" Chrome process is only used for a single operation and then killed again. It's really only a development feature as we kept having a bunch of zombies on our dev machines, it's pretty tricky to kill external processes when the BEAM is shutdown, especially with abort from iex. Not so much an issue on production, of course.

from chromic_pdf.

Well only putting it in on_demand mode fixed the problem, seemed like just changing the pool size didn't fix it, but I think it was in some bad state, kept retrying.
Once the fix is out I will give it a go, and stop using on_demand mode.

from chromic_pdf.

seemed like just changing the pool size didn't fix it

🤔 didn't fix it = you still had high CPU load?

Released 0.7.1 now, pls give it a try.

from chromic_pdf.

@benonymus closing this for now, feel free to re-open if you still experience issues

from chromic_pdf.