[macOS] Lag when piping stdout about cli HOT 5 OPEN

Wow, that's some wonderful investigation you've done.

Yes, it I can reproduce the behavior on Linux if I switch the code to the subprocess behavior used on MacOS.

I'm no expert in the low level OS details either, but should I guess (based on some investigations), the difference seems to be that the > version is a command output redirection while the piped (|) version seem to wait for the whole process (and subprocesses) to finish. The Popen.communicate() waits for the subprocess to finish (as written here).

Come to think of it, I really don't know why the process.communicate() command is used there - it does not seem to pass any value to the subprocess, does not do anything with the possible result returned from the subprocess. And there seems to be no info in the method docs that you have to wait for the subprocess to finish or join it back or anything.

The only thing I can see it does is wait for the subprocess to finish (which seems to be against the purpose of daemonizing the update fetcher).

But feel free to correct me if I am wrong - @isidentical and @jkbrzt (authors of the commit).

Unless you see a use for it, I would suggest just to remove the process.communicate() call, that should fix the problem.

from cli.

I have tried to reproduce your result but so far without meaningful success. Nothing on the scale presented by your result. My results seem to vary by server response time.

⬢[lukas@toolbox ~]$ time http https://restcountries.com/v3.1/alpha/cz?fields=currencies,capital > /dev/null

real	0m0.617s
user	0m0.259s
sys	0m0.050s
⬢[lukas@toolbox ~]$ time http https://restcountries.com/v3.1/alpha/cz?fields=currencies,capital | cat > /dev/null

real	0m0.641s
user	0m0.279s
sys	0m0.056s

⬢[lukas@toolbox ~]$ time http https://restcountries.com/v3.1/alpha/cz > /dev/null

real	0m0.683s
user	0m0.279s
sys	0m0.043s
⬢[lukas@toolbox ~]$ time http https://restcountries.com/v3.1/alpha/cz | cat > /dev/null

real	0m0.626s
user	0m0.271s
sys	0m0.047s

I have tried with fully up to date Fedora Workstation 38, httpie 3.2.2-2, time 1.9-20, both from Fedora repositories.

Could you please double-check that the issue still exists and provide some more details to your setup, like your OS (and version), httpie version and more details what your server response looks like (valid json?, response length? etc).

from cli.

Thanks for looking into it. I tested it again against an httpbin server running on localhost and hitting http://localhost:8000/get with httpie 3.2.2:

• On MacOS Ventura 13.2.1 (Darwin Kernel Version 22.3.0) the piped version is ~4-5x slower
• On Linux (#35~20.04.1-Ubuntu SMP) there is not significant difference.

So it manifests on macOS only; I'll edit the title.

from cli.

Thanks for taking the time to double-check. Unfortunately, not having an access to Mac I won't be able to test and look into it further.

On the other hand, at the very least, you've managed to narrow the problem down a bit and save the next person looking some time figuring out where the problem is 😄

from cli.

After several hours of debugging, it turns out this is unrelated to potential buffering differences between MacOS and Linux as I had original suspected; it's this code. Commenting it out (or just the process.communicate() line) does away with the lag.

That's as far as I got. I don't have an explanation of why this affects only the piped version; the interactive non-piped version also calls process.communicate() but it apparently finishes(?) much faster. And I don't know enough about the low-level OS-specific process spawn details in this module.

I have to say, I am kind of shocked to find out that an HTTP client CLI includes a "feature" that:

• will regularly warn you about the new update by checking the latest version on PyPI.
• is enabled by default and if you want to disable it you have to add/edit a config file, can't do it from the CLI.
• requires an ad-hoc (and unsurprisingly buggy) daemons module instead of delegating to a library designed specifically for this.

IMO the whole httpie.internal subpackage needs to go or at least be extracted as a separate project. Checking for new releases has nothing to do with sending requests to HTTP servers.

from cli.