Moving away from httplib about urllib3 HOT 4 OPEN

Even though urllib3 doesn't use httplib too much, a full rewrite of something this fundamental can be involved. I would highly recommend these links for the lazy:

• httplib docs
• httplib source
• urllib3's Connection Pool (first major usage of httplib)
• urllib3's Response (second major usage)

As for the design thus far, the enhancements are tentatively broken into creating specialized Managers (aka Pool Managers) that implement specialized behavior (e.g., proxying, caching, etc.) and lower-level httplib-replacement classes like Connection, Request (HTTPRequest), and Response (HTTPResponse). Connection manages basic socket behaviors and more advanced transport behaviors like handshaking. Request/Response mostly handle parsing/serialization stuff.

General patterns for customization are inheritance (i.e., HTTPSRequest could be a child of HTTPRequest. Other requests like FTPRequest should be possible), and type passing (i.e., telling Manager type Foo to use Connection type Bar).

As for enhancement-specific links, I recommend:

• benoitc's http-parser - Modern and much improved handling of the response parsing side. Ported from mongrel/node.js for use in gunicorn, it has pure Python and Cython versions available in the linked repo.
• py-nonblocking-http - A Google project from 2010 that provides a non-blocking-socket-compatible replacement for httplib's Connection class. While not directly applicable on the whole, it does demonstrate:
  • How to replace 90% of httplib
  • How to unify HTTP and HTTPS Connections, if that is desirable
  • Some of the complexities of handshaking and redirects in a non-httplib context

You'll note that it is still quite opaque and monolithic, so, again, just for reference.

from urllib3.

@makuro Thanks for organizing this and writing it up. :)

from urllib3.

This is so exciting.

from urllib3.

Still hopeful for this someday but it'd be a huge project. Maybe with a sponsor or something.

✨

from urllib3.