http://urllib3.readthedocs.org/en/latest/
urllib3 Documentation
Highlights
- Re-use the same socket connection for multiple requests, with optional client-side certificate verification. See: HTTPConnectionPool andHTTPSConnectionPool
- File posting. See: encode_multipart_formdata()
- Built-in redirection and retries (optional).
- Supports gzip and deflate decoding. See: decode_gzip() and decode_deflate()
- Thread-safe and sanity-safe.
- Tested on Python 2.6+ and Python 3.2+, 100% unit test coverage.
- Works with AppEngine, gevent, eventlib, and the standard library io module.
- Small and easy to understand codebase perfect for extending and building upon. For a more comprehensive solution, have a look at Requests which is also powered by urllib3.
Getting Started
Installing
pip install urllib3 or fetch the latest source from github.com/shazow/urllib3.
Usage
By default, urllib3 does not verify your HTTPS requests. You’ll need to supply a root certificate bundle, or use certifi
For more on making secure SSL/TLS HTTPS requests, read the Security section.
urllib3’s responses respect the io framework from Python’s standard library, allowing use of these standard objects for purposes like buffering:
Upgrading & Versioning
urllib3 uses a compatibility-based versioning scheme (let’s call it compatver). For the user, they indicate the required decision for upgrading.
Given a version A.B.C:
C. Strictly backwards-compatible, usually a bug-fix. Always upgrade.
B. Possibly partially incompatible, usually a new feature or a minor API improvement. Read the changelog and upgrade when ready.
A. Major rewrite and possibly breaks everything. Not really an upgrade, basically a new library under the same namespace, decide if you want to switch.
For example, when going from urllib3 v1.2.3 to v1.2.4, you should always upgrade without hesitation. When going from v1.2 to v1.3, you should read the changes to make sure they’re not going to affect you.
Components
urllib3 tries to strike a fine balance between power, extendability, and sanity. To achieve this, the codebase is a collection of small reusable utilities and abstractions composed together in a few helpful layers.
PoolManager
The highest level is the PoolManager(...).
The PoolManager will take care of reusing connections for you whenever you request the same host. This should cover most scenarios without significant loss of efficiency, but you can always drop down to a lower level component for more granular control.
A PoolManager is a proxy for a collection of ConnectionPool objects. They both inherit from RequestMethods to make sure that their API is similar, so that instances of either can be passed around interchangeably.
ProxyManager
The ProxyManager is an HTTP proxy-aware subclass of PoolManager. It produces a single HTTPConnectionPool instance for all HTTP connections and individual per-server:port HTTPSConnectionPool instances for tunnelled HTTPS connections:
ConnectionPool
The next layer is the ConnectionPool(...).
The HTTPConnectionPool and HTTPSConnectionPool classes allow you to define a pool of connections to a single host and make requests against this pool with automatic connection reusing and thread safety.
When the ssl module is available, then HTTPSConnectionPool objects can be configured to check SSL certificates against specific provided certificate authorities.
Again, a ConnectionPool is a pool of connections to a specific host. Trying to access a different host through the same pool will raise a HostChangedErrorexception unless you specify assert_same_host=False. Do this at your own risk as the outcome is completely dependent on the behaviour of the host server.
If you need to access multiple hosts and don’t want to manage your own collection of ConnectionPool objects, then you should use a PoolManager.
A ConnectionPool is composed of a collection of httplib.HTTPConnection objects.
Timeout
A timeout can be set to abort socket operations on individual connections after the specified duration. The timeout can be defined as a float or an instance ofTimeout which gives more granular configuration over how much time is allowed for different stages of the request. This can be set for the entire pool or per-request.
See the Timeout definition for more details.
Retry
Retries can be configured by passing an instance of Retry, or disabled by passing False, to the retries parameter.
Redirects are also considered to be a subset of retries but can be configured or disabled individually.
See the Retry definition for more details.
Stream
You may also stream your response and get data as they come (e.g. when using transfer-encoding: chunked). In this case, method stream() will return generator.
Completely consuming the stream will auto-close the response and release the connection back to the pool. If you’re only partially consuming the consuming a stream, make sure to manually call r.close() on the response.