LychnisA scratchpad for thoughts on my journey through life

1.1   Client-side HTTP in Python

There are thousands of web frameworks that are written in Python: Django, Mochikit, etc. There are many web application servers: SkunkWeb, CherryPy, etc. 28% of the talks in this conference are about the server side in some way or other, so this is probably a forte of Python. But if the server side is so well represented, howcome the client side is so badly represented?

Of course, if all you need to do is retrieve a document, there are plenty solutions: urllib, http.client, PycURL, etc. None of them support any more complex use cases very well.

1.2   Outline of the presentation

1. First, we will tell a little about our application. Knowing our requirements is important to understand why the current solutions are inadequate.
2. Then, we will discuss the current state of the Python standard libraries and some external libraries. We will not cover every possible implementation there is, because there are quite a lot of them. The discussion will involve some code fragments.
3. Then, we will quickly outline next steps that could be taken, and future plans from our side.
4. Finally, this session ends in an interactive discussion, where there is room for questions and wild ideas. If there is a need for it, discussion will continue in the open space.

By the way, if, at some point during the presentation, you know that some package or module exists somewhere that does what we need, please shout it out!

2.1   Client

We have written a Python client that talks to Python server via HTTP. Although, HTTP in a corporate environment means that you have to deal with proxies, authentication (basic, kerberos, NTLM), SSL or not, IP-whitelisting or not, etc.

Usually, we sell the software as a complete, standalone product, sometimes as a SaaS (Software as a Service) or PaaS (Platform as a Service) solution. The standalone product has to integrate with the corporate environment (think: file/backup policies, database connections, user authentication and authorization).

The client application (running on the user’s Windows desktop) connects to the server. The "service" solution must connect to the internet: proxies with authorization.

The client application is the visualization and data management tool that users see. This is programmed in Python, using (amongst others): py2exe for distribution, PyQt for the user interface, pyOpenGL for the visualizations, some custom C++ libraries for data management on top of Qt.

2.2   Server

Mostly Python, some key parts in C++ using Boost-Python.

Some parts, e.g. the calculation and text processing pipelines, have been written in C++. The wrapping of this code is done using Boost-Python.

The server uses a custom HTTP/1.1 server because we wanted an small, extensible, server that can at the same time manage processes. The solution involves some POSIX message queue magic.

2.3   Protocol

3.1   Use of the API

Understand body in file-like objects

Instead of requiring the client to retain the entire request body in memory, allow it to pass a file-like object (i.e. with a suitable read method) as the body of the request.

Use file-like objects for server response

The converse is also true: when reading enormous amounts of data in the response from the server, it’s never wise to keep that in memory.

Authentication schemes

It must be possible to implement other authentication schemes besides Basic (think: Digest, Kerberos, NTLM). Whether or not these implementations are part of the library doesn’t matter. (See RFC 2617 for Basic and Digest mechanisms.)

Dynamic authentication

We do not know beforehand what the authentication will be; for e.g. Kerberos or Basic authentication, quite different values need to be passed, but only after we get our first response with a WWW-Authenticate header.

Server SSL certificate validation

When the server presents a security certificate, the application should be enabled to ask the user to verify and validate it.

3.2   HTTP protocol quirks

Keepalive

The client must maintain an open socket until it is explicitly closed by either the calling code or the server. This should be done by default when using HTTP 1.1, otherwise, heed the contents of the Connection response header.

HTTP/0.9 support

Seriously, how many non-trivial HTTP/0.9 servers are in use at this time?

Other commands than GET

The client must be able to do other HTTP commands, such as PUT or OPTIONS. The library must understand the functional difference between these request methods, and never blindly assume that a redirect changes the request method. This may be so with POST, but definitely not with PUT. Another example is that OPTIONS may or may not have a body in both the request and the response.

Use and understand chunked encoding

HTTP 1.1 clients are required to be able to receive chunked encoded messages from the server. We must be able to send chunked encoded messages to the server as well.

3.3   Understanding the server

Redirect handling

The server sends redirect status codes in response to various situations, sometimes the request body needs to be resent to the new location (e.g. in case of PUT).

Heed 100 Continue status

We explicitly ask the server for a 100 Continue response using the
Expect: 100-continue header in our request. The client library
must signal the application when we are ready to send the request
body.

SSL

The client must be able to talk to the server over SSL or TLS.

Client SSL certificate authentication

The client must be able to authenticate itself to the server using an SSL certificate.

3.4   Managing server connections

Communication via a proxy

The library must be able to transparently talk to a HTTP proxy. This proxy may require authentication, SSL certificate validation, etc., as with a plain request directly to the origin server, and the library must be able to handle it all. The main problem is acquiring proxy configuration for all the platforms that Python supports.

Communication over unix domain sockets

For unit testing purposes, we use unix domain sockets (AF_UNIX) to communicate to an embedded test server. It would be nice if the library supported this somehow.

Multiple simultaneous open connections to the server

Multiple connections will be used, and preferrably the library should offer some control over which connection is used for a request (for example, by placing requests on a specific instance, and coupling instances to a socket).

3.5   Library usage

Cross-platform

Of course, the code must work on all platforms (win32, linux x86, linux x86-64, linux itanium, solaris, hp-ux). This should not be much of an issue, but watch out for request size limits on 32-bit platforms.

Pythonic

We’re writing Python code, and code using the library should preferably "look and feel" like python. What the library is made of doesn’t matter.

4.1   http.client (former httplib)

The library requires you to explicitly pass the request method, so in theory it supports them all, however the logic only appears to have native support for GET and POST.

http.client always sends the entire request in one go, completely ignoring 100 Continue statuses in the response. To support 100 Continue, 68 lines in the HTTPResponse class need to be duplicated, as shown in figure 1. The image is for the HTTPResponse class, found in httplib (Python 2.5). However, since this class had very little changes in py3k (checked against Python 3.1 alpha), so the situation hasn’t changed at all.

Figure 1: The code of httplib.HTTPResponse that needs to be duplicated to support 100 Continue; the area in red is the code that’s actually different.

4.2   urllib (former urllib2)

urllib is basically a chain of handlers for response status codes, built on top of http.client. This means that most of the incapabilities of http.client are shared.

The way to get urllib to execute another request besides GET and POST, is to make get_method of the Request instance return the method you want. For example:

request.get_method = lambda: 'PUT'

This kind of works; however, we use a redirect (301) from a general resource to a specific resource in response to the PUT (this is explicitly described as a possibility in section 9.6 of RFC 2616). This is not handled gracefully by urllib, which always discards the body and changes the request to GET in response to a redirect response. This is a valid assumption if you’re working with the POST-redirect-GET pattern.

The proper way to handle this for PUT however, is to create a redirect handler, and insert that in the handlers chain below the original redirect handler. Re-implementing this feature requires us to duplicate some of the code in HTTPRedirectHandler.

4.3   httplib2

httplib2 is a project to improve some of the shortcomings of httplib. The website lists project goals, and the most important (to us) is that it supports Keep-Alive. The rest of the library seems to be focused more on caching strategies rather than advanced HTTP usage, but it does acknowledge that there may be other use cases.

However, it uses the socket module’s builtin SSL support. It only provides some authentication mechanisms and no extension is possible, so no Kerberos authentication at this time. It has no direct support for 100 Continue.

4.4   bzrlib’s _urllib2_wrappers.py

Bazaar has quite similar goals to us: transfer possibly large amounts of data in several requests over the same socket connection, supporting outlandish authentication and network setups. They have written a series of connect handlers on top of urllib2 to manage this. For example, they have implemented proper redirect handling.

However, one of the developers has expressed the desire to re-implement this in a different library, should it become available; mostly because the existing code is a mess.

4.5   PycURL

Even though the API is not very pythonic, this library is also usable for other languages.

Its documentation is complete, but rather unstructured because everything is done through setopt and getinfo. But it’s nice in that it allows you to pass custom read and write methods for the body. In general, it offers very detailed control over the messages, but you need to understand HTTP.

4.6   Side-by-side comparison

Key:

Notes: