How to send ZIP-compressed, streaming AJAX response

This blog post explains how to send a ZIP-compressed HTTP response to an AJAX (XMLHttpRequest) request in a streaming way. By streaming we mean sending a prefix of the HTTP response in a way that the JavaScript running in the web browser gets notified soon that a partial HTTP response has arrived, and it can react to the bytes it has already received. With streaming all output buffering must be disabled (or output buffers must be flushed explicitly) so even a single extra HTTP response byte can make it to the JavaScript, without having to wait for other bytes to arrive first.

First make sure that streaming works without compression. For that, call all the relevant flush methods on the server. For Chrome (tested on Chrome 22), you also have to send the HTTP response header X-Content-Type-Options: nosniff, otherwise Chrome would buffer the first 1024 bytes of the HTTP response body before notifying the JavaScript. Firefox (tested on Firefox 14) doesn't buffer anything even without this header. I haven't tried Internet Explorer, Safari or other browsers. In your JavaScript onreadystatechange event handler, make sure that you process the .response even if .readyState is not 4. (In fact, partial streaming responses arrive with .readyState 3, and the final response data arrives with .readyState 4.)

Once streaming works without compression, enable compression by specifying Content-Encoding: deflate, and compress the HTTP response body in the ZLIB format (RFC 1950, also called as /FlateEncode). Please note that there is a similar format named GZIP (which is emitted by e.g. Java's GZIPOutputStream), but don't use that, because that produces an output which is about 10 bytes larger than for ZLIB. In Python, zlib.compress(body) can be used. In Java, use the DeflaterOutputStream or the Deflater class. You have to make more changes to make the compressor flush its output. In Python, you have to use the flush method on zlib.compressobj(). Call the .flush(1) method, where 1 corresponds to Z_PARTIAL_FLUSH (see more here). Z_SYNC_FLUSH also works, but it makes the output size larger than with Z_PARTIAL_FLUSH. (Even Z_PARTIAL_FLUSH makes the output larger than without flushing, because it forcibly closes the Huffman block, and starting a new Huffman block requires some extra bytes which describe the Huffman table of the new block.) In Java, you need at least Java 1.7 (because Java 1.6 and earlier doesn't have flushing capabilities in Deflater). If you use DeflaterOutputStream, specify the flushing mode (true for Z_SYNC_FLUSH, false is useless) as the 2nd argument of the constructor, and then call its .flush() method when you want to propagate the HTTP response accumulated so far to the web browser.

Don't send a compressed HTTP response body if the HTTP request headers don't contain Accept-Encoding with an item deflate.

Firefox 14 and Chrome 22 don't need any additional client-side (JavaScript) changes for receiving ZIP-compressed streaming HTTP response parts. These browsers recognize Content-Encoding HTTP response header, and decompress the HTTP response body on the fly, in a streaming way, without extra buffering if Z_PARTIAL_FLUSH or Z_SYNC_FLUSH was used by the HTTP server. I haven't tried other browsers, but most probably they also work except for some possible increased latency because of flushing issues.