API Gateway vs. gzip: ERR_CONTENT_DECODING_FAILED

tl;dr: API Gateway may treat binary as text and mangle the content, such as when our backend server returned Content-Type: text/html; charset=UTF-8 with Content-Encoding: gzip responses.  My workaround for Apache 2.4: SetEnvIf x-amzn-apigateway-api-id .+ no-gzip.  (If the request contains the header with any value, don’t gzip the response.)



I got an API backend working as expected when connecting directly to it, then moved on to testing it within Amazon AWS.  I created a new API in API Gateway, because I wanted to set it up as an HTTP service proxy integration instead of fully defining/transforming/etc. the requests and responses in Swagger.

(Related: I don’t really think JSON is a powerful enough representation for the task Swagger + API Gateway wants to perform.  There is simply too much redundancy in the end product.)

The proxy integration immediately started failing in all browsers, with a brand-new error message.  Firefox rendered it as “Content Decoding Error: The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.”  Google Chrome claimed it could not reach the site, but was clearly lying, because in the light gray please-don’t-notice-this area, it said ERR_CONTENT_DECODING_FAILED.  You have to reach the site to fail to decode its response, though…

Safari had a similar error I didn’t write down, and all three browsers’ developer tools were of little use.  The “invalid response” was being discarded, so I couldn’t learn anything about it.

I resorted to turning off all the forward-secrecy cipher suites (DHE/ECDHE) in Firefox, loading the private key into Wireshark, and decrypting a packet trace, where I could finally look at the entity-body enclosed in the response.

1F EF BF BD 08 …

Hmm.  Based on the gzip file format, I had been expecting a stream starting 1F 8B 08 … and those middle bytes look suspiciously like a UTF-8 encoding of something.  That turns out to be U+FFFD REPLACEMENT CHARACTER.

Something is processing the gzip stream as UTF-8 text, “helpfully” replacing invalid UTF-8 bytes with 3-byte replacement characters, and making the response fully non-compliant.  It’s no longer a valid gzip stream, and at 1400+ bytes, the unmodified Content-Length: 885 header is also wrong after this process.

I tried configuring text/html as a “binary” type from API Gateway, and set everything as “convert to binary,” but none of it helped. (In fact, convert-to-binary would cause errors later, and have to be restored to default pass-through behavior.)

However, I’d also done a tcpdump on the servers, simultaneously with the Wireshark trace, so I found out that API Gateway adds some request headers like x-amzn-apigateway-api-id.  If API Gateway is going to insist on mangling binary, then we simply avoid passing binary to it:

SetEnvIf x-amzn-apigateway-api-id .+ no-gzip

There!  Now we send back real text, API Gateway doesn’t replace anything (nothing is ‘improperly’ encoded), and things work.