gzip, Brotli, Zstd HTTP Compression for Devs Who Set content-encoding: identity by Accident

Your nginx config has gzip on;. Your app returns JSON. The response body is still 35KB uncompressed. No errors, no warnings — compression is just silently not happening.

This is usually one of four misconfigurations. But first: how the negotiation actually works.

How HTTP Compression Negotiation Works

Two headers, nothing else. The client announces what it can decompress in Accept-Encoding. The server picks an algorithm, compresses the body, and declares its choice in Content-Encoding:

GET /api/data HTTP/1.1
Host: example.com
Accept-Encoding: gzip, deflate, br, zstd

HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: br
Vary: Accept-Encoding

The Vary: Accept-Encoding header is non-optional if you care about CDN caching correctness. Without it, a CDN might cache a Brotli-compressed response and serve it to a client that only advertised gzip in Accept-Encoding. That client then attempts to decompress Brotli as gzip and gets garbage. nginx’s gzip_vary on; adds this automatically.

Content-Encoding: identity is technically valid — it means “no encoding” — but nobody sets it explicitly. The actual failure mode is the opposite: no Content-Encoding header at all when you expected one.

Verify Compression Is Actually Working

Before debugging config, confirm the problem:

# Check headers only
curl -sI -H "Accept-Encoding: gzip, br, zstd" https://example.com/api/data   | grep -i "content-encoding\|vary"

# Compare compressed vs uncompressed size
curl -so /dev/null -w "uncompressed: %{size_download} bytes
" https://example.com/api/data
curl -so /dev/null --compressed -w "compressed:   %{size_download} bytes
" https://example.com/api/data

--compressed sends Accept-Encoding: deflate, gzip, br, zstd automatically and decompresses the response. If both numbers match, compression isn’t firing. If you want to inspect all response headers and what each one means in context, the HTTP Header Analyzer will annotate them including Vary, Content-Encoding, and cache-control directives.

gzip vs Brotli vs Zstd

Three algorithms are practically relevant for HTTP today. Benchmark numbers below are from the Zstd official benchmarks on the Silesia corpus — a standard dataset of mixed real-world files (HTML, source code, PDFs, databases), tested on a Core i7-9700K. Pure JSON or plain text payloads typically compress better.

Algorithm	Level	Ratio	Compress	Decompress
gzip	1 (fast)	2.74x	69 MB/s	380 MB/s
gzip	6 (default)	2.97x	29.9 MB/s	360 MB/s
gzip	9 (max)	3.10x	18 MB/s	360 MB/s
Brotli	4	3.18x	104 MB/s	440 MB/s
Brotli	11 (max)	3.74x	0.4 MB/s	440 MB/s
Zstd	1 (fast)	2.88x	430 MB/s	1,380 MB/s
Zstd	3 (default)	3.01x	320 MB/s	1,350 MB/s
Zstd	19 (max)	3.40x	17.5 MB/s	1,380 MB/s

gzip is the baseline. Level 6 is the right on-the-fly choice — spending 65% more CPU to move from level 6 to level 9 gains you about 4% better ratio. Not worth it for dynamic responses. Pre-compressed static files are a different calculation.

Brotli genuinely outperforms gzip at comparable CPU cost at levels 4-6, and decompresses ~20% faster. The reason: Brotli has a static dictionary tuned for web content — HTML entities, HTTP field names, JavaScript keywords. It gets better ratios than a general-purpose compressor on the same material. Level 11 is only viable for pre-compressed static assets; at 0.4 MB/s compression speed, you’d compress about 25MB per minute. That’s a build step, not a request handler.

Zstd is the speed story. Default level (3) matches gzip’s ratio but compresses 10x faster and decompresses nearly 4x faster. The main limitation is browser support: Chrome 118+ (October 2023), Firefox 126+ (May 2024), Safari 18+ (late 2024). It’s not universal enough to use as the sole algorithm yet, but if your server negotiates properly, adding Zstd costs you a few config lines and helps clients that advertise it. Zstd at level 19 approaches Brotli-11 in ratio without the catastrophic compression speed penalty, making it more usable for on-the-fly work at high compression demands.

Browser and Client Support

Algorithm	Chrome	Firefox	Safari	Edge	Node.js
gzip	All	All	All	All	Built-in (zlib)
deflate	All	All	All	All	Built-in (zlib)
Brotli (br)	51+	44+	11+	15+	v10.16+
Zstd	118+	126+	18+	118+	v21+

One important quirk: br and zstd only appear in Accept-Encoding over HTTPS connections. Browsers intentionally don’t advertise them over plain HTTP — it’s a defense against MITM attacks that could inject encoding headers. If you’re testing on http://localhost and wondering why you only see gzip, deflate, that’s why. Test via HTTPS or use curl directly (curl doesn’t apply this restriction).

Four Misconfigurations That Silently Break It

1. gzip_proxied is missing (nginx reverse proxy)

nginx’s gzip module compresses responses it generates itself. For proxied requests (upstream app to nginx to client), you need gzip_proxied — otherwise nginx only compresses responses from its own content handlers, not from proxy_pass upstreams.

# This is NOT enough when nginx is a reverse proxy:
gzip on;
gzip_types text/plain application/json application/javascript text/css;

# You need this too:
gzip_proxied any;

Most nginx setups are reverse proxies. Most tutorials omit gzip_proxied. These two facts explain a lot of silently uncompressed responses.

2. MIME type not in gzip_types

nginx’s default gzip_types is text/html only. JSON, CSS, JavaScript, SVG — all uncompressed unless explicitly listed:

gzip_types
  text/plain
  text/css
  text/xml
  text/javascript
  application/json
  application/javascript
  application/xml
  application/rss+xml
  image/svg+xml;

nginx matches on the base MIME type, so application/json covers application/json; charset=utf-8. No need to list charset variants separately.

3. An intermediate proxy strips Accept-Encoding

AWS ALB, misconfigured Cloudflare Workers, and some API gateway setups strip or rewrite Accept-Encoding before it reaches your origin. The server never sees the header, defaults to no compression, and everyone downstream thinks the feature is broken when the real problem is the middleware. No error appears anywhere in the chain.

Debug by comparing origin response vs CDN response:

# Via CDN/proxy
curl -sI -H "Accept-Encoding: gzip, br" https://example.com/api/data

# Direct to origin (bypassing CDN via --resolve or direct IP)
curl -sI -H "Accept-Encoding: gzip, br" --resolve "example.com:443:ORIGIN_IP" https://example.com/api/data

If the origin returns Content-Encoding: gzip directly but the CDN response has no Content-Encoding, the CDN is stripping something — more likely stripping Accept-Encoding inbound so the origin never compresses in the first place.

4. The upstream app compresses, then nginx tries to compress again

If your Node.js/Go/Python app compresses the response body and sets Content-Encoding: gzip, nginx should skip double-compression — but this depends on header timing. If the upstream sends the header mid-stream or nginx’s detection races, you can end up with double-compressed garbage that clients fail to decode.

The clean fix: let nginx own all compression. Remove compression middleware from your app (express’s compression module, Go’s gzip.Handler, etc.), return plain responses, and let nginx compress at the edge. Same performance gains, no double-compression risk.

Working Configs

nginx

gzip on;
gzip_vary on;         # adds Vary: Accept-Encoding automatically
gzip_proxied any;     # compress responses from proxied upstreams
gzip_comp_level 6;
gzip_min_length 256;  # skip tiny responses where overhead isn't worth it
gzip_types
  text/plain
  text/css
  text/xml
  text/javascript
  application/json
  application/javascript
  application/xml
  application/rss+xml
  image/svg+xml;

# Brotli requires the ngx_brotli module
# https://github.com/google/ngx_brotli
brotli on;
brotli_comp_level 4;
brotli_static on;     # serve pre-compressed .br files when they exist
brotli_types
  text/plain
  text/css
  application/json
  application/javascript
  image/svg+xml;

Apache

LoadModule deflate_module modules/mod_deflate.so
AddOutputFilterByType DEFLATE text/html text/plain text/css   application/json application/javascript image/svg+xml
Header append Vary Accept-Encoding

# mod_brotli requires Apache 2.4.26+
LoadModule brotli_module modules/mod_brotli.so
AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/css   application/json application/javascript image/svg+xml

Caddy

Caddy enables gzip and Brotli by default. To add Zstd explicitly:

example.com {
  encode gzip zstd br
  reverse_proxy localhost:3000
}

No MIME type lists, no gzip_proxied edge cases, correct Vary handling out of the box. The honest answer to “which server has the least compression-related misconfiguration surface area” is Caddy.

Testing Compression on Your Own Payloads

Benchmark numbers on the Silesia corpus tell you relative performance, but your specific payload matters more. A repetitive JSON API response with consistent field names compresses differently than minified JavaScript or mixed HTML. These tools let you test specific payloads in-browser without spinning up a local compression server:

• Gzip / Zlib / Deflate Tester — paste your payload, see compressed size and ratio immediately
• Brotli Compression Encoder/Decoder — test Brotli compression at different quality levels
• Zstandard (Zstd) Compression Tool — Zstd encode/decode in the browser

Useful when deciding whether to pre-compress static assets at Brotli-11 or just let nginx handle gzip on-the-fly. Paste your actual response payload, compare the ratios, and make the call with real numbers.

Wrapping Up

If responses aren’t compressed and curl -sI confirms no Content-Encoding, the fix is almost certainly one of the four misconfigs above — most likely gzip_proxied any; for nginx, or a CDN eating your Accept-Encoding header. Check origin directly before blaming your server config.

For the algorithm choice: gzip-6 is fine for dynamic API responses and carries nearly zero config risk. Add Brotli for static assets — pre-compress at level 11 during your build step, serve with brotli_static on, and let nginx fall back to gzip for clients that don’t advertise br. Zstd is worth adding now; the config cost is trivial and its browser footprint is growing fast. Offering all three with correct Vary: Accept-Encoding handling is the right posture for anything new.

You may also like