Last active
August 29, 2015 14:18
-
-
Save fiddyspence/379a9c82cce23dc2c71d to your computer and use it in GitHub Desktop.
Forge gzip lies
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| spenceware:~ fids$ curl -o foo https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs | |
| % Total % Received % Xferd Average Speed Time Time Time Current | |
| Dload Upload Total Spent Left Speed | |
| 100 7175 100 7175 0 0 11957 0 --:--:-- --:--:-- --:--:-- 11958 | |
| spenceware:~ fids$ file foo | |
| foo: ASCII text, with very long lines, with no line terminators | |
| spenceware:~ fids$ curl -o foo https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs | |
| % Total % Received % Xferd Average Speed Time Time Time Current | |
| Dload Upload Total Spent Left Speed | |
| 100 7175 100 7175 0 0 11523 0 --:--:-- --:--:-- --:--:-- 11535 | |
| spenceware:~ fids$ file foo | |
| foo: ASCII text, with very long lines, with no line terminators | |
| spenceware:~ fids$ curl -o foo https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs | |
| % Total % Received % Xferd Average Speed Time Time Time Current | |
| Dload Upload Total Spent Left Speed | |
| 100 601 0 601 0 0 988 0 --:--:-- --:--:-- --:--:-- 986 | |
| spenceware:~ fids$ file foo | |
| foo: gzip compressed data, from Unix, last modified: Fri Apr 10 09:12:39 2015 | |
| spenceware:~ fids$ mv foo foo.gz | |
| spenceware:~ fids$ gunzip foo.gz | |
| spenceware:~ fids$ cat foo | |
| {"puppetlabs/nodejs":[{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.1.0.tar.gz","version":"0.1.0","dependencies":[["puppetlabs/apt",">= 0.0.2"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.1.1.tar.gz","version":"0.1.1","dependencies":[["puppetlabs/apt",">= 0.0.3"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.2.0.tar.gz","version":"0.2.0","dependencies":[["puppetlabs/apt",">= 0.0.3"],["puppetlabs/stdlib",">= 2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.2.1.tar.gz","version":"0.2.1","dependencies":[["puppetlabs/apt",">= 0.0.3"],["puppetlabs/stdlib",">= 2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.3.0.tar.gz","version":"0.3.0","dependencies":[["puppetlabs/apt",">= 0.0.3"],["puppetlabs/stdlib",">= 2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.4.0.tar.gz","version":"0.4.0","dependencies":[["puppetlabs/apt",">= 0.0.3"],["puppetlabs/stdlib",">= 2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.6.0.tar.gz","version":"0.6.0","dependencies":[["puppetlabs/apt",">=0.0.3"],["puppetlabs/stdlib",">=2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.6.1.tar.gz","version":"0.6.1","dependencies":[["puppetlabs/apt",">=0.0.3"],["puppetlabs/stdlib",">=2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.7.0.tar.gz","version":"0.7.0","dependencies":[["puppetlabs/apt",">=0.0.3"],["puppetlabs/stdlib",">=2.0.0"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-nodejs-0.7.1.tar.gz","version":"0.7.1","dependencies":[["puppetlabs/apt",">=0.0.3"],["puppetlabs/stdlib",">=2.0.0"]]}],"puppetlabs/stdlib":[{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.0.0.tar.gz","version":"2.0.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.1.3.tar.gz","version":"2.1.3","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.2.0.tar.gz","version":"2.2.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.2.1.tar.gz","version":"2.2.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.3.0.tar.gz","version":"2.3.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.3.1.tar.gz","version":"2.3.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.3.2.tar.gz","version":"2.3.2","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.3.3.tar.gz","version":"2.3.3","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.4.0.tar.gz","version":"2.4.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.5.0.tar.gz","version":"2.5.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.5.1.tar.gz","version":"2.5.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-2.6.0.tar.gz","version":"2.6.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.0.0.tar.gz","version":"3.0.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.0.1.tar.gz","version":"3.0.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.1.0.tar.gz","version":"3.1.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.1.1.tar.gz","version":"3.1.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.2.0.tar.gz","version":"3.2.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.2.1.tar.gz","version":"3.2.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-3.2.2.tar.gz","version":"3.2.2","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.1.0.tar.gz","version":"4.1.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.2.0.tar.gz","version":"4.2.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.2.1.tar.gz","version":"4.2.1","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.2.2.tar.gz","version":"4.2.2","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.3.0.tar.gz","version":"4.3.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.3.2.tar.gz","version":"4.3.2","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.4.0.tar.gz","version":"4.4.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.5.0.tar.gz","version":"4.5.0","dependencies":[]},{"file":"/system/releases/p/puppetlabs/puppetlabs-stdlib-4.5.1.tar.gz","version":"4.5.1","dependencies":[]}],"puppetlabs/apt":[{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-0.0.2.tar.gz","version":"0.0.2","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-0.0.3.tar.gz","version":"0.0.3","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-0.0.4.tar.gz","version":"0.0.4","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.0.0.tar.gz","version":"1.0.0","dependencies":[["puppetlabs/stdlib","2.x"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.0.1.tar.gz","version":"1.0.1","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.1.0.tar.gz","version":"1.1.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.1.1.tar.gz","version":"1.1.1","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.2.0.tar.gz","version":"1.2.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.3.0.tar.gz","version":"1.3.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.4.0.tar.gz","version":"1.4.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.4.1.tar.gz","version":"1.4.1","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.4.2.tar.gz","version":"1.4.2","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.5.0.tar.gz","version":"1.5.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.5.1.tar.gz","version":"1.5.1","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.5.2.tar.gz","version":"1.5.2","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.6.0.tar.gz","version":"1.6.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.7.0.tar.gz","version":"1.7.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]},{"file":"/system/releases/p/puppetlabs/puppetlabs-apt-1.8.0.tar.gz","version":"1.8.0","dependencies":[["puppetlabs/stdlib",">= 2.2.1"]]}]}spenceware:~ fids$ | |
| irb(main):022:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:02 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app01-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):023:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:03 GMT", "content-type"=>"application/json;charset=utf-8", "content-length"=>"7175", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app03-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding"} | |
| irb(main):024:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:04 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app02-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):025:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:08 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app04-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):026:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:09 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app02-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):027:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:10 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app04-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):028:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:12 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app04-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| irb(main):029:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:13 GMT", "content-type"=>"application/json;charset=utf-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app04-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding", "content-encoding"=>"gzip"} | |
| And then a broken response | |
| irb(main):030:0> open("https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs") {|f| f.meta } | |
| => {"server"=>"nginx", "date"=>"Fri, 10 Apr 2015 08:33:14 GMT", "content-type"=>"application/json;charset=utf-8", "content-length"=>"7175", "connection"=>"keep-alive", "status"=>"200 OK", "cache-control"=>"private, must-revalidate, max-age=0", "x-node"=>"forge-app03-prod", "x-revision"=>"923bd24", "x-content-type-options"=>"nosniff", "vary"=>"Accept-Encoding"} | |
| irb(main):031:0> | |
| spenceware:~ fids$ curl -o foo https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/nodejs | |
| % Total % Received % Xferd Average Speed Time Time Time Current | |
| Dload Upload Total Spent Left Speed | |
| 100 601 0 601 0 0 1014 0 --:--:-- --:--:-- --:--:-- 1013 | |
| spenceware:~ fids$ file foo | |
| foo: gzip compressed data, from Unix, last modified: Fri Apr 10 14:50:51 2015 |
Without Accept-Encoding:
$ for i in {1..20}; do curl -s -o output "https://forge.puppetlabs.com/api/v1/releases.json?module=rcole
man/motd"; file output; done
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminators
output: ASCII text, with very long lines, with no line terminatorsWith Accept-Encoding:
$ for i in {1..20}; do curl -s -H "Accept-Encoding: gzip,deflate" -o output "https://forge.puppetlabs.com/a
pi/v1/releases.json?module=rcoleman/motd"; file output; done
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:49 2015
output: gzip compressed data, from Unix, last modified: Fri Apr 10 11:24:48 2015
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This should be fixed now.
I think the response headers were always correctly indicating whether or not the response body was gzipped. However, it's obviously undesirable for it to be inconsistent in whether or not you get a gzipped response when you say you can accept one, and even less desirable for it to send a gzipped response to a request that did not say it could accept one. You should now always receive a plain text response unless you pass an appropriate "Accept-Encoding" header.