Socrata HTTP Gripes

data.acgov.org.txt

% curl -sIL 'https://data.acgov.org/api/geospatial/8e4s-7f4v?method=export&format=Original'

HTTP/1.1 405 Method Not Allowed
Server: nginx
Date: Tue, 13 Jan 2015 01:52:36 GMT
Connection: keep-alive
Access-Control-Allow-Origin: *
X-Error-Code: method_not_allowed
X-Error-Message: HEAD not allowed
X-Socrata-Region: production
Age: 0

data.sfgov.org.txt

% curl -sIL https://data.sfgov.org/download/kvej-w5kb/ZIPPED%20SHAPEFILE

HTTP/1.1 302 Found
Server: nginx
Date: Tue, 13 Jan 2015 01:51:36 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=Edge,chrome=1
Cache-Control: no-cache
X-Request-Id: a79b2f973b7f46e4292a4dd1ca6b555f
Set-Cookie: _socrata_session_id=BAh7BkkiD3Nlc3Npb25faWQGOgZFRiIlNmQ5NGQ2YTNiN2I4Y2ZjN2UyMzRjN2Y0MGMwODM5YjU%3D--d8c873bdb9117ef5c10134a8ebf188095cfeea17; path=/; HttpOnly; secure
Set-Cookie: logged_in=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Location: http://apps.sfgov.org/datafiles/view.php?file=sfgis/eas_addresses_with_units.zip
Status: 302
Vary: Accept-Encoding,User-Agent
X-Socrata-Region: production
Age: 0

HTTP/1.1 200 OK
Date: Tue, 13 Jan 2015 01:51:37 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10
Pragma: public
Expires: 0
Cache-Control: must-revalidate, post-check=0, pre-check=0
Content-Disposition: attachment; filename=eas_addresses_with_units.zip;
Content-Transfer-Encoding: binary
Content-Length: 8300106
Content-Type: application/download

@migurski - This just finally crossed my desk. Hopefully I can help, or at least we can commiserate :)

In the former case, you're right, we don't support the HTTP HEAD method. This is a big request of mine and I'm pushing our engineering team to include it our big platform release in a few months. It's something that I'd make a lot of use of too.

Along with that, we also need to provide you the correct content-types. Even if you did a full GET on that URL, you'd still just get application/octet-stream, which is pretty much useless, unless you want to look at the Content-Disposition header too. Unfortunately the proper mime type for a zipped Shapefile is still "application/zip", but at least that'd get you one step closer. We're going to fix that too.

In the latter case, that's actually an "External Dataset" that San Francisco has provided - basically it's just a link in the catalog. By the time you get redirected to apps.sfgov.org, it's unfortunately out of our hands and returning the proper content type is something that they'd need to fix. I've asked if our CS team can reach out to the city to let them know about that one, but you might have channels of your own too.

I currently try to download datasets from SFgov using CURL or WGET. I receive a file, but when I try to unzip it, it converts into a .cpgz file, which converts into a .zip file, which converts into a .cpgz file, ...

Is this related to the above mentioned problem?

Thanks @chrismetcalf, very useful background! I’ll see if I can follow up with SF to get a better HTTP response out of them.