Shell script to convert DC building permit data to BLDS format

dc-blds.sh

# Field names from BLDS spec
echo "IssuedDate,PermitNum,PermitType,PermitTypeDesc,StatusCurrent,OriginalAddress1,Description,ContractorCompanyName,StatusDate,OriginalCity,OriginalState,OriginalZip" > dc-permits-2014.csv

# Get permit data from ESRI endpoint and convert to CSV
curl -s "http://maps2.dcgis.dc.gov/dcgis/rest/services/FEEDS/CDW_Feeds/MapServer/5/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&maxAllowableOffset=&geometryPrecision=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&returnDistinctValues=false&f=pjson" \
| jq .features[] \
| jq .attributes \
| jq '[.ISSUE_DATE, .PERMIT_ID, .PERMIT_TYPE_NAME, .PERMIT_SUBTYPE_NAME, .APPLICATION_STATUS_NAME, .FULL_ADDRESS, .DESC_OF_WORK, .PERMIT_APPLICANT, .LASTMODIFIEDDATE, .CITY, .STATE, .LATITUDE, .LONGITUDE]' \
| jq @csv \
| sed 's/\\//g;s/^.//;s/.$//' \
>> dc-permits-2014.csv

Data obtained from here.

Goal is to convert to a format that matches (as closely as possible) the one outlined here.

Note - above script assumes the jq utility is installed.

At minimum, three new fields from the BLDS standard need to be added - PermitClassMapped, StatusCurrentMapped and PermitTypeMapped.

Additional questions:

• Are these permits for residential or commercial properties? Possible to tell from ZONING field?
• Possible to get more than 1,000 records at a time? (exceededTransferLimit is set to true in response).
• Need to convert value of ISSUE_DATE field from esriFieldTypeDate to something more meaningful.