Skip to content

Instantly share code, notes, and snippets.

@emiliom

emiliom/ioos_csw_obs_ndbc.ipynb

Created July 26, 2018 02:22
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save emiliom/8d75e50c1ab76375771d62156619e2dc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save emiliom/8d75e50c1ab76375771d62156619e2dc to your computer and use it in GitHub Desktop.
Download ZIP
NDBC problems in IOOS Catalog CSW searches and results
Display the source blob
Display the rendered blob
Raw
ioos_csw_obs_ndbc.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tests of NDBC search and results from IOOS Catalog CSW endpoint"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Can we get data near the [Coastal Endurance](http://oceanobservatories.org/array/coastal-endurance/) array?\n",
"\n",
"Emilio Mayorga, 7/25/2018"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- See NDBC SOS instructions page: https://sdf.ndbc.noaa.gov/sos/\n",
"- The **CSW datetime filters** don't work for NDBC endpoints. Specifying a datetime filter causes the search not to return any NDBC results/records. I looked at [ioos_tools.ioos import fes_date_filter](https://github.com/pyoceans/ioos_tools/blob/master/ioos_tools/ioos.py#L76) to see if a timezone code or offset was specified; it isn't, but my tests adding a \"Z\" UTC suffix did not lead to success either.\n",
"- The NDBC keywords (subject terms) don't include any CF standard names. They include instead the sensor urn's that are **not** used, plus the NDBC parameter names. This should be changed. But, the `GetObservation` requests actually do take CF standard names (see `observedProperty` below; not as MMI url's, though; but I haven't tested that), so even if only the GetObservation \"scheme\" references are changed, searching the CSW endpoint on CF standard names would probably work."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import xml.dom.minidom\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Set up and instantiate IOOS Catalog CSW endpoint"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from owslib.csw import CatalogueServiceWeb\n",
"from ioos_tools.ioos import get_csw_records\n",
"\n",
"\n",
"endpoint = 'https://data.ioos.us/csw'\n",
"\n",
"csw = CatalogueServiceWeb(endpoint, timeout=60)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Set up CSW filter elements that will be reused\n",
"Also, all search requests will include a literal text filter on \"NDBC\", to filter out non-NDBC results."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from owslib import fes"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"fesliteral_kw = dict(\n",
" wildCard='*',\n",
" escapeChar='\\\\',\n",
" singleChar='?',\n",
" propertyname='apiso:AnyText'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(a) Bounding box corresponding to an area offshore the Pacific NW (Washington and Oregon)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"min_lon, max_lon = -127, -123.75\n",
"min_lat, max_lat = 43, 48\n",
"\n",
"bbox = [min_lon, min_lat, max_lon, max_lat]\n",
"crs = 'urn:ogc:def:crs:OGC:1.3:CRS84'\n",
"\n",
"bbox_crs = fes.BBox(bbox, crs=crs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(b) Time span: today - 20 days."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from datetime import datetime, timedelta\n",
"\n",
"\n",
"dt = 20\n",
"\n",
"date = datetime.utcnow()\n",
"start = date - timedelta(days=dt)\n",
"stop = date\n",
"\n",
"from ioos_tools.ioos import fes_date_filter\n",
"\n",
"# double check the logic of the deafault 'overlaps' constraint. Looks weird\n",
"begin, end = fes_date_filter(start, stop)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Test CSW request with conventional spatial, temporal and variable filters"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(c) Variables: `sea_water_temperature`"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# List must have >1 element for or_filt statement to work\n",
"# Repeat the parameter\n",
"obs_parameters = ['sea_water_temperature', 'sea_water_temperature']"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"or_filt = fes.Or(\n",
" [fes.PropertyIsLike(literal=('*%s*' % val), **fesliteral_kw)\n",
" for val in obs_parameters])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Assemble the `fes` filter with `a`+`b`+`c`."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"filter_list = [\n",
" fes.And(\n",
" [\n",
" bbox_crs,\n",
" begin, end,\n",
" or_filt,\n",
" fes.PropertyIsLike(literal='*ndbc*', **fesliteral_kw),\n",
" ]\n",
" )\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 0 records.\n",
"\n"
]
}
],
"source": [
"get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000, esn='full')\n",
"\n",
"records = '\\n'.join(csw.records.keys())\n",
"print('Found {} records.\\n'.format(len(csw.records.keys())))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. CSW request with conventional spatial and temporal filters, and an NDBC parameter name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(c) Variables: `sea_water_temperature` and `Water Temperature`"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# List must have >1 element for or_filt statement to work\n",
"obs_parameters = ['sea_water_temperature', 'Water Temperature']"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"or_filt = fes.Or(\n",
" [fes.PropertyIsLike(literal=('*%s*' % val), **fesliteral_kw)\n",
" for val in obs_parameters])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Assemble the `fes` filter with `a`+`b`+`c`."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"filter_list = [\n",
" fes.And(\n",
" [\n",
" bbox_crs,\n",
" begin, end,\n",
" or_filt,\n",
" fes.PropertyIsLike(literal='*ndbc*', **fesliteral_kw),\n",
" ]\n",
" )\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 0 records.\n",
"\n"
]
}
],
"source": [
"get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000, esn='full')\n",
"\n",
"records = '\\n'.join(csw.records.keys())\n",
"print('Found {} records.\\n'.format(len(csw.records.keys())))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. CSW request with conventional spatial filter, no temporal filters, and an NDBC parameter name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(c) Variables: `sea_water_temperature` and `Water Temperature`"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# List must have >1 element for or_filt statement to work\n",
"obs_parameters = ['sea_water_temperature', 'Water Temperature']"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"or_filt = fes.Or(\n",
" [fes.PropertyIsLike(literal=('*%s*' % val), **fesliteral_kw)\n",
" for val in obs_parameters])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Assemble the `fes` filter with `a`+`b`+`c`."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"filter_list = [\n",
" fes.And(\n",
" [\n",
" bbox_crs,\n",
" # begin, end,\n",
" or_filt,\n",
" fes.PropertyIsLike(literal='*ndbc*', **fesliteral_kw),\n",
" ]\n",
" )\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 28 records.\n",
"\n"
]
}
],
"source": [
"get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000, esn='full')\n",
"\n",
"records = '\\n'.join(csw.records.keys())\n",
"print('Found {} records.\\n'.format(len(csw.records.keys())))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**It worked. We have results.** List all the records found:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[SATURN Estuary Station #03]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_seto3\n",
"[SATURN Estuary Station #04 at Mott Basin]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_sefo3\n",
"[SATURN Estuary Station #07, Baker Bay, WA]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46116\n",
"[SATURN Offshore Station #10, Columbia River, South Jetty, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46127\n",
"[STONEWALL BANK - 20NM West of Newport, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46050\n",
"[Tansy Point, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_tano3\n",
"[TILLAMOOK, OR - 85 NM WNW of Tillamook, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46089\n",
"[UMPQUA OFFSHORE, OR (139)]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46229\n",
"[Astoria Canyon, OR (179)]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46248\n",
"[Buoy CB-06 - West of Coos Bay, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46128\n",
"[Buoy NH-10 - West of Newport, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46094\n",
"[Cape Arago, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_caro3\n",
"[CAPE ELIZABETH- 45NM NW of Aberdeen, WA]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46041\n",
"[Cha'Ba La Push]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46119\n",
"[Clatsop Spit, OR (162)]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46243\n",
"[COLUMBIA RIVER BAR - 20NM West of Columbia River Mouth]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46029\n",
"[Columbia River Entrance - SATURN Station #02 Offshore]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46096\n",
"[Desdemona Sands Light, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_dmno3\n",
"[Destruction Island, WA]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_desw1\n",
"[Grays Harbor, WA (036)]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46211\n",
"[Grays Point, WA]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_gptw1\n",
"[Jetty A, USCG Station Disappointment]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_jtaw1\n",
"[Lakeside, OR (231)]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46260\n",
"[Newport, OR]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_nwpo3\n",
"[OOI Newport Shelf]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46097\n",
"[OOI Waldport Offshore]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46098\n",
"[OOI Westport Offshore]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46100\n",
"[OOI Westport Shelf]\n",
" sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46099\n"
]
}
],
"source": [
"for key, value in list(csw.records.items()):\n",
" print(u'[{}]\\n {}'.format(value.title, key))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Summarize `scheme` references and `geolinks.sniff_links` results across all records"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use `geolinks.sniff_link` to find more information about the services available."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"from geolinks import sniff_link\n",
"\n",
"\n",
"df = []\n",
"for key, rec in csw.records.items():\n",
" df.append(pd.DataFrame(rec.references))\n",
"\n",
"df = pd.concat(df, ignore_index=True)\n",
"df['geolink'] = [sniff_link(url) for url in df['url']]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The services found using `geolinks`:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"OGC:SOS 151\n",
"Name: geolink, dtype: int64"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['geolink'].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All the schemes in the list."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"WWW:LINK - text/csv 95\n",
"OGC:SOS 28\n",
"OGC:SOS:DescribeSensor 9\n",
"OR 7\n",
"WA 3\n",
"OR (179) 1\n",
"USCG Station Disappointment 1\n",
"OR (162) 1\n",
"Baker Bay 1\n",
"OR (231) 1\n",
"WA (036) 1\n",
"OR (139) 1\n",
"OR - 85 NM WNW of Tillamook 1\n",
"Columbia River 1\n",
"Name: scheme, dtype: int64"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['scheme'].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Compare the joint occurrence of `geolinks` and `scheme` results."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"geolink scheme \n",
"OGC:SOS Baker Bay 1\n",
" Columbia River 1\n",
" OGC:SOS 28\n",
" OGC:SOS:DescribeSensor 9\n",
" OR 7\n",
" OR (179) 1\n",
" OR (139) 1\n",
" OR (162) 1\n",
" OR (231) 1\n",
" OR - 85 NM WNW of Tillamook 1\n",
" USCG Station Disappointment 1\n",
" WA 3\n",
" WA (036) 1\n",
" WWW:LINK - text/csv 95\n",
"dtype: int64"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.groupby(['geolink', 'scheme']).size()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Examine one of the records"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"# cswrec1 = csw.records['sdf.ndbc.noaa.gov-urn_ioos_station_wmo_46050']\n",
"cswrec1 = csw.records['sdf.ndbc.noaa.gov-urn_ioos_station_wmo_sefo3']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**pretty print complete record xml**"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<?xml version=\"1.0\" ?>\n",
"<csw:Record xmlns:apiso=\"http://www.opengis.net/cat/csw/apiso/1.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\" xmlns:csw=\"http://www.opengis.net/cat/csw/2.0.2\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:dct=\"http://purl.org/dc/terms/\" xmlns:dif=\"http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/\" xmlns:fgdc=\"http://www.opengis.net/cat/csw/csdgm\" xmlns:gco=\"http://www.isotc211.org/2005/gco\" xmlns:gmd=\"http://www.isotc211.org/2005/gmd\" xmlns:gml=\"http://www.opengis.net/gml\" xmlns:inspire_common=\"http://inspire.ec.europa.eu/schemas/common/1.0\" xmlns:inspire_ds=\"http://inspire.ec.europa.eu/schemas/inspire_ds/1.0\" xmlns:ogc=\"http://www.opengis.net/ogc\" xmlns:os=\"http://a9.com/-/spec/opensearch/1.1/\" xmlns:ows=\"http://www.opengis.net/ows\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\" xmlns:sitemap=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:soapenv=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:srv=\"http://www.isotc211.org/2005/srv\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xmlns:xs=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n",
" <dc:identifier>sdf.ndbc.noaa.gov-urn_ioos_station_wmo_sefo3</dc:identifier>\n",
" <dc:title>SATURN Estuary Station #04 at Mott Basin</dc:title>\n",
" <dc:type>dataset</dc:type>\n",
" <dc:subject>Weather</dc:subject>\n",
" <dc:subject>National Data Buoy Center</dc:subject>\n",
" <dc:subject>NDBC</dc:subject>\n",
" <dc:subject>Moored Buoy</dc:subject>\n",
" <dc:subject>C-MAN</dc:subject>\n",
" <dc:subject>DART</dc:subject>\n",
" <dc:subject>TAO</dc:subject>\n",
" <dc:subject>Air Temperature</dc:subject>\n",
" <dc:subject>Barometric Pressure</dc:subject>\n",
" <dc:subject>Conductivity</dc:subject>\n",
" <dc:subject>Ocean Currents</dc:subject>\n",
" <dc:subject>Salinity</dc:subject>\n",
" <dc:subject>Water Level</dc:subject>\n",
" <dc:subject>Water Temperature</dc:subject>\n",
" <dc:subject>Waves</dc:subject>\n",
" <dc:subject>Winds</dc:subject>\n",
" <dc:subject>urn:ioos:sensor:wmo:sefo3::ct1</dc:subject>\n",
" <dc:subject scheme=\"http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#MD_TopicCategoryCode\">climatologyMeteorologyAtmosphere</dc:subject>\n",
" <dct:references scheme=\"OGC:SOS:DescribeSensor\">https://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&amp;version=1.0.0&amp;procedure=urn:ioos:station:wmo:sefo3&amp;request=DescribeSensor&amp;outputFormat=text/xml;subtype=&quot;sensorML/1.0.1&quot;</dct:references>\n",
" <dct:references scheme=\"WWW:LINK - text/csv\">http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&amp;version=1.0.0&amp;offering=urn:ioos:station:wmo:sefo3&amp;request=GetObservation&amp;observedProperty=urn:ioos:sensor:wmo:sefo3::ct1&amp;responseFormat=text/csv&amp;eventTime=2018-07-24T15:01:30/2018-07-24T17:01:30</dct:references>\n",
" <dct:references scheme=\"OGC:SOS\">http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&amp;acceptVersions=1.0.0&amp;request=GetCapabilities</dct:references>\n",
" <dct:modified>2018-07-24</dct:modified>\n",
" <dct:abstract>National Data Buoy Center SOS\n",
"\n",
"This station provides the following variables: Urn:ioos:sensor:wmo:sefo3::ct1</dct:abstract>\n",
" <dc:date>2018-07-24</dc:date>\n",
" <dc:language>eng; USA</dc:language>\n",
" <ows:BoundingBox crs=\"urn:x-ogc:def:crs:EPSG:6.11:4326\" dimensions=\"2\">\n",
" <ows:LowerCorner>46.2 -123.76</ows:LowerCorner>\n",
" <ows:UpperCorner>46.2 -123.76</ows:UpperCorner>\n",
" </ows:BoundingBox>\n",
"</csw:Record>\n",
"\n"
]
}
],
"source": [
"cswrec1_parsedxml = xml.dom.minidom.parseString(cswrec1.xml)\n",
"print(cswrec1_parsedxml.toprettyxml(indent=' '))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`scheme` references and `geolinks.sniff_link` results**"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(3,\n",
" [{'scheme': 'OGC:SOS:DescribeSensor',\n",
" 'url': 'https://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&procedure=urn:ioos:station:wmo:sefo3&request=DescribeSensor&outputFormat=text/xml;subtype=\"sensorML/1.0.1\"'},\n",
" {'scheme': 'WWW:LINK - text/csv',\n",
" 'url': 'http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&offering=urn:ioos:station:wmo:sefo3&request=GetObservation&observedProperty=urn:ioos:sensor:wmo:sefo3::ct1&responseFormat=text/csv&eventTime=2018-07-24T15:01:30/2018-07-24T17:01:30'},\n",
" {'scheme': 'OGC:SOS',\n",
" 'url': 'http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&acceptVersions=1.0.0&request=GetCapabilities'}])"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(cswrec1.references), cswrec1.references"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Code to test an SOS GetObservation example/url\n",
"Just to confirm that it fails, then to show the corrected url that succeeds."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"import requests as r"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&offering=urn:ioos:station:wmo:sefo3&request=GetObservation&observedProperty=urn:ioos:sensor:wmo:sefo3::ct1&responseFormat=text/csv&eventTime=2018-07-24T15:01:30/2018-07-24T17:01:30'"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getobs_url = [ref['url'] for ref in cswrec1.references if 'text/csv' in ref['scheme']][0]\n",
"getobs_url"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"resp = r.get(getobs_url)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<?xml version=\"1.0\" ?><ows:ExceptionReport version=\"1.0.0\" xml:lang=\"en\" xmlns:ows=\"http://www.opengis.net/ows/1.1\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.opengis.net/ows/1.1 owsExceptionReport.xsd\"> \n",
" <ows:Exception exceptionCode=\"InvalidParameterValue\" locator=\"observedProperty\"> \n",
" <ows:ExceptionText>urn:ioos:sensor:wmo:sefo3::ct1</ows:ExceptionText> \n",
" </ows:Exception> \n",
" <ows:Exception exceptionCode=\"InvalidParameterValue\" locator=\"eventTime\"> \n",
" <ows:ExceptionText>2018-07-24T15:01:30/2018-07-24T17:01:30</ows:ExceptionText> \n",
" </ows:Exception> \n",
" <ows:Exception exceptionCode=\"InvalidParameterValue\" locator=\"eventTime\"> \n",
" <ows:ExceptionText>2018-07-24T15:01:30/2018-07-24T17:01:30</ows:ExceptionText> \n",
" </ows:Exception> \n",
"</ows:ExceptionReport>\n"
]
}
],
"source": [
"print(xml.dom.minidom.parseString(resp.content).toprettyxml(indent=' ', newl=''))"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"# Manually compose new, corrected GetObservation url\n",
"# Changed observedproperty and eventTime\n",
"new_getobs_url = 'http://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&offering=urn:ioos:station:wmo:sefo3&request=GetObservation&observedProperty=sea_water_temperature&responseFormat=text/csv&eventTime=2018-07-24T15:01:30Z/2018-07-24T17:01:30Z'"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"resp = r.get(new_getobs_url)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[b'station_id,sensor_id,\"latitude (degree)\",\"longitude (degree)\",date_time,\"depth (m)\",\"sea_water_temperature (C)\"',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:06:00Z,0.00,21.14',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:12:00Z,0.00,21.16',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:18:00Z,0.00,21.13',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:24:00Z,0.00,21.14',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:30:00Z,0.00,21.13',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:36:00Z,0.00,21.13',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:42:00Z,0.00,21.11',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:48:00Z,0.00,21.09',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T15:54:00Z,0.00,21.09',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:00:00Z,0.00,21.09',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:06:00Z,0.00,20.88',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:12:00Z,0.00,20.95',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:18:00Z,0.00,20.92',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:24:00Z,0.00,20.90',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:30:00Z,0.00,21.00',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:36:00Z,0.00,21.03',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:42:00Z,0.00,21.05',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:48:00Z,0.00,21.07',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T16:54:00Z,0.00,21.01',\n",
" b'urn:ioos:station:wmo:sefo3,urn:ioos:sensor:wmo:sefo3::ct1,46.204,-123.759,2018-07-24T17:00:00Z,0.00,21.08']"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"resp.content.splitlines()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Description of the GetObservations request failures, problems, and fix\n",
"\n",
"For reference see NDBC SOS instructions page: https://sdf.ndbc.noaa.gov/sos/\n",
"\n",
"#### This is a request sample present in a CSW record reference\n",
"```\n",
"https://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&request=GetObservation&responseFormat=text/csv\n",
"&offering=urn:ioos:station:wmo:sefo3&observedProperty=urn:ioos:sensor:wmo:sefo3::ct1&eventTime=2018-07-24T15:01:30/2018-07-24T17:01:30\n",
"```\n",
"It fails with errors on `observedProperty` and `eventTime`\n",
"\n",
"#### This request (modified from the one above) works\n",
"```\n",
"https://sdf.ndbc.noaa.gov/sos/server.php?service=SOS&version=1.0.0&request=GetObservation&responseFormat=text/csv\n",
"&offering=urn:ioos:station:wmo:sefo3&observedproperty=sea_water_temperature&eventtime=2018-07-24T15:01:30Z/2018-07-24T17:01:30Z\n",
"```\n",
"- `observedproperty` must a \"variable\" name, not a sensor urn; CF standard name strings generally work\n",
"- `eventTime` string must be in UTC, with \"Z\" suffix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"_draft": {
"nbviewer_url": "https://gist.github.com/45efe49edf6ff6f653ee9a41481ee665"
},
"gist": {
"data": {
"description": "IOOS_data_demo.ipynb",
"public": true
},
"id": "45efe49edf6ff6f653ee9a41481ee665"
},
"kernelspec": {
"display_name": "Python [conda env:IOOS]",
"language": "python",
"name": "conda-env-IOOS-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment