Get data sources that are used by ArcGIS Server map services with Python - Jupyter notebook

get_mapservice_sources.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:100% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often when having multiple map services published on ArcGIS Server one needs to find out what feature classes a certain map service refers to. This can be done in ArcGIS Desktop by connecting to the ArcGIS Server, right-clicking the service, and choosing [Service Workspaces](http://server.arcgis.com/en/server/latest/publish-services/linux/reviewing-service-workspaces-in-arcgis-for-desktop.htm). The dialog box provides you details on whether the data is copied or referenced as well as where exactly the data that is served by the service data is stored. The same information can be obtained from [ArcGIS Server Manager](http://server.arcgis.com/en/server/latest/publish-services/linux/reviewing-service-workspaces-in-manager.htm), a web based application.\n",
    "\n",
    "However, when in need to automate this process, your only choice is to use the [ArcGIS REST API](http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/The_ArcGIS_REST_API/02r300000054000000/) which provides access to a resource called [Service Manifest](http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#//02r3000001vt000000). \n",
    "\n",
    "This is basically a `.json` file that is stored in the _arcgisserver_ input directory. So, for instance there is a service _Landmarks_ that is published in the folder _Parks_; the manifest JSON file would be stored here:\n",
    "\n",
    "`C:\\arcgisserver\\directories\\arcgissystem\\arcgisinput\\Parks\\Landmarks.MapServer\\extracted\\manifest.json`\n",
    "\n",
    "This JSON file gives you a whole lot of useful information about the service resources such as whether the data is referenced or has been copied at the publishing time, what datasets are served (i.e., feature classes names), database connection strings, and map documents that were used for authoring the service. If you are not comfortable consuming resources over the network using the REST API, you could just crawl through the Windows folders of your _arcgisserver_ input folder and read the information stored within the JSON manifest files. \n",
    "\n",
    "However, a more complete representation of this information is available through an XML file which is stored at the same folder with the same name yet with the `.xml` extension. Again, you can open up this file and read/parse through it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have many map services you would like to do this thing for, you probably would like to write a Python script that will get this XML metadata file and then parse it. When you are logged into the Administrator Directory, you can get the `.xml` file from the URL:\n",
    "\n",
    "`http://localhost:6080/arcgis/admin/services/Parks/Landmarks.MapServer/iteminfo/manifest/manifest.xml`\n",
    "\n",
    "As getting this file requires authenticating on ArcGIS Server, you could consult the help page [Scripting ArcGIS Server administration](http://server.arcgis.com/en/server/latest/administer/linux/example-stop-or-start-all-services-in-a-folder.htm) to see how to get an ArcGIS Server token using Python.\n",
    "\n",
    "If you are comfortable installing 3rd party Python packages, you could just use [ArcREST](https://github.com/Esri/ArcREST/) and [requests](http://docs.python-requests.org/en/master/). When you are logged into ArcGIS Server Administrator Directory, you can then submit a `GET` request using Python requests package to get back this XML file as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "metadata_url = '{0}/iteminfo/manifest/manifest.xml'.format(service_url)\n",
    "xml_data = requests.get(metadata_url,params={'token':token}).content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you got the XML string, you can use the built-in `xml` Python module to parse the XML data. Here is an example of parsing the XML file rather than string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xml.dom.minidom.Document instance at 0x039538C8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xml_file = r\"C:\\arcgisserver\\directories\\arcgissystem\\arcgisinput\\CountiesMap.MapServer\\extracted\\manifest.xml\"\n",
    "\n",
    "from xml.dom.minidom import parse\n",
    "dom = parse(xml_file)\n",
    "#to get pretty print of XML\n",
    "#print dom.toprettyxml(indent='\\t', newl='\\n', encoding=None)\n",
    "dom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DATABASE=C:\\GIS\\Temp\\test.gdb\n"
     ]
    }
   ],
   "source": [
    "#get the place from where the data was published\n",
    "source = dom.getElementsByTagName('OnPremiseConnectionString')[0]\n",
    "print source.firstChild.nodeValue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'C:\\\\GIS\\\\Temp\\\\test.gdb\\\\counties']\n"
     ]
    }
   ],
   "source": [
    "#get path to every item that was published\n",
    "datasets = dom.getElementsByTagName('Datasets')[0].childNodes\n",
    "paths = [dataset.getElementsByTagName('OnPremisePath') for dataset in datasets\n",
    "         if hasattr(dataset,'tagName')]\n",
    "fcs = [path[0].firstChild for path in paths]\n",
    "fcs_values = [fc.nodeValue for fc in fcs]\n",
    "print fcs_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, working with XML as if you are working with JSON is much nicer. There is a great 3rd parthy Python module called [xmltodict](https://github.com/martinblech/xmltodict) which will convert the XML string into a Python dictionary, so you can work with the XML structure as if you have a JSON structure.\n",
    "\n",
    "This is what it takes to convert XML into a Python dict and then access the data you need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import xmltodict\n",
    "from collections import OrderedDict\n",
    "data = dict(xmltodict.parse(xml_input=xml_string,encoding='UTF-8'))\n",
    "\n",
    "#data\n",
    "#{u'SVCManifest': OrderedDict([(u'@xmlns:typens', u'http://www.esri.com/schemas/ArcGIS/10.1')..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigating around the `data` object is really just about navigating around a large Python dictionary. \n",
    "\n",
    "Getting the list of feature classes that are served by service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'C:\\\\GIS\\\\Temp\\\\test.gdb\\\\counties']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service_items = []\n",
    "datasets = data['SVCManifest']['Databases']['SVCDatabase']['Datasets']['SVCDataset']\n",
    "if isinstance(datasets,list):\n",
    "    for ds in datasets:\n",
    "        service_items.append(ds['OnPremisePath'])\n",
    "elif isinstance(datasets,OrderedDict):\n",
    "    service_items.append(datasets['OnPremisePath'])\n",
    "\n",
    "service_items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting path of map document published:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\GIS\\Temp\\CountiesMap.mxd\n"
     ]
    }
   ],
   "source": [
    "resources = data['SVCManifest']['Resources']['SVCResource']\n",
    "if isinstance(resources,list):\n",
    "    for res in resources:\n",
    "        print res['OnPremisePath']\n",
    "elif isinstance(resources,OrderedDict):\n",
    "    print resources['OnPremisePath']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The whole script that I've written really quickly is available below. You can use it on your ArcGIS Server installation to get information on which feature classes are used by your map services.\n",
    "\n",
    "As you see, using external Python packages and modules can help you write less code and get work much faster. Happy scripting!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}