Data Science Academy -- Oct 14, 2016

README.md

CARTO

Oct 14, 2016

Andy Eschbacher, [email protected], @MrEPhysics

This link is here: http://bit.ly/carto-data-sci-oct14

What is CARTO?

• It depends on how you use it ;)
  • Builder -- Cloud-based Analysis and mapping platform
  • Engine -- APIs for building custom applications
• Geo data matters

Maps I like

• Urban Reviewer
• Tweets mentioning sunrise
• Carto Calculator
• Evictions in San Francisco
• L Train project
• Other projects
• NYC Henge

Data Science projects

• Animated maps to compliment analysis: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/
• Our L Train analysis: https://carto.com/blog/looking-at-the-l/
• Ben Wellington:
  • Illegal Parking Ticketing

---

Demo analysis workflow

Importing the data

First we need to sync a new dataset. Search for "usgs earthquake spreadsheet" to find the page, or use the following dataset of all recorded seismic events from the past 30 days:

http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

This is updated every 15 minutes, so it's a great candidate for making a map which updates based on measurements in the world. Let's import that into our accounts. Even cooler: the map we publish will be updated automatically, and the underlying analysis changes dynamically based on the new points added to the dataset.

Building the analysis workflow

To build our analysis workflow, let's focus on the output we would like.

Visualize significant, recent earthquake events by communicating the total population who are living within 50 miles of the epicenter.

We can build this analysis workflow in the Builder by chaining together some of the tools.

1. We need to restrict our analysis to California (within 50 miles of the borders)
2. Filter to only show significant events
3. Find the number of people living with 50 miles of the earthquake events
4. Visualize the points by the number of people (proportional symbol map)

Getting California

There are many ways to get the boundary of California. One ways is just grab all the states from Natural Earth Data. Another ways is to grab it from my CARTO account:

http://eschbacher.cartodb.com/api/v2/sql?q=SELECT%20*%20FROM%20ne_50m_admin_1_states&format=shp&filename=us_states

Copy that link, and import it just like you did for the earthquake dataset above. Finally, we can do some Analysis with the Builder to get it into the state that we're interested in (50 miles outside of current borders).

Finding Earthquakes that happened in or around California

We can again use the Builder to find all earthquakes which intersect California or are within 50 miles of its border by using the "Aggregate intersection" method.

Filtering out non-significant events

Using the "Filter by column value" analysis method, we can surface only the significant earthquakes (by magnitude).

Finding 50 mile buffers

Using the "Create areas of influence" tool, we can get the regions that are within a 50 mile distance of the epicenter of an earthquake.

Enrich with the Data Observatory

This analysis allows us to attach valuable information from sources such as the US Census. For our demo we will use Total Population that are within 50 miles of the epicenter of the earthquakes.

Visualize by the number of people

---

CARTO's Data Science toolkit

Signing up

• Get on GitHub Student Developer Packs if you can
• "student accounts": https://carto.com/signup?plan=academy

Our APIs

• SQL API (write sql, get data via HTTP requests)
• Data Services API
  • Geocoding, routing, etc.
  • Nice demos here: https://carto.com/location-data-services/
• Data Observatory API
  • Functions for getting measures (World censuses, Zillow, etc.)
  • Functions for getting boundaries
  • Catalog of available data: https://cartodb.github.io/bigmetadata/index.html
  • Find things like median income at our location: SELECT OBS_GetMeasure(CDB_LatLng(40.7529671,-73.9927947), 'us.census.acs.B19013001'::text)
  • Explain Subway stops: https://jkrauss.carto.com/builder/18644096-65c3-11e6-adbe-0ee66e2c9693/embed
• Maps API
  • For fetching map tiles for use with Google Maps API, Leaflet.js, OpenLayers, etc.
  • Generating dynamically made image previews (aka, static maps)
  • For example: http://mamataakella.carto.com/api/v1/map/static/named/tpl_25aaad68_7134_11e6_afae_0e3ebc282e83/1600/1600.png?zoom=5&lat=40.7529777&lon=-73.994722
• Import API
  • Send data to your CARTO account
  • Setup sync tables (responsive to upstream changes to data so you can create dynamic maps)
• Spatial Analysis
  • geo-statistics using PySAL, scikit-learn, and others
  • tons more coming soon

JavaScript Libraries

• Carto.js -- workhorse library for building custom applications off of CARTO's platform
• Torque.js -- create spatio-temporal maps from your time-series data

Programmatic access / R and Python workflows

• r2cartodb: https://rpubs.com/walkerke/r2cartodb
• CARTO's Python Client: https://github.com/CartoDB/carto-python
• R's read.csv and Panda's read_csv work with our SQL API when the format flag is set to csv
  • Format: http://eschbacher.carto.com/api/v2/sql?q=SELECT * FROM awesome_table&format=csv
  • Use with Plotly: https://plot.ly/ipython-notebooks/cartodb/

---

Demos with JavaScript

• DC Elevation App -- see index.html on this gist
• Charts!
  • We made this at a little meetup I run last week

index.html

<!DOCTYPE html>
<html>
  <head>
    <title>CartoDB Workshop | CartoDB.js</title>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
    <meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
    <link rel="shortcut icon" href="http://cartodb.com/assets/favicon.ico" />
    <style>
      html, body, #map {
        height: 100%;
        padding: 0;
        margin: 0;
      }
      button {
         display: inline-block;
         padding: 10px 20px 10px 20px;
         background: #ff5a5f;
         border-radius: 3px;
         color: #eee;
         font-weight: 400;
         font-size: 90%;
         margin: 0px 5px 0px 5px;
         z-index: 100;
         position: absolute;
         top: 20px;
         right: 20px;
      }
      button.hidden {
        display: none;
      }
    </style>
    <link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/3.14/themes/css/cartodb.css" />
    <!--[if lte IE 8]>
      <link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/3.14/themes/css/cartodb.ie.css" />
    <![endif]-->
  </head>
  <body>
    <div id="map"></div>
    <button class="hidden">Clear filter</button>

    <!-- include cartodb.js library -->
    <script src="http://libs.cartocdn.com/cartodb.js/v3/3.15/cartodb.js"></script>

    <script>
      function main() {
          var map = new L.Map('map', {
            zoomControl: false,
            center: [38.893, -77.03],
            zoom: 13
          });

          var layer = L.tileLayer('http://{s}.basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png',{
              attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors, &copy; <a href="http://cartodb.com/attributions">CartoDB</a>'
            }).addTo(map);

        cartodb.createLayer(map, 'https://team.carto.com/u/eschbacher/api/v2/viz/5f709a68-f9e3-11e4-abb5-0e4fddd5de28/viz.json')
            .addTo(map)
            .done(function(layer) {
                // Do further things here
                var lots = layer.getSubLayer(0);
                lots.setInteractivity('cartodb_id, elevation');
                lots.setInteraction(true);
                lots.on('featureOver', function(e, pos, pixel, data) {
                    console.log("Elevation: " + data.elevation + " feet.");
                });
                lots.on('featureClick', function(e, pos, pixel, data) {
                    var newSql = "SELECT * FROM dc_tax_lots WHERE elevation >= "+(data.elevation - 30)+" AND elevation <= "+(data.elevation + 30);
                    console.log("SQL: " + newSql);
                    lots.setSQL(newSql);
                    $('button').removeClass('hidden');

                    $('button').click(function(){
                        $('button').addClass('hidden');
                        lots.setSQL("SELECT * FROM dc_tax_lots");
                    });
                });
            })
            .error(function(err) {
                console.log('error error');
            });
      }
      window.onload = main;
    </script>
  </body>
</html>