How we built Now?

post.md

History of foursquare and swarmapp. Use of foursquare.

We built an app for showing live foursquare check-ins.

Key Ingredients

The now recipe can be broken into two key pieces: the backend worker and the user interface.

1. Backend Worker: (How do we fetch data from Twitter and Foursquare?)
   • We use the Meetup's long-polling RSVP stream endpoint to get realtime RSVPs.
     Note: This demo wouldn't have been possible without meetup supporting an API endpoint to extract the RSVP information.
   • We then store this data in appbase.io, which provides us a convenient way to query both historical data and realtime feeds - we call it the streaming DB ;).
2. User Interface:
   • Querying appbase.io for meetup RSVP feeds by cities and topics.
   • The UI / UX logic maintained with a jQuery frontend. And we use typeahead for displaying the search interface.

What is NOW?

Now shows live Foursquare checkins from people across the globe. It has some neat filters that allow you to select a city and the check-in categories.

{<1>}

Image: Let's check for RSVPs in the all the singles tagged meetups in San Francisco.

Now when a person RSVPs for a singles meetup, you would see it show up on the now feed. Now don't get any ideas for stalking unsuspecting people. It's a cool way to find the most happening meetups!

A fun fact: While building this app, we noticed that there are about 120 RSVPs on meetup.com every minute, that's 2 every second. Or in 30 days, we would see over 5 million RSVPs - that's a good data set size to query in realtime ;).

Deep Dive

Now that we know what now does, let's get into the meat of how the app works.

Backend Worker

Our backend worker is a simple Node.JS code that keeps running forever on a digitalocean droplet.

How do we get the data from Twitter + Foursquare APIs?

A diagram on the workflow.

// Replace the code below.

http.get(meetup_url, function(res) {
	res.on('data', function(chunk) { // 'data' event handler is called everytime there is a new meetup RSVP
		var data = JSON.parse(chunk);
		meetup_data.push(data);  // capture the RSVP responses in an array
	});
});
 

// Replace the code below.

appbaseRef.index({
	type: DATA_TABLE,            // collection to store the data into
	body: meetupData[dataCount]
}).on('data', function(res) {
	console.log("successfully indexed one meetup RSVP");
})

Meetup provides us a JSON object for every RSVP. We then write this data into appbase.io as soon as it arrives. appbase.io is built as a streaming layer on ElasticSearch, and provides a continuous query interface for streaming JSON results.

A RSVP JSON data looks like this:

<Paste an example JSON response data>

User Interface

The User Interface is a small frontend that queries appbase.io for realtime checkins based on either city name or map geolocation coordinates.

{<2>}

Image: Showing meetups by city and topic tags - try it live.

The app directory structure looks like this:

// Change the app structure to reflect the Now app

now/
|_ _ _ _ app/
|        |_ _ _ _ app.js
|        |_ _ _ _ request.js
|        |_ _ _ _ variable.js
|_ _ _ _ index.html

The codebase can be accessed at the now github repo.

.js

This is the entry point of the codebase and brokers responsibility to variable.js and request.js.

data-contoller.js

// update what data-controller file does.

Creates a meetup_request prototype which is responsible for three different queries.

1. Fetching the list of the 1000 most popular cities and topic where meetups happen,
2. A realtime query fetching live meetup data from the selected cities and topics,
3. Pagination showing older RSVPs when a user scrolls to the bottom of the meetup feed.

index.js

Instantiates a meetup prototype object which is responsible for the tag selection logic when a user ticks a city / topic and displaying a new meetup RSVP inside the feed.

{<3>}

A sneakpeak into continuous queries

Here's a sneakpeak into how the continuous query for fetching live meetup data works.

We use the appbase-js library (available via bower and npm) for making continuous queries over the meetup feed data.

appbaseRef = new Appbase({
    url: "https://qz4ZD8xq1:[email protected]",
    appname: "meetup2"
})

appbaseRef.streamSearch({
    type: "meetup",
    size: 100,
    body: {
      "query": {
        "filtered": {
          "filter" : {
            "terms" : { 
              "group_city_simple" : ["San Francisco", "Ahmedabad"]
            }
          }
        }
      }
	}
}).on('data', function(res) {
   if (res.hits.hits) {
      // feeds are returned in the field 'res.hits.hits'
      console.log("initial feed size: ", res.hits.hits.length);
   } else {
      // realtime updates with one new RSVP
      console.log("new data feed: ", res._source);
   }
})

streamSearch() provides a data handler that returns:

1. the matching records to our query of find meetups with Social or Singles topics,
2. New records as RSVPs in Social or Singles topics arrive.

Summary

We created now as a way to find realtime checkins around the globe.

1. We use the Twitter streaming API for finding all the tweets with a swarmapp link, which we then send to the Foursquare API for getting the check-in JSON response.
2. We use appbase.io to transform this into queriable live data.
3. We put together a small frontend that allows neat filtering based on city selection and categories or by browsing the map view.

Without further ado, here are the important links: now worker code, now and the demo.