Say, you want to save your D3 application in a CouchDB database. This is just a proof of concept that this is possible. I use the »Focus + Context« diagram by Mike Bostock (http://bl.ocks.org/1667367) as an example.
Prerequisites:
- A running CouchDB (see http://couchdb.apache.org/; in my Windows installation CouchDb is started as a Windows service when Windows starts).
- An installation of couchapp (how to install couchapp, see http://couchapp.org/page/index).
- Your favorite editor.
The steps are quite simple.
- In your favorite workspace create a new folder and name it say d3apps1.
- Within this folder create a file named .couchapprc. This file will be used by couchapp to create the CouchDB database.
- Open this file in your text editor and add an empty object: {}. Nothing more is required for couchapp to create the database!
- In the d3apps1 folder create a subfolder named _attachments.
- Copy index.html and sp500.csv of the »Focus + Context« example into the subfolder _attachments.
Now you should have the following folder/file structure:
d3apps1
_attachments
index.html
sp500.csv
.couchapprc
Open a command line window and navigate to your d3apps1 folder. In the command line window issue the command (of course, couchapp must be on your path; otherwise you must supply the full path to couchapp.bat):
couchapp push d3apps1
That's it. Now in your browser, you can navigate to http://127.0.0.1:5984/d3apps1/_design/d3apps1/index.html and admire your d3 application.
Of course, this just shows that you can integrate and serve (!) D3 applications from a CouchDB without any middleware. There is still one external dependency: the d3 library, that is loaded from GitHub. This example doesn't use any features of CouchDB to store and deliver the data. This is just preliminary work for putting the data of my own applications into a CouchDB database. I thought this might be useful for others. How the d3 library and the data can be integrated into CouchDB is shown in part 2.
This one is easy. First, make a copy of your complete d3apps1 folder and store it in your workspace using the name d3apps2. Then just move a copy of d3.v3.min.js into the _attachments folder. Then it should look as follows:
d3apps2
_attachments
d3.v3.min.js
index.html
sp500.csv
.couchapprc
Delete the references to the external d3 library files and insert a reference to the local d3.v3.min.js file. The beginning of your index.html file should look as follows (with or without the out-commented lines):
<!DOCTYPE html>
<meta charset="utf-8">
<!--
<script src="http://mbostock.github.com/d3/d3.js?2.7.2"></script>
<script src="http://mbostock.github.com/d3/d3.csv.js?2.7.2"></script>
<script src="http://mbostock.github.com/d3/d3.time.js?2.7.2"></script>
-->
<script src="d3.v3.min.js"></script>
From within your d3apps2 folder, push your code into the CouchDB:
couchapp push d3apps2
Now D3 will be loaded from CouchDB.
First, under _attachments create a new file called import.html. Then add the following code to import.html:
<html>
<head>
<script src="/_utils/script/jquery.js"></script>
<script src="/_utils/script/jquery.couch.js"></script>
<script src="d3.v3.min.js"></script>
</head>
<body>
<script>
// Adapt the values to your needs
var file = "sp500.csv",
dbName = "d3apps2",
docId = "sp500",
source = "FocusAndContext";
d3.csv(file, function(data) {
var doc = {
_id : docId,
souce : source,
data : data
// Add any fields you need
};
$.couch.db(dbName).saveDoc(doc, {
success:function (data) {
console.log("New doc created");
},
error:function (status) {
console.log("error:", status);
}
});
});
</script>
</body>
</html>
From within your d3apps folder, push your code into the CouchDB:
couchapp push d3apps2
Navigate to your import file:
http://127.0.0.1:5984/d3apps2/_design/d3apps2/import.html
That's it. The file is loaded and imports your data into CouchDB. You may control your import by navigating to
http://127.0.0.1:5984/_utils/database.html?d3apps2
There you find a new document named "sp500". If you click on it, you can drill down to individual data.
A Warning: This import file is just a quick hack to get data into the CouchDB. There's no error checking. You can execute this import only once. If something goes wrong, you have to delete "sp500" manually using Futon, the rudimentary database manager of CouchDB, fix the error and try again. After the data has been imported successfully, it's no longer needed in this application. But don't yet delete it. You need it for the third part. And keep the import.html file, because you might want to import additional datasets into your CouchDB database, using the same import pattern.
How you can access the data in CouchDB from your D3 application, is shown in part 3.
First, make a copy of your complete d3apps2 folder and store it in your workspace using the name d3apps3. Your folder structure should look as follows:
d3apps3
_attachments
d3.v3.min.js
import.html
index.html
sp500.csv
.couchappr<br>
To access the data in your d3apps3 CouchDB, you need to change your index.html file. At the beginning you must insert two jquery scripts:
<!DOCTYPE html>
<meta charset="utf-8">
<script src="/_utils/script/jquery.js"></script>
<script src="/_utils/script/jquery.couch.js"></script>
<script src="d3.v3.min.js"></script>
<style>
The d3.csv function is no longer needed. Instead you retrieve the data from the database. Replace the function
d3.csv("sp500.csv", function(data) {
...
});
completely by following code:
// This function replaces the d3.csv function.
$.couch.db("d3apps3").openDoc("sp500", {
success : function (doc) {
var data = doc.data;
data.forEach(function(d) {
d.date = parseDate(d.date);
d.price = +d.price;
});
x.domain(d3.extent(data.map(function(d) { return d.date; })));
y.domain([0, d3.max(data.map(function(d) { return d.price; }))]);
x2.domain(x.domain());
y2.domain(y.domain());
focus.append("path")
.data([data])
.attr("clip-path", "url(#clip)")
.attr("d", area);
focus.append("g")
.attr("class", "x axis")
.attr("transform", "translate(0," + height + ")")
.call(xAxis);
focus.append("g")
.attr("class", "y axis")
.call(yAxis);
context.append("path")
.data([data])
.attr("d", area2);
context.append("g")
.attr("class", "x axis")
.attr("transform", "translate(0," + height2 + ")")
.call(xAxis2);
context.append("g")
.attr("class", "x brush")
.call(brush)
.selectAll("rect")
.attr("y", -6)
.attr("height", height2 + 7);
},
error : function (status) {
console.log("Doc not found");
console.log("*** error:", status);
}
});
Don't forget: Change dbName in your import.html file to d3apps3!
From within your d3apps3 folder, push your code into the CouchDB:
couchapp push d3apps3
Import your data into the CouchDB database by navigating to:
http://127.0.0.1:5984/d3apps3/_design/d3apps3/import.html
If the import has been successful, the sp500.csv file in the _attachments folder is no longer needed. If you haven't done so, delete it now.
That's it. Now your data will be loaded from the CouchDB database.
On your local machine, navigate to
http://127.0.0.1:5984/d3apps3/_design/d3apps3/index.html
You can watch the result here.
If you followed this tutorial, you now should have three CouchDB databases: d3apps1, d3apps2, d3apps3. They demonstrate different degrees of integration of an D3 application into a CouchDB database. The last one (d3apps3) has no dependencies on external files whatsoever (no libraries, no data files). Everything is contained in the database. Neither middleware nor a framework is required to access and display the data. The application and the data can be moved around, copied, replicated, deployed, etc. just by applying these operation to the database file. The only requirement is that the recipient has CouchDB running (which is available for all major operating systems).
This tutorial is the result of about a week of researching and figuring out, how d3 can be integrated with a database. Of course, this is a simple application, and the data is contained in a single document. The application has been taken verbatim from Mike Bostock's example, and no 'real' database features (records, selections, filters, etc.) are used. To show that this is not a toy application, some issues must be solved. It must be shown
- that one can work with many documents (records). Because I'm interested in (historical) timelines, my target at the moment is around 50,000 to 100,000 documents.
- that one can integrate richer content (pictures, videos, audio files, etc.). There are many applications that use CouchDB to deliver content of this type.
- that one can build a richer user interface. Again, there are many application that feature rich interfaces with CouchDB.
Stay tuned!
Loading static data from a csv is fine, but really to use the power of CouchDB you'll want to be using views as the data source for your visualisation. This allows you to easily modify your dataset by adding/removing/editing documents.
First copy the app from a previous step over into d3apps4:
cp -r d3apps1 d3apps4
Next you'll need to make a view. Execute couchapp generate view d3apps4 pricetimeseries to create the view boiler plate. Remove the reduce function as we're not going to use that in this example (e.g. rm d3apps4/views/pricetimeseries/reduce.js). The reduce function is actually ideal for this kind of thing, you could have daily records being aggregated into single monthly records, but for simplicities sake we'll skip it for now.
The map function (d3apps4/views/pricetimeseries/map.js) should look like:
function(doc) {
if (doc.date && doc.price){
emit(doc.date, doc.price);
}
}
Next we need to change the javascript in d3apps4/_attachments/index.html to use the json from the view instead of the static csv. There are lots of ways to do this (jquery.couch.js, sag.js to name a couple) but d3 has json support baked in, so lets use that directly. Once online, the view is available at _view/pricetimeseries (in relation to index.html) so we can replace the d3.csv() function with:
d3.json("_view/pricetimeseries", function(viewdata) {
// We just want rows from the view in the visualisation
data = viewdata["rows"];
data.forEach(function(d) {
// the key holds the date, in seconds
d.date = new Date(d.key);
d.price = +d.value;
});
// rest of the visalisation code
As you can see we've had to pick out the rows from the view and change the data parsing to take into account the different format of the view data (see below). You can see the full index.html in this gist.
Once you've made these changes push the couchapp up into the database with
couchapp push d3apps4
If you visit your app now your visualisation won't have any data in it. We need to import the csv into CouchDB documents so that the view can process them. The gist contains a python script, import_csv.py, to do just this. You'll need the excellent requests library installed to use it. Copy the import_csv.py into d3apps4 and go into that directory. Run the script via python import_csv.py, if everything works you should see <Response [201]>.
The script reads and uploads the csv data with each row being an individual document. It converts the data ("Sep 2012") into milliseconds during the import. This is to make sorting the date simpler - we could do that in the view or the d3 code but it's simpler to do it at acquisition time.
Once the data is uploaded go visit the application at http://127.0.0.1:5984/d3apps4/_design/d3apps4/index.html. You should see the visualisation, rendering data from the view.