Server Setup 2: more setup.

Databases and App Servers.md

Welcome to Serverside.

Alright. So we setup a server on the previous installment of Will writes a markdown doc. Today we're gonna setup a database, setup nginx to reverse proxy to a nodejs script, and make a nodejs script for serving pages from that database.

Setup a Database.

So the only new piece of software we'll need is our database. What's a database, I'm gonna let wikipedia handle this one.

A database is an organized collection of data. The data is typically organized to model aspects of reality in a way that supports processes requiring information. For example, modelling the availability of rooms in hotels in a way that supports finding a hotel with vacancies.

from: en.wikipedia.org/wiki/Database

Wait that doesn't help at all! A database is a way to store data to retrieve it in a useful way later. Its like a spreadsheet vs. a text document. It becomes easier to see why thats useful over flat files as the data grows more complex, and you need to access it multiple ways.

Why use a database over flat files? Well, reasons. Its up to you when to decide the complexity is worth it. Moving on.

There's some options, like last time I'm just going to be dictatorial and pick one for us, but for sake of completeness, there's two big buckets of databases you'll run across these days.

• SQL databases: MySQL, Microsoft SQL, Others
• NoSQL databases: MongoDB, Redis, Cassandra, Others

As you can probably surmise from NoSQL being named in opposition to SQL, SQL was ascendant for most of recent history, NoSQL only came onto the scene as a viable option in the last few years. SQL is good for sets of data you need to make complicated comparisons across, that you don't necessarily know in advance. Think about needing to get all the comments for a blog post vs. all the comments by one user, SQL (really relational databases) are good at that. Also in practice SQL databases are stable and durable, which means you have to try harder to lose everything.

So why use NoSQL then, well a few reasons. NoSQL databases commonly are configurable in multi server environments in such a way to allow you to keep the system up even if individual instances fail. NoSQL also doesn't require a structure be defined before hand, commonly, which is a boon in development. What you may lose out on in those big systems is consistency (changes can take time to propagate in big arrays), and flexibility in querying. If I wanted to get all the comments by a user and they were stored on a per blog post level I'd need to look at all the blog posts, or more realistically, store references to the comments in both places. Its a thing.

That said, in the mythical real world, you should pick a datastore that fits your application, in reality even in these two big buckets there's a lot of diversity in how they work, and its even reasonable in a more complicated application to use more than one. This is one of the reasons I do small personal projects with them, to have an idea how to use them so I can better assess them for real work.

So! We're going to use MongoDB, which is a document based NoSQL database. What that means for us is we can store things that look like JSON objects that you might be familiar with without deciding how to structure them in advance. This makes interfacing MongoDB with Javascript relatively easy. It also does some other cool stuff.

Don't take this as an endorsement of MongoDB for all or even your specific application, just. We have to use one okay? Okay.

Alright, down to business, get yourself logged in to your server. We'll use our old friend apt-get to install MongoDB.

In Terminal

sudo apt-get install mongodb-org

Have a look at the changes, then go ahead and tell it Yes. Once that installs go ahead and try and start it up.

In Terminal

sudo service mongod start

Assuming that completed successfully, Mongo is running, lets connect to it from command line and poke around a little. We can connect to the database thats running using a command line utility called mongo. Go ahead and type that in. You can also invoke mongo with options in order to, for instance, connect to a database on another server. But by default it does the right thing and connects to the local database on the standard port.

So you should see something close to this.

MongoDB shell version: 2.6.5
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2014-11-21T13:19:08.055-0800 [initandlisten] 
2014-11-21T13:19:08.055-0800 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> 

Alright, now for brevity and to avoid too much goofy formating, I'm gonna paste a log of some commands to show you basically how to interact with Mongo.

> use newDB
switched to db newDB
> db
newDB
> db.users.insert({name:'will',password:'password'})
WriteResult({ "nInserted" : 1 })
> db.users.find()
{ "_id" : ObjectId("546fad1037f148b1d0c3d5f4"), "name" : "will", "password" : "password" }
> db.users.insert({name:'bob',password:'password'})
WriteResult({ "nInserted" : 1 })
> db.users.count()
2
> db.users.find()
{ "_id" : ObjectId("546fad1037f148b1d0c3d5f4"), "name" : "will", "password" : "password" }
{ "_id" : ObjectId("546fad8c37f148b1d0c3d5f5"), "name" : "bob", "password" : "password" }
> db.users.find({name:"will"})
{ "_id" : ObjectId("546fad1037f148b1d0c3d5f4"), "name" : "will", "password" : "password" }
> db.users.find({name:{$ne:'will'}})
{ "_id" : ObjectId("546fad8c37f148b1d0c3d5f5"), "name" : "bob", "password" : "password" }

See, pretty easy. MongoDB is very accommodating. We make a database just by naming what database we want to use and we make a collection of stored data on that db (now loaded as the db object) just by making up a property name we like. Now we can insert and retrieve things from it using methods in a way that feels very similar to interacting with Javascript objects. Cool right?

A couple of things to notice. MongoDB automatically is assigning unique ID's to things when we insert them. Also, the find method by default returns everything in the collection, but you can pass it parameters that are tested against the values in the collection. We can retrieve will by name, or in the second instance, retrieve everyone without that name. Lets focus on that one for a second.

> db.users.find({name:{$ne:'will'}})

See that weird thing, $ne? Thats a query operator in Mongo. This one is 'not equal'. What this line is really saying is 'in db.users, find me all the things with a name thats NOT EQUAL to will'. There's all the ones you'd expect, greater than, less than or equal to, etc. And some ones you wouldn't expect like $where which tests against a JS function, or $exists that checks if a field is there. There's even boolean ones, like $and.

What find returns looks like a flat list of items, but what it returns really is more useful than that. It returns what Mongo calls a cursor. Cursors contain that list of documents, and contain functionality to iterate over the results. More on that when we get into node.

I know, its a lot to take in, but the important thing right now is to know whats possible in this context. In the name of tidying up. Lets get rid of that database we just made, and logout.

> db.dropDatabase()
{ "dropped" : "newDB", "ok" : 1 }
> exit
bye	

Reconfiguring Nginx to reverse proxy.

Now we're back out at our command line. Going back to my thesis, we've setup our database, now we need to muck with Nginx some more. Lets open up the site configuration we made last time.

In Terminal

> cd /etc/nginx
> cd /sites-available
> ls
my_site
> sudo nano my_site

It probably looks similar to this.

server {
    root /home/newusername/www;

    location / {
    }

}

We're gonna add a subpath that forwards to the node app we're gonna write, this way the root folder of the server still serves your static stuff. In reality this may not be what you want, but its what we're doing.

server {
    root /home/newusername/www;

    location / {
    }
    location /app/ {
    	proxy_pass http://127.0.0.1:8080;
	}
}

So, as you can probably see, if a request hits SERVER_URL/app/, its going to route it to 127.0.0.1:8080. My choice of /app/ as that path was arbitrary, it could be /dave/ or even /. If you're new to networking (I know you aren't Alex but we're doing these for posterity) 127.0.0.1 is the loopback address, that means its always the machine you are using. The :8080 means connect on port 8080. Specific applications attach themselves to certain ports as a way of keeping their connections separate while still allowing other applications access to the network. Our node script will use 8080.

Thats all the changes we need, go ahead and save that with CNTL+O and quit with CNTL+X and restart nginx.

In Terminal

> sudo /etc/init.d/nginx restart

Assuming that went [OK] going to SERVER_URL/app should do... well nothing yet. We're gonna get to that now.

Node time.

Alright. So we have requests from the internet getting routed away from the filesystem and we have a database eager to store things for us, now we just need the thing in the middle that makes sense of what the user wants and talks to the database. This obviously depends on what your application does, but in general there's two sorts of things we're going to do.

1. Serve Pages Using Data from the Database.

Someone is accessing your blog at /app/Some-Post-Title, our app needs to see 'Oh they want Some-Post-Title', ask the database if it knows anything by that name. If there is, it sends it to the user, probably formated into a nice HTML representation. If there isn't a post by that name, or there's some problem completing that operation, we need to send an error reflecting what went wrong to the user.

2. Accept Input and Check it Against or Store it in the Database.

Someone is trying to register for your site, or post a comment. We need to process and store that information in a way we can retrieve it later, assuming the content they gave us is valid. If it is we save it, and then we need to send an acknowledgment of success. If its not valid data, or there's another problem, we need to send an error. This goes for login forms too, with the added caveat that they check data against the server and then allow that user access to privileged areas. We'll deal with that in the future, but its an extension of these same principles.

There's a lot of ways we can do this (this is a theme in web dev specifically, and in your life more broadly, I'm sure). We're going to use nodejs, but you can make web applications in Ruby, or Python, or a million other things.

So, to business. Lets setup a node project in your Home directory.

In Terminal

> cd ~
> mkdir code
> cd code
> mkdir example
> cd example
> npm init

That launches the npm Wizard that helps you setup a new project, just follow along. If you don't have an answer for any of the fields, just hit enter to continue. Once thats done there's a couple of node modules we're going to need. Thats what npm is for, as the Node Package Manager it allows you to manage libraries your code depends on a per application basis. Its sort of like apt-get for nodejs. We're going to get a couple of packages I know we'll need, in reality you can do this as the project evolves, it doesn't have to happen all up front. We're going to install a mongodb library and express, a web framework.

In Terminal

> npm install --save express
and some console spam later
> npm install --save mongoskin
and some console spam later
> npm install --save jade

It'll download and install those in your project folder now, so you can use them with nodejs. Easy. Alright, so lets make a little hello world in node.

Download index.js and put it in the root folder of your example folder, and create another folder in example called 'views', put home.jade in there.

From the examples folder, run the index.js file with node.

In Terminal

> node index.js

And go to SERVER_URL/app/

Will ran out of time to write this, so we'll cover ways to modify the node file live in person.

home.jade

doctype html
html(lang="en")
	head
		title="Mongo Test - #{title}"
	body
		h1 #{title}
		#post
			| #{body}

index.js

var express = require('express')	//for the server bits
var app = express();
var mongo = require('mongoskin');	//to talk to the database
var jade = require('jade');		//to render the template

//configure express to use jade for templates
app.set('views', __dirname + '/views');
app.set('view engine', 'jade');

// mongo url to our database, and to a database named site.
var url = 'mongodb://localhost:27017/site';

//connect to db
var db = mongo.db(url, {native_parser:true});
//set up posts as a collection
db.bind('posts');

//upsert, insert, or if a similarly named post exists, update the existing one
db.posts.update({name:'home'},{		//named home
	name:'home',			//new data
	title:'A test title.',
	body:'Frankfurter boudin tail chicken swine meatball. Hamburger pastrami ground round chuck pancetta brisket rump landjaeger tongue short loin shank shoulder pork chop salami bresaola. Kielbasa tri-tip chuck capicola pork chop pork loin corned beef. Spare ribs ham hock filet mignon flank strip steak ribeye leberkas, kevin meatball andouille. Tongue flank prosciutto beef alcatra, short loin bresaola landjaeger.'
}, {upsert:true}, function(err, result){	//yes an upsert
	console.log(err,result)			//after the upsert
});

//when someone goes to root
app.get('/', function(req, res){
	//get the thing named home from posts
	db.posts.findOne({name:'home'},function(err,result){
		if(err) res.status(404).send(JSON.stringify(err));	//if there was a problem
		else res.render('home',result);				//otherwise render the thing (an empty result isn't an err)
	})	
});

//server listen to 8080
app.listen(8080);