ft todo

ft-todo.md

A few things on my to-do list.

1/ Implement cluster API - http://nodejs.org/api/cluster.html

Can we get a simple demo application working with cluster based deployments on ft platform?

Some notes here - https://gist.github.com/jedi4ever/6079180

Clustering seems useful for, a) using multiple processors on the same machine, b) not having to rely on doing a forever restarts when you deploy, c) no-downtime deploys, and possibly d) dynamic scaling.

or http://blog.keithcirkel.co.uk/load-balancing-node-js/ - perhaps it's not the answer right now?

2/ Profiling and instrumentation

Node on FT Platform doesn't give us a huge amount of information about what it's doing - cpu, memory, leaks etc. so would be useful to investigate what information we can get from the boxes and expose it.

Eg, http://blog.nodetime.com/2012/07/detecting-memory-leaks-in-nodejs-apps.html

Likewise, the presentation tier is going to be very dependent on making HTTP calls to the service layer but contains no instrumentation by default (dns lookup time, response times).

3/ Express is ok but largely feels like syntax sugar over the base node stuff. Plus it feels like at the end of it's life time.

What would a simple node HTTP server from scratch look like? What are we missing out on?

• http://koajs.com/
• https://medium.com/code-adventures/4ba9e7f3e52b

4/ Offline build processes are horrible. We should aim to make everything dynamic, on demand.

5/ Dependencies. Rhys had a nice idea for visualising npm dependencies, and changes, licenses, and general activity around our packages.

It freaks me out every time I run npm install that we are wedded to hundreds of small, unvetted (?), modules without a clear idea of their status or whether they are maintained.

6/ Docker. See if this makes deployment simpler (or at least less about transfering and unpacking lots of little files).

I actually implemented a basic Cluster implementation for the metadata-cache app on Heroku. Did a brief load test and there wasn't a significant difference between clusters vs dynos (although admittedly this was hardly thorough, and would be interesting to do with proper instrumentation). The main benefit I thought was the forever aspect (i.e. refork when one of the cluster instances crashes, whereas heroku would just stay crashed) - but presumably forever on FT platform does the same.

Also it had the unfortunate downside of maxing out the number of processes that Nodetime (tier 1) could handle so couldn't get any decent instrumentation on it.

I quite like the idea of exploring alternatives to Express JS. Maybe it would be useful to do a retrospective (so to speak) of Fast FT (and any other node projects like Responsive News feed) pertaining to Node.JS/Express and the good/bad things you found using it - so we know what to look for with alternatives?

Sorry @adgad - I'm not getting notifications on my gists, so I missed this. Grab me for a chat tomorrow.

I actually implemented a basic Cluster implementation for the metadata-cache

ok, that's interesting. I remember when Nuno came to talk to us and said it's quicker/simpler just to spin up a handful of instances on the same box and have your load balancer just fire requests at the different ports.

I quite like the idea of exploring alternatives to Express JS
it would be useful to do a retrospective

ok - that's a good idea. Lets chat next week in more detail... in the meantime feel free to stick your notes here.

Express notes I've been keeping ...

• I don't like mixing static (css, js) and dynamic file (html) serving from the same process. It's ok in development but possibly fatal in production.
• I want a dynamic asset pipeline!
• I think if we had a lot of small (micro!) apps we would end up implementing the same boilerplate code again and again (origami, express, config, half a dozen middleware, health/metrics, tests etc.). How can we fix this?
• You need to restart express when you change a file
• Routes should be sharable between client and server.
• Feels like it should make much better use of events (Eg, I'm going to send a response event) so you can plug stuff in to it.
• TJ has left.
• If this becomes more widespread @ FT there's probably some simple tooling to be written.

I want a dynamic asset pipeline!

What's that?

TJ has left.

So sad. But Go does look good/tempting.

I want a dynamic asset pipeline!

Like the build service.

I've been admiring Hugo and Hexo. Both have something like that.

Consider using Koa over express. Also I like Swig for templating.