What's the big deal with Node.js and `await`?

whats-the-big-deal.md

What's the big deal with Node.js and await?

So I really don't like most new things. The only reason I even got involved with Node.js early on was because of its sweeping promise: "One language, spoken by developers across the world." By consolidating parallel efforts, it would accelerate the pace of innovation, and quickly lead to more transformative and disruptive developer tools. It would create tremendous value for software businesses by unlocking efficiencies in the hiring and implementation process. And best of all everyone would waste less time on boring stuff.

Still, there was a problem. While it's true most developers have touched some JavaScript callbacks up there in browserland, in the bowels of the application server, there tends to be a lot more asynchronous things going on. And that causes all sorts of issues. All those callbacks also make for a way steeper learning curve for developers (especially if they're also front-end developers learning to write server-side code for the first time.) In fact, that learning curve was what led some bright-eyed early adopters to abandon the technology altogether. Some of the HackerNews contingent even denounced it outright. ("Node.js? Please, no. Everyone knows it has callback hell.")

But incredibly, it didn't matter. Node.js grew like bonkers. Its focus on performance attracted enterprise workloads from the likes of PayPal and Walmart, and its developer appeal combined with cost-savings allowed it to woo major brands like MasterCard while still impressing startups like Uber.

Meanwhile, its core libraries quickly grew stable, the NPM ecosystem flourished, and Node core earned the respect of many open-source developers. In retrospect, it feels like there was a lucky cocktail of several different factors helping it along: Not just Ryan Dahl's genius engineering on top of V8, but the simultaneous innovation of NPM, which was destined to become the most popular package manager in the world.

But I've been less thrilled about some other things that have hit the JavaScript world since. I'm not a big fan of const. ES6 modules? Not my fav.

Arrow functions? Destructuring? Spread operators? New syntax for string literals? Pretty neat... but even more stuff for new JavaScript developers to learn before they can be productive. Because of all these new additions, now there's twice as much variety of core JavaScript syntax floating around. And that means it all needs to be taught.

Needless to say, I'd been holding on to ample skepticism about new ES features.

But then one day that all changed. Because this happened:

var orgs = await Organization.find();

I can't begin to stress how important this is!

Node.js apps and libraries written with await are faster to design and safer to edit. They have fewer bugs; and even when they do throw errors, those Error instances come with much better stack traces, and they no longer risk crashing the process by going unhandled in asynchronous callbacks or promise chains.

Consider the following if statement. It's similar to a normal if statement, except that either the "then", the "else", or both sides have asynchronous logic. If we aim to do this without await, it can be a bit involved to get the two parallel universes to "converge"-- i.e. do the same thing afterwards regardless of which path was taken. The real obstacle is that we're using callback functions as a mechanism for flow control, which means it's not obvious how to do even the most basic sorts of flow control. Fortunately, at Sails Co., we developed a strategy for recognizing this "if…then…finally" pattern and solving it generically, so that a hand-rolled, custom lock doesn't have to be built every time.

For now, let's aim to do this without await, and use our special strategy: a "pocket function" (aka "self-calling function-- or to the cool kids, an "IIFE"):

// (before `await`)
((proceed)=>{
  if (Math.random() > 0.5) {
    return proceed();
  }

  setTimeout(()=>{
    var result;
    try {
      result = JSON.parse(foo);
    } catch (err) {
      return proceed(err);
    }
    return proceed(undefined, result)
  }, 1000);

})((err, resultMaybe)=>{
  if (err) {
    return res.serverError(err);
  }
  return res.send(resultMaybe);
});

The approach above of using a pocket function works OK, and at least you don't waste time designing the control structure from scratch. But it is still fragile, and very error-prone. Remember, the slightest typo in an asynchronous callback could crash the entire process! That's why we have to use the try block-- since JSON.parse() can hypothetically throw, and since we're in an asynchronous callback, we have to write code in such a way that any thrown errors could be caught and dealt with.

Of course, this extreme edge case handling is a risk in and of itself. Since it's very unlikely to run, it usually doesn't get tested as well, leaving it a potential hotbed for copy/paste mistakes, logic errors, or even simple typos, with potentially disastrous consequences.

Plus, it's hard to rearrange-- imagine how many lines of code would be affected just to simply move the JSON.parse() out to the very top.

Think of the potential merge conflicts! Not to mention the cognitive load and distraction of figuring all that stuff out and holding all those concerns in your head. Consider how that's affecting your productivity as a software developer! Finally, roll that up one more level across all the developers on your team and consider how it might cumulatively affect the productivity of your employer, or your business.

And yet... we still use Node.js. It's that good. (Again, not always the tool itself-- the vision. One language to rule them all. Just imagine someone trying to make the claim: "I've heard that Google Chrome, Firefox, Edge, Safari, and Opera are all switching to Haskell. They said they were tired of the W3C.")

Despite having to deal with all that--the risk to stability, the technical debt for future maintainers, the distraction of literally hand-rolling your own "if" statements--people still use Node.js. "Callback hell" is an overwhelmingly strong argument against Node.js/Express/Sails.js (and a valid criticism), but despite that, the number of developers, companies, and governments using the technology is growing at a record pace.

As-is, earlier this year, before support for await was added to core, Node.js was already set to outpace the market share of Java by the end of 2018. One more time: Node.js is set to outpace the market share of Java in less than a year, in spite of this callback problem.

But now, check this out.

Here's what the same code looks like now that we don't have to do all that IIFE nonsense anymore:

// (now, with `await`)
if (Math.random() > 0.5) {
  return res.send();
}

await sails.helpers.flow.pause(1000);
var result = JSON.parse(foo);
             //^ Note: No `try` block necessary here!
             // If it throws, it'll just go straight up to the outer logic,
             // where it can be handled naturally.  (For example, both Sails
             // and Express automatically catch these uncaught errors.  In Sails,
             // they go to res.serverError() automatically.)
return res.send(result);

This is an obvious improvement. But, as we alluded to above, there are more nuanced advantages still. Let's recap:

First of all, it's safe. If it throws, it won't crash the entire process. And it doesn't require the invention of custom control flow structures-- we just use an if statement (something that exists in literally any common programming language).

Next, there's no extraneous error handling. So none of the risk of introducing accidents that comes along with it, either. And there's less danger of doing weird things with try/catch blocks -- since without the fragile scope of asynchronous callbacks to protect, you need unleash only a fraction (if even any at all) of the try statements you would have to whip out otherwise.

So finally, we're down to 4 lines of... pretty normal-looking code.

No matter what programming language or background you come from, JavaScript still might not be your cup of tea, even without "callback hell". But I'm not sure it matters, because now, for the first time, writing safe, server-side JavaScript code is as simple as writing server-side code in any other mainstream language. How could you not use it by default?

And this is why await is such a big deal for Node.js. It's not just another ES* feature to put in our toolbox-- it's the end-all-be-all solution to the greatest challenge Node.js has ever faced. When Node 8 was released as LTS 4 weeks ago (Oct 2017), it removed the last remaining roadblock holding back massive, worldwide, mainstream adoption of Node.js and JavaScript. People seem to be approaching await cautiously; as if they're scarcely able to believe it is really possible. (I know that's how I felt the first few days.) But eventually, word will get around, teams will start to measure the dramatic benefit to time-to-market, user experience, and code quality.. New tutorials will be released. And faster than we think, Node.js will become the platform of choice across every industry, every job market, and every "Intro to Computer Science" course.

-m

If you have feedback about this post, or another take on await and its impact on Node.js, I'd love to hear your thoughts. You can find me on Twitter @mikermcneil.

I agree on the usage of async/await I've been waiting for it for quite some time (thanks to typescript anyways I didn't have to wait too much) What i'm still unsure is exception handling since with promises as long as we return the promise exceptions propagate to the last catch
but on async functions I think I might have steped into cases where the exception is swallowed and just warn in the console that in future versions it will crash, how do you handle this kind of situations?
lets say something like this

async function P1() {
  let result = JSON.parse(""); // this won't crash the process but I'm sure it won't propagate
  // some more logic here
  return result;
}

async function anotherFn() {
  // as far as I'm aware if the async function inside trycatch **throws** explicitly it will, else it won't be catch
  try {
     let finalresult = await P1();
     console.log(finalresult);
  } catch(error) {
    console.error(error);
  }
}

anotherFn();

I'm sure the developer has to be aware of exceptions but perhaps someone just checks in code like that
any ideas on what kind of safeties should we enforce to prevent code like that?

@AngelMunoz I think of async functions as a separate thing tbh. await works with any “thenable”:

var parley = require(‘parley’);

await parley((done)=>{
  setTimeout(()=>{
    return done(undefined, result);
  }, 500);
});

^^^ that’s how waterline methods work. Parley returns a “Deferred”, not a promise. But it’s “thenable”, so it works. Similarly parley takes care of all the potential issues that it can (throwing, double invocation of the cb, automatic stack trace management if you pass in a “flaverr.omen()”, etc)

That said, nothing wrong with using vanilla async functions — they’re great if all their internals support await. Parley is good when not all of the internals support await, or you need to target an audience who doesn’t necessarily have support for await in their js interpreter (eg backwards compatibility in sails for node 6)

One last thing about Deferreds and try/catch: if you haven’t already, check out .intercept() and .tolerate() (see recent conversation in sails gitter from evening thu dec 21, 2017 for context— sorry im on my phone so don’t have a link handy)

What's wrong with const?

@isomorphisms Nothing, really! Except... well, I don't like how it's confusing for new developers being introduced to JavaScript :(

For example:

const foo = { phone: '512-555-5555' };
foo.bar = 'insanity';
console.log(foo.bar);
//=> 'insanity'

...so, wait-- it's called "const", but it's not a constant?

I know you can work around this with .freeze(), and writing your own custom recursive loop to "deep freeze" your own deeply-nested true constants-- and I realize there are minor performance benefits to be had from using const, long term-- but still, personally, I don't think it's worth it. I prefer to teach using var in general, and using let within if or for blocks where it's necessary to avoid potential unintended consequences/mistakes due to hoisting. I think using let in general would also be a perfectly valid approach.

But that's neither here nor there- all I'm trying to argue in this post is that await is way more important