Hi, everyone, my name is Forrest Norvell, I’m a developer at npm, Inc., and I’m here to talk to you about the next version or two of JavaScript. I’m truly sorry that I’m what stands between you and lunch, but this is a talk I’ve wanted to give for a long time, and I hope you find it valuable.
I also compare you all to cows at least once in here, so that’s a thing.
ECMAScript 6, the standard that defines the next version of the JavaScript language, is probably going to be ratified towards the end of this year. It describes the most sweeping set of changes to the language in at least a decade, and is the end point of a lot of work and at least one fiasco.
As a Node developer who’s been writing fairly involved JavaScript full-time for the last 3 years (and been using JavaScript as a full-stack developer for at least a decade before that), I have a lot of friends who are also Node developers. As ES6 gets closer to standardization, a lot of these friends are skeptical or concerned about what they see as needless complication and wrong-headed design decisions.
I can see why you might feel that way – there’s a lot that’s going to change as the new features roll out, and it’s not clear how all of this new stuff is going to combine – but every risk can be viewed as an opportunity. I feel comfortable saying that everything that’s in ES6 as of now has a good reason to be there.
In this talk, I’m going to spend a little time talking about the trauma which shaped the development process of ES6, and then talk about what we can do as a community to figure out how to use ES6 effectively. Ultimately, the point of the talk is this: it’s up to us as developers to make ES6 a success, and we can do that if we work together and spread what we learn as widely and quickly as we can.
ES6 wasn’t developed in a vacuum. There are a lot of parties involved, and I think it’s helpful to map out who everybody is, and how they relate (or don’t) to each other. I group the contributors into three categories: standards committees, stakeholders (or implementers), and the developer community.
First, the committees, and in particular TC39. It’s impossible to discuss the development of JavaScript without mentioning TC39, or to give it its full name, Ecma International Technical Committee 39. TC39 is the steward responsible for ES-262, the ECMAScript standard. The two most important figures on TC39 are its chairman, Brendan Eich (who really doesn’t need any further description), and the editor of ES-262, Allen Wirfs-Brock. Without them, and the other members of TC39, the web platform as we know it would not exist. The membership of TC39 is largely self-selecting – a point, as Rick Waldron forcefully reminded me the other night, worth keeping in mind when you look at its makeup. The various member companies of Ecma nominate delegates to join TC39, and subsets of the delegates attend the individual face-to-face meetings where the bulk of the standardization work is done.
Turning to stakeholders, a language specification on its own isn’t very interesting, unless [points at self] you’re some kind of programming language nerd. You need implementations to actually do anything, and that’s where stakeholders enter the picture. You can also think of these teams as implementers, but I call them stakeholders because I want to be clear that the owners of the specs and the owners of the implementations are interdependent – neither can get a new version of the language standardized without buy-in from the others.
The important stakeholders have names that should be familiar to you – Mozilla, Google, Microsoft, Apple, Adobe – but there’s an interesting trend, which is the increasing importance of library developers in the standardization process. JQuery, Ember, Angular, and the AMD & RequireJS have all contributed substantially to ES6, and several of the most active members of TC39 come from that world. This is a new and very important development, because it brings the standard closer to us as developers.
You’ll note that I didn’t mention Node, and that’s because with certain significant exceptions, Node just uses an implementation of JavaScript, it doesn’t implement JavaScript itself. Despite what a lot of Node users think, it has very little control over its own language runtime. This is something that only seems weirder the longer you use Node, but strictly restricting itself to being a consumer of V8 has been key to its rapid development.
All of this work and wrangling is pointless without us, the JavaScript community. All TC39 can do is let us loose with the standard and see what we do with it. You can think of us as a herd of cows, let loose in high pasture. Over time, we’ll figure out the best ways to get around and the easiest ways to do things, and end up laying down down trails. If TC39 is doing its job, it will be watching what paths we’ve made, and come in behind us and tend and pave them for our convenience and safety.
This paints a picture of a tidy little ecosystem, with its own feedback loops and checks and balances. And, with some allowances made for the inescapable politics and wrangling that comes from any activity involving a lot of people with conflicting goals, that’s pretty much what’s actually happening.
With that out of the way, let’s talk about what happened six years ago, the last time a lot JavaScript was going through a lot of changes. What was the ES4 fiasco? Put another way, what made some of you out there a little skeptical about JavaScript’s odds of successfully being updated?
Standards bodies are strange beasts, with minds of their own, but they do have principles and goals – stated or unstated – and sometimes those goals fall into conflict. ECMAScript4 was a very ambitious, stakeholder-led project, spread out over many years. The stakeholders had many goals, among them being to add “proper” classes to JavaScript’s prototypal object system, to unify JavaScript with Flash’s ActionScript, and to generally take what had more or less accidentally turned out to be a cornerstone of the web and put it on a more solid footing.
At the same time, TC39 was working on ECMAScript 3.1, an incremental refinement of ECMAScript 3. The problem that most of you have probably encountered with branchy development is that multiple groups working on multiple versions of a single thing at the same time end up generating a lot of overhead and conflict when it comes time to figure out how to reconcile everything into a final end product. Ultimately, what sunk ES4, along with the extremely ambitious nature of the changes, was the fact that the team working on ES4 had essentially created an entirely new thing and called it JavaScript. They completely lost the plot, and the blamestorming was ferocious and sustained.
It’s hard to overstate how traumatic this was – years of spec work, stakeholder discussions, and implementation effort had gone into ES4, and while it was probably a relief to not have to argue about it anymore, the literal cost of all that wasted effort was staggering.
So the stakeholders sat down and hammered out a set of principles to keep this from happening again, which resulted in what’s been called ECMAScript Harmony. Originally “Harmony” was more or less a version code-name for ES6, but over time it’s come to refer to “the version of ECMAScript under development”, regardless of the version number, and ultimately “the set of principles governing the development of the standard.”
Broadly put, Harmony takes these principles as its goals:
- Don’t break the web. New features can’t come at the cost of breaking existing code. Whole search engines exist just to support this goal, and even a very small number of breaking sites is potentially enough to sink new features.
- New language features require concrete implementations. I would actually prefer that JavaScript follow the IETF’s lead here and require two complete implementations of a feature before calling it standard, although I know that’s unlikely to happen.
- Keep the language pleasant for casual developers. Look at all the things we’ve done over the past couple days, and it should be clear why keeping the language simple and easy enough for newcomers to get up and running quickly is so valuable. This may make working with the language tedious for power users, but whatever, we can write Haskell on our own time.
- Improve interoperation. Observe existing standards (explicit and de facto) whenever possible. This is another way of saying, “pave the cowpaths.” (© Rick Waldron)
- Preserve harmony. If no consensus can be found around a feature among the implementors on TC39 (and to an extent the broader stakeholder community), it doesn’t go in.
It’s important to note that while this all sounds pretty straightforward, it’s not something that is universal or even typical for language standards. Anyone who’s watched the evolution of C++ or Java knows what I mean. It’s easy to get wrapped up in the needs of implementors and standards committee wonkery, and to lose sight of any language’s primary constituency, which is people who need to write code to get things done.
Part of the motivation for ES4’s development was the hard-won emergence of the “web platform”. A lot of you probably remember the bad old days of the browser wars, when the majority of our time as web developers was spent trying to coax our code to do the same thing in multiple browsers, without crashing. One of the ways we got past this was by collectively deciding what sane, conventional JavaScript should look like.
Two things really helped firm up JavaScript as a language for serious software development.
On the one hand, jQuery is important because it took the pretty much ridiculous world of DOM programming and turned it into something that was actually accessible, as well as pretty much defining its own abstracted web platform.
On the other, Douglas Crockford wrote JavaScript: The Good Parts, which is in a lot of ways (some of them kind of uncomfortable) the Ten Commandments of JavaScript programming.
Why were they important? Because they signaled the arrival of a widely-shared consensus about what was good and bad in JavaScript, and indirectly defined a set of conventions that experts could point to and call “good JavaScript.”
It’s worth pointing out that this isn’t to say that either of these was the absolute, objective best at what they did. MooTools & YUI continue to have their partisans, and I personally find Dave Herman’s Effective JavaScript to be a less idiosyncratic, and more dogma-free presentation of what make for high-quality JavaScript. It’s just to say that, again, we see the language community blazing trails for their own use.
Conventions are basically already de facto standards, so it shouldn’t be too surprising that at several points developers have just gone ahead and made their own standards. Several of these standards have turned out to be successful enough to cause headaches for TC39 when they try to fold them into ECMAScript, and it’s pretty illustrative to look at how the tension between ad hoc standards and the language design play out.
First off, browserify. ES6 introduces a module system, which is interesting
mostly because it highlights that JavaScript didn’t have a module system
before. A package system and namespaces were proposed for ES4, and were in
fact one of the most contentious parts of the language.
That doesn’t mean that people didn’t want one. CommonJS was a community effort that is one of the first tangible contributions of the server-side JavaScript community, and came out of the Rhino community (anyone remember Rhino?). It defines not just a module system but a set of common modules.
Node pretty much ignored the standard library part, but it grabbed the idea of
the module system, and this has been key to Node’s success. It has even come
back to the browser, in a twisted way, through substack’s browserify, which
allows you to write your browser code out of the same kind of small,
single-purpose modules that are practically the Node paradigm, and then bundle
them for use in the browser. This end-to-end integration is a powerful signal
that CommonJS modules are an important tool for the community.
Browserify isn’t the only attempt by developers to bring modular code to JavaScript. AMD started in the browser and grappled with the problem of asynchronous module resolution head-on. It has also become a full-stack tool, although I think it hasn’t caught on in Node as much because Node modules already existed, and because it’s more finicky to use.
In practice, AMD and CommonJS are very different. While there are frameworks, like UMD, that bridge the gap, they’re not precise mappings, and so they’re not very widely used. This highlights an important point – any standard module system is going to have to make choices, and any attempts to offer everything implemented by multiple, potentially conflicting, implementations will either have to drop features or introduce complexity. This is something that you see over and over in standardization efforts, and helps explain why standards turn out to be supersets of what language communities are using. Standards end up being bloated for a reason.
The last, best example of community standards I want to discuss is Promises/A+. You thought you were going to escape this conference without a big talk about promises, didn’t you? No such luck.
For a language that is predominantly used asynchronously (even before Node!), JavaScript doesn’t have a hell of a lot of support for asynchrony. This is a pain point for everybody, pretty much regardless of how long they’ve been writing JavaScript, and an industry has grown up to propose, debate, and vilify competing styles of asynchronous control flow.
Promises, love them or hate them, are widely popular, and as a result Promises/A+ is probably the most significant standard to be defined purely by the community. It’s simple, clear, and narrowly scoped, and has done a great job of driving adoption, to the point where in the JavaScript-speaking world, “promise” is nearly synonymous with Promises/A+. This is probably the only reason promises successfully made it into ES6, given how late they were introduced to the standardization process.
This movement from browser wars through conventions into standards pretty much is the visible aspect of the maturation of JavaScript as a language, and seems like a pretty important reason why JavaScript’s popularity has continued to grow for so long.
Speaking of which, I want to drop in an aside here. I haven’t always been a JavaScript developer. In fact, I spent about 15 years writing Java for a living, and I studied computer science in school. I was (and remain) a big fan of programming language theory, but probably my favorite class in school was actually my software engineering class. Mostly because I love watching train wrecks, which is largely what studying the history of software engineering turns out to be.
Anyway, a few of you may remember that Java was the first programming language that was going to revolutionize the browser (remember applets?) only to find its real success on the server side. And as soon as I started writing server-side Java, I realized it was a nearly perfect tool for software engineering. Why? Because it was just dumb enough.
Java 1.0 and 1.1 were about as straightforward as an imperative object-oriented language could be. It was very difficult for people to get too clever with their code and write the kind of write-only gibberish that Perl developers were so fond of golfing out in the 90s.
However, this changed. Over time developers demanded, and then got, stacks and stacks of new features in the language. Each of those features introduced complexity (and in some cases compounded existing technical debt in the language’s design), and around the time that generics and annotations got added to the language, I decided that the complexity had gotten burdensome enough that I was no longer really interested in dealing with it.
This is a way of saying that the most important piece of tooling in your language of choice is the language itself – its syntax and features are its user experience, and like all user experiences, you want to make sure you don’t clutter that UX with unnecessary, contradictory, or confusing features.
However, as far as I can tell, the primary purpose of language standardization is to include everything, all the time, forever. Each of these columns is a single thread. I may have cherry-picked a little bit for effect, but these are all discussions about ES6 features that went on for days. People were arguing about promises for months.
This is one of the chief reasons Harmony is so important. Doing this stuff out in the open, and discussing things fully (and beyond) does a pretty good job of getting all the issues where everyone can see them. Discussion gets kinda aggro at times, but the spirit of Harmony has mostly held true, and acts as a guard against the most pathological forms of standards committee necrosis.
So let’s talk about the public mechanics of ECMAScript’s development.
The starting place is the draft spec itself, which is most easily consumed via Jason Orendorff’s HTML conversion. Allen is cranking out new drafts as quickly as he can now, as finishing the language of the spec is the most substantial obstacle to finishing the standard, and Jason has pretty much completely automated the process of producing the HTML draft. Standards language is not always super user-friendly, and Allen’s style takes a little getting used to, but this is the ground truth for ES6, so it’s worth your time if you’re interested.
Rick’s notes on the face-to-face meetings are at least as important as his work on johnny-five, in my opinion. It takes me more or less an entire day to go through each set of notes, published shortly after each face-to-face meeting, but aside from the spec itself, this is the most valuable resource for understanding what’s happening to JavaScript.
I think you either need to be working on ECMAScript standardization for your day job or be sort of a glutton for punishment to actually subscribe to es-discuss. There is an incredible amount of detail sometimes, but also endless and sometimes tedious bikeshedding, and a lot of discussions that don’t go anywhere interesting. Fortunately, you don’t need to subscribe to the list, because Forbes Lindsay has put a fabulous and simple UI around the list on esdiscuss.org.
If you find issues when reviewing the specs, or when reading the meeting notes, you can (and should) file them on the ECMAScript bug tracker. This is basically your API into the spec’s editorial process. You should use it.
Tests are no substitute for a formal language standard, but test262 is a great resource for people who want to get involved in validating the spec and contributing to the evolution of the language. Without actual code exercising these features, it’s really hard to say whether a given chunk of the design is any good or not. It’s also critical to have some way of evaluating how much of the spec a given runtime has implemented.
test262 was only recently been put up on GitHub, and just this week completely open-sourced, so it’s ready to take pull requests. Many people put a lot of work into convincing Ecma to do something it’s never done before by opening this process up to the public, so please do take advantage of it.
Believe it or not, ES6 is basically done. The work that remains is mostly on the specification itself, and refining the standard as feedback comes back from stakeholders about how easy (or hard) it is to implement all of the new features in the language.
The draft is slated to be ratified and promoted to version 6.0 of EcmaScript by the end of this year. TC39 has largely moved on to ES7 (with some features being pushed from 6 to 7), and the other standards bodies are rapidly incorporating ES6 features into their own specs.
The big open question is when the first complete, standards-compliant implementation of ES6 will be available. I’m hopeful it will take less than a year (especially because the due date for ES7 is sometime in 2016), but if any of you want to place bets on how long it will take, I’ll be happy to lay you odds.
Finally, one of the great products of Harmony is increased attention to improving the specification process itself. Going forward, new features will be developed as strawman proposals, with a champion responsible for gathering consensus on how features should work, and with an example implementation (and – maybe – tests?) before the feature even gets brought back to TC39. This is an excellent place for developers who are passionate about features to get involved and contribute directly to the development of JavaScript.
What’s the story today?
These are Juriy Zatsev's compatibility charts for ES6. Don’t try to read them, this is just to give you a feel of how much work remains before ES6 is done in various browsers. The picture is actually slightly worse than it appears here, because these just say that the features are there, not that they actually work.
It’s probably not visible, but Spidermonkey is doing the best of any of the browser runtimes. This is scrolled further down in that same chart.
Even in cases where features are available today, there are cases where the standard is still in flux (as is the case with some edge cases around generators) or the implementation is just straight-up buggy (as in the case of promises in Chrome Canary and development versions of Node).
Given that implementation is spotty, how do you get a feel for how all this works today?
As I mentioned, Firefox is in the best shape of any of the major browsers when it comes to ES6 readiness, although it still has a way to go.
If you want to try polyfills and shims, you can lash together a more or less working environment with es6-shim (for built-in constructors and objects), traceur (for syntax), and es6-module-transpiler or es6-module-loader (for modules and module loaders), but even all of that is incomplete and lags behind the standard, so there's no out-of-the-box playground for ES6.
One really interesting strategy, which James Long talked about on Wednesday, is to use a tool like Sweet.js to test individual new features as macros and to see how well they work and what backwards compatibility risk they have. Macros can’t do everything – anything that depends on runtime functionality that isn’t exposed by the language, like WeakMaps with their special garbage-collection semantics, is going to be difficult or impossible to do with macros – but they can do a lot, less invasively than a complete transpiler would.
Let’s talk in a little more detail about how standards and world of active web platform development interact. I’m going to use promises as an example, because they clearly illustrate a lot of factors at work and also because maybe we’ll all finally be able to stop talking about them all the time when ES6 is done. Then we can move on to arguing about private symbols and Object.observe instead.
So TC39 isn’t the only player in web platform standards. The W3C and WHATWG play substantial roles as well, and as there is still only one standard language in the web platform, sometimes these other committees end up creating their own standards that require support from JavaScript. Historically this has led to some weirdness, like, oh, the DOM, with basic bits of platform functionality just dropped into the runtime as globals. Now, at least, stakeholders try to get their features integrated into the language.
Here’s a pretty straightforward example: for a bunch of new APIs, the WHATWG team needed a standard way to do asynchronous computation, so they came up with something called DOMFutures, that were pretty much an extraction of a pre-existing pattern. As I’ve already discussed, the JavaScript community had pretty much solved this same problem with Promises/A+, and so the discussion eventually turned to how to reconcile what WHATWG wanted with the existing spec – a clear-cut example of looking at the existing cowpath and figuring out how to pave it.
Pretty hot on the heels of Promises making it into ES6 – too hot, in fact – the Chrome team announced their intent to ship promises in V8. I think it’s interesting that one of the biggest implementation drivers is actually other standards – Chrome needed Promises and WeakMaps for new DOM APIs, so they moved promises from Blink to V8, and moved them both from behind a feature flag.
There are risks to this strategy. In this case, the promises implementation that V8 is using has several bugs (which are clearly called out and documented in Promises/A+, for what it’s worth), and for unrelated reasons this version of V8 got dropped into a development build of Node.js, in such a way that it’s entirely possible that the next major version of Node, 0.12, will include these buggy promises. If people start building significant code bases on top of buggy promises, it could cause real pain later when Node includes a build of V8 that fixes them.
It’s not a good idea to put draft language features into runtimes that are intended to be used in production, but that’s web development for you.
Of all the arguments around promises on es-discuss, the longest (and arguably most tedious) debate was over whether ES6 should support flattening promises, monadic promises, or both. In addition, the W3C had built a standard for DOMFutures, and Promises/A+ had pretty much taken over within the JS developer community. Promises are really a pretty simple construct, and they threatened to turn into something really cumbersome in ES6. All of the parties in these debates had at least a little history and computer science on their sides, and none of them had a complete understanding of the entire argument.
Another example of this that has generated a lot of heat is the new module system. People have very strong opinions about AMD and Node’s module system – there are very deeply ingrained patterns around both, and in the case of Node an ecosystem of nearly unprecedented scale, and this has led to skepticism and a lot of very blunt talk about the new module system, which will have to tread carefully if anyone wants to see it widely used in one or both of the browser and Node. The module system was one of the last pieces to be finished (work on module loaders still isn’t entirely done!), in large part because Dave Herman and Sam Tobin-Hochstadt had to work so hard to accommodate all these competing constituencies.
There can really be only one of each of these kinds of features in the standard if it’s going to deserve the name (Node isn’t Scala, for which I’m very thankful), and a lot of what fueled the arguments was the desire of all of the people in the debate to see their preferred paradigm win. Beyond a certain point, it became pure politics. Who cares whether or not this is a good thing; this is how the process works, and having an understanding of that makes you better-equipped to participate in the process, and to figure out how to effectively champion the pieces you care about.
At various points in the development of ES6 I got concerned about how much of the discussion of new features was framed in terms of how those features worked in other languages. ES6 is a magpie’s nest of a language, with shiny bits and pieces picked from many languages and not as much consideration as I’d like as to how these pieces will fit. We’re all going to have to figure that out together, and we need to be careful, or we’re going to end up being stuck with some really gross stuff for a while. We need to be free to look at these features in terms of how they relate to JavaScript, and clinging to some other language’s model really isn’t helpful.
Part of the reason I’m here in front of you today is that I spent much of the last two years banging my head against JavaScript’s limitations in order to help Node users figure out how to make their applications faster. I had to learn pretty much all of the JavaScript just to do my job. Server-side JavaScript is just different than browser code – there’s more code, it’s running for much longer, and – what’s up, #pdxnode – it runs closer to the metal.
There are a lot of features in ES5 that don’t run that fast in V8 and Node. In fact, you can recognize people who’ve spent a lot of time analyzing performance of Node apps because their JavaScript looks more or less like C. Some of this is because V8 is still optimized for Chrome, some of this is a result of a carefully collected body of lore about how V8’s JIT will optimize things, but some of it is because features like Function.bind are just doing way too much stuff – they were designed by people who waved away concerns about whether all of this could be optimized. Filing spec bugs and adding useful performance tests that aren’t part of meaningless benchmarking games is our best hope for trying to get these performance issues addressed. That’s right, I added this slide just for Trevor, if he’s around.
To pull this all together, there are a lot of basic, powerful abstractions in ES6, some of which can be taken further in subsequent versions of the language – put them all together, and see how it feels to compose and combine them. It’s time to find the new cowpaths.
Think really hard about "what's good" and try to separate it from "what's cool". Being clever is gratifying in the short time, but building an ecosystem with solid foundations will be much more rewarding in the long run.
We don't need to use all the new features at once, and if we do, we run the risk of creating conventions we're going to find incredibly toxic to work with in a few years. Some of us love shiny things and gadgets and hopping from crevice to crevice like mountain goats. The rest of look at that mountainside and see not only no trail, but certain and terrible death. As trailblazers, it’s our responsibility to remember the needs of the people who will come after us. Remember: “keep the language pleasant for casual developers.”
Something you learn when you study a foreign language is that when you are fluent in a language, you think in that language instead of mentally translating as you go along. You stop noticing which words are loan words and which words sound funny and struggling to juggle conjugations and genders in your head, and think instead in terms of getting across concepts and feelings.
I want us to get to a point where it is possible to be fluent in idiomatic JavaScript, and that we as a community, spanning the browser and the back end, can all code in a way that is intelligible and consistent to each other. JavaScript is a very expressive language, and ES6 adds a lot more expressiveness into the mix. I will know that ES6 is a success when people talk about JavaScript primarily in terms of JavaScript, and not in terms of how it compares to other languages, even with the new features that originally came from somewhere else.
To get there is going to require critical thinking and judicious editing. If
some of the new features turn out to be obviously bad, deprecate them quickly
and loudly and move on. with is still there, but nobody's forcing you to use
it.
It took over 10 years between the introduction of JavaScript and the publishing of The Good Parts. It doesn’t have to take that long again. Work with the goal in mind of converging on a new style of "Effective JavaScript" using a new set of "Good Parts", but treat it all as a process rather than a product. "Good Parts" is best when it talks concretely about what has and hasn't worked, and less effective when it tries to establish a new dogma. We should strive to be descriptive as much as we can, and to save the opinion-based flaming for Twitter and IRC. Ultimately, we’re all in this together.
Some of this may sound a little surprising to those of you who know me; I am an opinionated smartass, and fond of saying things like “your favorite programming language sucks.” At the end of the day, JavaScript is just another imperfect tool made by imperfect beings. This is a polite way of saying that JavaScript is kind of terrible. But it gets things done, as this conference shows.
I think my message to you is hopeful: ES6 is almost done, and that means that it is now our responsibility as a community to figure out how to use it most effectively, and to make it a success. There is a lot of stuff in there, and I have strong opinions about which bits of it are absolutely terrible. But I – and more importantly, all of you – have the power to sort through everything and figure out which bits are useful and which bits are nasty, together. We are JavaScript, and its future is under our control. I urge yourself to take advantage of that control, and to help each other out along the way.
Thank you.
“Pave the cowpaths” is actually one of the principles included in the HTML Design Principles document, which really applies to any Web-exposed language (i.e. not just HTML).