A few years ago, when the Frontend developer tooling craze hadn't yet started, techniques for building and optimizing JavaScript applications varied. Many were using shell scripts (cat to concatenate files etc.), tools provided by the web framework (like Play Framework's templates), or some generic Java-based tools. There might have been some manual operations (especially around minifying for production builds), and most of the tooling was not JavaScript-based.
Then Frontend developers started to notice that it was really hard to reason about lots of files that each rely on some globals (making the inclusion order significant) and expose a "namespace" (global variable) for other files to use. Then came AMD (Asynchronous Module Definition), which enabled making modules that define their dependencies and that explicitly exposed their exports. AMD modules were used together with the require.js tool and the r.js optimizer. The A in AMD (asynchronous) was a clear design choise to allow smaller builds that use async module loading. However, because of this the syntax was cumbersome, for example:
define([
'jquery',
'underscore',
'backbone',
'app.mod1',
'app.mod2',
// ...
'app.modn'
], function (
$,
_,
Backbone,
mod1,
mod2,
// ...
modn
) {
VAR CONSTANT = 'foobar';
function foobar() {
// ...
}
return {
CONSTANT: CONSTANT
export1: function () {
// ...
},
foobar: foobar
};
});but as you notice, the syntax has a lot of noise and especially the connection from dependency names to the dependency parameters of the module is really hard to follow when the number of dependencies grows. As for the asynchronous module loading, usually it wasn't even needed since applications were mostly built into a single file anyways...
The AMD/require.js setup worked, but mostly because there wasn't much other options.
At the same time with the Frontend developers using AMD modules, the Node.js folks had their own module system based on CommonJS modules. The Node module system used by NPM (bundled with Node) was clearly only suited for the Backend, since it allowed using different versions of the same package in different parts of the application (read: your code vs. your dependencies). This was great for the Backend in terms of interoperability, but an instant no-go for the Frontend since all those different versions would have to be sent over the wire.
There was also effort to combine the two packaging systems into something called UMD (Universal Module Definition). This increased the interoperability, but the syntax was even more verbose than AMD. UMD ended up being embraced only in some popular libraries that were useful in different runtimes.
During this period, it was quite normal to spend several days to get a proper build setup (with test runners etc.). Yeoman fixed some of this pain, but application scaffolding had its own problems, mainly developers didn't really understant the build process anymore.
We already had several JavaScript-based tools for different parts of the build process, and Ben Alman just decided to group those into separate tasks used by his new Grunt build tool. This happened some time during the AMD popularity and Grunt spread really fast.
Grunt was a great tool, but pretty much relearned all the lessons from ancient build tools during its first few years. The biggest problem with Grunt IMO was when the Gruntfile started growing too big. Having separate static configuration objects and tasks and multitasks as functions made the build process really hard to grasp and reading a Gruntfile wasn't straightforward. Even more, making custom tasks was hard since you had to read the input from some temp directory and write the output somewhere else, and this ended up being a huge PITA. Many custom tasks were simple one-liners as a shell command, but making a Grunt task out of them was hard. Many times you ended up writing a Grunt task that simply executed a shell command, breaking OS interoperability at the same time.
Gulp was born out of frustration with Grunt configuration and the pain in composing different tasks into pipelines. Gulp was essentially a thin wrapper around Node streams than could be piped forward, making composing build pipelines easy. Gulp also embraced code over static configuration object for full hackability.
At some point, most JavaScript packages started using CommonJS modules and NPM as a package manager, and Browserify became the simple tool to build these modules into a single application. While AMD allowed for development-time asynchronous module loading, Browserify always built the modules into a single bundle, and was fast enough that this step apparently didn't end up being a problem.
Now that we had Browserify and a working set of tools usable from the command line, most applications didn't even need a build tool like Grunt or Gulp. You could simply define scripts in your package.json file that could be run with npm run (see http://substack.net/task_automation_with_npm_run ). This was worse for OS interoperability (since you were using shell command and Unix pipes), but the build process was simple and easy to follow (until it grew big enough, after which some people switched to using e.g. Make if they continued being fond of the command line tools).
Build tools had matured to the point where common good patterns were recognized and useful defaults were apparent. CommonJS had won the module war and UI frameworks had started embracing modules and components. Webpack became the modern tool that came with good defaults and something called hot module replacement. Live reloading tools were nothing new, but hot replacing modules became the new hotness. This was taken even further with React Hot Loader that hotloads single UI components.
We've seen hot loading being super useful in LISP-land, but not embracing immutability and having state all over the place will definitely bring a set of problems into the game. So the usefulness in a large JavaScript application is to be seen.
The Webpack configuration format is also super ugly when adding several phases to a build pipeline, with possible arguments for each phase.
The cool thing with Webpack is that you can still bundle your build into separate bundles if the need arises in the future. Also, including style files for a UI component within the JavaScript file is awesome (should also work with Browserify)!
As mentioned above, CommonJS modules won and NPM became the package manager for all JavaScript packages, as well as some non-JavaScript code runnable with Node. There was a brief moment when Bower was popular, but the quote "Bower, also known as Curl" isn't too far away from the truth. Bower only helped in downloading packages from the command line instead of searching the correct versions from random websites. This was really helpful, but the usefulness ended there.
Having CommonJS packages in NPM ensures that they are usable and interoperable in Frontend as well as Backend, and many packages are tested in both browser and Node environments.
There were also lots of other package managers (aka. "pakettimankeli" in Finnish). See my old presentation:
http://kpuputti.github.io/perkele.js/examples/javascript-package-managers/index.html
(Sorry for the crappy slide framework, just use arrows to move forward/back)
Babel is a JavaScript transpiler, turning ES6/ES2015/ES7/ES2016/ESwhatever code into something that can be run also in older JavaScript runtimes. Babel is easy to include in your build pipeline, and works great. The progress is really fast and new language features are usable early.
However, it is easy to go too far with Babel, since it also implements experimental features that might not end up as standards, and could be dropped at any point. It is therefore recommended to carefully consider which features you include in the transpilation.
The goal of using a tool like Babel is that some day you can just drop the tool when all your runtimes support the features you use in transpilation. But I'm sure there will always be useful upcoming features not yet in browsers... Cat and mouse, you know.
Choose package.json, Makefile, Grunt, Gulp, or Webpack for your build process. Use CommonJS or ES6 modules. Use Babel. Distribute through NPM.