- Intro
- Basics
- Event Module
- HTTP Module
- NPM
- ENVIRONMENT
- Configuration
- Project structure
- Route Params - Parameter validation
- Middleware
- Debugging
- Template Engines
- Asynchronous Code
- MongoDB / Mongoose
- DB Modelling
- DB Transactions
- Authentication & Authorization
- Error Handling
- Production
- Testing
Node.js is a C++ program which wraps a V8 engine in .exe program.
- sync = blocking
- async = non-blocking
Node.js monitors this queue in order to get new events for processing.
-
global is same as window in the browser
-
Declared variables live only inside the created file. Variables are not added to the global scope.
-
In order to export a variable we need to use modules.
- module is just a JSON object which wraps the file:
{ ..., exports: {}, ... }
- Whatever we add to the exports object is going to be exported to the outside.
- For loading a module from other file we use
require()
. We what to use this import statement together withconst
. That way we can be sure that we do not override this import. - Node.js wraps whole file in IIFE function (module wrapper function):
(function(exports, require, module, __filename, __dirname) { // content of the file });
- Code is wrapped inside module wrapper function, but it's not execute immediately.
- The first argument
exports
is the reference tomodule.exports
. - If we do not provide import
require()
with./
or../
, node.js is going to look around if there is a file with the same name. If there isn't any, node.js will look forward inside node_modules directory.
It's used as a signal that something is happening.
- It uses EventEmitter to emit new event.
const EventEmitter = require('events'); const emitter = new EventEmitter(); emitter.emit(<event name>, <event data>);
- Listener:
emitter.on(<event name>, function() { ... });
- We can extent class
EventEmitter
in order to inherit all it's properties.
- Extends
EventEmitter
class - This module includes
createServer
method:const http = require('http'); http.createServer((req, res) => { if (req.url === '/') { res.write('Hello world'); res.end(); } }
- It also includes method for working with API REST points:
.get()
,.post()
,.put()
,.patch()
,.delete()
,- ...
- All dependencies are saved as flat modules inside
node_modules
. - If some module has dependency which is already in
node_modules
but it's different version, that dependency module is placed inside parent module.
4 . 13 . 6
^ ^ ^
| | |
major minor patch
- major version can have breaking changes
Prefixes:
^
... it is going to update all versions except major (=4.x
)~
... it is going to update all versions to patch version (=4.13.x
)<empty>
... it is going to leave module version as it is
npm list
returns list of all node modules--depth=0
returns only first modules
npm view <module name>
returns all information about this modulenpm update
shows all versions which are outdated and eligible for updatenpm version <major|minor|patch>
bumps version for 1 based on semantic
Reading variable:
const port = process.env.PORT || 3000;
Exporting variable:
export PORT=3001
process.env.NODE_ENV // undefined by default
or
app.get('env'); // 'development' by default
- Packages for loading configuration files:
npm i rc
npm i config
Project structure:
.
+-- config/
| +-- default.json
| +-- development.json
| +-- production.json
default.json
example:
{
"name": "Basic Node.js Application"
}
Read variables in node.js from config
directory:
const config=require('config');
console.log(config.get('name'));
Declare variable:
> export app_password=1234
.
+-- config/
| +-- ...
| +-- custom-environment-variables.json
Loading secrets in config files:
{
"name": "Basic Node.js Application",
"mail_password": "app_password"
}
.
+-- config/
+-- middleware/
| +-- logger.js
+-- models/
| +-- course.js
| +-- customer.js
+-- node_modules/
+-- public/
+-- routes/
| +-- courses.js
| +-- customers.js
| +-- home.js
+-- views/
+-- index.js
+-- package.json
Example of index.js:
...
const home = require('./routes/courses');
const courses = require('./routes/courses');
app.use('/', home);
app.use('/api/courses', courses);
Example of routes/home.js:
const express = require('express');
const router = express.Router();
app.get('/', (req, res) => {
res.render('index', { title: 'My Express App', message: 'Hello' });
});
module.exports = router;
- Normal parameters:
req.params
- Query parameters:
req.query
Example:
app.get('/api/courses/:id', (req, res) => {
console.log(req.params.id);
});
- Best packing for parameter validation is
joi
. - Best practice for validation complex passwords is joi-password-complexity
Middleware is a function that interfere with response. It responses to the client or forwards control to next middleware function.
- Methods such as
.get()
or.post()
are also middleware functions.
request processing pipeline
+--------------------------+
request -> | json() -> route() | -> response
+--------------------------+
Each middleware function needs to be in the own file.
app.use(function(req, res, next) {
console.log('logging');
next(); // if next() is not called request will hang
});
express.json()
express.urlencoded() // express.urlencoded({extended: true})
express.static('<path to static directory>)
- serving static files
helmet
- helps to secure application with HTTP headersmorgan
- logger for HTTP requests
> npm i debug
Placing debug prints:
const startupDebugger = require('debug')('app:startup');
const dbDebugger = require('debug')('app:db');
// ...
startupDebugger('Morgan loaded');
dbDebugger('Connected to the db');
Setting debug level:
export DEBUG=app.startup
export DEBUG=app.startup,app.db
export DEBUG=app.*
# - or -
DEBUG=app.db nodemon index.js
Engines:
- Pug
- Mustache
- EJS
Using Pug:
app.set('view engine', 'pug');
app.set('views', './views'); // default views location
index.pug:
html
head
title= title
body
h1= message
Rendering view inside the controller:
app.get('/', (req, res) => {
res.render('index', {title: 'My Express App', message: 'Hello'});
});
Non blocking code
Dealing with asynchronous code:
-
Callbacks:
console.log('Before'); getUser(1, (user) => { getRepositories(user.github, (repos) => { getCommits(repo, (commits) => { // ... }); }); }); console.log('After'); function getUser(id, callback) { setTimeout(() => { console.log('Reading from a db'); callback({id: id, username: 'matox'}); }); }
- This leads to the callback hell
- Solution for callback hell problem:
console.log('Before'); getUser(1, getRepositories); console.log('After'); function getRepositories(user) { getRepositories(user, getCommits); } function getCommits(repos) { getCommits(repo, displayCommits); } function displayCommits(commit) { console.log(commits); } function getUser(id, callback) { setTimeout(() => { console.log('Reading from a db'); callback({id: id, username: 'matox'}); }); }
-
Promises:
- Promise is an object that holds the eventual result of an asnychronous operation, it promises you that it is going to give you a result of an asynchronous operation
- Initial state of the promise is Pending. After the async operation is completed, promise is going to return Fulfilled or Rejected promise.
const p = new Promise((resolve, reject) => { setTimeout(() => { resolve(1); // pending => resolved, fulfilled // reject(new Error('message')); // pending => rejected }, 2000); }); p .then(result => console.log('Result', result)); .catch(err => console.log(error.message));
-
Settled Promises:
- Useful for tests
- Example:
const p = Promise.resolve({ id: 1 }); p.then(result => console.log(result));
-
Parallel Promises:
- If any on the promises in the array is rejected, whole
.all()
is going to be rejected.
const p1 = new Promise((resolve, reject) => { setTimeout(() => { console.log('Async operation 1'); // resolve(1); reject(new Error('error')); }, 2000); }); const p2 = new Promise((resolve) => { setTimeout(() => { console.log('Async operation 2'); resolve(2); }, 2500); }); Promise.all([p1, p2]) .then(result => console.log(result));
- We can use
Promise.race()
if we want to only wait for the first promise to complete.
- If any on the promises in the array is rejected, whole
-
Async / await:
- We need to prefix our function with
async
and our statement withawait
. - In async code we need to wrap our code in try-catch block in order to catch exceptions.
async function displayCommits() { try { const user = await getUser(1); const repos = await getRepositories(user.username); const commits = await getCommits(repos[0]); console.log(commits); } catch (err) { console.log(err); } } displayCommits();
- We need to prefix our function with
Mongoose is an abstraction over mongoDB driver.
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/playground')
.then(() => console.log('Connected to MongoDB'))
.catch((err) => console.error('Could not connect to MongoDB', err));
Types:
- String
- Number
- Date
- Buffer
- Boolean
- ObjectID - for assigning unique identifier
- Array
const courseSchema = new mongoose.Schema({
name: String,
author: String,
tags: [ String ],
date: {
type: Date,
default: Date.now
},
isPublished: Boolean
});
- Only applicable for Mongo DB
const courseSchema = new mongoose.Schema({
name: {
type: String,
required: true,
minlength: 5,
maxlength: 255,
// match: /regex-+/
},
category: {
type: String,
required: true,
enum: ['web', 'mobile', 'network'],
lowercase: true,
// uppercase: true,
trim: true
},
author: String,
tags: {
type: Array,
validate: {
validator: function(v) { // Sync validator
return v && v.length > 0;
},
message: 'A course should have at least one tag.'
}
},
tagsAsync: {
type: Array,
validate: {
isAsync: true,
validator: function(v, callback) { // Async validator
setTimeout(() => {
// Do some async work
const result = v && v.length > 0;
callback(result);
}, 1000);
},
message: 'A course should have at least one tag.'
}
},
date: {
type: Date,
default: Date.now
},
isPublished: Boolean,
price: {
type: Number,
required: function() { return this.isPublished; }, // We cannot use arrow functions at this point
min: 10,
max: 200,
get: v => Math.round(v), // Custom getter
set: v => Math.round(v) // Custom setter
}
});
const Course = mongoose.model('Course', courseSchema);
const course = new Course({
name: 'Node.js Course',
category: 'web',
author: 'matox',
tags: ['node', 'backend'],
isPublished: true,
price: 15
});
async function createCourse() {
const course = new Course({
name: 'Node.js Course',
category: 'web',
author: 'matox',
tags: ['node', 'backend'],
isPublished: true,
price: 15
});
try {
course.validate((err) => { // Manually trigger validation
if (err) { console.log(err); }
});
const result = await course.save();
console.log(result);
}
catch (ex) {
console.log(ex.message);
}
}
createCourse();
async function getCourses() {
const courses = await Course
.find({ author: 'matox', isPublished: true })
.limit(10)
.sort({ name: 1 })
.select({ name: 1, tags: 1 }); // what should query return
console.log(courses);
}
getCourses();
Operators:
- eq (equal)
- ne (not equal)
- gt (greater than)
- gte (greater than or equal to)
- lt (less than)
- lte (less than or equal to)
- in
- nin (not in)
async function getCourses() {
const courses = await Course
.find({ price: { $gte: 10, $lte: 20 } })
// .find({ price: { $in: [15 , 20, 25] } })
.limit(10)
.sort({ name: 1 })
.select({ name: 1, tags: 1 }); // what should query return
console.log(courses);
}
getCourses();
Operators:
- or
- and
async function getCourses() {
const courses = await Course
.find()
.or([ {author: 'matox'}, { isPublished: true } ])
.and([ { price: { $lt: 10 }, isPublished: true } ])
.limit(10)
.sort({ name: 1 })
.select({ name: 1, tags: 1 }); // what should query return
console.log(courses);
}
getCourses();
Short format:
.select({ name: 1, author: 1 })
->.select('name author')
.sort({ price: -1 })
->.sort('-price')
async function getCourses() {
const courses = await Course
// Starts with "ma"
.find({ author: /^ma/ })
// Ends with "ox", case insensitive
.find( { author: /ox$/i} )
// Contains "at"
.find({ author: /.*at.*/i })
.limit(10)
.sort({ name: 1 })
.count(); // returns count
console.log(courses);
}
getCourses();
async function getCourses() {
const pageNumber = 2;
const pageSize = 10;
const courses = await Course
.find({ author: /^ma/ })
.skip((pageNumber - 1) * pageSize)
.limit(pageSize)
.sort({ name: 1 })
.count(); // returns count
console.log(courses);
}
getCourses();
-
Query first:
- findById()
- Modify it's properties
- save()
async function updateCourse(id) { const course = await Course.findById(id); if (!course) return; course.isPublished = true; course.author = 'Another Author'; // - or - course.set({ isPublished: true, author: 'Another Author' }); const result = course.save(); console.log(result); }
-
Update first:
- Update directly
- Optionally: get the updated document
async function updateCourse(id) { const result = await Course.update({ _id: id}, { $set: { author: 'matox', isPublished: false } }); console.log(result); }
-
FindByIdAndUpdate:
async function updateCourse(id) { const result = await Course.findByIdAndUpdate(id, { $set: { author: 'matox', isPublished: false } }, { new: true }); // so we get new model instead of the old one console.log(result); }
- DeleteOne
async function deleteCourse(id) { const result = await Course.deleteOne({ _id: id }) console.log(result); }
- DeleteMany
async function deleteCourse(id) { const result = await Course.deleteMany({ _id: id }) console.log(result); }
- FindByIdAndRemove
.findByIdAndRemove()
is going to returnnull
if there isn't any document to delete
async function deleteCourse(id) { const course = await Course.findByIdAndRemove(id); console.log(course); }
Trade off between query performance vs consistency
- Using references (Normalization) -> CONSISTENCY
let author = {
name: 'matox'
};
let course = {
author: 'id'
};
- Using Embedded Documents (Denormalization) -> PERFORMANCE
let course = {
author = {
name: 'matox'
}
};
- Hybrid
let course = {
author = {
name: 'matox',
// 50 other properties
}
};
let course = {
author: {
id: 'ref',
name: 'matox'
}
};
We can bind two documents with one id, similar as in relation database.
Example:
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/playground')
.then(() => console.log('Connected to MongoDB...'))
.catch(err => console.error('Could not connect to MongoDB...', err));
const Author = mongoose.model('Author', new mongoose.Schema({
name: String,
bio: String,
website: String
}));
const Course = mongoose.model('Course', new mongoose.Schema({
name: String,
author: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Author'
}
}));
async function createAuthor(name, bio, website) {
const author = new Author({
name,
bio,
website
});
const result = await author.save();
console.log(result);
}
async function createCourse(name, author) {
const course = new Course({
name,
author
});
const result = await course.save();
console.log(result);
}
async function listCourses() {
const courses = await Course
.find()
.populate('author', 'name -_id')
.populate('category', 'name')
.select('name author');
console.log(courses);
}
createCourse('Node Course', '5c07f6a41fcbb182b26fcb7b');
- We must always edit embedden documents by it's parent (We cannot edit them directly)
- Example:
const mongoose = require('mongoose'); mongoose.connect('mongodb://localhost/playground') .then(() => console.log('Connected to MongoDB...')) .catch(err => console.error('Could not connect to MongoDB...', err)); const authorSchema = new mongoose.Schema({ name: String, bio: String, website: String }); const Author = mongoose.model('Author', authorSchema); const Course = mongoose.model('Course', new mongoose.Schema({ name: String, author: { type: authorSchema, required: true } })); async function createCourse(name, author) { const course = new Course({ name, author }); const result = await course.save(); console.log(result); } async function listCourses() { const courses = await Course.find(); console.log(courses); } async function updateAuthor(courseId) { const course = await Course.update({ _id: courseId }, { $set: { 'author.name': 'matox' } }); // Update directly in the DB } async function deleteAuthor(courseId) { const course = await Course.update({ _id: courseId }, { $unset: { 'author': '' } }); // Update directly in the DB } //updateAuthor('5c07faa46acc7d84023b8654'); deleteAuthor('5c07faa46acc7d84023b8654');
Example:
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/playground')
.then(() => console.log('Connected to MongoDB...'))
.catch(err => console.error('Could not connect to MongoDB...', err));
const authorSchema = new mongoose.Schema({
name: String,
bio: String,
website: String
});
const Author = mongoose.model('Author', authorSchema);
const Course = mongoose.model('Course', new mongoose.Schema({
name: String,
authors: {
type: [authorSchema],
required: true
}
}));
async function createCourse(name, authors) {
const course = new Course({
name,
authors
});
const result = await course.save();
console.log(result);
}
async function addAuthor(courseId, author) {
const course = await Course.findById(courseId);
course.authors.push(author);
course.save();
}
async function removeAuthor(courseId, authorId) {
const course = await Course.findById(courseId);
const author = await course.authors.id(authorId);
author.remove();
course.save();
}
createCourse('C Programming Language', [
new Author({ name: 'Brian Kernighan' }),
new Author({ name: 'Dennis Ritchie' })
]);
addAuthor('5c07fe7cd0e37384d96879f2', new Author({ name: 'Samuel P. Harbison' }))
removeAuthor('5c07fe7cd0e37384d96879f2', '5c07ff5f577d4e8511057b91');
- In Mongo DB objectID is represented with 24 characters (12 bytes)
- 4 bytes - timestamp
- 3 bytes - machine identifier
- 2 bytes - process identifier
- 3 bytes - counter
- With that counter we can represent 16M different transactions
- If we create more than 16M transactions in the same time, on the same machine, in the same process we can get counter overflow
- We dont have an objectID which garentees uniquness
- Create custom id:
const mongoose = require('mongoose'); const id = new mongoose.Types.ObjectID(); console.log(id);
- Get timestamp from objectID:
const mongoose = require('mongoose'); const id = new mongoose.Types.ObjectID(); console.log(id.getTimestamp());
- Validate objectID:
const mongoose = require('mongoose'); const isValid = new mongoose.Types.ObjectID.isValid('1234'); console.log(isValid);
- Using with __Joi__package:
const Joi = require('joi'); const Joi.objectId = require('joi-objectid')('Joi'); // ... function validateRental(rental) { const schema = { customerId: Joi.objectId().required(), movieId: Joi.objectId().required() };
A group of operations that should be performed as one. Either all of this operations will complete or all will roll back.
- In MongoDB we use Two Phase Commits: more
- We can simulate transactional behaviour with package fawn
- Example:
const mongoose = require('mongoose'); const Fawn = require('fawn'); Fawn.init(mongoose); new Fawn.Task() .save('rentals', rental) .update('movies', { _id: movie._id }, { $inc: { numberInStock: -1 } }) .run(); // this is a must res.send(rental);
- ObjectID is generated by driver, so we don't need to wait for database to assign us an ID.
Authentication is a process of identifing if the user is who they claim they are.
Authorization is determening if user has the right permission to perform certain actions.
- Install package bcrypt:
npm i bcrypt
- Example password encrypting:
const bcrypt = require('bcrypt'); const salt = await bcrypt.genSalt(10); user.password = await bcrypt.hash(user.password, salt)
- Example password decrypting:
const bcrypt = require('bcrypt'); const validPassword = await bcrypt.compare(req.body.password, user.password); if (!validPassword) res.status(400).send('Invalid email or password.');
- Install package jsonwebtoken:
npm i jsonwebtoken
- It's a very bad practice to store JWT token unprotected in our database
- It's important to use HTTPS for sending tokens around
- Example:
const jwt = require('jsonwebtoken'); const config = require('config'); if (!config.get('jwtPrivateKey')) { console.error('FATAL ERROR: jwtPrivateKey is not defined'); process.exit(1); } const token = jwt.sign({ _id: user._id }, config.get('jwtPrivateKey');
- Sending the token in headers:
res.header('x-auth-token', token).send(_.pick(user, ['_id', 'name', 'email']));
- Information Expert principle: Information expert should be responsible for generating the token. So we should place token generation code to the user service.
userSchema.methods.generateAuthToken = function() { const token = jwt.sign({ _id: this._id }, config.get('jwtPrivateKey')); return token; }
- Example:
const jwt = require('jsonwebtoken'); const config = require('config'); module.exports = function (req, res, next) { const token = req.header('x-auth-token'); if (!token) return res.status(401).send('Access denied. No token provided.'); try { const decoded = jwt.verify(token, config.get('jwtPrivateKey')); req.user = decoded; next(); } catch (ex) { res.status(400).send('Invalid token.'); } }
- In order to protect route we need to add middleware function after the url
- Example:
const auth = require('../middleware/auth'); router.post('/', auth, async (req, res) => { const { error } = validate(req.body); if (error) return res.status(400).send(error.details[0].message); let genre = new Genre({ name: req.body.name }); try { genre = await genre.save(); res.send(genre); } catch(ex) { console.error(ex.message); } });
- Admin middleware:
module.exports = function (req, res, next) { // 401 Unauthorized // 403 Forbidden if (!req.user.isAdmin) return res.status(403).send('Access denied.'); next(); }
- Protecting routes with role based middleware:
router.delete('/:id', [auth, admin], async (req, res) => { const genre = await Genre.findByIdAndDelete(req.params.id); if (!genre) return res.status(404).send('The genre with the given ID was not found.'); res.send(genre); });
- Error Middleware:
module.exports = function(err, req, res, next) { // Log the exception res.status(500).send('Internal error'); }
- Implementation:
const error = require('./middleware/error'); app.use(error);
- AsyncMiddleware:
module.exports = function asyncMiddleware(handler) { return async (req, res, next) => { try { await handler(res, res); } catch (ex) { next(ex); } } }
- Implementation:
const asyncMiddleware = require('../middleware/async'); router.get('/', asyncMiddleware(async (req, res, next) => { const genres = await Genre.find().sort('name'); res.send(genres); }));
- Popular library in Node.js is winston
- It has different "transports":
- Console
- File
- Http
- MongoDB
- CouchDB
- Redis
- Loggly
- Log levels:
- error
- warn
- info
- verbose
- debug
- silly
- Configuration:
winston.add(winston.transports.File, { filename: 'logfile.log' });
winston.add(winston.transports.MongoDB, { db: 'mongodb://localhost/vidly', level: 'error' }); ```
- Reporting an error:
winston.error(err.message, err);
Only for sync code, if an error appears inside Promise, this error is not going to be caught
- Example:
process.on('uncaughtException', (ex) => { console.log("WE GOT AN UNCAUGHT EXCEPTION!"); winston.error(ex.message, ex); }); winston.handleExceptions(new winston.transports.File({ filename: 'uncaughtExceptions.log' }));
Only for async code, for all promises that were not caught by
.catch()
statement.
- Example:
process.on('unhandledRejection', (ex) => { console.log("WE GOT AN UNHANDLED REJECTION!"); winston.error(ex.message, ex); process.exit(1); });
winston.handleExceptions(
new winston.transports.Console({ colorize: true, prettyPrint: true }),
new winston.transports.File({ filename: 'uncaughtExceptions.log' })
);
process.on('unhandledRejection', (ex) => {
throw ex;
});
- We need to add production middlewares:
const helmet = require('helmet'); const compression = require('compression'); module.exports = function(app) { app.use(helmet()); app.use(compression()); }
- Preparing for Heroku:
"scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start": "node index.js" }, "engines": { "node": "10.8.0" },
- Deploying:
heroku create heroku config:set vidly_jwtPrivateKey=1234 heroku config:set NODE_ENV=production git push heroku master
- Test types:
- Unit tests
- Integration tests
- End-to-end tests
- Number of test cases for specific test should be => the number of return paths
- We use package Jest for testing our app
describe('absolute', () => {
it('should return a positive number if an input is positive', () => {
const result = lib.absolute(1);
expect(result).toBe(1);
});
it('should return a positive number if an input is negativee', () => {
const result = lib.absolute(-1);
expect(result).toBe(1);
});
it('should return a zero if an input is zero', () => {
const result = lib.absolute(0);
expect(result).toBe(0);
});
})
describe('greet', () => {
it('should return the greeting message', () => {
const result = lib.greet('matox');
expect(result).toMatch(/matox/);
expect(result).toContain('matox');
})
});
describe('getCurrencies', () => {
it('should return supported currencies', () => {
const result = lib.getCurrencies();
expect(result).toEqual(expect.arrayContaining(['EUR', 'AUD', 'USD']));
});
});
describe('getProduct', () => {
it('should return the product with the given id', () => {
const result = lib.getProduct(1);
// expect(result).toEqual({ id: 1, price: 10 }); // equals exact object
expect(result).toMatchObject({ id: 1, price: 10 }); // some properties are matching
expect(result).toHaveProperty('id', 1);
});
});
describe('registerUser', () => {
it('should throw if username is falsy', () => {
const args = [null, undefined, NaN, '', 0, false];
args.forEach(a => {
expect(() => { lib.registerUser(a) }).toThrow();
});
});
it('should return a user object if valid username is passed', () => {
const result = lib.registerUser('matox');
expect(result).toMatchObject({ username: 'matox' });
expect(result.id).toBeGreaterThan(0);
});
});
describe('applyDiscout', () => {
it('should apply 10% discount if customer has more than 10 points', () => {
db.getCustomerSync = function(customerId) {
console.log('Fake reading customer...');
return { id: customerId, points: 20 };
}
const order = { customerId: 1, totalPrice: 10 };
lib.applyDiscount(order);
expect(order.totalPrice).toBe(9);
})
});