File management

FileManagement.md

File management with node.js

Files, they're used as a basic persistence layer by a lot of people, but it often gets fuzzy when they're accessed in multiple different places.

The problem

Let's take a simple bank implementation as example, where we have one file getting our balance, and one file is used for transactions (e.g. payments). Our data is stored in a file called bank.json:

// File: balance.js
// Nice shortcut node! Now I don't need to read my file and parse it manually!
const accounts = require('bank.json');

module.exports = (user) => {
  const account = accounts[user.id];
  if (!account) {
    return user.inform('You have no account with us, sorry.');
  }
  user.inform(`Your balance is ${account.balance}`);
}

// File: pay.js
// More shortcuts

const accounts = require('bank.json');

module.exports = (user, otherUser, amount) => {
  const account = accounts[user.id];
  if (!account) {
    return user.inform('You have no account with us, sorry.');
  }
  
  if (account.balance < amount) {
  	return user.inform('You seem to have insufficient funds to pay this.');
  }
  
  // Transiently open a new account, so we can move the money around
  let otherAccount = accounts[otherUser.id];
  if (!otherAccount) {
  	otherAccount = accounts[otherUser.id] = {
      name: otherUser.name,
      balance: 0
    };
  }
  // Move the money around
  account.balance -+ amount;
  otherAccount.balance += amount;
  // Ow, we need to save it to ensure the transfer is not lost!
  fs.writeFile('bank.json', JSON.stringify(accounts), (err) => {
  	if (err) {
      console.log('Failed to save the accounts to JSON');
    } else {
      user.inform(`You paid ${otherUser.name} £${account.balance}`);
    }
  });
}

Easy as pie, right? ... But what does the balance say after I paid someone? It won't update my balance! Why, you might ask? Because it has a completely separate idea of what bank.json contains, as it only reads it once on startup.

A first solution to the problem

A simple solution to this problem, is to read it every time someone runs a command. We remove the const coins = require('bank.json') completely from the file, and add the following right after the export

module.export = (user) => {
  // Add this:
  const accounts = JSON.parse(fs.readFileSync('bank.json', 'utf8'));
...

This will make sure to read the file from disk every time, which ensures we see the last thing the transaction has managed to write to disk. Great, right?!

One huge downside of this, is that every time someone wants to request their balance, we have to read the whole file from disk again. As you might already know, reading from disk is slow. Very slow at times.

("But but it's only taking a few milliseconds to read!", sadly that's really slow in computer terms, and the milliseconds can easily become seconds if the file grows large)

A better alternative

A better solution would be separating the file loading completely from the two other files, and introduce a new file:

// File: bank-api.js
// We can use this trick again, always nice
const accounts = require('bank.json');
module.exports.getAccount = (user) => {
  return accounts[user.id];
};

module.exports.save = (errorHandler) => {
  fs.writeFile('bank.json', JSON.stringify(accounts), err => {
    errorHandler(err); // Pass error to caller
  }
};

Obviously you can expand on this (i.e. not allowing access to account and just exposing an API), but that is for a different writeup. As node require is cached (i.e. if called twice, the same instance is returned), you can simply include it in both files (balance.js and pay.js) and it should work as is.

Injecting the "dependencies^*"

Rather than relying on Node to deal with it for us, especially when we want to expand things later on, it might be beneficial to depend on "dependency injection". As the name implies, it depends on passing whatever a module might depend on, directly to the module, rather than having node handle it nicely for us.

In the case of JavaScript, the best way to do this is to export a function instead of the API itself, and have that function initialise the API for us. This might seem really complex, so let me demonstrate it instead with our balance.js file:

// File: balance.js
// Instead of exporting the function which deals with the whole process of telling
//  our user their balance, we export a function which returns a function (quite
//  meta, I know). This works as the function we return can use anything we define
//  in the function we export.
module.exports = (bankAPI) => {
  // Here we have a bankAPI parameter defined
  return (user) => {
    // And as this is in bankAPI's scope, we can even call it in the function
    //  return. Pretty neat, huh?
    const account = bankAPI.getAccount(user);
    if (!account) {
      return user.inform('You have no account with us, sorry.');
    }
    user.inform(`Your balance is ${account.balance}`);
  }
}

Which means that if we want to let our user see their balance, e.g. in the main file, we import the module with the usual require, but instead of returning the function which does this, it gives us a function which, in exchange for the bankAPI instance, gives us this function (I know, it's weird, ask me on Discord or wherever you got this from). Let me just demostrate it instead:

// File: e.g. index.js
// Load our bank API
const bankAPI = require('./bank-api.js');
// require('./balance.js') will now return a function, and we call it on the spot
//  so instead of returning the function, it will return what that function returns
//  which is our balance handler
const tellBalance = require('./balance.js')(bankAPI);
// Now we passed bankAPI to balance, and it will actually manage accounts though it
//  rather than requiring bankAPI itself.
tellBalance(someUser); // Tells someUser his balance

^*Dependencies can mean anything externally which a module depends on, whether it's a file, an API file, a different module, a database, anything really