firestore to json & json to firestore

readme.md

The origin of this gist: https://blog.cloudboost.io/copy-export-a-cloud-firestore-database-388cde99259b

The schema is for example only, it's must be modified according to the firestore database.

Quoting Bruno Braga

"this schema is how your DB is organized in a tree structure. You don't have to care about the Documents but you do need to inform the name of your collections and any subcollections, in this case we have two collections called users and groups, the all have their documents, but the collection users has its own subcollections, friends and groups, which again have their own subcollection, messages."

For any doubt, add it here, but let me know by email that the question has been added, I usually go unnoticed when someone comments here.

[email protected]

firestore2json.js

const admin = require('firebase-admin');
const fs = require('fs');

const serviceAccount = require('../../../../../../Private/myschool-data_transfer-key.json');

admin.initializeApp({ credential: admin.credential.cert(serviceAccount) });

const schema = require('./schema').schema;

const firestore2json = (db, schema, current) => {
  return Promise.all(
    Object.keys(schema).map(collection => {
      return db
        .collection(collection)
        .get()
        .then(data => {
          let promises = [];
          data.forEach(doc => {
            if (!current[collection]) current[collection] = { __type__: 'collection' };
            current[collection][doc.id] = doc.data();
            promises.push(
              firestore2json(
                db.collection(collection).doc(doc.id),
                schema[collection],
                current[collection][doc.id]
              )
            );
          });
          return Promise.all(promises);
        });
    })
  ).then(() => current);
};

firestore2json(admin.firestore(), { ...schema }, {}).then(res =>
  fs.writeFileSync('./src/native_web/firebase/local_db.json', JSON.stringify(res, null, 2), 'utf8')
);

json2firestore.js

const admin = require('firebase-admin');
const fs = require('fs');

const serviceAccount = require('../../../../../../Private/myschool-data_transfer-key.json');

admin.initializeApp({ credential: admin.credential.cert(serviceAccount) });

const schema = require('./schema').schema;

const json2firestore = (_JSON, db, schema) => {
  return Promise.all(
    Object.keys(schema).map(collection => {
      let promises = [];
      Object.keys(_JSON[collection]).map(_doc => {
        const doc_id = _doc;
        if (_doc === '__type__') return;
        let doc_data = Object.assign({}, _JSON[collection][_doc]);
        Object.keys(doc_data).map(_doc_data => {
          if (doc_data[_doc_data] && doc_data[_doc_data].__type__) delete doc_data[_doc_data];
        });
        promises.push(
          db
            .collection(collection)
            .doc(doc_id)
            .set(doc_data)
            .then(() => {
              return json2firestore(
                _JSON[collection][_doc],
                db.collection(collection).doc(doc_id),
                schema[collection]
              );
            })
        );
      });
      return Promise.all(promises);
    })
  );
};

json2firestore(
  JSON.parse(fs.readFileSync('./src/native_web/firebase/local_db.json', 'utf8')),
  admin.firestore(),
  { ...schema }
).then(() => console.log('done'));

schema.js

exports.schema = {
	all_users: {},
	feedback: {},
	reports: {},
	schools: {
		grades: {},
		notifications: {},
		publications: {},
		users: {},
	}
};

Sorry to ask some very naive question. I am new to Firestore (and GCP actually). I need to copy some collections from Firestore to Cloud Storage or Big Query. I am guessing I can modify these scripts for that purpose but have no idea what to do with clearly:

1> Where is the export_db.js script copying data to? It has "fs.writeFileSync" in the end but not so sure where it is writing to.
2> import_db.js script is writing back to Firestore?

Again sorry to ask very basic questions. Thanks!

@keeyong Do not apologize for asking, I apologize for taking so long to respond. 1) All the collections that are marked in the schema.js file will be transformed into a .json file (using the fs library) where you choose to save it. 2) Recently modify the names of the files to make it more understandable.

If you have any other questions, I would like to help you. Good luck!

Hello, thank you so much.. i used it and works perfect! but always receive this message:

The behavior for Date objects stored in Firestore is going to change
AND YOUR APP MAY BREAK.
To hide this warning and ensure your app does not break, you need to add the
following code to your app before calling any other Cloud Firestore methods:

const firestore = new Firestore();
const settings = {/* your settings... */ timestampsInSnapshots: true};
firestore.settings(settings);

With this change, timestamps stored in Cloud Firestore will be read back as
Firebase Timestamp objects instead of as system Date objects. So you will also
need to update code expecting a Date to instead expect a Timestamp. For example:

// Old:
const date = snapshot.get('created_at');
// New:
const timestamp = snapshot.get('created_at');
const date = timestamp.toDate();

Please audit all existing usages of Date when you enable the new behavior. In a
future release, the behavior will change to the new behavior, so if you do not
follow these steps, YOUR APP MAY BREAK.

I try to put "timestampsInSnapshots: true" the code continue working but it still appears the message... :(

@SistemasIntelygenz I think this is gonna be useful for you gdg-tangier/vue-firestore#11 (comment)

Thanks so much for the handy script. This skips the users collection. Any chance as how we can traverse the users collection and get all the documents inside users?

Thanks

@ukris Recently add a readme file with more information, sorry for the delay in answering, I honestly do not know about your comment.

Thank you for the very useful script. It works perfectly, the only issue I have is with references.

If you write them to json, it won't be recoverable.

Do you have have an idea how to solve this? maybe post processing?

Thank you

For anyone who needs this, I solved it by converting the references to an object and restoring the object on restore.

Code is a little hacky, I appologize.

firestore2json:

'strict';

const firebase = require('firebase-admin');
var serviceAccount = require("../config/firebase-development.json"); // source DB key

const localFile = require('./schema').localFile
const admin = require('firebase-admin');
const fs = require('fs');
const app = admin.initializeApp({ credential: admin.credential.cert(serviceAccount) });
const schema = require('./schema').schema;
const firestore = app.firestore()

firestore.settings({ timestampsInSnapshots: true })


/**
 * Converts DocumentReferecnces to {type:'reference', path:''}
 * on restore has to be made 'undone';
 *
 * @param document
 * @returns {*}
 */
const convertReferences = function (document) {
    for (var key in document) {
        const val = document[key];

        switch (true) {
            case (val instanceof firebase.firestore.DocumentReference): // its a reference
                document[key] = { type: 'reference', path: val.path };
                break;
            case (typeof(document[key]) === 'object'):
                document[key] = convertReferences(document[key]);
                break;
        }
    }
    return document;
}


const firestore2json = (db, schema, current) => {
    return Promise.all(Object.keys(schema).map(collection => {
        return db.collection(collection).get()
            .then(data => {

                let promises = [];
                data.forEach(doc => {
                    if (!current[collection]) current[collection] = { __type__: 'collection' };
                    let data = doc.data();
                    data = convertReferences(data);
                    current[collection][doc.id] = data;

                    const subdocument = firestore2json(db.collection(collection).doc(doc.id), schema[collection], current[collection][doc.id]);
                    promises.push(subdocument);
                });
                return Promise.all(promises);
            });
    })).then(() => current);
};

const firestore2localCache = async () => {
    const data = await firestore2json(admin.firestore(), { ...schema }, {})
    const json = JSON.stringify(data, null, 2)
    fs.writeFileSync(localFile, json, 'utf8')
    return localFile;
}

module.exports = {
    firestore2localCache
}

json2firestore

const admin = require('firebase-admin');
const fs = require('fs');


const localFile = require('./schema').localFile
const schema = require('./schema').schema;
let firestore;


/**
 * restores references from json to firestore;
 *
 * @param document
 * @returns {*}
 */
const restoreReferences = (document) => {

    for (var key in document) {
        const val = document[key];
        //
        console.log(val && val.type && val.type === 'reference'? val.path : '..');

        switch (true) {
            case (val && val.type && val.type === 'reference'):
                document[key] = firestore.doc(val.path);
                break;
            case (typeof(document[key]) === 'object'):
                document[key] = restoreReferences(document[key]);
                break;
        }
    }
    return document;


}


const json2firestore = (_JSON, db, schema) => {
    return Promise.all(Object.keys(schema).map(collection => {
        let promises = [];
        if (!_JSON[collection]) {
            return Promise.resolve();
        }

        Object.keys(_JSON[collection]).map(_doc => {
            const doc_id = _doc;
            if (_doc === '__type__') return;
            let doc_data = Object.assign({}, _JSON[collection][_doc]);
            Object.keys(doc_data).map(_doc_data => {
                if (doc_data[_doc_data] && doc_data[_doc_data]['__type__']) delete doc_data[_doc_data];
            })

            doc_data = restoreReferences(doc_data)

            const recursion = db.collection(collection).doc(doc_id)
                .set(doc_data).then(() => {
                    return json2firestore(_JSON[collection][_doc], db.collection(collection).doc(doc_id), schema[collection]);
                });
            promises.push(recursion)
        });
        return Promise.all(promises);
    }));
};


const localeCacheToFirestore = async function (tenant) {
    const tenantConfig = '../config/firebase-' + tenant;
    let serviceAccount;
    const jsonData = fs.readFileSync(localFile, 'utf8');

    if (!jsonData) {
        throw new Error('JSON not readable in ' + localFile);
    }
    const json = JSON.parse(jsonData);
    if (!json) {
        throw new Error('JSON is invalid in ' + localFile);
    }

    try {
        serviceAccount = require(tenantConfig);
    } catch (e) {
        console.error(e);
        throw new Error('tenant config not readable in ' + tenantConfig);
    }

    app = admin.initializeApp({ credential: admin.credential.cert(serviceAccount) });
    firestore = app.firestore()
    firestore.settings({ timestampsInSnapshots: true })

    await json2firestore(json, admin.firestore(), { ...schema })

    console.log('done');


}


module.exports = {
    localeCacheToFirestore
}

schema.js

// @docs  https://gist.github.com/sturmenta/cbbe898227cb1eaca7f85d0191eaec7e
exports.schema = {
    courses: {},
     lectures: {
         slides: {}
     },
};
exports.localFile = '../tmp/local_db.json';

HI,
Sorry to ask very basic question, but I am not sure how to use this script..
I put it inside a new folder in my desktop, then I open a terminal (moving into the desktop folder), then I run npm install firebase-admin then I run node firestore2json.js but without changing anything in the script... 1/ is it the good method ? - 2/I am sure I have to change something inside the code to access to my own database, but I do not know what to change ? I have download the new private key from firebase but also do not know how to use it.. Thanks for help

hello @YLM-UJM since you answered me via email that you could solve your problem, it would be good if you could add the solution here, if it is useful for someone, thank you very much!

The export leaves timestamps as objects:
"createdAt": { "_seconds": 1588657677, "_nanoseconds": 660000000 },

The following has no effect on storing or parsing of the timestamps:
firestore.settings({ timestampsInSnapshots: true });

Am I missing something? thanks!

Workaround for dates:

const fixDates = (document) => {
  for (var key in document) {
    const val = document[key];

    if (val != null && val["_seconds"] != null && val["_nanoseconds"] != null) {
      document[key] = new admin.firestore.Timestamp(
        val["_seconds"],
        val["_nanoseconds"]
      );
    }
    if (val != null && val.constructor === Object) {
      fixDates(val);
    }
  }
  return document;
};

Then add doc_data = fixDates(doc_data);
right before setting the data

Workaround for dates:

const fixDates = (document) => {
  for (var key in document) {
    const val = document[key];

    if (val != null && val["_seconds"] != null && val["_nanoseconds"] != null) {
      document[key] = new admin.firestore.Timestamp(
        val["_seconds"],
        val["_nanoseconds"]
      );
    }
    if (val != null && val.constructor === Object) {
      fixDates(val);
    }
  }
  return document;
};

Then add doc_data = fixDates(doc_data);
right before setting the data

Its working well! Thanks a lot @klivin

So amazing to see that this is still helping people @sturmenta thanks for keeping it up to date.

@sturmenta can you please make a package with this ? I could improve a few things for handling very large collections (1M+ documents)

export / import command line

Project to export
Goto -> project settings -> Service account -> Generate new private key -> save it as exportedDB.json

Project to import
Goto -> project settings -> Service account -> Generate new private key -> save it as importedDB.json

run these 2 commands from the folder where u saved the files

Export:
npx -p node-firestore-import-export firestore-export -a exportedDB.json -b backup.json

Import:
npx -p node-firestore-import-export firestore-import -a importedDB.json -b backup.json

Hope it helps

Hi @sadortun , I am bruno the initial creator of this script :)

can you expand on your use case and why https://firebase.google.com/docs/firestore/manage-data/export-import does not fix your problem?

I would gladly tackle this issue and create a package if it seems like something that could benefit others :)

Workaround for dates:

const fixDates = (document) => {
  for (var key in document) {
    const val = document[key];

    if (val != null && val["_seconds"] != null && val["_nanoseconds"] != null) {
      document[key] = new admin.firestore.Timestamp(
        val["_seconds"],
        val["_nanoseconds"]
      );
    }
    if (val != null && val.constructor === Object) {
      fixDates(val);
    }
  }
  return document;
};

Then add doc_data = fixDates(doc_data); right before setting the data

This was not working for document with arrays and dates inside. Here the code fixed:

const fixDates = (document) => {
    for (var key in document) {
        const val = document[key];
        if (val != null && val["_seconds"] != null && val["_nanoseconds"] != null) {
            document[key] = new admin.firestore.Timestamp(
                   val["_seconds"],
                   val["_nanoseconds"]
            );
        }
        if (val != null && val.constructor === Object) {
            fixDates(val);
        }
        if (val != null && val.constructor === Array) {
            for(var arrayKey of val){
                fixDates(arrayKey);
            }
        }
    }
    return document;
};