Removing Sensitive Data from Git History

Removing Sensitive Data from Git History.md

Introduction

Sometimes developers may accidentally add secrets or other sensitive information into files to git repositories (and in some cases, the entire file contains secrets which should be removed). This document describes the steps to perform to ensure that these secrets can be masked going back through the git history

Masking Secrets in Files

To mask secrets in files, we will use git-filter-repo package. In Macbook, this can be installed via the following command:

$ brew install git-filter-repo

Create an expressions file which contains the sensitive phrases and the respective phrase that they should be replaced with sepeated by == eg:

passw0rd==>p*****d
passw0rd1a==>p*****d1*

Now, run the following commands to keep track of the remote branches, pull the latest changes from the remote branch:

# Note the the Remote branch $REMOTE_BRANCH
$ git remote -v
$ git pull
$ git filter-repo --replace-text /tmp/expressions.txt  --force

# This following step is required only if the remote branch `origin` is not specified
$ git remote add origin $REMOTE_BRANCH

# Push the changes to remote branch
$ git push origin master -f

This should mask the 2 secrets specified in the /tmp/expressions.txt file with its masked counterparts specified after ==> in the same file.

Removing entire file from git

To remove sensitive files, we will leverage the git-filter-repo package once again

Say, we wanted to remove a file called pass.txt, we execute the following commands:

# Note the the Remote branch $REMOTE_BRANCH
$ git remote -v
$ git pull
$ git filter-repo --invert-paths --path pass.txt

# This following command is required only if the remote branch `origin` is not specified
$ git remote add origin $REMOTE_BRANCH

# Push the changes to remote branch
$ git push origin master -f

Masking secrets in Past Commit Messages

To mask secrets in the commit histories, one can use git-rebase.

First, identify the commit messages which needs to be wiped in a given branch say master

$ git checkout master
$ git log -v

Identify the commit whose message needs to be modified eg for the fifth message from top, specify HEAD~5

$ git rebase -i HEAD~5

In the editor that opens, specify reword instead of pick against the commit which contains the secret that should be changed.

On the next screen, replace the secret with the masked value and save.

Now push the changes to the origin

git push -f origin master