A guide to back up and recover 2FA tokens from FreeOTP (Android)

freeotp_backup.md

Backing up and recovering 2FA tokens from FreeOTP

Backing up FreeOTP

Using adb, create a backup of the app using the following command:

adb backup -f freeotp-backup.ab -apk org.fedorahosted.freeotp

org.fedorahosted.freeotp is the app ID for FreeOTP.

This will ask, on the phone, for a password to encrypt the backup. Proceed with a password.

Manually extracting the backup

The backups are some form of encrypted tar file. Android Backup Extractor can decrypt them. It's available on the AUR as android-backup-extractor-git.

Use it like so (this command will ask you for the password you just set to decrypt it):

abe unpack freeotp-backup.ab freeotp-backup.tar

Then extract the generated tar file:

$ tar xvf freeotp-backup.tar
apps/org.fedorahosted.freeotp/_manifest
apps/org.fedorahosted.freeotp/sp/tokens.xml

We don't care about the manifest file, so let's look at apps/org.fedorahosted.freeotp/sp/tokens.xml.

Reading tokens.xml

The tokens.xml file is the preference file of FreeOTP. Each <string>...</string> is a token (except the one with the name tokenOrder).

The token is a JSON blob. Let's take a look at an example token (which is no longer valid!):

<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
	<!-- ... -->
	<string name="Discord:[email protected]">{&quot;algo&quot;:&quot;SHA1&quot;,&quot;counter&quot;:0,&quot;digits&quot;:6,&quot;imageAlt&quot;:&quot;content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674&quot;,&quot;issuerExt&quot;:&quot;Discord&quot;,&quot;issuerInt&quot;:&quot;Discord&quot;,&quot;label&quot;:&quot;[email protected]&quot;,&quot;period&quot;:30,&quot;secret&quot;:[122,-15,11,51,-100,-109,21,89,-30,-35],&quot;type&quot;:&quot;TOTP&quot;}</string>
</map>

Let's open a python shell and get the inner text of the XML into a Python 3 shell. We'll need base64, json and html in a moment:

>>> import base64, json, html
>>> s = """{"algo":"SHA1","counter":0,"digits":6,"imageAlt":"content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674","issuerExt":"Discord","issuerInt":"Discord","label":"[email protected]","period":30,"secret":[122,-15,11,51,-100,-109,21,89,-30,-35],"type":"TOTP"}"""

We decode all those HTML entities from the XML encoding:

>>> s = html.unescape(s); print(s)
{"algo":"SHA1","counter":0,"digits":6,"imageAlt":"content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674","issuerExt":"Discord","issuerInt":"Discord","label":"[email protected]","period":30,"secret":[122,-15,11,51,-100,-109,21,89,-30,-35],"type":"TOTP"}

What we specifically need from this is the secret. It's a signed byte array from Java... Let's grab it:

>>> token = json.loads(s); print(token["secret"])
[122, -15, 11, 51, -100, -109, 21, 89, -30, -35]

Now we have to turn this into a Python bytestring. For that, these bytes need to be turned back into unsigned bytes. Let's go:

>>> secret = bytes((x + 256) & 255 for x in token["secret"]); print(secret)
b'z\xf1\x0b3\x9c\x93\x15Y\xe2\xdd'

Finally, the TOTP standard uses base32 strings for TOTP secrets, so we'll need to turn those bytes into a base32 string:

>>> code = base64.b32encode(secret); print(code.decode())
PLYQWM44SMKVTYW5

There we go. PLYQWM44SMKVTYW5 is our secret in a format we can manually input into FreeOTP or Keepass.

Hi, this guide is quite old and I have since moved to iOS. But that string you get in the last step is the secret you can import in your authenticator app. You can use it as an alternative to the QR Code.

Long story short:

Install abe (choose your preferred aur installer):

yaourt -S android-backup-extractor-git

Save this python script to get-token.py:

#!/usr/bin/env python

import base64, json
import xml.etree.ElementTree as ET

verbose = False

root = ET.parse ('org.fedorahosted.freeotp/sp/tokens.xml').getroot()
for secrets in root.findall ('string'):
    name = secrets.get ('name')
    if name == 'tokenOrder':
        continue

    secret_json = secrets.text
    print ("secret name: {}".format(name))
    if verbose: print ("secret json: {}".format(secret_json))
    token = json.loads(secret_json);
    token_secret = token["secret"]
    if verbose: print("token secret: {}".format(token_secret))
    secret = bytes((x + 256) & 255 for x in token_secret)
    if verbose: print("token secret bytes {}".format(secret))
    code = base64.b32encode(secret)
    print("token secret base64: {}".format(code.decode()))

Get your token:

adb backup -f freeotp-backup.ab -apk org.fedorahosted.freeotp
abe unpack freeotp-backup.ab freeotp-backup.tar
tar xvf freeotp-backup.tar
cd apps/
../get-token.py
secret name: token name
token secret base64: PLYQWM44SMKVTYW5

For those who can't get the abe working because you are on windows machine(unfortunately) like me. Here is what works fine with me.
Taken from link https://thevaliantway.com/2018/08/freeotp-migration/

*Note : you need to use git bash or any similar command line which allows running linux commands

Multiline:
dd if=freeotp-backup.ab bs=1 skip=24 > compressed-data
printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - compressed-data | gunzip -c > decompressed-data.tar
tar -xvf decompressed-data.tar

One liner:
(printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" && dd if=freeotp-backup.ab bs=1 skip=24) | gunzip -c | tar -xvO apps/org.fedorahosted.freeotp/sp/tokens.xml > tokens.xml

---

The method above works well until the time you try to add entry manually to FreeOTP of FreeOTP+(my case) .
Problem is >>> Issuer is a mandatory field in the app.
Add any random text like redhat and then edit it later to remove it if you want.

The alternative that was suggested on various sites be used(did not work for me) is QR code generator which doesn't need the issuer: https://freeotp.github.io/qrcode.html
The issue ^^ here is counter value can't be put manually and defaults to 4.

If anyone's switching to FreeOTP+ it's also possible to directly import all tokens using the JSON format.
script modified from @albfan's:

#!/usr/bin/env python3
import base64, json, sys
import xml.etree.ElementTree as ET

tokens = []
token_order = []

root = ET.parse("apps/org.fedorahosted.freeotp/sp/tokens.xml").getroot()
for secrets in root.findall("string"):
    name = secrets.get("name")
    if name == "tokenOrder":
        continue

    #print("secret name:", name)
    tokens.append(json.loads(secrets.text))
    token_order.append(name)

json.dump({"tokenOrder": token_order, "tokens": tokens}, sys.stdout)

Copy the output into a json file, transfer it to the device and import it inside FreeOTP+.

Hi, this is my solution

Install deps, plug phone and dump backup

apt install adb maven qrencode
adb backup -f ./freeotp.ab -noapk org.fedorahosted.freeotp
git clone https://github.com/nelenkov/android-backup-extractor.git
cd android-backup-extractor
mvn clean package
cd ..
java -jar android-backup-extractor/target/abe.jar unpack freeotp.ab tokens

print out values

import base64
import json
import html
import re
import os


with open('tokens') as a:
    b = a.read()
    
    start, end = b.index('<map>'), b.index('</map>')
    for i in b[start:end].splitlines()[1:]:
        res = re.findall(r'(:?<.*>)(.*)(:?<.*>)', i)
        if res and len(res[0]) > 1:
            s = html.unescape(res[0][1])
            data = json.loads(s)
            if isinstance(data, dict):
                for k,v in list(data.items()):
                    if k == 'secret':
                        val = bytes((x + 256) & 255 for x in v)
                        code = base64.b32encode(val).decode()
                        data['secret_decoded'] = code
                    if not data.get('issuerInt'):
                        data['issuerInt'] = data['issuerExt']

                # otpauth://totp/ACME%20Co:[email protected]?secret=HXDMVJECJJWSRB3HWIZR4IFUGFTMXBOZ&issuer=ACME%20Co&algorithm=SHA1&digits=6&period=30
                _url = "otpauth://totp/{issuerInt}:{label}?secret={secret_decoded}&issuer={issuerInt}&algorithm={algo}&digits={digits}&period={period}".format(**data)
                print(f"{k}: {v}")
                print(_url)
                os.system(f'qrencode -o "token.png" "{_url}"')
                os.system("display token.png")
                    
                print()

We'll get something like this

algo: SHA1
counter: 0
digits: 6
issuerExt: Slack
issuerInt: Slack
label: [email protected]
period: 30
secret: THATSECRET
type: TOTP

and also it will display a QRcode image to easily import the OTP in your mobile phone.

Which adb version does not bother with the android:allowBackup="false" setting in the AndroidManifest.xml?
allowBackup was set to false since 2016 but apparently people were able to backup their FreeOTP tokens, so there must be something going on on newer adb verions? (Android 11 btw)

neither way from above did yield any backup, usually the file is 47 bytes long and contains ANDROID BACKUP.. none .. as ascii.
using a password is much more bloated up but the extracted data is just 0s.

and a 'full' backup does not include the freeotp data.

---

update

So I remembered it correctly that I was once trying out the adb backup method. Found a older backup and by timestamp also the matching adb platform tools in the downloads folder of the laptop.

platform-tools_r31.0.3-windows.zip

was the one that worked back then, and... to my surprise... it also worked now.

tried these versions:
platform-tools_r33.0.2-windows.zip (latest as for now)
platform-tools_r31.0.0-windows.zip
platform-tools_r29.0.0-windows.zip
platform-tools_r28.0.0-windows.zip
platform-tools_r27.0.0-windows.zip
platform-tools_r26.0.0-windows.zip

without any luck. did use the platform tools from August 2021 (v31.0.3), same command, but a correct backup file. :)

With the notes by sfan5 the generated json is imported without any issues into FreeOTP+

I cannot tell why those other versions didn't work, but it might help someone who does struggle creating a backup on a non rooted device.
maybe v31.0.3 was bugged and didn't bother about the allowBackup=false manifest, but go figure, time to get rid of FreeOTP and move to FreeOTP+

Thanks for the hint with AndroidManifest.xml and allowBackup=false.
r26 to r33 didn't worked for me.
After reading this hint i used a version from 2016: platform-tools_r24.0.3-windows.zip worked liked a charm :).
For extracting the xml you can use https://rawgit.com/viljoviitanen/freeotp-export/master/export-xml.html (https://github.com/viljoviitanen/freeotp-export/blob/master/export-xml.html)

To extract freeotp-backup.ab without "Android Backup Extractor", just run this:

( printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" ; tail -c +25 freeotp-backup.ab ) |  tar xfvz -

You can ignore the gzip warring: unexpected end of file, it will still extract the file.

Credits to: https://stackoverflow.com/questions/18533567/how-to-extract-or-unpack-an-ab-file-android-backup-file

And thanks @albfan and @sfan5, your Json parser works great for FreeOTP+
https://gist.github.com/kontez/05923f2fc208c6bbe3de81f28de571db?permalink_comment_id=3955026#gistcomment-3955026

If anyone's switching to FreeOTP+ it's also possible to directly import all tokens using the JSON format. script modified from @albfan's:

#!/usr/bin/env python3
import base64, json, sys
import xml.etree.ElementTree as ET

tokens = []
token_order = []

root = ET.parse("apps/org.fedorahosted.freeotp/sp/tokens.xml").getroot()
for secrets in root.findall("string"):
    name = secrets.get("name")
    if name == "tokenOrder":
        continue

    #print("secret name:", name)
    tokens.append(json.loads(secrets.text))
    token_order.append(name)

json.dump({"tokenOrder": token_order, "tokens": tokens}, sys.stdout)

Copy the output into a json file, transfer it to the device and import it inside FreeOTP+.

Hi Sfan5,

kannst du mir helfen ich bin total am verzweifeln. Ich will ein OTP auf ein neues Handy übertragen, ich wollte das jetzt so machen wie du, indem ich es in OTP+ direkt übertrage, und dort kann ich ja vermutlich ein Backup machen davon weil OTP+ das ja zulässt.

Ich verstehe aber das alles hier nicht so, ich hab von der Materie auch nicht so viel Ahnung. Wie und wo gebe ich den von dir angegeben Code ein? Ich würde mich sehr freuen, wenn du mir das erklären könntest. Danke!!

Edit: Kann es sein das ich für das alle hier ein Linux Systen brauche? Ich versuch das grad alles in Windows Powershell