Android-Incremental-Update

Proof-of-concept(s) for implementing incremental update on private Android apps without using the Google Play store.

One of the difficulties of maintaining a private app (which does not publish to the Google Play store) is to implement a updating scheme. With Google Play store, the logic of app update is handled by the ingenious minds at Google, and for us developers, whenever we build a new version, we simply upload the latest APK to the store, and we done. However with a private app, we'll have to do it ourselves.

Problem with full APK download

Sounds simple enough!😎 Just setup a http server, upload our latest apk, let the client download it, boom! Profit. Here I have 2 versions of an app which prints its current version name on the screen.

Version 1.0.0	Version 1.6.9

I've also add some flavour text and an image to the version 1.6.9.

Works fine. The new version shows the up-to-date version number and our new image.

Problem is, we've downloaded the whole APK package, which is 9.1MB, while the new resources are only ~400KB in size. We've wasted so much bandwidth!

And that's just for our little demo app! Imagine a full app with bunch of resources, the APK size can go up quickly. Typical size of APKs range from 10MB to 50MB, with some apps even exceeding that. Thats kinda big!

Obviously we don't want our user to download the whole APK every time the app updates, and thats when Incremental Updates comes in!

Enter Incremental Updates

Instead of downloading the whole app again, we want to only download the new stuff and combine it into the existing program.

Basic idea is to compare two versions, identify the differences and create a update patch for our users. In the previous example, version 1.6.9 includes an additional image and some code changes. We want to create a patch consisting of those changes and ship it to our users. Since APKs (and all files really) are just bunch of binary numbers, if we compute the binary difference between 2 versions, we get ourselves a patch file. I believe thats how bsdiff work. On the client side, we apply the patch by adding the difference to the old APK, and we a patched version of the app ready to be installed.

Easier said than done, though! As Google doesn't provide any libraries for these kind of binary comparison algorithms nor any incremental update APIs for us Android developers, we have 2 options: to implement this feature ourselves, or use a third-party library.

Not going to reinvent the wheel, of course, so might as well just use a library.

One thing to note, is that most of these libraries are written in C/C++, which means we'll have to utilize Android NDK to call these C functions from our Kotlin code. Time to go low level :-)

In the next sections I'll try out different libraries I found on the Internet and share the results.

bsdiff

bsdiff is a simple library consist of 2 core programs: bsdiff and bspatch. These 2 do exactly what they sound.

It is worth mentioning that there's a report mentioning a vulnerability present in bsdiff which could lead to a buffer overflow attack. This vulnerability was associated with version 4.3, and has been patched. Though that means all the prebuilt executables pre-4.3 are outdated, and we have to build the program ourselves.

Alright, let's build:

Dependencies:

cmake
gcc (or other C compiler)

git clone https://github.com/mendsley/bsdiff
cd bsdiff

# dependency
git clone https://github.com/enthought/bzip2-1.0.6 bzip2

Create a CMakeLists.txt in the current dir with the following contents:

# CMakeLists.txt
cmake_minimum_required(VERSION 3.21.0)
project(bsdiff)

include(ExternalProject)
ExternalProject_Add(bzip2
        SOURCE_DIR ${CMAKE_SOURCE_DIR}/bzip2
        CONFIGURE_COMMAND ""
        BUILD_IN_SOURCE ON
        BUILD_COMMAND make
        INSTALL_COMMAND ""
)

include_directories(${CMAKE_SOURCE_DIR}/bzip2)

add_compile_definitions(BSDIFF_EXECUTABLE)
add_executable(bsdiff bsdiff.c)
target_link_libraries(bsdiff ${CMAKE_SOURCE_DIR}/bzip2/libbz2.a)

add_compile_definitions(BSPATCH_EXECUTABLE)
add_executable(bspatch bspatch.c)
target_link_libraries(bspatch ${CMAKE_SOURCE_DIR}/bzip2/libbz2.a)

This will configure the build for 2 executables: bsdiff and bspatch.

Next run:

mkdir build
cd build
cmake ..
make

Now we can see how bsdiff performs. Here I've prepared 2 versions of APKs.

# create patch
# bsdiff: usage: ./bsdiff oldfile newfile patchfile
./bsdiff 1.0.0.apk 1.6.9.apk patch

# then apply patch
# bspatch: usage: ./bspatch oldfile newfile patchfile
./bspatch 1.0.0.apk patched.apk patch

Essentially we're creating a new patch file patch, then applying the patch and output the patched version as patched.apk.

sha256sum checks out, proving that the patched version patched.apk is identical to the updated version 1.6.9.apk. Cool!

And the size of patch is only 445KB, much better than sending the full APK!

Works on my machine! But...

Now do it in Android

In practice, we'll have a server to run bsdiff, send the patch to client(Android), and run bspatch on it. So far we've only built the program targeting our own machines (Linux x64 in my case). We need to build bspatch for Android.

Dependencies

Android NDK
- Can download in Android Studio
- or from Official Site
cmake
wget

Setting up NDK

Android Studio: File -> Add C++ to Module
Download source code

cd app/src/main/cpp

wget https://github.com/devkitPro/bzip2/archive/main.zip -O bzip2.zip
unzip bzip2.zip
rm bzip2.zip

wget https://github.com/mendsley/bsdiff/archive/master.zip -O bsdiff.zip
unzip bsdiff.zip
rm bsdiff.zip

Add CMakelists.txt with the following content:

cmake_minimum_required(VERSION 3.22.0)

project("NativeProject")

# header file paths
include_directories(bzip2-main)
include_directories(bsdiff-master)

# compile bzip2
file(GLOB bzip2source bzip2-main/*.c)
add_library(bzip2 STATIC ${bzip2source})

# compile bspatch
add_compile_definitions(BSPATCH_EXECUTABLE)
add_library(bspatch STATIC bsdiff-master/bspatch.c)
target_link_libraries(bspatch bzip2)

# compile our code
add_library(native SHARED patch.c)
target_link_libraries(native bspatch log)

We build bzip2, bspatch, and finally our native C code. Add a patch.c source file if you don't have it. The project should now compile.

bspatch from Kotlin

We can now write our JNI code to call bspatch from the Kotlin space.

Create a Kotlin file Native.kt

package com.example.inc_update

object Native {
    init {
        System.loadLibrary("native")
    }

    external fun patch(oldFile: String, newFile: String, patch: String): Int
}

Here We define an object Native which has an external function patch(), which takes 3 args:

oldFile: file to be patched
newFile: the patched file
patch: the patch containing binary difference between 2 versions

Create a corresponding function in C.

Modify patch.c:

#include <jni.h>
#include <android/log.h>
#include "bspatch.h"

/* Wrapper for the bspatch cli */
JNIEXPORT int JNICALL
Java_com_example_inc_1update_Native_patch(JNIEnv *env, jobject thiz,
                                          jstring old_file,
                                          jstring new_file,
                                          jstring patch) {
    // convert jstring into C-string
    char *oldCStr = (char *) (*env)->GetStringUTFChars(env, old_file, NULL);
    char *newCStr = (char *) (*env)->GetStringUTFChars(env, new_file, NULL);
    char *patchCStr = (char *) (*env)->GetStringUTFChars(env, patch, NULL);

    /* bspatch cli usage: bspatch oldfile newfile patchfile */
    char *argv[] = {
            "bspatch",  // program name only useful when calling with cli; useless here
            oldCStr,
            newCStr,
            patchCStr,
    };

    int argc = sizeof(argv) / sizeof(char *);
    return bspatch_main(argc, argv);
}

Here we just write a wrapper of the bspatch cli (main() of bspatch), so we can just call it without big modification of the bsdiff code.

We still have to change the signature of the bspatch tho. Bspatch is originally a cli program and uses main() as function name. Better change it to something else (like bspatch_main) to avoid confusion.

in bsdiff-master/bspatch.c, change main to bspatch_main

-   int main(int argc,char *argv[]) {...}
+   int bspatch_main(int argc,char *argv[]) {...}

in bsdiff-master/bspatch.h, add signature for bspatch_main

+   int bspatch_main(int argc,char * argv[]);

and we can finally call bspatch in Kotlin. Let's test by putting some APKs in the app-specific files directory (something like /data/data/<package_name>/files/) and do some patching in that directory. In MainActivity for example:

override fun onCreate(savedInstanceState: Bundle?) {
    // some code

    val oldFile = File(filesDir, "1.0.0.apk").absolutePath
    val newFile =
        File(
            filesDir,
            "patched.apk"
        ).absolutePath // not exist yet, bspatch will create this if successful
    val patch = File(filesDir, "patch").absolutePath

    val result = Native.patch(
        oldFile = oldFile,
        newFile = newFile,
        patch = patch,
    )

    Log.d("Kotlin Space", "Native.patch() return $result!")
}

We push the files 1.0.0.apk and patch to files directory using adb and we have a folder structure similar to this:

After running the program we should see the patched new file being generated. If we check the hashes of patched.apk and 1.6.9.apk they should be identical.

Updating our App

Now we've got our latest (patched) apk, its time to update!

Checkout this article for how to install APKs programmatically. We'll be following this guide to write a simple UI that prompt the user to update.

First add some permissions so APKs can be installed:

Add in manifest:

<uses-permission android:name="android.permission.REQUEST_INSTALL_PACKAGES" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

<application>
    ...
    <provider
        android:name="androidx.core.content.FileProvider"
        android:authorities="${applicationId}.provider"
        android:exported="false"
        android:grantUriPermissions="true">
        <meta-data
            android:name="android.support.FILE_PROVIDER_PATHS"
            android:resource="@xml/provider_paths" />
    </provider>
</application>

Add a provider_paths.xml

<?xml version="1.0" encoding="utf-8"?>
<paths>
    <files-path
        name="files"
        path="." />
</paths>

Add following in MainActivity.onCreate()

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
    if (!packageManager.canRequestPackageInstalls()) {
        startActivityForResult(
            Intent(Settings.ACTION_MANAGE_UNKNOWN_APP_SOURCES)
                .setData(Uri.parse(String.format("package:%s", packageName))), 1
        )
    }
}

Lets say we have the 1.0.0 installed and uploaded the patch at /sdcard/patch.

The base apk of the app is located at /data/app/<package_name>-<hash>/base.apk. - It is under a folder named package name appended with a random hash. - The base.apk is the apk file of the app
The patch file contains differences between 2 versions of the app. - Here we just upload it through adb; In practice, download it from a server or something.

Both of these paths require elevated permissions to write to, therefore we'll have to copy it and do the patching somewhere else. In code we'll copy base.apk and patch to the app-specific folder before calling the native patch function:

val baseApk = File(applicationInfo.publicSourceDir)
val baseCopy = File(filesDir, "base.apk")   // copy of base apk
baseApk.copyTo(baseCopy, overwrite = true)

val patch = File(Environment.getExternalStorageDirectory(), "patch")
val patchCopy = File(filesDir, "patch")
patch.copyTo(patchCopy, overwrite = true)

val newFile =
    File(filesDir, "patched.apk") // not exist yet, bspatch will create this if successful

val oldFilePath = baseCopy.absolutePath
val newFilePath = newFile.absolutePath
val patchPath = patchCopy.absolutePath

val result = Native.patch(
    oldFile = oldFilePath,
    newFile = newFilePath,
    patch = patchPath,
)

To install the apk:

val installIntent = Intent(Intent.ACTION_VIEW).apply {
    setDataAndType(
        FileProvider.getUriForFile(
            this@MainActivity,
            BuildConfig.APPLICATION_ID + ".provider",
            newFile
        ), "application/vnd.android.package-archive"
    )
    addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
    addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION)
}

try {
    startActivity(installIntent)
} catch (e: ActivityNotFoundException) {
    e.printStackTrace()
}