Sync3story: How we fixed a reformat

Intro

The Sync 3 reformat package is the special Ford's package that allows to perform a clean OS installation on the Sync 3 APIM board.

The reformat does a lot of work. It wipes and creates a new partition table on the APIM's eMMC, getting "release" OS and apps packages from the USB drive, verifies them using the built-in certificate and, if they are valid, writes these packages on the APIM's eMMC.

The reformat is the internal Ford package and it should never be publicity available. Should... 😆
So someone leaked these packages and made it publicity.

The problem

The problem is that after flashing a reformat package we got Error failed to partition eMMC...
This error is caused by Ford's decision to change the eMMC model in the new APIM boards, which in turn resulted in incompatibility "new" eMMCs with "old" reformat package.

So we have the bricked APIM with a funny red screen. What's now?
The only solution I could find was using eMMC reader and flash either a dump form APIM with a new eMMC or if we had a new reformat that supports new eMMC...

Ah, forget about it, let's do it our way 💪

Search for a solution

First of all let's see whats inside the reformat package:

$ tar -tvf 1u5t-14g386-cb.tar.gz 
-rwxrwxrwx 0/0             514 2017-08-24 22:35 1U5T-14G386-CB.sh
-rwxrwxrwx 0/0           21292 2017-08-24 10:06 MLO
-rwxrwxrwx 0/0         9279956 2017-08-24 10:06 QNX-IFS-REFORMAT
-rwxrwxrwx 0/0            2441 2017-09-22 20:12 Version.der

We already know that MLO is the bootloader and QNX-IFS-REFORMAT is the OS kernel+rootfs.

Since we see a changes pictures and the error message on screen, we can assume that the bootloader is working correctly and we get stuck somewhere inside the QNX-IFS-REFORMAT.

Time to take a closer look at QNX-IFS-REFORMAT. dumpifs will do the most job for us.

Unpack ifs: dumpifs -xr QNX-IFS-REFORMAT-DEV

Search for the error message:

$ grep -rl "Error failed to partition eMMC"
usr/sbin/update_radio.sh

Thats it! update_radio.sh is the main script that handles reformating work.

Here is the part where we get an eMMC error:

echo "Repartitioning eMMC..." >> $LOG_FILE
echo "Repartitioning eMMC..." > $DISPLAY
partition >> $LOG_FILE

if [ $? -ne 0 ]; then
	echo "Failed to partition eMMC..." >> $LOG_FILE	
	echo "Error failed to partition eMMC..." > $DISPLAY
	exit_reformat
fi

partition ... who the hell are you 🧐

$ find -name "partition*" -exec readelf -h {} \;
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1

Something is becoming clearer. partition is the some utility that returns an error and causes to reformat to fail.
BTW output of this tool is redirected to LOG_FILE:

Please insert USB stick...
Reformat install start
Searching for update packages...
Thu Jan 01 00:00:19 UTC 1970
Found update packages on USB stick...
Repartitioning eMMC...
ERROR: Unknown platform
DBG: Size: Platform{61071360]-nand[62095360]
Failed to partition eMMC...
ERROR: calculating NAND size error: -1
Reformat install end
Thu Jan 01 00:00:19 UTC 1970

ERROR: Unknown platform - here it is, the first error string from partition utility.

Hydra, you're up! 🐉
I'm not going to describe the entire process of reverse engineering partition tool, but I'll just give a picture.

Summarizing the analysis: this function gets some value from the eMMC and compares it with four values that calculated based on built-in values. Built-in values are organized into four structures like this:

struct flash_info {
	uint32_t super_value_1;
	uint32_t super_value_2;
	uint32_t super_value_3;
}

These built-in values used for detect the APIM type: 8/16/32/64gb.
If the val from eMMC doesn't match to any of the four built-in values, then we will get Unknown platform error.

The next string from log file reports that partition expects some value that equals to 61071360, but gets 62095360 from the new eMMC chip.

TL;DR...
Ok, let's patch built-in table values to make them match to the new eMMC chip params.

(0xECE0 * 0x40 * 0x20)/2 == 62095360 Done!

Packing things back: mkifs buildfile.bld patched_QNX-IFS-REFORMAT.ifs.

The result

Unfortunately, if the APIM, already bricked with the old reformat, it requires removal from the car 🤷

We will use sync3flash tool to recover the APIM. Follow the link to find detailed instructions.

Connect APIM thought USB, short pads, and run command: sudo ./sync3flash -i patched_QNX-IFS-REFORMAT.ifs -m MLO

We did it!

Now we have the fixed "old" QNX-IFS-REFORMAT package which supports the new 64Gb eMMC.

Outro

I want to thank everyone who helped in the search the solution, shared info, debugged and tested 🙏

IgorStr
Sanek2033
p0w3r_0ff
AuRoN89
Kacpi
AlexeyBalmix