Skip to content

Instantly share code, notes, and snippets.

@dcow

dcow/embedded-android.md

Last active December 3, 2016 12:25
Show Gist options
  • Save dcow/0dd5f8b45b5a9fd38abb8f3b16f130d2 to your computer and use it in GitHub Desktop.
Save dcow/0dd5f8b45b5a9fd38abb8f3b16f130d2 to your computer and use it in GitHub Desktop.
Download ZIP
Android Dev Con 2016
Raw
embedded-android.md

"Embedded" Android

http://www.slideshare.net/opersys/embedded-android-workshop-with-nougat

... android history and platform overview

libc

  • bionic
  • bastardized but mostly works
    • (I believe it's bsd's libc)

hal

  • hardware abstraction layer
    • biggest divergance from linux

init

  • only userspace process started by kernel
  • android has its own init semantics
    • namely global properties and triggers (executed in response to changes)

shell

  • toybox
    • busybox is gpl so android doesn't use it
    • toybox has many parts written by the original busybox author

zygote

  • vm bootstrap (parent) process

services

  • $ service list
  • system vs app
    • system is boot to boot
    • app service lifecycle is managed by the activity manager
  • binder (ipc mechanism)

java

  • openjdk now

kernel

  • low memory killer
    • when memory fills up system kills apps
    • system has communicate memory conditions
    • custom oom killer driver (lmk)

boot process

  • cpu gets power
  • first instruction address
  • bootloader
    • modern cpus usually have internal booloaders which handle various bootstrap things
    • like secure boot
  • kernel starts
    • init

== identical to standard linux boot so far ==

  • daemons
  • zygote
    • tailored for android user experience
    • hot vm
    • about 4k classes loaded
    • when you start an activity the activity manager asks zygote to fork
      • that's why the launcher is not the parent process for apps

bootloader

storage layout

  • boot image -- kernel _ ram disk
  • system partition -- /system
  • data partition -- /data
  • cache partition -- /cache
  • vendor partition -- /vendor
  • recovery partition -- alternate boot image

boot

  • kernel
    • as mentioned, standard start (pretty hardware dependent)
  • android init

linux kernel internals

  • hardware components
  • kernel right on top
  • exposes userspace
    • syscalls and signals (communication)
  • hardware support
    • HALs
    • one HAL for each hardware type
    • HAL modules are shared libraries
    • still backed by a service unless otherwise specified

native user-space

  • /system => system components or otherwise firmware
  • /data => user data
  • /cache => cache and ota update also:
  • /dev
  • /proc
  • /sys
  • /sbin
  • /mnt
  • etc..

android java

  • oracle (sun) java
    • java language + jvm + jdk libs
  • android java
    • java language + art + apache harmony | openjdk

art (android runtime)

  • vs dalvik
  • 64 bit
  • multi-core
  • aot (vs jit)
  • better debugging
    • profiling/stack/crashes
  • largely replaced dalvik

"java is nice but sometimes you have to do real work" jni

  • java native interface
  • call gate for other languages
  • usage: native app from code
  • tools: ndk

system services

  • the android os

  • 95 or so on modern android

  • 5 or 6 new every major release
  • $ service list will not show app services, only system
  • many are separate services
  • some are combined
  • some written in c some in java
    • because performance
      • story: flight sims
      • devs would insert random 10/20ms delays if the test pilots were jerks
      • moral: humans are sensative even to that type of latency (think video/audo sync)
      • garbage collection sucks
  • some services are relaunched when they're killed
    • e.g. phone, but not email
  • android:persistent=true for platform apps
  • activity manager starts persistent apps
    • just their application is loaded into memory
    • from there you can register your system service
  • apps/services run as different users
  • app api: getSystemService(id)

activity manager

  • big thing kinda sucks but necessary

binder

  • cobra/com-like ipc
  • binder comes from bos/90s
  • palm wanted to stop using bos
    • engineers wanted to open-source it before it got killed
  • everyone at palm hated it
    • except the palm engineers who liked it
  • android binder inspired by "open binder" the binder source drop
  • kernel-supported
    • /dev/binder
    • /sys/kernel/debug/binder/*
  • "object-oriented os on top of normal os"
  • original open-binder was network capable so you could do rpc to remote resources
    • android not so much )=
  • vs sockets:
    • sockets: port, feed data structures
    • no pre-defined binder ids
      • no port 80, or 443, etc.
    • cursory name resolution
    • ask for service named: "foo"
    • it is actual remote method invocation
      • feed method params then feed function name
      • android parcelable is the way to feed things through binder

binder driver

  • driver gets an init call on boot
  • doesn't init hardware only does stuff in ram
  • service manager talks to binder driver
    • service manager is just a directory service
    • remember: service list
  • service manager declares itself as the context manager
  • anyone talking to binder id:0 is talking to the service manager
  • service manager serves as "dns" for system services
  • system services must register with the service manager
  • app does:
    • get service
    • looks up service via binder context service
    • returns handle to service
    • app calls into service using handle
    • activity manager handles ids for some system services
  • every service has a dump() function
  • $ dumpsys dumps all system services
  • $ dumpsys statusbar would just do statusbar (obviously)
  • context manager is "first come first serve" usually registered very early in init
  • for fun:
    • $ service call statusbar 1|2 (expand/collapse status bar)
  • binder explorer
  • extra
    • kmsg failed (in-kernel dmsg impl)
    • future: binder on top of bus1?
      • linux plumbers conference last year

Hardware Abstraction Layer

  • often closed source from vendors
  • idea is that as a vendor all you need to provide is a hal module
    • framework and apps use your HAL
  • few dozen integration points.. not as simple as it could be

AOSP tools and location

content

  • see slide 65
  • changes for your hardware happen in the "device" tree
  • libhardware/include/hardware is the hal
  • kernel is not part of the aosp
  • there is a dependency on kernel headers, though

building

  • ubuntu friendly
  • follow the instructions
  • . or source the build/buildsetup.sh
  • choose your lunch options (haha)
  • and make
  • croot takes you to the top of the android tree
  • godir file.name takes you to the directory that contains the file
  • mm rebuilds just the part of the tree you are in
  • m snod (snod is the system image target)
  • remember:
  • hmm
  • make help

build system architecture

  • NOT RECURSIVE
  • "recursive make considered harmful"
  • Android.mk
    • build "module"
    • rules
      • every Android.mk is self contained
      • (vs recursive where parents matter)
      • can load/include other Android.mk..
  • PRODUCT_PACKAGES is configured with packages that should be built
    • list begins at embedded.mk
    • included by base.mk
  • "object oriented build system with multiple inheritance" --guy who wrote it
    • barf
  • packages/apps/gallery/Android.mk
    • include BUILD_PACKAGE
      • determines what it's building
    • LOCAL_PACKAGE_NAME = "apkname"
  • frameworks/native/c2dm/service/Android.mk
    • include BUILD_EXECUTABLE
    • LOCAL_MODULE = "soname"
  • more LOCAL things exist ^^ just examples
  • ninja
    • layer under make for module generation and and build progress caching

output

  • [aosp]/out/target/product/generic/
  • kernel
    • prebuilt/android-arm/kernel/kernel-qemu
    • emulator overrides kernel and initrd

adb

  • use the help or something
  • it's the debug bridge

add a new device

  • device/acme/coyotepad/
    • AndroidProducts.mk
      • add to PRODUCT_MAKEFILES := $(LOCAL_DIR)/full_coyotepad.mk
    • full_coyotepad.mk
      • specifies options such as name, device, model, etc.
    • BoardConfig.mk
      • hardware options, kernel, arch, camera/audio etc.
    • Android.mk
  • menu integration:
    • vendorsetup.sh
      • add_lunch_combo full_[combo-level]
  • Android.mk as a filter so that it only gets picked up when building for the hardware
  • lunch combos
    • [device name]-[eng|userdebug|user]
  • levels
    • eng has root
      • and extra tools
    • userdebug is normal user by default but allows root
    • user: production build
      • no root

adding new apps

  • vendor local
    • device/acme/coyotepad
    • Android.mk
    • PRODUCT_PACKAGES
  • global
    • [aosp]/packages/apps
    • [aosp]/build/target/product/core.mk
  • overlays are used to provide your own resources (or even sources)
    • DEVICE_PACKAGE_OVERLAYS := device/acme/coyotepad/overlay
  • tools
    • system[/core]/
    • frameworks/[base|native]/cmds/
    • external/

Kernel

Native Android User-Space

  • acct for scheduler

  • props

    • property service
    • /data/property

  • etc

    • fstab: normal use
    • selinux policies
    • android rc

android rc

  • system/core/rootdir/init.rc
    • import
      • "run level" strings
    • not command line instructions, internal init language
    • init binary determines orders
    • hardware-specific init files can exist
      • init process process all rc files and runs all "sections" together
      • early init is the only thing that doesn't have something run before it
    • service declarations
      • service name and then a string
      • service here means init service not system service (binder)
      • traditional daemons +"launchctl-like" options: oneshot, last will instructions
      • classes: core, main, default (if none is specified)
      • classes are like the init levels
      • this is how most user-space daemons start
    • example: boot animation depends on surface flinger
      • so the boot animation service is disabled initially
      • then it is enabled by surface flinger after it has started
      • special properties correspond to init services
  • stop zygote kills the userspace
    • surface flinger brings back the boot animation when the connection to the window manager dies
  • start zygote brings it back
    • bootanimation is swapped back out for the windowmanager once it is restarted

binary utilities

  • am => activity manager
  • pm => package manager
  • wm => window manager
  • svc => binder service registry
  • ... see slides for more (cf. 124...)

app home

  • /data/data
  • permission model: uid sandbox
    • each app is its own uid
  • multi-user
    • user 0: /data/data
    • user n: /data/data/userN

etc/passwd on android

  • defined in a header file
    • static mapping to strings since you don't create kernel/system users dynamically
  • app ids are deltas from 10000
    • each android user gets 100000 app uids
  • unlisted shit goes to root as 644

adb

  • usb
  • tcp/ip
  • different log buffers
  • bugreport

shell

  • was old netBSD crap shell
  • moved to korn BSD shell
  • now toybox

getting java up

  • required:
    • service manager
    • zygote
  • package manager
    • finds dex files and processes them
  • system things
    • imput methods
    • persistent apps
    • home screen (home intent)
    • BOOT_COMPLETED intent
    • APPWIDGET_UPDATE intent
  • framework is now ready for interactions
  • naitive daemons
    • exist for privelege separation
    • rild exception to HAL model
    • system talks to rild and rild talks to HAL
      • system can crash without bringing the ril down
      • doesn't quite actually end up working that way
      • point is mission critical components are housed outside of android right on the kernel

observe system services

  • look for pid and logcat
  • dumpsys

android cross reference (opersys)

implement service

  • see slide 155
Raw
platform-debugging.md

Android Platform Debugging and Development

http://www.slideshare.net/opersys/android-platform-debugging-and-development-64674937

Put a breakpoint in the platform code.

Architecture Basics

  • Not focused on the baseband
  • Only main processor

Linux Kernel

  • Logger driver
  • otherwise pretty standard

Systrace

  • focused on ftrace functionality

System Servivces

  • some in c, some in java
    • human interaction things usually in c (gc bad)

Nexus 9

  • serial port is audio jack
    • !! I did not konw this...

uboot is network capable

  • can do netboot

Studio recognize android sources

  • get sources
  • get studio
  • [aosp]$ make idegen && development/tools/idegen/idegen.sh
  • res.java (if it complains)
    • delete it from your board output directory
    • out/target/product/generic/obj/GYP/shared_intermediates
  • can't build with studio, of course

Native Utilities

  • schedtop
  • librank
  • procmem
  • procrank
  • showmap
  • latencytop framework
  • dumpsys
  • service aanndd
  • getprop
  • logcat
  • dmsg (kernel logs)
    • timestamp different from logcat

Helper commands when in aosp source

  • hmm
  • make help

Symbolic debugging

  • slide 31

Native ptrace

  • gdbserver

ddms

  • implements jdwp
  • jdb

threads

  • threads with stars are ones started by the framework
  • also notice binder threads
  • one main thread

debug system app: java

Start monitor

  • add remote run configurations
    • port 8700
  • apply and debug

monitor: attach to process

  • debug
  • look for green bug

native debugging

target side

  • gdbserver --attach localhost:2345 30

  • symbols in: out/target/product.../symbols/... (on host)
  • start app for debuggin with gdbserver prepended

    • gdbserver localhost:2345 service list

  • forward the port on the host
    • $ adb forward tcp:2345 tcp:2345

host side

  • load file first then attach on host side
  • $ prebuilts/gcc/linux-x86/arm/armeabi-4.7/bin/arm-eabi-gdb
    • need to use gdb compiled with support for the target arch
  • (gdb) file out/target/product/generic/symbols/system/bin/service
  • (gdb) target remote localhost:2345

ftrace

  • makes systrace work
    • systrace is python script running on the host
    • atrace is native android binary
    • systrace calls atrace via ADB
    • atrace uses ftrace to capture kernel events
  • need to download or compile a kernel with ftrace enabled
  • filters are cool
  • all functions traceable
  • events dir
    • sched evvents (scheduler)
  • trace_marker
  • add a marker to each trace line
  • (apparently the browser support is only for chrome)

perf on Android on ARM

  • uses performance counters for:
    • page faults
    • branch prediction outcomes
    • cache hits/misses
  • dependent on manufacturer support...
Raw
trust-zone.md

Android Trust Zone

Jonathan Levin

http://technologeeks.com/files/ATZ.pdf

Recap of ARMv7 ARMv8

  • ELS - exception levels
  • Android Implementations
    • Samsung
    • Qualcomm (terrible)

Notion of

  • trusted os
  • trust"let"

Trust Zone

Trust boundaries

  • user
  • kernel
  • hardware
  • Historically if you are kernel you are god
  • need hardware backed security
    • confine some part of the processor to be isolated
    • arm64 has exception levels

aarch64

  • EL3 only exists in secure world
  • EL2 kernel

Rings

  • user mode

  • root (artificial separation)

    • because the processor doesn't know
    • relies on kernel to protect access

  • kernel mode

    • physically enforced by hardware

  • intel

    • ring: 0, 1, 2, 3

  • arm

    • EL

Hardware enforced gate

  • intel: syscall
  • arm: svc

TrustZone is an extra ring

  • protect hardware form rootkits, etc.

RTFM - arm

  • nice illustrations
  • normal world vs secure world

intel

  • boot into real mode
    • access to everything
  • kernel drops the proc out of real mode
    • into protected mode
    • no more physical memory acces
    • virtual addresses

aarch32

  • boot into secure world
    • can run full os in this mode
    • certain hardware elements such as trusted modules only accessable to secure world
      • secure storage
      • crypto hardware
      • etc. *after secure world: drop out into normal world
    • android time!

Android TrustZone

  • key generation (keystore, gatekeeper)
    • non-secure world only gets tokens
    • public keys accessible
    • secret unlocking (e.g. passwords) can be throttled or auto-wiped
  • DRM
  • Hardware backed entropy
    • PRNG code
  • NFC (Android Pay)
  • kernel and boot chain integrity
    • secure boot

This puts us close to apple (apples is still better)

  • put keystore in hardware trustzone
  • now you need code running on the device to in order to bruteforce
    • because you tie secret with key from keystore in trustzone

samsung knox

  • client certificate management (CCM)
    • store certs in trustzone
    • tampered device = no certs
  • periodic kernel measurement (PKM)
    • similar to iOS KPP: periodically check kernel page hashes
  • realtime kernel protection (RKP)
    • intercept events from kernel using traps to secure monitor (SMC)

Implementation

aarch32

  • co-processor register
    • secure configuratoin register
    • 0 to 7 lsb are the only that matter
    • 0 only really matters: NS bit (not secure)
  • once NS is set to 1 virtually no way back other than:
    • trust zone interrupt handlers
      • important because they can't be patched
    • or SMC instruction
  • MMU enforces memory separation between worlds

Entering TrustZone

  • when you boot into trusted zone: setup "monitor vector"
    • standard exception vector (1 instruction wide, usually branch/jump)

Voluntary Transition: SMC

  • SMC only valid whie in [super/hyper]visor mode
    • (requires the os to be in kernel mode or higher)
  • most arm field 0, ios field 17

Recap: exception handling

  • exception levels
    • EL0: user
    • EL1: kernel
    • EL2: unused (hypervisor)
    • EL3: secure monitor (trust zone)
  • arm processors can run 3 operating systems concurrently, side-by-side
  • SVC -> kernel,hypervisor (EL1,2)
  • SMC (EL1,2) -> trustzone (EL3)
  • side note: secondary bootloader is still in trust zone

ELx registers

  • tuned to each exception level
    • separate SP_ELx, too
  • x0-x30 + PC + LR: standard
  • access to lower ELx can be trapped by higher ELx

64bit

  • s/secure world/EL3
  • s/exception vector/vbar
    • vbar is BIG
    • 128 bytes of instructions for each entry
    • good for cpu cache
  • vbar has adjacent entries for each exception level

Android & TrustZone *trust zone images are ELF (so is hypervisor)

  • Boot

== Levin's disassembler ==

  • disarm: good arm disassembler
  • newandroidbook.com/tools/disarm

Application (path)

  • keystore
    • hal
  • gatekeeper
    • hal
  • tzdaemon
  • kernel
    • tzdriver
  • SMC
  • TZ OS

Trusty

  • Google's attempt to standardize TEE Oses
    • source.android.com/security/trusty/index.html
    • baked into Linux kernel tree: /driver/trusty/
  • Used by Nvidia
  • Based on lk (similar to aboot) and provides
    • gatekeeper, keymaster, NVRAM modules
    • kernel driver
    • LK base (little kernel)
    • Trusty OS

Linux Kernel Support

  • generic TZ driver integrated into 3.10
  • use long term kernel
    • Levin recommends 4.4.35

Android Vulnerabilities (TrustZone)

  • plenty (see slide, also google's website)

bits-please.blogspot.com

  • rips apart trustzone
  • grabs full disk encryption keys
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment