We're running a lot of emulators for testing on our CI. Right now, we're using swiftshader_inderect renderer. But we have tesla p40 on our servers and want to utilize it because CPU usage is our bottle neck.
I'm using command:
/opt/android-sdk/emulator/emulator -avd emulator_22 -sdcard /sdcard.img -verbose -gpu host -snapshot ci -no-snapshot-save -no-boot-anim -no-audio -qemu
for running gpu accelerated emulator and everething works great.
But when I tried to run a lot more emulators (about 6) I found X11 lags (about 20 FPS) and white screen of deth on 7th emulator with a lot of buffer allocations errors:
D/EGL_emulation(20592): eglCreateContext: 0xa381eac0: maj 2 min 0 rcv 2
E/EGL_emulation(20592): rcCreateColorBuffer returned 0
E/EGL_emulation(20592): tid 20644: eglCreatePbufferSurface(930): error 0x3003 (EGL_BAD_ALLOC)
E/libEGL (20592): call to OpenGL ES API with no current context (logged once per thread)
E/libEGL (20592): call to OpenGL ES API with no current context (logged once per thread)
E/libEGL (20592): call to OpenGL ES API with no current context (logged once per thread)
F/libc (20592): Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 20644 (RenderThread)
I/DEBUG ( 1152): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG ( 1152): Build fingerprint: 'Android/sdk_google_phone_x86/generic_x86:5.1.1/LMY48X/5056529:userdebug/test-keys'
I/DEBUG ( 1152): Revision: '0'
I/DEBUG ( 1152): ABI: 'x86'
I/DEBUG ( 1152): pid: 20592, tid: 20644, name: RenderThread >>> com.android.systemui <<<
I/DEBUG ( 1152): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
E/DEBUG ( 1152): AM write failure (32 / Broken pipe)
I/DEBUG ( 1152): eax 00000000 ebx b700e508 ecx 00000000 edx 323d300c
I/DEBUG ( 1152): esi a38131d4 edi 00000000
I/DEBUG ( 1152): xcs 00000073 xds 0000007b xes 0000007b xfs 000000a7 xss 0000007b
I/DEBUG ( 1152): eip b74c30c0 ebp a33ef90c esp a33ef8e8 flags 00210246
I/DEBUG ( 1152):
I/DEBUG ( 1152): backtrace:
I/DEBUG ( 1152): #00 pc 000140c0 /system/lib/libc.so (strchr+112)
I/DEBUG ( 1152): #01 pc 000371de /system/lib/libhwui.so (android::uirenderer::Extensions::findExtensions(char const*, android::SortedVector<android::String8>&) const+62)
I/DEBUG ( 1152): #02 pc 000372bf /system/lib/libhwui.so (android::uirenderer::Extensions::Extensions()+127)
I/DEBUG ( 1152): #03 pc 00038455 /system/lib/libhwui.so
I/DEBUG ( 1152): #04 pc 000273f7 /system/lib/libhwui.so (android::uirenderer::Caches::Caches()+87)
I/DEBUG ( 1152): #05 pc 0006e085 /system/lib/libhwui.so
I/DEBUG ( 1152): #06 pc 00086419 /system/lib/libhwui.so
I/DEBUG ( 1152): #07 pc 00081112 /system/lib/libhwui.so
I/DEBUG ( 1152): #08 pc 00086f13 /system/lib/libhwui.so
I/DEBUG ( 1152): #09 pc 00086d00 /system/lib/libhwui.so
I/DEBUG ( 1152): #10 pc 0008850e /system/lib/libhwui.so
I/DEBUG ( 1152): #11 pc 00089cc9 /system/lib/libhwui.so (android::uirenderer::renderthread::RenderThread::threadLoop()+153)
I/DEBUG ( 1152): #12 pc 00016a5e /system/lib/libutils.so (android::Thread::_threadLoop(void*)+398)
I/DEBUG ( 1152): #13 pc 00071ca2 /system/lib/libandroid_runtime.so (android::AndroidRuntime::javaThreadShell(void*)+98)
I/DEBUG ( 1152): #14 pc 0001617a /system/lib/libutils.so (thread_data_t::trampoline(thread_data_t const*)+122)
I/DEBUG ( 1152): #15 pc 000211a8 /system/lib/libc.so (__pthread_start(void*)+56)
I/DEBUG ( 1152): #16 pc 0001c529 /system/lib/libc.so (__start_thread+25)
I/DEBUG ( 1152): #17 pc 000130f6 /system/lib/libc.so (__bionic_clone+70)
I/DEBUG ( 1152):
I/DEBUG ( 1152): Tombstone written to: /data/tombstones/tombstone_03
as I understand, this error about lack of gpu resources.
But nvidia-smi (and other gpu performance analysis tools) show me about 15% of gpu memory usage and about 5-25% GPU utilization.
I've tried to reproduce it on 2 environments and had about the same results:
Environment 1:
Nvidia Tesla p40
Xorg version: 1.19.2
Nvidia driver version: 418.56
OS: Linux 4.9.0-4-amd64, Debian 9
Results:
Maximum emulators count: 6
GPU memory usage: about 10%
GPU utilization: 5-35%
Environment 2:
Nvidia RTX 2080
Xorg version: 1.19.6
Nvidia driver version: 415.27
OS: Linux 4.20.14-042014-generic, Linux Mint 19.1 Tessa
Results:
Maximum emulators count: 8
GPU memory usage: about 25%
GPU utilization: 5-30%
The main questions:
- Why we can't use gpu accelerated emulators with
-no-windowflag? - Why we have frame buffer allocations problems with low GPU utilization and memory pressure on our video cards? Maybe I can provide another GPU performance metrics for understanding, what the real bottle neck of GPU accelerated emulators usage?
Thank you, feel fre for asking me any additional information, that you want.