capture web audio.md

1. Capture entire app audio? [nope]

Android 10 (2019) introduced AudioPlaybackCapture api

https://android-developers.googleblog.com/2019/07/capturing-audio-in-android-q.html

Not many devices are on android 10 though.

Capture speaker output

AudioSource.REMOTE_SUBMIX

https://developer.android.com/reference/android/media/MediaRecorder.AudioSource.html#REMOTE_SUBMIX

Can't use. Requires difficult steps by user (and probably rooted phone): https://stackoverflow.com/a/45881427

Example app?

I didn't find any app that can record other apps audio.

There is only Mobizen app which claims to record internal sound.

It only works on Samsung and LG devices though: https://www.online-tech-tips.com/smartphones/how-to-record-internal-audio-and-video-on-an-android-smartphone/

I tried on pocofone and it didnt record any audio.

Low quality hack

Hack recording with the Visualizer class. https://stackoverflow.com/a/30157272

Would have to check with reese (legal implications) bcus this might get Google angry and potentially app suspensed.

2. Capture WebView audio? [maybe possible]

WebView + id of its MediaSession + custom AudioEffect = we get audio data

WebView allows application to control its MediaSession. source: <put doc link here>
with the MediaSession id we can apply audio effects e.g. android framework BassBoost
and im almost certain this means the audio data is accessible. source: ?
so we could make a custom AudioEffect that captures the audio data as it passes through unchanged. source: ?, else make some native MyAudioEffect.cpp

2. Capture GeckoView audio? [possible]

where is the audio buffers?

~~Audio playback:~~ ~~https://github.com/mozilla/gecko-dev/blob/master/mobile/android/geckoview/src/thirdparty/java/com/google/android/exoplayer2/audio/MediaCodecAudioRenderer.java#L359~~

GeckoView developer: most (all?) audio from Gecko goes through cubeb which is backed by OpenSL in most cases https://searchfox.org/mozilla-central/source/media/libcubeb/src/cubeb_opensl.c

How to hook into the audio buffers

Not sure, might be possible to hook in without editing the source? Otherwise will need to modify and then build geckoview from source.

How to get geckoview source:

git clone --depth 100 [email protected]:mozilla/gecko-dev.git
cd gecko-dev/mobile/android/geckoview
# <now open the current directory in your IDE>

Android Studio had some errors when I opened that folder. The Mozilla repo probably has a setup script that I didn't run.

2. With AudioSession id, how to get audio data?

Description of AudioSession id: https://developer.android.com/reference/android/media/AudioManager.html#generateAudioSessionId()

Implement a new AudioEffect?

NOT POSSIBLE? james34602/JamesDSPManager#9 (comment)

https://developer.android.com/reference/kotlin/android/media/audiofx/AudioEffect.html

Not sure if Android framework will let you use a custom created AudioEffect. Reading the docs it sounds like AudioEffects might be limited to what the device has pre-installed (e.g. code by Samsung).

First check if it is possible to use an AudioEffect coded in our application (not android system).

There is probably code of all audio effects in AOSP. Search AOSP source code here: https://cs.android.com/

Other way to get audio data?

Surely its possible with android NDK? The audio session id is the secure thing that allows the application to access the data.

This github thread might give useful info: james34602/JamesDSPManager#9 (comment)

Conclusion

Full app audio capture is not viable in 2020
WebView doesn't have an audio capture API
With an AudioSession id it should be possible to get the audio data
WebView does expose its AudioSession id
GeckoView does expose its AudioSession id
GeckoView doesn't have an audio capture API
GeckoView source has (HLS content only) playback method with ByteBuffer of audio data

troy-lamerton/capture web audio.md