Skip to content

Instantly share code, notes, and snippets.

@dmlap

dmlap/source-buffer-management.md

Last active March 19, 2016 19:58
Show Gist options
  • Save dmlap/26809ba67487ad413d47 to your computer and use it in GitHub Desktop.
Save dmlap/26809ba67487ad413d47 to your computer and use it in GitHub Desktop.
Download ZIP
Associating virtual to real SourceBuffers by examining the kinds of active video and tracks.
Raw
source-buffer-management.md

Media Configurations

The type, kind, and number of active tracks are enough information to figure out which SourceBuffers need to be active and what data to filter out of transmuxer output before appending.

available streams tracks kinds source buffer mime types
video+audio main video/mp2t
video+audio,audio main,alternative video/mp2t, application/octet-stream;codecs=mp4a
video,audio main,main video/mp2t;codecs=avc1, application/octet-stream;codecs=mp4a
video+audio,video,audio main,alternative,alternative video/mp2t, application/octet-stream;codecs=mp4a

Algorithms

The steps and responsibilities for common MediaSource input configurations are listed below. Steps are prefixed with the party responsible for performing them:

  • App: the application developer
  • VJS: video.js
  • HLS: videojs-contrib-hls
  • MSE: videojs-contrib-mse

Configuring a MediaSource with combined video and audio

video+audio

  1. HLS: create a media source
  2. HLS: create a source buffer with type video/mp2t

Configuring a MediaSource with backwards-compatible alternate audio support

video+audio,audio

  1. HLS: create a media source
  2. HLS: create a source buffer with type video/mp2t
  3. HLS: create a source buffer with type application/octet-stream; codecs=mp4a.40.2 (It doesn't look like Apple has defined a mime type for packed audio with AAC, their examples use application/octet-stream or text/plain)

Configuring a MediaSource with separate video and audio streams

video,audio

  1. HLS: create a media source
  2. HLS: create a source buffer with type video/mp2t; codecs=avc1....
  3. HLS: create a source buffer with type application/octet-stream; codecs=mp4a...

Enabling an alternate audio track

  1. App: change an audio track with kind alternative to enabled
  2. VJS: fire a change event at the AudioTrackList object
  3. MSE: determine the appropriate source buffer configuration by querying the set of enabled audio and video tracks
  4. MSE: disable audio input is on the primary source buffer
  5. MSE: enable audio input on the audio-only source buffer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment