bash scripts to create VOD HLS stream with ffmpeg almighty (tested on Linux and OS X)

README.md

running:

bash create-vod-hls.sh beach.mkv

will produce:

    beach/
      |- playlist.m3u8
      |- 360p.m3u8
      |- 360p_001.ts
      |- 360p_002.ts
      |- 480p.m3u8
      |- 480p_001.ts
      |- 480p_002.ts
      |- 720p.m3u8
      |- 720p_001.ts
      |- 720p_002.ts
      |- 1080p.m3u8
      |- 1080p_001.ts
      |- 1080p_002.ts  

origin: http://docs.peer5.com/guides/production-ready-hls-vod/

create-vod-hls.sh

#!/usr/bin/env bash

set -e

# Usage create-vod-hls.sh SOURCE_FILE [OUTPUT_NAME]
[[ ! "${1}" ]] && echo "Usage: create-vod-hls.sh SOURCE_FILE [OUTPUT_NAME]" && exit 1

# comment/add lines here to control which renditions would be created
renditions=(
# resolution  bitrate  audio-rate
#  "426x240    400k    64k"
  "640x360    800k     96k"
  "842x480    1400k    128k"
  "1280x720   2800k    128k"
  "1920x1080  5000k    192k"
)

segment_target_duration=4       # try to create a new segment every X seconds
max_bitrate_ratio=1.07          # maximum accepted bitrate fluctuations
rate_monitor_buffer_ratio=1.5   # maximum buffer size between bitrate conformance checks

#########################################################################

source="${1}"
target="${2}"
if [[ ! "${target}" ]]; then
  target="${source##*/}" # leave only last component of path
  target="${target%.*}"  # strip extension
fi
mkdir -p ${target}


key_frames_interval="$(echo `ffprobe ${source} 2>&1 | grep -oE '[[:digit:]]+(.[[:digit:]]+)? fps' | grep -oE '[[:digit:]]+(.[[:digit:]]+)?'`*2 | bc || echo '')"
key_frames_interval=${key_frames_interval:-50}
key_frames_interval=$(echo `printf "%.1f\n" $(bc -l <<<"$key_frames_interval/10")`*10 | bc) # round
key_frames_interval=${key_frames_interval%.*} # truncate to integer

# static parameters that are similar for all renditions
static_params="-c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0"
static_params+=" -g ${key_frames_interval} -keyint_min ${key_frames_interval} -hls_time ${segment_target_duration}"
static_params+=" -hls_playlist_type vod"

# misc params
misc_params="-hide_banner -y"

master_playlist="#EXTM3U
#EXT-X-VERSION:3
"
cmd=""
for rendition in "${renditions[@]}"; do
  # drop extraneous spaces
  rendition="${rendition/[[:space:]]+/ }"

  # rendition fields
  resolution="$(echo ${rendition} | cut -d ' ' -f 1)"
  bitrate="$(echo ${rendition} | cut -d ' ' -f 2)"
  audiorate="$(echo ${rendition} | cut -d ' ' -f 3)"

  # calculated fields
  width="$(echo ${resolution} | grep -oE '^[[:digit:]]+')"
  height="$(echo ${resolution} | grep -oE '[[:digit:]]+$')"
  maxrate="$(echo "`echo ${bitrate} | grep -oE '[[:digit:]]+'`*${max_bitrate_ratio}" | bc)"
  bufsize="$(echo "`echo ${bitrate} | grep -oE '[[:digit:]]+'`*${rate_monitor_buffer_ratio}" | bc)"
  bandwidth="$(echo ${bitrate} | grep -oE '[[:digit:]]+')000"
  name="${height}p"
  
  cmd+=" ${static_params} -vf scale=w=${width}:h=${height}:force_original_aspect_ratio=decrease"
  cmd+=" -b:v ${bitrate} -maxrate ${maxrate%.*}k -bufsize ${bufsize%.*}k -b:a ${audiorate}"
  cmd+=" -hls_segment_filename ${target}/${name}_%03d.ts ${target}/${name}.m3u8"
  
  # add rendition entry in the master playlist
  master_playlist+="#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},RESOLUTION=${resolution}\n${name}.m3u8\n"
done

# start conversion
echo -e "Executing command:\nffmpeg ${misc_params} -i ${source} ${cmd}"
ffmpeg ${misc_params} -i ${source} ${cmd}

# create master playlist file
echo -e "${master_playlist}" > ${target}/playlist.m3u8

echo "Done - encoded HLS is at ${target}/"

It seems this script only outputs the filename on the next line after each #EXT-X-STREAM-INF line and this seems to 'break' any m3u8 playlist file because the entire URL to the file has to exist, no? (reference from Apple https://developer.apple.com/documentation/http_live_streaming/example_playlists_for_http_live_streaming/creating_a_master_playlist)

On other words, I cannot get this to work with this example:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p.m3u8

It seems to only work if I cite the URLs to the files:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
http://mydomain.com/360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
http://mydomain.com/720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
http://mydomain.com/1080p.m3u8

This is my extended version of this script that supports auto generate bitrates based on video source resolution, and handle cases when video after scaling has dimensions are not divisible by 2 that will cause error at run time.

Link: https://gist.github.com/maitrungduc1410/9c640c61a7871390843af00ae1d8758e

You are a life saver ! Works great !
Tho thanks to OP for first answer !

Tho i wonder for exampe how dose sfari determines which resultion to take from the playlist?
By screen size? by network speed? latency etc... ?

P.S
How can i force it to split to even more chunks ? it dose like 3 chunks for 22 seconds video
Can i force it into more ?

Hi, thanks for you script. I need help with other problem. I have one video with multiple quality and multiple audios. The audio have the same quality. How i can modify script for this.
Example
video1 --> 1080 | 720 | 480
audio --> english | spanish (same quality)
It possible?
Thanks!

Replace Line Number 39 :

static_params="-c:a copy -map 0:a -map 0:v -c:v h264 -profile:v main -crf 19 -sc_threshold 0

-c:a copy >> copy audio without convert
-c:v copy >> copy video without convert
-c:av copy >> copy audio and video without convert
-map 0:a >> all audio
-map 0:v >> all videos
-map 0:s >> all subtitles

when i try to convert the video it gives me error
bash create-vod-hls.sh vid.mp4 Executing command: ffmpeg -hide_banner -y -i vid.mp4 -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 60 -keyint_min 60 -hls_time 4 -hls_playlist_type vod -vf scale=w=640:h=360:force_original_aspect_ratio=decrease -b:v 800k -maxrate 856k -bufsize 1200k -b:a 96k -hls_segment_filename vid/360p_%03d.ts vid/360p.m3u8 -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 60 -keyint_min 60 -hls_time 4 -hls_playlist_type vod -vf scale=w=842:h=480:force_original_aspect_ratio=decrease -b:v 1400k -maxrate 1498k -bufsize 2100k -b:a 128k -hls_segment_filename vid/480p_%03d.ts vid/480p.m3u8 -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 60 -keyint_min 60 -hls_time 4 -hls_playlist_type vod -vf scale=w=1280:h=720:force_original_aspect_ratio=decrease -b:v 2800k -maxrate 2996k -bufsize 4200k -b:a 128k -hls_segment_filename vid/720p_%03d.ts vid/720p.m3u8 -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 60 -keyint_min 60 -hls_time 4 -hls_playlist_type vod -vf scale=w=1920:h=1080:force_original_aspect_ratio=decrease -b:v 5000k -maxrate 5350k -bufsize 7500k -b:a 192k -hls_segment_filename vid/1080p_%03d.ts vid/1080p.m3u8 Unrecognized option 'hls_playlist_type'. Error splitting the argument list: Option not found

Thanks for the script. But when I run this, here's my error message.

./create-vod-hls.sh 022505043294641312470435.mp4

[libx264 @ 0x55a878bcb800] [Eval @ 0x7fff9d53be00] Undefined constant or missing '(' in 'keyint_min'
[libx264 @ 0x55a878bcb800] Unable to parse option value "-keyint_min"
[libx264 @ 0x55a878bcb800] Error setting option g to value -keyint_min.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x55a878bca840] Qavg: 119.779
[aac @ 0x55a878bca840] 2 frames left in the queue on closing
[aac @ 0x55a878b9c080] Qavg: 119.779
[aac @ 0x55a878b9c080] 2 frames left in the queue on closing
[aac @ 0x55a878bcf800] Qavg: 119.779
[aac @ 0x55a878bcf800] 2 frames left in the queue on closing
[aac @ 0x55a878ba8e40] Qavg: 119.779
[aac @ 0x55a878ba8e40] 2 frames left in the queue on closing

did you solve this? I have the same error

hi, is there any way to include codec info as well in manifest?
Currently it is like:

#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080```

What i want is :

#EXT-X-STREAM-INF:BANDWIDTH=767744,AVERAGE-BANDWIDTH=720351,CODECS="avc1.4d401f,mp4a.40.5",RESOLUTION=640x480,FRAME-RATE=29.970

Is there a way to get CODECS in the manifest? If I use the following command:

ffmpeg -y -i ~/work/sample-videos/flowers.mp4  \
  -preset slow -g 48 -sc_threshold 0 \
  -map 0:0 -map 0:1 -map 0:0 -map 0:1 \
  -s:v:0 640x360 -c:v:0 libx264 -b:v:0 365k \
  -s:v:1 960x540 -c:v:1 libx264 -b:v:1 2000k  \
  -c:a copy \
  -var_stream_map "v:0,a:0 v:1,a:1" \
  -master_pl_name master.m3u8 \
  -f hls -hls_time 6 -hls_list_size 0 \
  -hls_segment_filename "v%v/fileSequence%d.ts" \
  v%v/prog_index.m3u8

CODECS are auto included for h264(for above command) but the above command does not support resolution based naming of segments.

Thanks

I tried to get all the segments for 10 seconds except for the first one which will be of 5 seconds by passing-hls_init_time 5 -hls_time 10 -hls_playlist_type VOD but seems like hls_time is being ignored all together, i.e. all the segments are of 5 seconds instead of 10. Am I doing something wrong? Can someone help?

Hi @mrbar42. Thank you for great script! Why resolution is 842x480 but by standard it is 854x480? from here https://support.google.com/youtube/answer/6375112?co=GENIE.Platform%3DDesktop&hl=en

Hi @mrbar42. Thank you for great script! Why resolution is 842x480 but by standard it is 854x480? from here https://support.google.com/youtube/answer/6375112?co=GENIE.Platform%3DDesktop&hl=en

There are many iterations of 480p; use the one relevant to your most prevalent client device.
https://en.wikipedia.org/wiki/480p#Resolutions

Thanks for the script. But when I run this, here's my error message.
./create-vod-hls.sh 022505043294641312470435.mp4
[libx264 @ 0x55a878bcb800] [Eval @ 0x7fff9d53be00] Undefined constant or missing '(' in 'keyint_min'
[libx264 @ 0x55a878bcb800] Unable to parse option value "-keyint_min"
[libx264 @ 0x55a878bcb800] Error setting option g to value -keyint_min.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x55a878bca840] Qavg: 119.779
[aac @ 0x55a878bca840] 2 frames left in the queue on closing
[aac @ 0x55a878b9c080] Qavg: 119.779
[aac @ 0x55a878b9c080] 2 frames left in the queue on closing
[aac @ 0x55a878bcf800] Qavg: 119.779
[aac @ 0x55a878bcf800] 2 frames left in the queue on closing
[aac @ 0x55a878ba8e40] Qavg: 119.779
[aac @ 0x55a878ba8e40] 2 frames left in the queue on closing

did you solve this? I have the same error

I change L:35 to awk

key_frames_interval="$(awk  "BEGIN { rounded = sprintf(\"%.0f\", ${key_frames_interval}); print rounded }")"

check this single comment multi output => https://github.com/masterofthesith/ffmpeg

How about having differntial segments accoding to the duration of the original video. For example, having shorter segments for the first 10% duration of video and then longer segments for the rest 90%?

I was looking for a code to convert video files in a directory where it has multiple subfolders for each video file. I want to run a batch code so it can convert the video files and save them where the original file was there.

Example:

: tree movies/
movies/
├── subfolder1
│ └── movie1.mp4
├── subfolder2
│ └── movie2.mp4
└── subfolder3
└── movie3.mp4

And so on.

I want the output to be the same as my main structure but in .m3u8 format.

I have used a code where it can convert each movie manually but for some reason, I can use the .m3u8 file or link only in that server I can't make it work all over the internet.

for i in .mp4; do
ffmpeg -i "$i" -c copy -map 0 -hls_time 3 -hls_list_size 0 -f hls "${i%.}.hls"
done

Thanks

@DravitLochan i am also thinking like you to initial hls video size size is small like first few are divided in 2 sec chunk and then after 10 sec we make it to 5 sec chunk.
so have you achieved this solution ?
so something we can do 5 chunk are in increasing order like 1s,2s,3s,4s,5s and then all remain are 5s/10sec chunk

Isn't it easier to create a master playlist file using FFMPEG itself?
Using -master_pl_name param

Ref: https://stackoverflow.com/a/71985380/6513289

Great script to get started, doesn't play in Safari though, might be the codec this script is using.