Bash scripts to create VOD HLS stream with ffmpeg (Extended version)

create-vod-hls.sh

#!/usr/bin/env bash
START_TIME=$SECONDS
set -e

echo "-----START GENERATING HLS STREAM-----"
# Usage create-vod-hls.sh SOURCE_FILE [OUTPUT_NAME]
[[ ! "${1}" ]] && echo "Usage: create-vod-hls.sh SOURCE_FILE [OUTPUT_NAME]" && exit 1

# comment/add lines here to control which renditions would be created
renditions=(
# resolution  bitrate  audio-rate
  "426x240    400k    128k"
  "640x360    800k     128k"
  "842x480    1400k    192k"
  "1280x720   2800k    192k"
  "1920x1080  5000k    256k"
)

segment_target_duration=10       # try to create a new segment every 10 seconds
max_bitrate_ratio=1.07          # maximum accepted bitrate fluctuations
rate_monitor_buffer_ratio=1.5   # maximum buffer size between bitrate conformance checks

#########################################################################

source="${1}"
target="${2}"
if [[ ! "${target}" ]]; then
  target="${source##*/}" # leave only last component of path
  target="${target%.*}"  # strip extension
fi
mkdir -p ${target}

# ----CUSTOM----
sourceResolution="$(ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 ${source})"
# echo ${sourceResolution}
arrIN=(${sourceResolution//x/ })
sourceWidth="${arrIN[0]}"
sourceHeight="${arrIN[1]}"

echo ${sourceWidth}
echo ${sourceHeight}

sourceAudioBitRate="$(ffprobe -v error -select_streams a:0 -show_entries stream=bit_rate -of csv=s=x:p=0 ${source})"
sourceAudioBitRateFormatted=$((sourceAudioBitRate / 1000))
# ----END CUSTOM----

key_frames_interval="$(echo `ffprobe ${source} 2>&1 | grep -oE '[[:digit:]]+(.[[:digit:]]+)? fps' | grep -oE '[[:digit:]]+(.[[:digit:]]+)?'`*2 | bc || echo '')"
key_frames_interval=${key_frames_interval:-50}
key_frames_interval=$(echo `printf "%.1f\n" $(bc -l <<<"$key_frames_interval/10")`*10 | bc) # round
key_frames_interval=${key_frames_interval%.*} # truncate to integer

# static parameters that are similar for all renditions
static_params="-c:a aac -ar 48000 -c:v h264 -profile:v main -crf 19 -sc_threshold 0"
static_params+=" -g ${key_frames_interval} -keyint_min ${key_frames_interval} -hls_time ${segment_target_duration}"
static_params+=" -hls_playlist_type vod"

# misc params
misc_params="-hide_banner -y"

master_playlist="#EXTM3U
#EXT-X-VERSION:3
"
cmd=""
resolutionValid=0
prevHeight=0
for rendition in "${renditions[@]}"; do
  # drop extraneous spaces
  rendition="${rendition/[[:space:]]+/ }"

  # rendition fields
  resolution="$(echo ${rendition} | cut -d ' ' -f 1)"
  bitrate="$(echo ${rendition} | cut -d ' ' -f 2)"
  audiorate="$(echo ${rendition} | cut -d ' ' -f 3)"

  audioBitRateFormatted=${audiorate%?} # remove "k" at the last index

  # take highest possible audio bit rate
  if [ $audioBitRateFormatted -gt $sourceAudioBitRateFormatted ]; then
      audiorate=${sourceAudioBitRateFormatted}k
  fi

  # calculated fields
  width="$(echo ${resolution} | grep -oE '^[[:digit:]]+')"
  height="$(echo ${resolution} | grep -oE '[[:digit:]]+$')"
  maxrate="$(echo "`echo ${bitrate} | grep -oE '[[:digit:]]+'`*${max_bitrate_ratio}" | bc)"
  bufsize="$(echo "`echo ${bitrate} | grep -oE '[[:digit:]]+'`*${rate_monitor_buffer_ratio}" | bc)"
  bandwidth="$(echo ${bitrate} | grep -oE '[[:digit:]]+')000"
  name="${height}p"
  
  if [ $sourceHeight -le $prevHeight ]; then
      echo "video source has height smaller than output height (${height})"
      break
  fi

  widthParam=0
  heightParam=0

  if [ $(((width / sourceWidth) * sourceHeight)) -gt $height ]; then
    widthParam=-2
    heightParam=$height
  else
    widthParam=$width
    heightParam=-2
  fi
  
  cmd+=" ${static_params} -vf scale=w=${widthParam}:h=${heightParam}"
  cmd+=" -b:v ${bitrate} -maxrate ${maxrate%.*}k -bufsize ${bufsize%.*}k -b:a ${audiorate}"
  cmd+=" -hls_segment_filename ${target}/${name}_%03d.ts ${target}/${name}.m3u8"
  
  # add rendition entry in the master playlist
  master_playlist+="#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},RESOLUTION=${resolution}\n${name}.m3u8\n"

  resolutionValid=1
  prevHeight=${height}
done

if [ $resolutionValid -eq 1 ]; then
  # start conversion
  echo -e "Executing command:\nffmpeg ${misc_params} -i ${source} ${cmd}\n"
  ffmpeg ${misc_params} -i ${source} ${cmd}
  # create master playlist file
  echo -e "${master_playlist}" > ${target}/playlist.m3u8
  echo "Done - encoded HLS is at ${target}/"
else
  echo "Video source is too small"
  exit 1
fi

ELAPSED_TIME=$(($SECONDS - $START_TIME))

echo "Elapsed time: ${ELAPSED_TIME}"
echo "-----FINISH GENERATING HLS STREAM-----"

Thanks for putting this out. I'm currently working on a nodejs project that use dustjs as the templating engine, how do i generate hls manifest and dash manifest using the nodejs exec command to run shell script within a docker container?

Here's my simple video transcoder server that's uses NodeJS (Express + Multer) and generate HLS using exec command, the project is dockerized and can run with Docker.

You'll need an Azure storage account in order to run but you may modify my code to remove that part if not needed

Hey there, do you plan in doing a version that is S3 compatible ? Thank you and really appreciate this script you wrote.

@JohnTrabusca

I'm not having much time at the moment to make an S3 version for the server, you need to do it yourself. All you have to do is updating queue.js file, line 53 and 54: https://gitlab.com/maitrungduc1410/video-transcode-server/-/blob/master/queue.js#L54

You replace the content with something like this:

// declare these on top of queue.js
const { S3 } = require('aws-sdk')
const { createReadStream } = require('fs')
const s3 = new S3({
  endpoint: 'some endpoint',
  accessKeyId: 'accessKeyId',
  secretAccessKey: 'secretAccessKey',
});

// below is content for line 53 and 54
const uploadParams = {
  Bucket: 'some bucket',
  Key: file,
  Body: createReadStream(filepath),
  ACL: "public-read",
};

const data = await s3.upload(uploadParams).promise()

@JohnTrabusca

I'm not having much time to make an S3 version for the server, you need to do it yourself. All you have to do is updating queue.js file, line 53 and 54: https://gitlab.com/maitrungduc1410/video-transcode-server/-/blob/master/queue.js#L54

You replace the content with something like this:

// declare these on top of queue.js
const { S3 } = require('aws-sdk')
const { createReadStream } = require('fs')
const s3 = new S3({
  endpoint: 'some endpoint',
  accessKeyId: 'accessKeyId',
  secretAccessKey: 'secretAccessKey',
});

// below is content for line 53 and 54
const uploadParams = {
  Bucket: 'some bucket',
  Key: file,
  Body: createReadStream(filepath),
  ACL: "public-read",
};

const data = await s3.upload(uploadParams).promise()

Really appreciate the fast response, I will try to make this modifications. Really appreciate it :)

@JohnTrabusca you're welcome :)

@maitrungduc1410 I'm facing an issue where multer seems not to be saving the file. I'm running the app with pm2 and here's the error:

0|Encoder  | /home/encoder/video_transcode/storage/uploads/outputProcess_1d72f6b8f33c7e8069a360c3_1628878083521.mp4 storage/outputs/test-ppppp-80bee0336fd6dc147f863d731628878083716
0|Encoder  | JobID 5 has started
0|Encoder  | -----START GENERATING HLS STREAM-----
0|Encoder  | /home/encoder/video_transcode/storage/uploads/outputProcess_1d72f6b8f33c7e8069a360c3_1628878083521.mp4: No such file or directory
0|Encoder  | JobID 5 Failed
0|Encoder  | Error: Error when generating HLS stream!
0|Encoder  |     at ChildProcess.<anonymous> (/home/encoder/video_transcode/hls.js:21:16)
0|Encoder  |     at ChildProcess.emit (events.js:400:28)
0|Encoder  |     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)

What could be causing this ? I already set storage at 755 with Recursive and the files are own by the account I created not root.

Thanks in Advance.

EDIT:

The files does reach the server via POST

0|Encoder  | {
0|Encoder  |   fieldname: 'file',
0|Encoder  |   originalname: 'sample_1280x720_surfing_with_audio.mp4',
0|Encoder  |   encoding: '7bit',
0|Encoder  |   mimetype: 'video/mp4',
0|Encoder  |   destination: './storage/uploads/',
0|Encoder  |   filename: 'cf4eebbc6931ea7bdc960a66_1628884364395.mp4',
0|Encoder  |   path: 'storage/uploads/cf4eebbc6931ea7bdc960a66_1628884364395.mp4',
0|Encoder  |   size: 71753110
0|Encoder  | }

Fixed the issue, wasn't feeding the right mp4 to Generate since I had removed preprocessing.

Had a issue with this script because I use a charset with comma separator.

Fixed it like this:

key_frames_interval="$(echo `ffprobe ${source} 2>&1 | grep -oE '[[:digit:]]+(.[[:digit:]]+)? fps' | grep -oE '[[:digit:]]+(.[[:digit:]]+)?'`*2 | bc || echo '')"
key_frames_interval=${key_frames_interval:-50}
key_frames_interval=${key_frames_interval/,/\.}
key_frames_interval=$(bc -l <<<"$key_frames_interval/10")
key_frames_interval=${key_frames_interval/\./,}
key_frames_interval=$(echo `printf "%.1f\n" ${key_frames_interval}`)
key_frames_interval=${key_frames_interval/,/\.}
key_frames_interval=$(echo ${key_frames_interval}*10| bc) # round)
key_frames_interval=${key_frames_interval%.*} # truncate to integer

Had a issue with this script because I use a charset with comma separator.

Fixed it like this:

key_frames_interval="$(echo `ffprobe ${source} 2>&1 | grep -oE '[[:digit:]]+(.[[:digit:]]+)? fps' | grep -oE '[[:digit:]]+(.[[:digit:]]+)?'`*2 | bc || echo '')"
key_frames_interval=${key_frames_interval:-50}
key_frames_interval=${key_frames_interval/,/\.}
key_frames_interval=$(bc -l <<<"$key_frames_interval/10")
key_frames_interval=${key_frames_interval/\./,}
key_frames_interval=$(echo `printf "%.1f\n" ${key_frames_interval}`)
key_frames_interval=${key_frames_interval/,/\.}
key_frames_interval=$(echo ${key_frames_interval}*10| bc) # round)
key_frames_interval=${key_frames_interval%.*} # truncate to integer

@fivethreeo there is actually an easier fix, see my comment

Thanks for this script, it does a really good job.
I'm working on a project where, besides creating the HLS stream, I also want to encrypt it.
I already managed to do it with another ffmpeg script I made, but with this one, I don't see where I could insert the -hls_key_file keyinfo.drm parameter (where keyinfo.drm is a text file that contains my AES128 encryption key). Could you guide me on this ?

@dlobjoie This is what I've done for my script. Hope this might help you.

cmd+=" ${static_params} -vf scale=w=${widthParam}:h=${heightParam}"
cmd+=" -b:v ${bitrate} -maxrate ${maxrate%.*}k -bufsize ${bufsize%.*}k -b:a ${audiorate}"
# Add this line for encryption
cmd+=" -hls_key_info_file KEY_INFO_FILE"
cmd+=" -hls_segment_filename ${target}/${name}_%03d.html ${target}/${name}.m3u8"

@xubmuajkub Thank you very much. I had been looking for a long time.
Did you also automate the addition of the path to the key in the creation of the playlist?
If yes how did you do it?

@dlobjoie Just follow the step they do here https://hlsbook.net/how-to-encrypt-hls-video-with-ffmpeg/

@dlobjoie Just follow the step they do here https://hlsbook.net/how-to-encrypt-hls-video-with-ffmpeg/

Thanks @xubmuajkub that's where I started my HLS adventure a few months ago :)

@dlobjoie in my case, I just want a simple encryption so I only use 1 file for every video.

Hi @maitrungduc1410
Thanks for your amazing script.

Is there a way to add FRAME-RATE, CODECS, and AUDIO for every resolution?

In the case where we want multiple renditions at the same frame size - but at different bitrates - the script doesn't account for it, and the m3u8 points to the same manifest file, e.g.
`#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=400000,RESOLUTION=384x216
216p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=700000,RESOLUTION=512x288
288p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1100000,RESOLUTION=720x404
404p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1750000,RESOLUTION=720x404
404p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2750000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=4300000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5800000,RESOLUTION=1920x1080
1080p.m3u8

`

Is there a way to solve this; perhaps outputting each rendition and playlist in a separate folder?

Thanks!

Hi @maitrungduc1410 Thanks for your amazing script.

Is there a way to add FRAME-RATE, CODECS, and AUDIO for every resolution?

FRAME-RATE: not sure
CODECS: now supported: https://trac.ffmpeg.org/ticket/8904
AUDIO: seems possible: https://stackoverflow.com/questions/60017730/create-hls-streamable-audio-file-from-mp3

In the case where we want multiple renditions at the same frame size - but at different bitrates - the script doesn't account for it, and the m3u8 points to the same manifest file, e.g. `#EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:BANDWIDTH=400000,RESOLUTION=384x216 216p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=700000,RESOLUTION=512x288 288p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1100000,RESOLUTION=720x404 404p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1750000,RESOLUTION=720x404 404p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=2750000,RESOLUTION=1280x720 720p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1280x720 720p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=4300000,RESOLUTION=1920x1080 1080p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=5800000,RESOLUTION=1920x1080 1080p.m3u8

`

Is there a way to solve this; perhaps outputting each rendition and playlist in a separate folder?

Thanks!

my script is only for improving and fixing bugs based on the original script.

for your purpose you may need to write your own logics

Encode video not working in some Android Tv specially in Sony Bravia. Tv os version is 7. H.264 codec work on os version 6 or later. But it is not working. Anyone give me more insights ?