Skip to content

Instantly share code, notes, and snippets.

@o770
Last active October 2, 2024 08:11
Show Gist options
  • Save o770/81093049d0a8d62c687a713a3db707e3 to your computer and use it in GitHub Desktop.
Save o770/81093049d0a8d62c687a713a3db707e3 to your computer and use it in GitHub Desktop.
Batch transcoding in FFmpeg to save disk space.

This bash script is a batch processing tool that runs FFmpeg and works like this:

Files and directories are entered on the command line, and user options are entered into variables within the script. Directories are expanded recursively and files are optionally filtered by bitrate and/or video width. Output files are saved in the current working directory. Options include automatic downscaling, test mode, and more. If a key is pressed in the interval between encodings, the script may exit or continue after confirmation. Information and statistics for the input and output files are displayed along with a summary of any warnings and errors when the execution ends. With no arguments entered on the command line, the script usage and current configuration are displayed.

Usage: penc.sh [-t] file|directory...

Dependencies: AWK, GNU Coreutils, GNU Grep, FFmpeg, FFprobe.

Notes
The displayed input file names are always quoted in a format that can be reused as input, so for example if some files were skipped but no error occurred, only source files with a new encoding can be deleted with ease.
There is no guarantee that the displayed input media information is complete, as in the results of running FFprobe on the file: duplicate streams and tags are not shown.

Options
In the first group, the values of 13 variables are either read directly into FFmpeg options or specify which files are passed to FFmpeg and, to some extent, how the script is executed. There are 2 other groups of variables that occur later in the script execution and consist of entire sections of FFmpeg options for further command line customization.

ibitmin=<number>
Minimum input bitrate (kbit/s). Script default: Disabled.
All files are passed to FFmpeg if this value is 0 (zero), or a file is ignored if the bitrate is less than this value or unknown. If the bitrate is equal to or greater than this value, a file will be passed to FFmpeg. Mediainfo is run if installed, tags in the video stream are read, or FFprobe is run second and the bitrate is read with the lowest value chosen between the file container and the video stream to which no image, video thumbnail or cover is attached. If Mediainfo is installed but tags have not been read at this time, Mediainfo will run again and read the bitrate in the file container.

iwidthmin=<number>
Minimum input video width (pixel). Script default: Disabled.
All files are passed to FFmpeg if this value is 0 (zero), or a file is ignored if the video width is less than this value or unknown.

owidth=<number>
Maximum output video width (pixel). FFmpeg scale filter: (-vf scale=<number>:-2). Script default: Disabled.
The width to which the output video will be reduced if the input width is greater than this value. The aspect ratio is maintained and the height is adjusted to be divisible by 2. Zero disables automatic downscaling.

ow=<n|y>
FFmpeg global options: (-n, -y). Script default: n (do not overwrite).
A file that exists in the current directory and has the same name as an input file will be overwritten if this value is y for Yes, or will cause the input file to be ignored if this value is n for No.

oext=<file extension>
File extension for output files. Script default: mp4.

aenc=<copy|encoder>
FFmpeg audio copy or encoder: (-c:a copy or -c:a <encoder>). Script default: aac.

venc=<encoder>
FFmpeg video encoder: (-c:v <encoder>). Script default: libx264.
Tested by the script author and verified along with other options are libx264 and libx265.

vpreset=<preset>
Video encoder preset: (-preset <preset>). Script default: medium (x264 default).
One of the following presets for x264 or x265: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow or placebo.

vcrf=<crf>
Video encoder CRF: (-crf <crf>). Script default: 23 (x264 default).
Quality for constant quality encoding.

verbo=<[flags+]loglevel>
FFmpeg log level and flags (-loglevel). Script default: level+24 (warnings and errors with level prefix).
A level higher than Warning (24) disables the script's own warning and error reporting, and FFmpeg's output is only displayed rather than saved and printed along with the filenames later when the script's execution comes to an end.

x265v=<integer|string>
x265 encoder log and statistics (--log-level). Script default: 1 (warnings and errors).
See "verbo" for more information.

tdur=<duration>
FFmpeg output option: (-t). Script command line option and default: (-t), 3 seconds.
Syntax (one or the other): [-][<HH>:]<MM>:<SS>[.<m>...] or [-]<S>+[.<m>...]
Stop writing the output after its duration reaches the time expressed by this value. It will only take effect when the script is run with the -t option.

qdur=<seconds>
Linux Read command option: (-t). Script default: 3 seconds.
Time between file encodings to press any key and exit or continue execution after confirmation. Disabled in test mode.

The second and third groups of options can be edited and both consist of complete sections on the FFmpeg command line run by the script. Both groups with the previous variables are combined to form, in simplified notation, the following command line:
$ comffmpeg iop verbo <input file> [tdur] {vfds0 | vfds1} vop [x265p] aop <output file>.oext

The second group defines FFmpeg command invocation, global, input and output options. By default, the script invokes the Linux Nice command with no options to run FFmpeg with a default niceness value of 10, i.e. a lower priority load on the system; the FFmpeg banner is suppressed, encoding progress and statistics are printed, and stream selection is performed automatically:
comffmpeg=(nice ffmpeg)
iop=(-hide_banner -stats -"$ow")
aop=(-c:a "$aenc")
vop=(-c:v "$venc" -preset "$vpreset" -crf "$vcrf")
x265p=(-x265-params log-level="$x265v")

The third group defines the FFmpeg filtergraph with (1) and without downscaling (0). The null filter passes the video source unchanged to the output and is therefore only used as a placeholder in the code for unscaled encodings:
vfds1=(-vf scale="${owidth}":-2)
vfds0=(-vf null)

Finally, a summary is displayed that reports information about input and output files, errors, and the time elapsed since the script was started:

Warnings and errors
File-indexed FFmpeg and x265 warnings and errors. Displayed only if there is output with log set to Warning or lower. The output is duplicated to stderr.

Input file names
Files passed to FFmpeg. Only displayed if the script (not FFmpeg) skips a file and/or is exited by the user.

Total input size
Total size of the set of files passed to FFmpeg.

Total output size
Total size of output files.
Known issue: If a file exists in the current directory that causes FFmpeg to "skip" an input file because the file names are the same, the size of that file will also be counted towards the sum of the output size.

Elapsed time
Time elapsed since the start of script execution.

#!/bin/bash
# PENC - batch transcoding in FFmpeg to save disk space.
# Version 0.1
# 2024
# SPDX-License-Identifier: Unlicense
# set the value of each of the following 13 variables according to your preferences.
# 0 (zero) or bitrate in kbit/s below which the video is not re-encoded:
ibitmin=0
# 0 (zero) or the width in pixels below which the video is not re-encoded:
iwidthmin=0
# 0 (zero) or the width in pixels to which the video is scaled down:
owidth=0
# overwrite files? (y/n):
ow=n
# output file extension:
oext=mp4
# audio copy or encoder:
aenc=aac
# video encoder (libx264/libx265):
venc=libx264
# video encoder preset (-preset):
vpreset=medium
# video encoder crf (-crf):
vcrf=23
# ffmpeg verbosity (-loglevel):
verbo=level+24
# libx265 verbosity (log-level):
x265v=1
# duration of video output for testing (-t):
tdur=3
# time in seconds to press any key and quit the script (read):
qdur=3
# set command invocation, global, input and output options.
# command invocation:
comffmpeg=(nice ffmpeg)
# global and input options:
iop=(-hide_banner -stats -"$ow")
# audio options:
aop=(-c:a "$aenc")
# video options:
vop=(-c:v "$venc" -preset "$vpreset" -crf "$vcrf")
# x265 parameters:
x265p=(-x265-params log-level="$x265v")
# set filtergraph.
# downscaling yes:
vfds1=(-vf scale="${owidth}":-2)
# no downscaling:
vfds0=(-vf null)
# this script file name
scp="$(basename "$0")"
if (($# == 0)) ; then
echo "Usage: ${scp} [-t] file|directory..."
echo "Configuration: Minimum input width (px): ${iwidthmin/#0/Disabled}, Minimum input bitrate (kb/s): ${ibitmin/#0/Disabled}, Maximum output width (px): ${owidth/#0/Disabled}, Overwrite files (Y/N): ${ow^}"
if ((owidth == 0)) ; then
if [ 'libx265' = "$venc" ] ; then
echo "Command line: ${comffmpeg[*]} ${iop[*]} -v $verbo -i <input file> [-t $tdur] ${vfds0[*]} ${vop[*]} ${x265p[*]} ${aop[*]} <output file>.$oext"
else
echo "Command line: ${comffmpeg[*]} ${iop[*]} -v $verbo -i <input file> [-t $tdur] ${vfds0[*]} ${vop[*]} ${aop[*]} <output file>.$oext"
fi
else
if [ 'libx265' = "$venc" ] ; then
echo "Command line: ${comffmpeg[*]} ${iop[*]} -v $verbo -i <input file> [-t $tdur] {${vfds0[*]} | ${vfds1[*]}} ${vop[*]} ${x265p[*]} ${aop[*]} <output file>.$oext"
else
echo "Command line: ${comffmpeg[*]} ${iop[*]} -v $verbo -i <input file> [-t $tdur] {${vfds0[*]} | ${vfds1[*]}} ${vop[*]} ${aop[*]} <output file>.$oext"
fi
fi
exit
else
if [ '-t' = "$1" ] ; then
nquit=yes
uop=(-t "$tdur")
shift
fi
fi
# dependencies
type awk >/dev/null 2>&1 || { echo >&2 "AWK not installed - aborting."; exit 1; }
type grep >/dev/null 2>&1 || { echo >&2 "GREP not installed - aborting."; exit 1; }
type ffprobe >/dev/null 2>&1 || { echo >&2 "FFPROBE not installed - aborting."; exit 1; }
type ffmpeg >/dev/null 2>&1 || { echo >&2 "FFMPEG not installed - aborting."; exit 1; }
# read bitrate tags
if type mediainfo >/dev/null 2>&1 ; then
mi=yes
fi
# time
gettime() {
date '+%X'
}
# input video specifications
getspec() {
readarray -t speca < <(ffprobe -v quiet -show_entries stream=codec_name,sample_rate,channels -select_streams a:0 -of default=nw=1 -i "$@") # audio
readarray -t specv < <(ffprobe -v quiet -sexagesimal -show_entries format=bit_rate,duration:stream=codec_name,width,height,r_frame_rate,bit_rate,duration -select_streams V -of default=nw=1 -i "$@") # video stream and container
if [ -v mi ] ; then
ibitb=$(mediainfo --inform="Video;%BitRate%" "$@" | grep -oE '[0-9]+') # video stream
if [ -z "$ibitb" ] ; then
ibitb=$(printf "%s\\n" "${specv[@]}" | grep -E '^bit_rate=' | grep -oE '[0-9]+' | sort -n | head -n 1) # video stream or container
if [ -z "$ibitb" ] ; then
ibitb=$(mediainfo --inform="General;%BitRate%" "$@" | grep -oE '[0-9]+') # container
fi
fi
else
ibitb=$(printf "%s\\n" "${specv[@]}" | grep -E '^bit_rate=' | grep -oE '[0-9]+' | sort -n | head -n 1) # video stream or container
fi
ibitk="$(awk 'BEGIN { printf ("%i", '"${ibitb:-0}"' / 1000 ) }')" # bitrate in kilobit
icodecv=$(printf "%s\\n" "${specv[@]}" | grep -E '^codec_name=' | head -n 1 | cut -d'=' -f2-) # video codec
iwidth=$(printf "%s\\n" "${specv[@]}" | grep -E '^width=' | grep -oE '[0-9]+' | sort -nr | head -n 1) # video resolution width
iheight=$(printf "%s\\n" "${specv[@]}" | grep -E '^height=' | grep -oE '[0-9]+' | sort -nr | head -n 1) # video resolution height
ifpsi=$(printf "%s\\n" "${specv[@]}" | grep -E '^r_frame_rate=' | head -n 1 | cut -d'=' -f2- | grep -oE '[0-9]+/[0-9]+') # video framerate
ifpso="$(awk 'BEGIN { printf ( '"${ifpsi:-0}"' ) }')" # video framerate calc
idur=$(printf "%s\\n" "${specv[@]}" | grep -E '^duration=' | grep -oE '[0-9]+:[0-9]{2}:[0-9]{2}' | sort -r | head -n 1) # longest duration
icodeca=$(printf "%s\\n" "${speca[@]}" | grep -E '^codec_name=' | head -n 1 | cut -d'=' -f2-) # audio codec
isample=$(printf "%s\\n" "${speca[@]}" | grep -E '^sample_rate=' | head -n 1 | cut -d'=' -f2- | grep -oE '[0-9]+') # audio sample rate
ich=$(printf "%s\\n" "${speca[@]}" | grep -E '^channels=' | head -n 1 | cut -d'=' -f2-) # audio channels
isz="$(du -hs "$@" | cut -f 1)" # file size
}
# print skipped file specs
encn() {
echo "${scp} ($(gettime)): Input file name - SKIPPING:" "${ivid@Q}"
echo "${scp} ($(gettime)): Size: ${isz}, Bitrate: ${ibitk}kb/s, Duration: ${idur:-0}, Video: ${icodecv:-N/A}, ${iwidth:-0}x${iheight:-0}, ${ifpso:-0}fps, Audio: ${icodeca:-N/A}, ${isample:-0}Hz, ${ich:-0}ch"
echo
} >&2
# input file name without extension
getfname() {
basename "${@%.*}"
}
# encode video
enc() {
echo "${scp} ($(gettime)): Input file name:" "${ivid@Q}"
echo "${scp} ($(gettime)): Size: ${isz}, Bitrate: ${ibitk}kb/s, Duration: ${idur:-0}, Video: ${icodecv:-N/A}, ${iwidth:-0}x${iheight:-0}, ${ifpso:-0}fps, Audio: ${icodeca:-N/A}, ${isample:-0}Hz, ${ich:-0}ch"
echo "${scp} ($(gettime)): Output options:" "$@" "${vop[*]} ${aop[*]}"
echo "${scp} ($(gettime)): Press [q] to stop, [?] for help - encoding..."
printf "\\t\\t%s\\n" "${oname}.$oext"
if echo "$verbo" | grep -Eq '.*warning$|.*error$|.*fatal$|.*panic$|.*24$|.*16$|.*8$|.*0' ; then # no error summary with highest verbosity levels
if echo "$verbo" | grep -Eq '.*48|.*40' ; then
if [ 'libx265' = "$venc" ] ; then
"${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${x265p[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" # ffmpeg command line
else
"${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" # ffmpeg command line
fi
else # use error summary with only warnings and errors
if [ '0' = "$x265v" ] || [ 'error' = "$x265v" ] || [ '1' = "$x265v" ] || [ 'warning' = "$x265v" ] || [ '-1' = "$x265v" ] || [ 'none' = "$x265v" ] ; then
if [ 'libx265' = "$venc" ] ; then
readarray -t errenc < <("${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${x265p[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" 2>&1 | tee /dev/tty | grep -vE '^frame=.*') # ffmpeg command line and reading output
else
readarray -t errenc < <("${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" 2>&1 | tee /dev/tty | grep -vE '^frame=.*') # ffmpeg command line and reading output
fi
else
if [ 'libx265' = "$venc" ] ; then
"${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${x265p[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" # ffmpeg command line
else
readarray -t errenc < <("${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" 2>&1 | tee /dev/tty | grep -vE '^frame=.*') # ffmpeg command line and reading output
fi
fi
fi
else # no error summary
if [ 'libx265' = "$venc" ] ; then
"${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${x265p[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" # ffmpeg command line
else
"${comffmpeg[@]}" "${iop[@]}" -v "$verbo" -i "$ivid" "$@" "${vop[@]}" "${aop[@]}" "$opath"/"$oname"."$oext" # ffmpeg command line
fi
fi
echo
}
# encoding stats
getstats() {
if [ -n "${errencf[0]}" ] ; then # input file with warning or error output
echo "${scp} ($(gettime)): Warnings and errors:"
for err in "${errencf[@]}" ; do # every file with a warning or error
iterin2=$((iterin2 + 1)) # count every file and error
printf "\\t\\t%s\\n" "${err@Q}" # the file name
readarray -t errm < <(printf "%s\\n" "$(tr '\t' '\n' <<< "${errencm[$iterin2]}")")
for errl in "${errm[@]}" ; do
printf "\\t\\t%s\\n" "$errl" # an error or warning message
done
echo
done
fi >&2
if [ -n "${skipf[0]}" ] || [ q = "$1" ] && [ -n "${ividall[0]}" ] ; then # an input file skipped or user quits
echo "${scp} ($(gettime)): Input file names:"
printf "\\t\\t%s\\n\\n" "${ividall[*]@Q}"
fi
if [ -n "${ovidall[0]}" ] ; then # an output file
echo "${scp} ($(gettime)): Total input size: $(du -hc "${ividall[@]}" | tail -qn1 | cut -f 1) ($(du -bc "${ividall[@]}" | tail -qn1 | cut -f 1) bytes)"
echo "${scp} ($(gettime)): Total output size: $(du -hc "${ovidall[@]}" | tail -qn1 | cut -f 1) ($(du -bc "${ovidall[@]}" | tail -qn1 | cut -f 1) bytes) ($(awk 'BEGIN { printf "%.1f", '"$(du -bc "${ovidall[@]}" | tail -qn1 | cut -f 1)"' / '"$(du -bc "${ividall[@]}" | tail -qn1 | cut -f 1)"' * 100 }')%)"
echo "${scp} ($(gettime)): Elapsed time: $(awk '{printf("%02d:%02d:%02d\n",($1/60/60%24),($1/60%60),($1%60))}' <<< "$SECONDS")"
fi
}
# exit the script
uquit() {
echo "${scp} ($(gettime)): Press any key to exit the script."
if read -rst "$qdur" -N 1 ; then
echo "${scp}: Please confirm:"
select ans in "EXIT the script" "Cancel" ; do
case "$ans" in
'EXIT the script' )
getstats q
echo "${scp} ($(gettime)): Exiting..."
exit;;
'Cancel' )
break;;
esac
done 2>&1
fi
}
for ivid in "$@" ; do # each input path
if [ -d "$ivid" ] ; then # input directory
shopt -qs globstar # recursive pathname expansion if an input directory
ivid1="$(realpath -q "$ivid")" # resolved input path
opath="$(pwd)" # resolved output path
for ivid2 in "${ivid1}"/** ; do # each input file and directory
if [ -f "$ivid2" ] ; then # input file
iterout=$((iterout + 1)) # count input files
ivid="$ivid2"
getspec "$ivid"
if ((iwidthmin <= iwidth)) ; then
if ((ibitmin == 0)) ; then
oname="$(getfname "$ivid")" # output file name without extension
if [ ! -v nquit ] && ((iterout > 1)) ; then
uquit
if [ "$?" -eq 1 ] ; then # skip the file
iterout=$((iterout - 1)) # count input files
continue
fi
fi
ividall+=("$ivid") # all input files
if ((owidth > 0 && owidth < iwidth)) ; then # scale down the video
enc "${uop[@]}" "${vfds1[@]}"
else
enc "${uop[@]}" "${vfds0[@]}"
fi
if [ -n "${errenc[0]}" ] ; then # there is a warning or error
iterin1=$((iterin1 + 1)) # count the warning and error output
errencf+=("$ivid") # all input files with warning or error
readarray -t -O "$iterin1" errencm <<< "$(printf "%s\\t" "${errenc[@]}")" # all warnings and error messages
fi
if [ -f "${oname}.$oext" ] ; then
ovidall+=("${oname}.$oext") # all output files
fi
elif ((ibitmin <= ibitk)) ; then # higher input bitrate
oname="$(getfname "$ivid")" # output file name without extension
if [ ! -v nquit ] && ((iterout > 1)) ; then
uquit
if [ "$?" -eq 1 ] ; then # skip the file
iterout=$((iterout - 1)) # count input files
continue
fi
fi
ividall+=("$ivid") # all input files
if ((owidth > 0 && owidth < iwidth)) ; then # scale down the video
enc "${uop[@]}" "${vfds1[@]}"
else
enc "${uop[@]}" "${vfds0[@]}"
fi
if [ -n "${errenc[0]}" ] ; then # there is a warning or error
iterin1=$((iterin1 + 1)) # count the warning and error output
errencf+=("$ivid") # all input files with warning or error
readarray -t -O "$iterin1" errencm <<< "$(printf "%s\\t" "${errenc[@]}")" # all warnings and error messages
fi
if [ -f "${oname}.$oext" ] ; then
ovidall+=("${oname}.$oext") # all output files
fi
else # lower input bitrate
iterout=$((iterout - 1)) # count input files
skipf+=("$ivid") # all input files skipped
encn # print specs
continue
fi
else
iterout=$((iterout - 1)) # count input files
skipf+=("$ivid") # all input files skipped
encn # print specs
continue
fi
else # not a regular input file
continue
fi
done
elif [ -f "$ivid" ] ; then
iterout=$((iterout + 1)) # count input files
ivid="$(realpath -q "$ivid")" # resolved input path
opath="$(pwd)" # resolved output path
getspec "$ivid"
if ((iwidthmin <= iwidth)) ; then
if ((ibitmin == 0)) ; then
oname="$(getfname "$ivid")" # output file name without extension
if [ ! -v nquit ] && ((iterout > 1)) ; then
uquit
if [ "$?" -eq 1 ] ; then # skip the file
iterout=$((iterout - 1)) # count input files
continue
fi
fi
ividall+=("$ivid") # all input files
if ((owidth > 0 && owidth < iwidth)) ; then # scale down the video
enc "${uop[@]}" "${vfds1[@]}"
else
enc "${uop[@]}" "${vfds0[@]}"
fi
if [ -n "${errenc[0]}" ] ; then # there is a warning or error
iterin1=$((iterin1 + 1)) # count the warning and error output
errencf+=("$ivid") # all input files with warning or error
readarray -t -O "$iterin1" errencm <<< "$(printf "%s\\t" "${errenc[@]}")" # all warnings and error messages
fi
if [ -f "${oname}.$oext" ] ; then
ovidall+=("${oname}.$oext") # all output files
fi
elif ((ibitmin <= ibitk)) ; then # higher input bitrate
oname="$(getfname "$ivid")" # output file name without extension
if [ ! -v nquit ] && ((iterout > 1)) ; then
uquit
if [ "$?" -eq 1 ] ; then # skip the file
iterout=$((iterout - 1)) # count input files
continue
fi
fi
ividall+=("$ivid") # all input files
if ((owidth > 0 && owidth < iwidth)) ; then # scale down the video
enc "${uop[@]}" "${vfds1[@]}"
else
enc "${uop[@]}" "${vfds0[@]}"
fi
if [ -n "${errenc[0]}" ] ; then # there is a warning or error
iterin1=$((iterin1 + 1)) # count the warning and error output
errencf+=("$ivid") # all input files with warning or error
readarray -t -O "$iterin1" errencm <<< "$(printf "%s\\t" "${errenc[@]}")" # all warnings and error messages
fi
if [ -f "${oname}.$oext" ] ; then
ovidall+=("${oname}.$oext") # all output files
fi
else # lower input bitrate
iterout=$((iterout - 1)) # count input files
skipf+=("$ivid") # all input files skipped
encn # print specs
continue
fi
else
iterout=$((iterout - 1)) # count input files
skipf+=("$ivid") # all input files skipped
encn # print specs
continue
fi
else # not an input file or directory
echo >&2 "${scp} ($(gettime)): Input error - SKIPPING: $ivid"
fi
done
getstats
echo "${scp} ($(gettime)): Exiting..."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment