Skip to content

Instantly share code, notes, and snippets.

@Brainiarc7
Last active October 11, 2024 18:43
Show Gist options
  • Save Brainiarc7/18fca697891aea0e879f13ed092cb213 to your computer and use it in GitHub Desktop.
Save Brainiarc7/18fca697891aea0e879f13ed092cb213 to your computer and use it in GitHub Desktop.
Some snippets you can quickly adapt for use with FFmpeg and GNU Parallel for use for standard tasks.

Useful Examples of ffmpeg and GNU parallel on the command-line:

Transcoding FLAC music to Opus:

ffmpeg is a highly useful application for converting music and videos. However, audio transcoding is limited to a a single core. If you have a large FLAC archive and you wanted to compress it into the efficient Opus codec, it would take forever with the fastest processor to complete, unless you were to take advantage of all cores in your CPU.

parallel 'ffmpeg -v 0 -i "{}" -c:a libopus -b:a 128k "{.}.opus"' ::: $(find -type f -name '*.flac')

Transcoding Videos to VP9:

libvp9 has one glaring flaw in regards to encoding: it can only use about three cores at any given point in time. If you have an eight core processor and a dozen or more episodes of a TV series to transcode, you can use the parallel program to run several encode jobs on the host at the same time, provided you also have enough memory for that.

vp9_params="-c:v libvpx-vp9 -tile-columns 6 -frame-parallel 1 -rc_lookahead 25 -threads 4 -speed 1 -b:v 0 -crf 18"
opus_params="-c:a libopus -b:a 128k"
parallel -j 4 'ffmpeg -v 0 -i "{}" $vp9_params $opus_params -f webm "{.}.webm"' ::: $(find -type f -name '*.mkv')

Converting to H.264 with ffmpeg's VAAPI-based encoder:

Unlike NVENC, ffmpeg's VAAPI-based encoder is not limited to a maximum number of maximum simultaneous encodes based on the GPU's SKUs (As is the case at the present, consumer-grade NVIDIA GPUs can only do 2 simultaneous encodes whereas the Quadro and Tesla series have an unlimited number, dependent on available resources). In this case, you can take advantage of VAAPI as shown with GNU Parallel: (Note that you don't need to use -j$(nproc) here as its' the default if -j is undefined)

vaapi_params="-hwaccel vaapi -vaapi_device /dev/dri/renderD128 -c:v h264_vaapi -qp 19 -bf 2 -vf 'format=nv12,hwupload'"
opus_params="-c:a libopus -b:a 128k"
parallel -j 4 'ffmpeg -v 0 -i "{}" $vaapi_params $opus_params -f mkv "{.}.mkv"' ::: $(find -type f -name '*.mp4')

Converting to H.264 with FFmpeg's NVENC encoders:

The example below will encode all videos in the current directory from MKV to MP4 by launching two concurrent processes such that it will work on consumer-grade Nvidia GPUs that are limited to two simultaneous NVENC encode sessions only:

Note that the example below re-encodes a bunch of MKV files to MP4's with NPP scaling enabled and with high quality Dolby Digital surround sound with Dolby Pro-Logic II down-mixing enabled, with extensions for Dolby Pro-logic IIz and the Dolby headphone profile mode enabled. Adjust as needed. (My original files were encoded in DTS, with 5.1 channel mapping layout, and their audio tracks were massive, had to re-encode the audio stream to AC3 to cut down on the size).

parallel -j 2 --verbose 'ffmpeg -loglevel debug -i "{}" -c:a ac3 -b:a 448k -ac 6 -dsur_mode on -dmix_mode dplii -dsurex_mode dpliiz -dheadphone_mode on -copyright 1 \
  -filter:v hwupload_cuda,scale_npp=w=1920:h=800:format=nv12:interp_algo=lanczos,hwdownload,format=nv12 \
  -c:v h264_nvenc -preset:v slow -profile:v high -b:v 2250k -rc:v vbr_2pass -rc-lookahead:v 40 -surfaces 40 -bf 4 -refs 4 \
  -f mp4 "{.}.mp4"' ::: $(find -type f -name '*.mkv')

What of two-pass encodes based on the example above?

parallel -j 2 --verbose 'ffmpeg -loglevel debug -threads 4 -i "{}" -pass 1 -c:a ac3 -b:a 448k -ac 6 -dsur_mode on -dmix_mode dplii -dsurex_mode dpliiz -dheadphone_mode on -copyright 1 \
      -c:v h264_nvenc -preset:v slow -profile:v high -b:v 2250k -2pass 1 -rc:v vbr_2pass -rc-lookahead:v 40 -surfaces 40 -bf 4 -refs 4 \
      -f mp4 -y "/dev/null" && ffmpeg -loglevel debug -threads 4 -i "{}" -pass 2 -c:a ac3 -b:a 448k -ac 6 -dsur_mode on -dmix_mode dplii -dsurex_mode dpliiz -dheadphone_mode on -copyright 1 \
      -c:v h264_nvenc -preset:v slow -profile:v high -b:v 2250k -2pass 1 -rc:v vbr_2pass -rc-lookahead:v 40 -surfaces 40 -bf 4 -refs 4 -f mp4 -y "{.}.mp4"' ::: $(find -type f -name '*.avi')

The example above re-encodes a bunch of AVI files to MP4's with two-pass encode mode and with high quality Dolby Digital surround sound with Dolby Pro-Logic II down-mixing enabled, with extensions for Dolby Pro-logic IIz and the Dolby headphone profile mode enabled. Adjust as needed. (Original files were encoded in DTS, with 5.1 channel mapping layout).

Adjust the parallel -j value to a higher number if you own a Quadro or Tesla GPU, as these SKUs don't have the imposed limit mentioned above.

@Brainiarc7
Copy link
Author

Thanks for the insight, @ole-tange.

Is it OK if I reach out to you for more on GNU Parallel? I've found it to be incredibly useful, but daunting to use.

@l29ah
Copy link

l29ah commented Dec 25, 2020

Thanks to Ole for figuring the problem with the snippet, removing "s fixed it.

@Lukas-52
Copy link

I am currently running an encode using a adjusted version of your snippet for parallel 2 pass encoding, but I'm seeing very strange behavior.
Instead of running 3 Workers that work on one command until its done it seems like all 12 are running and just getting put to sleep and woken up again in short succession (at least that's what htop is telling me). This is also backed up by a massive increase in both RAM usage as well as disk IO compared to running three enoders in parallel the manual way (by using screen and commands containing the individual filenames).
This is the exact command i use:
parallel -j 3 'ffmpeg -i "{}" -c:v libaom-av1 -pix_fmt yuv420p -cpu-used 1 -b:v 600k -tune psnr -an -pass 1 -passlogfile "{}.log" -f webm -y "/dev/null" && ffmpeg -i "{}" -c:v libaom-av1 -pix_fmt yuv420p -cpu-used 1 -b:v 600k -tune psnr -an -pass 2 -passlogfile "{}.log" "{.}.webm"' ::: $(find -type f -name '*.avi')

Is this expected behavior or am i missing something? From what i understand parallel shouldn't run all possible workers at once and just altering between sleep and active process states right?

@arizvisa
Copy link

ole-tange, being the motherfuckin' man as usual.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment