Examples here use the default settings, see the VidStab readme on GitHub for more advanced instructions.
Here's an example video I made
brew install ffmpeg --with-libvidstab
This won't modify your video or create a new video, but it should create a new file called transform.trf
ffmpeg -i clip.mov -vf vidstabdetect -f null -
ffmpeg -i clip.mov -vf vidstabtransform=smoothing=5:input="transforms.trf" clip-stabilized.mov
This should create a new stabilized video called clip-stabilized.mov
Found on a forum here
ffmpeg -i Clip8.mov -i Clip8-vidstab.mov -filter_complex "[0:v:0]pad=iw*2:ih[bg]; [bg][1:v:0]overlay=w" merged.mov
This (-f null -) occurs because when performing step 1 (vidstabdetect) we do not (yet) need the final video file, but only "transforms.trf", necessary for step 2 (vidstabtransform).
By the way: "-f" specifies the output format (null in this case) for ffmpeg.