-
-
Save aik099/69f221d100b87cb29f4fb6c29d72838e to your computer and use it in GitHub Desktop.
// 1. Open the browser developper console on the network tab | |
// 2. Start the video | |
// 3. In the dev tab, locate the load of the "master.json" file, copy its full URL | |
// 4. Run: node vimeo-downloader.js "<URL>" | |
// (done automatically now) 5. Combine the m4v and m4a files with mkvmerge | |
const fs = require('fs'); | |
const url = require('url'); | |
const https = require('https'); | |
const { exec } = require('child_process'); | |
let masterUrl = process.argv[2]; | |
if (!masterUrl.endsWith('?base64_init=1')) { | |
masterUrl+= '?base64_init=1'; | |
} | |
getJson(masterUrl, (err, json) => { | |
if (err) { | |
throw err; | |
} | |
const videoData = json.video.sort((v1,v2) => v1.avg_bitrate - v2.avg_bitrate).pop(); | |
const audioData = json.audio.sort((a1,a2) => a1.avg_bitrate - a2.avg_bitrate).pop(); | |
const videoBaseUrl = url.resolve(url.resolve(masterUrl, json.base_url), videoData.base_url); | |
const audioBaseUrl = url.resolve(url.resolve(masterUrl, json.base_url), audioData.base_url); | |
processFile('video', videoBaseUrl, videoData.init_segment, videoData.segments, json.clip_id + '.m4v', (err) => { | |
if (err) { | |
throw err; | |
} | |
processFile('audio', audioBaseUrl, audioData.init_segment, audioData.segments, json.clip_id + '.m4a', (err) => { | |
if (err) { | |
throw err; | |
} | |
console.log('combining video and audio...'); | |
let combineCmd = 'ffmpeg -i ' + json.clip_id + '.m4v -i ' + json.clip_id + '.m4a -c copy ' + json.clip_id + '.mp4'; | |
exec(combineCmd, (err, stdout, stderr) => { | |
if (err) { | |
throw err; | |
} | |
console.log(`stdout: ${stdout}`); | |
console.log(`stderr: ${stderr}`); | |
console.log('removing video and audio in favor of combined version...'); | |
fs.unlink(json.clip_id + '.m4v', (err) => { | |
if (err) { | |
throw err; | |
} | |
fs.unlink(json.clip_id + '.m4a', (err) => { | |
if (err) { | |
throw err; | |
} | |
console.log('all done'); | |
}); | |
}); | |
}); | |
}); | |
}); | |
}); | |
function processFile(type, baseUrl, initData, segments, filename, cb) { | |
if (fs.existsSync(filename)) { | |
console.log(`${type} already exists`); | |
return cb(); | |
} | |
const segmentsUrl = segments.map((seg) => baseUrl + seg.url); | |
const initBuffer = Buffer.from(initData, 'base64'); | |
fs.writeFileSync(filename, initBuffer); | |
const output = fs.createWriteStream(filename, {flags: 'a'}); | |
combineSegments(type, 0, segmentsUrl, output, (err) => { | |
if (err) { | |
return cb(err); | |
} | |
output.end(); | |
cb(); | |
}); | |
} | |
function combineSegments(type, i, segmentsUrl, output, cb) { | |
if (i >= segmentsUrl.length) { | |
console.log(`${type} done`); | |
return cb(); | |
} | |
console.log(`Download ${type} segment ${i}`); | |
https.get(segmentsUrl[i], (res) => { | |
res.on('data', (d) => output.write(d)); | |
res.on('end', () => combineSegments(type, i+1, segmentsUrl, output, cb)); | |
}).on('error', (e) => { | |
cb(e); | |
}); | |
} | |
function getJson(url, cb) { | |
let data = ''; | |
https.get(url, (res) => { | |
res.on('data', (d) => data+= d); | |
res.on('end', () => cb(null, JSON.parse(data))); | |
}).on('error', (e) => { | |
cb(e); | |
}); | |
} |
@aik099 - great script, thanks very much. I was trying to figure this out from scratch using PowerShell (the task of downloading and assembling the m4s's). I had to throw in the towel, then fumbled some with similar python scripts that I found. Came across this, and it was easy and slick. In case you or anyone else finds this helpful - my initial URL has 5 quality options included:
/video/0257ac23,4bc54da6,c13d51c2,b17a5b4c,c5dba5c1/master.json?base64_init=1
You can see them comma-separated. In my case, the 4th one is for 1080p, the 5th is for 720p. I figured that out by reviewing the initial request's response ({"clip_id":
with an array of id's in it, each for different quality option). I then looked using Edge DevTools at one of the .m4s request URLs and figured out how to manipulate my initial URL that I fed your script, in order to get the quality of my choice.
Here's a segment's request URL:
/video/b17a5b4c/chop/segment-2.m4s
so I just removed the other options from my initial URL when feeding it to your script, and boom it successfully grabbed the 1080p:
/video/b17a5b4c/master.json?base64_init=1 HTTP/1.1
Before that, when I was sending it exactly as it was, it would grab the 720p (the last value in the comma separated list):
/video/0257ac23,4bc54da6,c13d51c2,b17a5b4c,c5dba5c1/master.json?base64_init=1 HTTP/1.1
@JeremyTBradshaw, interesting to know, that something in URL actually denotes the quality assortment.
In fact, I've changed
const videoData = json.video.pop();
const audioData = json.audio.pop();
lines on top into
const videoData = json.video.sort((v1,v2) => v1.avg_bitrate - v2.avg_bitrate).pop();
const audioData = json.audio.sort((a1,a2) => a1.avg_bitrate - a2.avg_bitrate).pop();
as per recommendation in https://gist.github.com/mistic100/895f6d17b1e193334882a4c37d0d7748#gistcomment-3123045 comment on original gist of this file. That should get you the best quality at the end if I've understood the point of your research correctly. I've incorporated these changes in this gist now.
Anyway, this script is useful, when the Vimeo video is so private, that upon page reload you won't see it. If the video is public or shared with a link, then you can use YouTube-Dl script (see http://yt-dl.org/), that can download all kind of stuff.
@aik099 trying to get this to work... the script seems to start ok but fails when running combineCmd
with :
...
combining video and audio...
/home/joel/workspace/hackvimeo/script.js:75
throw err;
^
Error: Command failed: ffmpeg -i e53f1dff-bb7f-449c-8f67-4a96e2acc2a7.m4v -i e53f1dff-bb7f-449c-8f67-4a96e2acc2a7.m4a -c copy e53f1dff-bb7f-449c-8f67-4a96e2acc2a7.mp4
ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10 (GCC)
configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-liblensfun --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librav1e --enable-libsmbclient --enable-version3 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libglslang --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzvbi --enable-lv2 --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
[m4v @ 0x561fe70f4dc0] Format m4v detected only with low score of 1, misdetection possible!
[mpeg4 @ 0x561fe70f69c0] illegal chroma format
[mpeg4 @ 0x561fe70f69c0] Marker bit missing at 8341 of 1226943056 before time_increment_resolution
[mpeg4 @ 0x561fe70f69c0] framerate==0
[mpeg4 @ 0x561fe70f69c0] illegal chroma format
[mpeg4 @ 0x561fe70f69c0] Marker bit missing at 8341 of 1226935664 before time_increment_resolution
[mpeg4 @ 0x561fe70f69c0] framerate==0
[mpeg4 @ 0x561fe70f69c0] illegal chroma format
[mpeg4 @ 0x561fe70f69c0] Marker bit missing at 8341 of 1226943056 before time_increment_resolution
[mpeg4 @ 0x561fe70f69c0] framerate==0
[mpeg4 @ 0x561fe70f69c0] header damaged
[m4v @ 0x561fe70f4dc0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[m4v @ 0x561fe70f4dc0] decoding for stream 0 failed
[m4v @ 0x561fe70f4dc0] Could not find codec parameters for stream 0 (Video: mpeg4, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, m4v, from 'e53f1dff-bb7f-449c-8f67-4a96e2acc2a7.m4v':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: mpeg4, none, 25 tbr, 1200k tbn, 25 tbc
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x561fe710b740] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x561fe710b740] moov atom not found
e53f1dff-bb7f-449c-8f67-4a96e2acc2a7.m4a: Invalid data found when processing input
seems like ffmpeg needs more info? is there some extra flags to give ffmpeg that could help?
EDIT: adding more info. Looking further into the ffmpeg
error I saw someone suggested the data may be encrypted. I coincidentally noticed this on my console while streaming the video in question
Adaptive Video Streaming Service by www.bitmovin.com instrument.js:110:45
Player Version 8.55.0 instrument.js:110:45
EmeEncryptionSchemePolyfill: No native encryptionScheme support found. Patching encryptionScheme support. instrument.js:110:45
Some forums related to encrypted video in general suggested a key might be available to decrypt but this is where i hit a wall with my knowledge. Could there be a decription key somewhere we could use? or is this a blocker related to DRM stuff?
@keponk, would youtube-dl approach, explained in https://gist.github.com/mistic100/895f6d17b1e193334882a4c37d0d7748#gistcomment-3579182 work for you?
@aik099 yes that worked! thanks for the tip :)
I know this is an older gist but I'd like to say
a) you are a legend
b) this is so ****ing helpful
c) long live aik009
The difference from the original version is that
ffmpeg
is used automatically to combinevideo only
andaudio only
files and then they're removed in favor of a combinedmp4
file.