How to properly record Youtube & Twitch live streams

howto_record_youtube_livestreams.md

How to record Youtube live streams:

• use livestream_saver to download from the first segment. Can also record membership-only streams by supplying it your cookies (uses yt-dlp to download)

• use ytarchive which basically does the same thing, except a bit better.

• use youtube_stream_capture. You can use cookies file to get member-only streams too. Be aware that this script currently fails to download chunks as soon as the stream has ended (this might be a bug).

• or use live-dl which does monitoring of streams too. This is a wrapper around streamlink and yt-dlp.

Download chat messages

If you know a program that downloads chat messages while the stream is live, please leave a comment.

Currently, the only reliable way of downloading chat messages is to use yt-dlp with the following arguments to generate a JSON file. The JSON file can be compressed afterward. For example: bzip2 *.json.

yt-dlp --skip-download --write-subs --sub-langs "live_chat" -o "%(upload_date)s [%(uploader)s] %(title)s [%(id)s].%(ext)s" <video_id>

Known downside: this only works after the video has become a VOD, it won't work during live stream.

This is an example script that does this automatically: glubsy/ytdl_batch/subs.py

How to record Twitch live streams:

• streamlink, to download from the segment being broadcast currently
• yt-dlp which downloads from the first segment BUT stops downloading once it reaches the currently broadcast segment (or maybe there is a way to avoid this, in which case, leave a comment).

A reliable way to get the best quality and not miss any data currently is this:

• (RECOMMENDED) run streamlink in a loop with save_livestream.py script to download [x% ----> 100%]. Setting the retry delay to about 1 minute reduces the chance of missing a big chunk of the beginning of a stream, so this solution should be enough for now.

• run yt-dlp to download from the beginning but lose the rest [0% ----> x%]. A script to handle this better would be nice.

• Note: better download the best video quality possible for two reasons. 1) Audio quality depends on video quality ("progressive" streams) 2) Lower resolutions on Twitch are very poorly compressed, so it is better to compress ourselves.

• Streams downloaded by streamlink-based solutions are in TS format which may cause issues with some players or when uploading to VOD services. You might want to convert them to mp4 (or whatever container you prefer) with ffmpeg like so: ffmpeg -i twitch_live.ts -c copy twitch_live.mp4

• If you get warnings from streamlink saying Encountered a stream discontinuity. This is unsupported and will result in incoherent output data, the resulting .ts file will be corrupted. This is because of ads served by Twitch. The only way to avoid this is to pay the streamer for a subscription, then pass the following argument to streamlink: --twitch-api-header 'Authorization=OAuth YOUR_TOKEN'. You can get the header from your web browser's devtool while logged into Twitch. You can pass this argument to the save_livestream.py script by calling it like this:

python3 livestream_scripts/save_livestream.py --author-name 'nanobites' https://www.twitch.tv/nanobites_ "--twitch-api-header 'Authorization=OAuth YOUR_TOKEN'"

Download chat messages

If you know a program that downloads chat messages while the stream is live, please leave a comment.

Once a stream has become a VOD, you may use TwitchDownloader. You can embed the emoticons inside the resulting chat log for archival purposes with the -E or --embed-images parameter. For example:

TwitchDownloader chatdownload --id $video_Id -E -o $video_Id.json

This is an example script that does this automatically: glubsy/ytdl_batch/subs.py

Video tracks concatenation

In order to concatenate the videos (and audio) you will most likely need to fix the container files.

Example case: we have two recordings of the same live stream, one partial from the start, and one full but with muted audio segments. Exactly same audio and video codecs (1080p x264/aac). First, split the full video file at the (shared) junction timestamp.

# ffmpeg -i full.mp4 -ss 1:06:36.0 -c copy full_split.mp4

To avoid the "Non-monotonous DTS in output stream", we have to fix the containers by recreating them (do it for all of them just in case), use "-fflags +igndts" to regenerate DTS based on PTS:

# ffmpeg -i full_split.mp4 -fflags +igndts -c copy -movflags +faststart full_split_dts_fix.mp4
# ffmpeg -i start.mp4 -fflags +igndts -c copy -movflags +faststart start_dts_fix.mp4

Then concatenate with the concat demuxer as normal (create list.txt with files to concatenate)

# ffmpeg -f concat -safe 0 -i list.txt -c copy -movflags +faststart final_full.mp4

Then compress the video, but keep audio (and reduce fps from 60 to 30 while we are at it):

# ffmpeg -i final_full -c:v libx264 -crf 30 -vf scale=iw/3:ih/3 -c:a copy -r 30 final_compressed.mp4

Cookies extraction

To extract from Firefox you may use this: https://github.com/rotemdan/ExportCookies (add-on page)

To extract from Chromium-based browsers: https://github.com/dandv/convert-chrome-cookies-to-netscape-format

If you use streamlink, you may want to avoid passing cookies as arguments, in which case you can create the following file ~/.config/streamlink/config.twitch:

twitch-api-header=Authorization=OAuth <auth_token value>

This is especially useful if you use the save_livestream.py script mentioned above.

As of 2023/07/20, the /live tab in Youtube is redirecting to /streams, which unfortunately breaks the following yt-dlp based solution:

while true; do yt-dlp --fragment-retries 50 -o '%(upload_date)s [%(uploader)s] %(title)s [%(height)s][%(id)s].%(ext)s' -ciw -f 'bestvideo+bestaudio' --add-metadata --embed-thumbnail --live-from-start --match-filter 'is_live' 'https://www.youtube.com/@inukaipuwin/live'; sleep $((10*60)); done

Great tutorial and script, but I have a suggestion. Add live chat downloading from twitch it would be perfect then.

@xEska1337 added sections on how to download chat messages.

I don't know of any program that downloads chat messages while the stream is broadcasting live, but I'm sure this can be done. This has been on my TODO list for a while now.

You can check this project https://github.com/MrBrax/LiveStreamDVR it can download chat live but it has big downside it requires opening your server to the internet

On the topic of chat download, https://github.com/xenova/chat-downloader works but it needs some work. As of today, it does not segment chat logs and does not download emoticons.

Do you have any recommendations for monitoring Twitch channels to start recording when they go live?

@Qwerty-Space as mentioned in the guide, I recommend https://github.com/mrwnwttk/livestream_scripts which is just a basic script that starts Streamlink every 40 seconds.
Although you could achieve the same thing with a systemd unit I suppose.

@glubsy Yeah I just wrote a little bash script and a systemd unit to run every minute. Requires yt-dlp.

#!/bin/bash

# Function to check if a Twitch channel is live
function is_live() {
  local channel_name=$1 # Store the channel name passed as an argument
  # store the channel link
  local channel_link="https://www.twitch.tv/$channel_name"
  # Fetch the HTML content of the Twitch channel page using curl
  local content=$(curl -s $channel_link)
  # Extract the title
  local description=$(echo "$content" | sed -E 's/.*"description":"([^"]+)".*/\1/')
  # Convert Unicode escape sequences to UTF-8 (e.g., \u003e -> >)
  local utf8_description=$(echo -e "$description" | sed 's/\\u\([0-9a-fA-F]\{4\}\)/\\x\1/g' | while read -r line; do echo -e "$line"; done)
  # Convert to a safe filename
  local filename=$(echo "$utf8_description" | tr -cd '[:alnum:][:space:]._-')

  # Check if the content contains the string "isLiveBroadcast"
  if [[ $content == *"isLiveBroadcast"* ]]; then
    echo "$channel_name is live" # Print a message if the channel is live
    echo "Steam Title:  $utf8_description"
    # Download the stream, converting the video to 30 FPS with a bitrate of 2500K
    yt-dlp -o "%(date)s $filename.%(ext)s" --postprocessor-args "-vf fps=30 -b:v 2500k" $channel_link
  else
    echo "$channel_name is not live :(" # Print a message if the channel is not live
  fi
}

# Check if the user provided a channel name as an argument
if [[ -z $1 ]]; then
  echo "Pass a channel name as an argument" # Prompt the user to provide an argument
else
  # Call the is_live function with the channel name in lowercase
  is_live "${1,,}" # The "${1,,}" syntax converts the input to lowercase
fi