Skip to content

Instantly share code, notes, and snippets.

@TheAMM
Last active July 8, 2024 14:26
Show Gist options
  • Save TheAMM/de48c152076fec4c0ba530ad09081f40 to your computer and use it in GitHub Desktop.
Save TheAMM/de48c152076fec4c0ba530ad09081f40 to your computer and use it in GitHub Desktop.
Twitter Media Source userscript
// ==UserScript==
// @name Twitter Media Source
// @namespace https://gist.github.com/TheAMM
// @downloadURL https://gist.github.com/TheAMM/de48c152076fec4c0ba530ad09081f40/raw/twitter_media_source.user.js
// @updateURL https://gist.github.com/TheAMM/de48c152076fec4c0ba530ad09081f40/raw/twitter_media_source.user.js
// @version 2.1.5
// @description Allows copying direct full-size image/video links on Twitter (with a #tweetid source suffix), downloading media, and navigating back to a Tweet from a media URL.
// @author AMM
// @match https://twitter.com/*
// @match https://x.com/*
// @match https://*.twimg.com/*
// @icon https://www.google.com/s2/favicons?sz=64&domain=twitter.com
// @grant none
// @inject-into auto
// ==/UserScript==
/*
== NOTE ==
This script functions by hooking the Twitter API calls, and requires "page" injection from your userscript manager.
ViolentMonkey is confirmed working on Firefox and Chrome, except in some mystery cases where it doesn't.
== INFORMATION ==
This userscript adds click listeners to media previews on Twitter.
You can SHIFT-CLICK and ALT-CLICK the small image or video previews visible on media tweets,
or the full-size image and video viewers when focusing a tweet (with /photo/1 or /video/1 in the URL).
SHIFT-CLICK to copy a direct media URL to your clipboard, such as:
https://pbs.twimg.com/media/Fswo6cLXwAA2khT.png:orig#1642732965254901761
https://pbs.twimg.com/media/FsyTAG2aEAA__CH.jpg:orig#1642849599730880513
https://video.twimg.com/ext_tw_video/1511313004780224512/pu/vid/1280x720/KEzdG4Bs6RIf_z9q.mp4#1511313503948525575
ALT-CLICK to download the media with a tidy filename, such as:
ricedeity - Fswo6cLXwAA2khT [Twitter-1642732965254901761].png
numenume_7 - FsyTAG2aEAA__CH [Twitter-1642849599730880513].jpg
namanoita175 - KEzdG4Bs6RIf_z9q [Twitter-1511313503948525575].mp4
When viewing a directly opened media file with a #tweetid suffix (for example, from the links above),
you may CTRL-CLICK the image or video to navigate back to the tweet, and ALT-CLICK to download the file
(however, this won't contain the username).
You may also press D to download or S to navigate to the source tweet, as for whatever reason Firefox doesn't
permit click listeners on the media elements.
== CHANGELOG ==
2024-07-08: 2.1.5
Fixed ads breaking the tweet ID matcher
2024-07-08: 2.1.4
Fixed window/unsafeWindow access for tampermonkey
2024-07-02: 2.1.3
Added toast notifications for actions, added DOM fallback when XHR hooking fails (photos and GIFs only).
2024-06-28: 2.0.3
Fix inverted tweet focus logic
2024-06-28: 2.0.2
Robusted media id extraction (fixed autoplay-disabled videos)
2024-06-27: 2.0.1
Included TweetWithVisibilityResults in XHR extractor
2024-06-27: 2.0.0
Overhauled the script, added video support.
Tweet data is now hooked from XHR calls instead of the DOM (which was only feasible for images).
The script keeps every seen media tweet in memory, which consumes memory, but it's not like Twitter isn't doing that.
2024-05-17: 1.4.4
xcom is here
2024-01-03: 1.4.3
Add keybinds to twimg.com media pages (direct image/video urls): t (lowercase) to go to tweet, s (lowercase) to save current image (video support tba)
2023-09-06: 1.4.2
Tread more carefully by just hiding the adverts?
2023-09-03: 1.4.1
Remove the verified/premium/whatever sidebar adverts
2023-07-31: 1.3
Deliberately fail Twitter's WebP check, so the sample format can be used to decude the original JPG/PNG
2023-03-09: 1.2
ALT-click on previews, full-sized overlay images or pbs.twimg.com images to download them with a useful filename,
like "ActualAMM - FaHQeO1VQAASEC4 [Twitter-1596999679115681792].jpg"
2023-02-19: 1.1
SHIFT-click a tweet image (preview or a full-sized overlay image) on Twitter to copy a #tweetid formatted link to it
2023-02-18: 1.0
Initial release
*/
(function() {
'use strict';
const log_label = '[TMS]';
const log_info = (...data) => console.log(log_label, ...data);
const log_warn = (...data) => console.warn(log_label, ...data);
const log_error = (...data) => console.error(log_label, ...data);
const log_debug = (...data) => console.debug(log_label, ...data);
// Override createElement (yeah, nice) to fail Twitter's WebP check (in main.js)
/*
const e = document.createElement("b");
e.innerHTML = "<object type=image/webp width=0><object type=image/webp data=data:i width=0>!</object>!</object>",
document.body.appendChild(e);
const webpSupported = !e.offsetWidth;
*/
const origCreateElement = document.createElement.bind(document);
document.createElement = (name) => {
if (name == "b") {
let elem = origCreateElement(name);
// Find the innerHTML property
let propertyDescriptor = null;
let proto = Object.getPrototypeOf(elem);
while (!propertyDescriptor && proto) {
propertyDescriptor = Object.getOwnPropertyDescriptor(proto, 'innerHTML');
proto = Object.getPrototypeOf(proto);
}
// Define a sabotaging setter
Object.defineProperty(elem, 'innerHTML', {
get: () => propertyDescriptor.get.call(elem),
set: function(value) {
if (value && value.startsWith("<object type=image/webp")) {
value = "!!";
}
propertyDescriptor.set.call(elem, value);
}
});
return elem;
}
return origCreateElement(name);
}
// Find tweets recursively from arrays and objects, and return an id-tweet map
const find_all_tweets = (value) => {
let tweet_map = new Map();
let queue = [value]
const clean_tweet = (tweet_result) => {
if (tweet_result.__typename == "TweetWithVisibilityResults") {
// For example, 1805976355559096692 from TweetDetail
tweet_result = tweet_result.tweet;
}
let cleaner_tweet = Object.assign(tweet_result.legacy, {});
cleaner_tweet.id_str = tweet_result.rest_id;
cleaner_tweet.source = tweet_result.source;
if (tweet_result.quoted_status_result) {
cleaner_tweet.quoted_status = clean_tweet(tweet_result.quoted_status_result.result);
}
// Yes the retweet is in legacy, quote is in result
if (cleaner_tweet.retweeted_status_result) {
cleaner_tweet.retweeted_status = clean_tweet(cleaner_tweet.retweeted_status_result.result);
delete cleaner_tweet.retweeted_status_result;
}
let user = ((tweet_result.core || {}).user_results || {}).result;
if (user) {
let cleaner_user = Object.assign(user.legacy, {});
cleaner_user.id_str = user.rest_id;
cleaner_user.is_blue_verified = user.is_blue_verified;
cleaner_tweet.user = cleaner_user;
}
return cleaner_tweet;
}
while (queue.length > 0) {
let item = queue.shift();
if (!item) {
continue;
} else if (item.__typename == "Tweet" || item.__typename == "TweetWithVisibilityResults") {
let cleaner_tweet = clean_tweet(item);
tweet_map.set(cleaner_tweet.id_str, cleaner_tweet);
if (cleaner_tweet.retweeted_status) {
tweet_map.set(cleaner_tweet.retweeted_status.id_str, cleaner_tweet.retweeted_status);
// Tweets quoting a tweet can be retweeted
if (cleaner_tweet.retweeted_status.quoted_status) {
tweet_map.set(cleaner_tweet.retweeted_status.quoted_status.id_str, cleaner_tweet.retweeted_status.quoted_status);
}
}
if (cleaner_tweet.quoted_status) {
tweet_map.set(cleaner_tweet.quoted_status.id_str, cleaner_tweet.quoted_status);
// Quoting quotes probably doesn't happen but play it safe
if (cleaner_tweet.quoted_status.quoted_status) {
tweet_map.set(cleaner_tweet.quoted_status.quoted_status.id_str, cleaner_tweet.quoted_status.quoted_status);
}
}
} else {
for (let child of Object.values(item)) {
if (Array.isArray(child) || typeof(child) === 'object') {
queue.push(child);
}
}
}
}
return tweet_map;
}
const urlsafe_atob = (base64) => atob(base64.replace(/_/g, '/').replace(/-/g, '+'));
const media_key_to_id = (key) => {
let uint8_array = Uint8Array.from(urlsafe_atob(key), c => c.charCodeAt(0));
return new DataView(uint8_array.buffer).getBigUint64(0, false).toString();
}
// Global store for all media tweets seen, as id-tweet
const TweetStore = new Map();
// Media id to media object map
const TweetMediaStore = new Map();
const XHRHookCounters = new Map();
// For debugging or manual shenanigans, you're welcome
let w = (typeof unsafeWindow == "undefined") ? window : unsafeWindow;
w.TweetStore = TweetStore;
w.TweetMediaStore = TweetMediaStore;
w.XHRHookCounters = XHRHookCounters;
const count_media = tweet => (((tweet.extended_entities || {}).media || []).length)
const XHR_readyStateHook = (xhr) => {
if (!xhr.responseURL) { return; }
let url = new URL(xhr.responseURL);
if (!url.host.match(/(.+?\.)?(twitter|x)\.com$/)) { return; }
// Check if response is supposed to be JSON
if (!(xhr.getResponseHeader("Content-Type") || "").startsWith("application/json")) {
return;
}
let data;
try { data = JSON.parse(xhr.response); } catch(e) {
log_warn("Failed parsing hooked JSON response:", e, xhr);
return;
}
XHRHookCounters.set(url.pathname, (XHRHookCounters.get(url.pathname) || 0) + 1);
// Find any media tweet objetcs
let all_tweets = find_all_tweets(data);
let media_tweets = Array.from(all_tweets.entries()).filter(([id, tweet]) => count_media(tweet) > 0);
if (media_tweets.length > 0) {
for (let [id, tweet] of media_tweets) {
TweetStore.set(id, tweet);
for (let media of tweet.extended_entities.media) {
media.tweet_id_str = id;
TweetMediaStore.set(media.id_str, media);
}
}
log_info(`Intercepted ${media_tweets.length} media tweets (out of ${all_tweets.size}) from ${url.pathname} (${TweetStore.size}T ${TweetMediaStore.size}M total held)`);
// log_debug(media_tweets);
}
};
// Hook the original send, adding our own callback
const XHR_send = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = function() {
let callback = this.onreadystatechange;
this.onreadystatechange = function() {
if (this.readyState == 4) {
try {
XHR_readyStateHook(this);
} catch(e) {
log_error("XHR hook failed:", e)
}
}
// Call original handler
if (callback) {
try {
callback.apply(this, arguments)
} catch(e) {
log_error('Original callback failed:', e);
throw e
}
}
}
return XHR_send.apply(this, arguments);
}
function killEvent(e) {
e.stopImmediatePropagation();
e.stopPropagation();
e.preventDefault();
return false;
}
const tweet_media_handler = (event) => {
let media_elem = event.currentTarget;
let media_url = media_elem.dataset.mediaUrl;
let tweet_id = media_elem.dataset.tweetId;
let media_filename = media_elem.dataset.filename;
let all_present = (tweet_id && media_url && media_filename);
if (event.shiftKey && !event.altKey && !event.ctrlKey) {
if (!all_present) {
show_notification("Can't copy link, missing data", 2000);
return killEvent(event);
}
let media_url_tweetid = `${media_url}#${tweet_id}`;
log_info("Copying", media_url_tweetid);
navigator.clipboard.writeText(media_url_tweetid);
show_notification(`Copied media link`, 1000);
return killEvent(event);
} else if (!event.shiftKey && event.altKey && !event.ctrlKey) {
if (!all_present) {
show_notification("Can't download media, missing data", 2000);
return killEvent(event);
}
download_file_from_url(media_filename, media_url);
show_notification(`Downloading <kbd>${media_filename}</kbd>`, 2000);
return killEvent(event);
}
}
const get_media_id_from_thumbnail = (image_url) => {
let media_key, media_id;
// Match for images
// https://pbs.twimg.com/media/Fw-DCLFaMAAb7mm?format=jpg&name=small
let image_match = image_url.match(/\/media\/([a-zA-Z0-9_\-]+)/);
media_key = image_match && image_match[1];
let media_type = 'photo';
// Match for GIF thumbnails
// https://pbs.twimg.com/tweet_video_thumb/FeZvQ8VakAAMlth.jpg
// https://pbs.twimg.com/tweet_video_thumb/GRFE0rgbkAAsDtw?format=jpg&name=small
if (!media_key) {
let key_match = image_url.match(/tweet_video_thumb\/([a-zA-Z0-9_\-]+)/);
media_key = key_match && key_match[1];
media_type = 'animated_gif'
}
if (media_key) {
media_id = media_key_to_id(media_key);
} else {
// Match for video thumbnails (the id is there directly, the "key" is per-variant)
// https://pbs.twimg.com/ext_tw_video_thumb/1578071056228970498/pu/img/GOFbqhnnjkS6FQYQ.jpg
// https://pbs.twimg.com/amplify_video_thumb/1661690860894027776/img/JS78i30E0XJ1tuST.jpg
let id_match = image_url.match(/_thumb\/(\d+)\//);
media_id = id_match ? id_match[1] : null;
media_type = 'video';
}
media_type = media_id ? media_type : null;
return {media_key, media_id, media_type};
}
let warnings = {};
function twitter_media_shenanigans() {
// Check if we are focused on a tweet, and grab the full-size viewer(s)
let tweet_match = document.location.pathname.match(/\/(.+?)\/status\/(\d+)/);
// An image or video is contained in each of these
let carousel_pages = tweet_match ? Array.from(document.querySelectorAll('#layers [data-testid="swipe-to-dismiss"]')) : [];
for (let elem of carousel_pages) {
if (elem._media_source_done) { continue; }
elem.dataset.username = tweet_match[1];
elem.dataset.tweetId = tweet_match[2];
}
// Find remaining tweet media on the entire page
let media_elements = Array.from(carousel_pages);
for (let tweet of document.querySelectorAll('[data-testid="tweet"]')) {
let tweet_media = tweet.querySelectorAll('[data-testid="tweetPhoto"]');
// We grab the tweet ID from the time-link (also, ads don't have a time-link)
let tweet_time = tweet.querySelector('a > time');
if (tweet_time && tweet_media.length > 0) {
let tweet_match = tweet_time.parentElement.pathname.match(/\/(.+?)\/status\/(\d+)/);
for (let elem of tweet_media) {
media_elements.push(elem);
if (elem._media_source_done) { continue; }
elem.dataset.username = tweet_match[1];
elem.dataset.tweetId = tweet_match[2];
}
}
}
for (let media_elem of media_elements) {
if (media_elem._media_source_done) { continue; }
let is_carousel = carousel_pages.includes(media_elem);
let media_id, media_key, media_type;
let video = media_elem.querySelector('[data-testid="videoPlayer"] video');
let image = media_elem.querySelector('img');
if (video) {
({media_id, media_key, media_type} = get_media_id_from_thumbnail(video.poster));
} else if (image) {
({media_id, media_key, media_type} = get_media_id_from_thumbnail(image.src));
} else {
// It's possible we have neither video or image while the page's loading, so check next mutation
continue;
}
media_elem._media_source_done = true;
if (!media_id) {
log_error("Failed to extract media ID from", video || image, "parent:", media_elem);
} else {
media_elem.dataset.mediaId = media_id;
let media = TweetMediaStore.get(media_id);
let media_url, media_name, media_ext;
if (media) {
if (media.type == 'photo') {
media_url = media.media_url_https + ':orig';
media_name = media.media_url_https.match(/\/media\/([a-zA-Z0-9_\-]+)/)[1];
media_ext = media.media_url_https.match(/\.(\w+)$/)[1];
} else {
// GIF and video
let mp4_variants = media.video_info.variants.filter(v => v.content_type == 'video/mp4');
let variant = mp4_variants.sort((a, b) => b.bitrate - a.bitrate)[0];
media_url = variant.url.split('?')[0];
// Pick the basename (which does not necessarily have the media key... but hysterical raisins)
[, media_name, media_ext] = media_url.match(/\/([a-zA-Z0-9_\-]+)\.(\w+)$/);
}
} else {
let warn_count = (warnings.missing_media || 0);
if (warn_count < 50) {
log_warn(`Missing media ${media_id} (${media_key}) for`, video || image);
warnings.missing_media = warn_count + 1;
if (warnings.missing_media >= 50) {
log_warn(`Ceasing further warnings about missing media.`)
}
}
if (XHRHookCounters.size == 0 && !warnings.no_intercept) {
warnings.no_intercept = true;
log_error("No API requests have been intercepted! Verify you're running with page injection mode");
show_notification("Unable to intercept API, DOM fallback only", 5000);
}
// Fallback handling
if (media_type == 'photo') {
// Check if we have an image, because stuff like 1808516050251890837
// have an amplify_video with a separate image (with its own media id) as a thumbnail
if (image) {
// Dragons: if we get webp thumbnails, this fucks up. For later, then!
media_ext = image.src.replace(/.+\//, '/').match(/(?:\.|format=)(\w+)/)[1];
media_url = `https://pbs.twimg.com/media/${media_key}.${media_ext}:orig`;
media_name = media_key;
}
} else if (media_type == 'animated_gif') {
media_ext = 'mp4';
media_url = `https://video.twimg.com/tweet_video/${media_key}.${media_ext}`;
media_name = media_key;
} else {
// video we can't do anything about
}
}
if (media_url) {
// For full-size image viewers, upgrade the image URL
if (is_carousel && image) { image.src = `${media_url}#${media_elem.dataset.tweetId}`; }
let media_filename = `${media_elem.dataset.username} - ${media_name} [Twitter-${media_elem.dataset.tweetId}].${media_ext}`;
media_elem.dataset.filename = media_filename;
media_elem.dataset.mediaUrl = media_url;
}
// Due to other useCapture listeners (I think), we can't properly kill the click event
// on videos, which results in unintended selections. Disable those.
media_elem.style.userSelect = 'none';
media_elem.addEventListener('click', tweet_media_handler, {capture:true});
}
}
// Find and hide the sidebar premium adverts (are these still relevant in 2024?)
// Right side
let premiumAside = Array.from(document.querySelectorAll('aside')).find(e => e.querySelector('a[href$=verified-choose]'));
if (premiumAside && !premiumAside.dataset.tmsHidden) {
premiumAside.parentNode.style.display = 'none';
premiumAside.dataset.tmsHidden = 'yes';
}
// Left side
let premiumNav = document.querySelector('nav > a[href$=verified-choose]')
if (premiumNav && !premiumNav.dataset.tmsHidden) {
premiumNav.style.display = 'none';
premiumNav.dataset.tmsHidden = 'yes';
}
}
function download_file_from_url(filename, media_url) {
log_info(`Downloading ${media_url} as "${filename}"`);
fetch(media_url).then(resp => {
if (resp.ok) { return resp.blob(); }
return Promise.reject(resp);
}).then(blob => download_blob(blob, filename));
}
function download_blob(blob, filename) {
if (typeof window.navigator.msSaveBlob !== 'undefined') {
window.navigator.msSaveBlob(blob, filename);
return;
}
const blobURL = window.URL.createObjectURL(blob);
const tempLink = document.createElement('a');
tempLink.style.display = 'none';
tempLink.href = blobURL;
tempLink.setAttribute('download', filename);
if (typeof tempLink.download === 'undefined') {
tempLink.setAttribute('target', '_blank');
}
document.body.appendChild(tempLink);
tempLink.click();
document.body.removeChild(tempLink);
setTimeout(() => { window.URL.revokeObjectURL(blobURL); }, 100);
}
function setup_cdn_listeners() {
let mediaElement = document.querySelector('img, video');
if (!mediaElement) { return; }
// Use tweet id from fragment
let fragment_match = document.location.hash && document.location.hash.match(/^#(\d+)$/);
let tweet_id = fragment_match[1] || null;
let goto_tweet = () => {
if (tweet_id) {
// TODO when this breaks, I guess
document.location = "https://twitter.com/i/status/" + tweet_id;
show_notification(`Navigating to tweet`, 1000);
}
}
let download_current_media = () => {
let twitter_source = tweet_id ? `Twitter-${tweet_id}` : 'Twitter';
let media_name, media_ext, media_url;
let image_match = document.location.pathname.match(/\/media\/([a-zA-Z0-9_\-]+?)(\.\w+)?(:.+)?$/);
let video_match = document.location.pathname.match(/\/([a-zA-Z0-9_\-]+?)(\.\w+)$/);
if (image_match) {
[, media_name, media_ext] = image_match;
if (!media_ext) {
// New-type URLs, with ?format=jpeg
media_ext = ('.'+ (new URLSearchParams(document.location.search).get('format')));
}
media_url = `https://${document.location.host}/media/${media_name}${media_ext}:orig`;
} else if (video_match) {
[, media_name, media_ext] = video_match;
media_url = document.location;
} else {
console.error("Unrecognized media url", document.location);
return;
}
let filename = `${media_name} [${twitter_source}]${media_ext}`;
download_file_from_url(filename, media_url);
show_notification(`Downloading <kbd>${filename}</kbd>`, 2000);
}
// Add mouse listeners to media element
mediaElement.addEventListener('click', e => {
if (e.ctrlKey && !e.altKey && !e.shiftKey) {
goto_tweet();
return killEvent(e);
} else if (!e.ctrlKey && e.altKey && !e.shiftKey) {
download_current_media();
return killEvent(e);
}
}, true);
// Firefox apparently doesn't do mouse events on video documents, so fallback keybinds it is
// d: download media
// s: go to source tweet
document.body.addEventListener('keypress', e => {
if (!e.ctrlKey && !e.shiftKey && !e.altKey) {
if (e.key == "s") {
goto_tweet();
return killEvent(e);
} else if (e.key == "d") {
download_current_media();
return killEvent(e);
}
}
})
}
const style_elem = document.createElement('style');
style_elem.innerText = `
.tms-notification-holder {
z-index:9001;
position: fixed;
left: 50%;
bottom: 5px;
transform: translate(-50%, 0);
margin: 0 auto;
}
.tms-notification {
width: fit-content;
padding: 4px 6px;
margin: 5px auto auto auto;
border-radius: 8px;
text-align: center;
font-family: TwitterChirp, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
font-size: 1.2em;
color: rgb(255,255,255);
background:rgb(20,70,100);
border: 1px solid rgb(90,90,90);
}
`;
document.body.appendChild(style_elem);
const notification_holder = document.createElement('div');
notification_holder.className = 'tms-notification-holder';
document.body.appendChild(notification_holder);
const show_notification = (content, timeout, confirm) => {
let elem = document.createElement('div');
elem.className = 'tms-notification';
elem.innerHTML = content;
while (notification_holder.childElementCount >= 3) {
notification_holder.removeChild(notification_holder.lastElementChild);
}
notification_holder.insertBefore(elem, notification_holder.firstElementChild);
if (timeout) {
setTimeout(() => {
try { elem.remove(); } catch(e) {};
}, timeout);
}
}
let last_failed = 0;
const safe_shenanigans = () => {
try {
twitter_media_shenanigans();
} catch (e) {
log_error("MutationObserver handler failed!");
console.error(e);
let now = Date.now();
if (now - last_failed > 10000) {
show_notification("MutationObserver handler failed, see console!", 5000);
last_failed = now;
}
}
}
// Check whether we're on Twitter, or on a CDN file
if (document.location.host.match(/^(.+?\.)?(twitter|x)\.com$/)) {
// Arguably inefficient but *you* go ahead and make a fine-tuned system for the obfuscated class mess and then have twitter break it
const observer = new MutationObserver(mutations => {
observer.disconnect()
observer.takeRecords()
safe_shenanigans();
observer.observe(document.body, { childList: true, subtree: true });
});
safe_shenanigans();
observer.observe(document.body, { childList: true, subtree: true });
} else if (document.location.host.match(/^(.+?\.)?twimg\.com$/)) {
setup_cdn_listeners();
}
})();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment