This is a collection of basic "recipes", many using twurl (the Swiss Army Knife for the Twitter API!) and jq to query the Twitter API and format the results. Also, some scripts to test or automate common actions.
-
-
Save andypiper/32bdb4c7f0b8d65385fc7c8fa3988083 to your computer and use it in GitHub Desktop.
twurl "/1.1/users/show.json?screen_name=andypiper" | jq '. | {id: .id_str, name: .name, screen_name: .screen_name, bio: .description, location: .location}'
{
"id": "786491",
"name": "andy piper (pipes)",
"screen_name": "andypiper",
"bio": "I'm the lead @TwitterDev @twitterapi platform team - find the best help via https://t.co/T2vkQoJ64f. Code, community, & respect. ⌨️ 🌈 🙇 #HeForShe 🏳️🌈",
"location": "Kingston upon Thames, London"
}
#!/bin/sh | |
# | |
# get an app-only auth token (Bearer Token) | |
# pass in consumer key and consumer secret as params | |
# requires: httpie and jq | |
while [ $# -ne 2 ] | |
do | |
echo "usage: bearer.sh <consumer key> <consumer secret>" | |
exit 1 | |
done | |
TOKEN=$(http --auth "$1:$2" --form POST 'https://api.twitter.com/oauth2/token' 'grant_type=client_credentials' -b | jq .access_token | tr -d \") | |
echo "Bearer $TOKEN" | |
exit 0 |
twurl -t -A "Content-Type: application/json" -d '{"id": "custom-543399461108469761","changes": [{ "op": "add", "tweet_id": "390897780949925889"},{ "op": "add", "tweet_id": "390853164611555329"}]}' "/1.1/collections/entries/curate.json"
Content-Type
will be ignored -> fail)
Get a Tweet with (poll data expansion) from the Twitter Developer Labs Tweets endpoint (supports multiple IDs)
twurl -j "/labs/2/tweets?ids=1155833627743272960&expansions=author_id,attachments.poll_ids"
{
"data": [
{
"attachments": {
"poll_ids": [
"1155833626535387136"
]
},
"author_id": "786491",
"id": "1155833627743272960",
"text": "If you're a developer using the Twitter API, what is you favourite tool to use to test endpoints? 🛠💻"
}
],
"includes": {
"polls": [
{
"id": "1155833626535387136",
"options": [
{
"position": 1,
"label": "twurl",
"votes": 11
},
{
"position": 2,
"label": "Postman",
"votes": 24
},
{
"position": 3,
"label": "Insomnia",
"votes": 9
},
{
"position": 4,
"label": "I roll my own!",
"votes": 10
}
]
}
],
"users": [
{
"id": "786491",
"name": "andypiper",
"username": "andypiper"
}
]
}
}
Get Tweet metrics for a Tweet that is owned by your auth'ed user (supports multiple IDs) Note that organic and non-public metrics are not available outside of a 30-day window from Tweet creation.
twurl -j "/labs/2/tweets/1266695547534168064?tweet.fields=non_public_metrics,organic_metrics,public_metrics"
{
"data": {
"id": "1266695547534168064",
"non_public_metrics": {
"impression_count": 1816
},
"organic_metrics": {
"impression_count": 1816,
"like_count": 11,
"reply_count": 3,
"retweet_count": 0
},
"public_metrics": {
"retweet_count": 0,
"reply_count": 3,
"like_count": 11,
"quote_count": 0
},
"text": "Ready to go @SummertonClub #SVWF 🥃 https://t.co/ztIqxZttCq"
}
}
The Engagement API is a commercial API that enables access to data about Tweet engagement (impressions, etc). Your application key can be granted permission for these endpoints if you've purchased access to the API. Contact Twitter here to discuss access.
twurl -H data-api.twitter.com -A "Content-Type: application/json" -X POST "/insights/engagement/totals" -d '{"tweet_ids": ["908115328856621056","908279258480594944","908296740691947520"],"engagement_types":["impressions","engagements","favorites"],"groupings": {"grouping name": {"group_by": ["tweet.id","engagement.type"]}}}'
{
"grouping name": {
"908115328856621056": {
"engagements": "16",
"favorites": "5",
"impressions": "556"
},
"908279258480594944": {
"engagements": "25",
"favorites": "4",
"impressions": "278"
},
"908296740691947520": {
"engagements": "3",
"favorites": "1",
"impressions": "48"
}
}
}
#!/bin/sh | |
# count members in Twitter list | |
twurl "/1.1/lists/show.json?slug=$1&owner_screen_name=andypiper" | jq '. | {count: .member_count}' |
#!/bin/sh | |
# remove from Twitter list | |
twurl -q -d "screen_name=$2&slug=$1&owner_screen_name=andypiper" "/1.1/lists/members/destroy.json" |
Fetch a v2 user timeline, with image attachments
twurl -j "/2/users/2244994945/tweets?max_results=100&expansions=attachments.media_keys&media.fields=url,media_key"
- match up the media_keys with the expansions to match image URLs to Tweets.
- currently, only images are supported for URL fields (videos and animated GIFs not yet available)
#!/bin/sh | |
# | |
# NB I prefer bearer.sh as it checks for number of params and outputs the right syntax for use in a header | |
# | |
# get an app-only auth token (Bearer Token) | |
# pass in consumer key and consumer secret as params | |
# expected JSON response of form {"access_token":"CCCCCCCC","token_type":"bearer"} | |
# | |
# remove | python -m json.tool if no Python installed (or, no need to prettify results) | |
# | |
curl -s -S -u "$1:$2" --data 'grant_type=client_credentials' 'https://api.twitter.com/oauth2/token' | python -m json.tool |
twurl -H publish.twitter.com "/oembed?url=https://twitter.com/andypiper&limit=5"
{
"url": "https://twitter.com/andypiper",
"title": "",
"html": "<a class=\"twitter-timeline\" data-tweet-limit=\"5\" href=\"https://twitter.com/andypiper\">Tweets by andypiper</a>\n<script async src=\"//platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>",
"width": null,
"height": null,
"type": "rich",
"cache_age": "3153600000",
"provider_name": "Twitter",
"provider_url": "https://twitter.com",
"version": "1.0"
}
Handy shell alias for making twurl
output more readable...
jsonator () {
ruby -rubygems -r pp -e 'require "json"; ARGF.each {|l| puts JSON.pretty_generate(JSON.parse(l))}'
}
jtwurl () {
twurl $@ | jsonator
}
(NB no longer needed with twurl
0.9.5 which has -j
option)
Two stages: create a welcome message, and assign it to the default rule.
twurl -A 'Content-type: application/json' /1.1/direct_messages/welcome_messages/new.json -d '{"name": "evil-welcome", "welcome_message": {"message_data": {"text": "Welcome! I am the evil version of @andypiper"}}}'
(take note of the id
value in the response to use in the follow-up call)
twurl -A 'Content-type: application/json' -X POST /1.1/direct_messages/welcome_messages/rules/new.json -d '{"welcome_message_rule": {"welcome_message_id": "966308520869187588"}}'
twurl "/1.1/statuses/user_timeline.json?count=5" | jq '[.[] | { text: .text, source: .source, time: .created_at}]'
[
{
"text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque interdum rutrum sodales. Nullam mattis fermen… https://t.co/kb5SVo14NR",
"source": "<a href=\"http://wibble.org\" rel=\"nofollow\">simpletester123</a>",
"time": "Wed Jul 26 10:12:53 +0000 2017"
},
{
"text": "test https://t.co/76eZnhqKj4",
"source": "<a href=\"http://wibble.org\" rel=\"nofollow\">simpletester123</a>",
"time": "Wed Jul 26 10:11:44 +0000 2017"
},
{
"text": "Well done @LastWeekTonight 👏🏻 https://t.co/DyHKpUKGy1",
"source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter for iPhone</a>",
"time": "Mon Jul 24 19:19:16 +0000 2017"
},
{
"text": "Test https://t.co/wBZ9lp9kCR",
"source": "<a href=\"https://fabric.io\" rel=\"nofollow\">PhotoboothDemoiOS</a>",
"time": "Fri Jun 30 12:25:50 +0000 2017"
},
{
"text": "just a test\nlook at this\n😎 https://t.co/gYEcopskOR",
"source": "<a href=\"https://fabric.io\" rel=\"nofollow\">PhotoboothDemoiOS</a>",
"time": "Thu Jun 29 17:26:21 +0000 2017"
}
]
Summarise a list of people:
twurl "/1.1/lists/members.json?slug=First-50&owner_screen_name=andypiper&count=5000&skip_status=1&include_entities=false" | jq '[.users[] | {id: .id_str, name: .name, screen_name: .screen_name}]' > peeps.json
Summarise a set of lists a user is a member of:
twurl "/1.1/lists/memberships.json?screen_name=andypiper&count=650" | jq '[.lists[] | {name: .name, description: .description, uri: .uri, people: .member_count, subscribers: .subscriber_count}]'
Get list of lists owned by a user
twurl "/1.1/lists/list.json?screen_name=andypiper" | jq '.[] | {slug: .slug, id: .id}'
#!/bin/bash | |
# convert UNIX epoch to timestamp (useful for rate limit resets) | |
perl -e "print scalar(localtime($1)), \"\n\"" |
http POST api.twitter.com/1.1/tweets/search/fullarchive/dev/counts.json 'authorization: Bearer YOUR_BEARER_TOKEN' query="#cheerlights" fromDate=202005020000 toDate=202006020000 bucket=day | jq '."totalCount"'
-> 921
Searches for the day counts for the string "#cheerlights" from May to June 2020 and prints the total (from the premium Full Archive Search endpoint)
NB the coderay pipe is optional, for syntax coloring
twurl -j "/labs/2/tweets?ids=1240791119560138752&tweet.fields=entities,context_annotations" | coderay -json
{
"data": [
{
"context_annotations": [
{
"domain": {
"id": "71",
"name": "Video Game",
"description": "A video game like Overwatch"
},
"entity": {
"id": "10045326731",
"name": "DOOM",
"description": "This entity includes all conversation about the franchise, as well as any individual installments in the series, if applicable.\t\t\t"
}
},
{
"domain": {
"id": "71",
"name": "Video Game",
"description": "A video game like Overwatch"
},
"entity": {
"id": "1004434935690612736",
"name": "DOOM Eternal",
"description": "DOOM Eternal"
}
}
],
"entities": {
"annotations": [
{
"start": 37,
"end": 48,
"probability": 0.4457,
"type": "Product",
"normalized_text": "DOOM Eternal"
}
]
},
"id": "1240791119560138752",
"text": "Wow, 2 minutes past midnight and the DOOM Eternal install pop ups, pop up. Impressive!"
}
]
}
twurl "/1.1/users/search.json?q=andypiper" | jq '[.[] | {id: .id_str, name: .name, screen_name: .screen_name}]'
[
{
"id": "786491",
"name": "Pipes",
"screen_name": "andypiper"
},
{
"id": "77825609",
"name": "Andy Piper",
"screen_name": "andypiper8"
},
{
"id": "455565439",
"name": "Andrew St",
"screen_name": "Andypip14"
},
{
"id": "332796322",
"name": "Andy Pipe",
"screen_name": "andypipe38"
},
{
"id": "424300699",
"name": "Felipe Basto",
"screen_name": "Andypipe98Basto"
},
{
"id": "2894986614",
"name": "Andy Piper",
"screen_name": "AndyAndypiper"
},
{
"id": "244145232",
"name": "andres felipe",
"screen_name": "andypipe7"
},
{
"id": "427650744",
"name": "andy pipe",
"screen_name": "andypipe3"
},
{
"id": "295296501",
"name": "ANDRES FELIPE BOTINA",
"screen_name": "andypipe2011"
},
{
"id": "3119979580",
"name": "andres cardenas",
"screen_name": "andypipe1207"
},
{
"id": "175910590",
"name": "Anderson Felipe",
"screen_name": "andypipe_"
},
{
"id": "2215376413",
"name": "andy.pip",
"screen_name": "andypip1"
},
{
"id": "156437723",
"name": "andres felipe pinzon",
"screen_name": "andypipe29"
},
{
"id": "4774876672",
"name": "andy",
"screen_name": "andypip69"
},
{
"id": "268190008",
"name": "Philip Piper",
"screen_name": "andypiper3971"
},
{
"id": "915079999",
"name": "Andy Piper",
"screen_name": "andypiper1985"
},
{
"id": "1388227916",
"name": "andres felipe garzon",
"screen_name": "andypipe22"
},
{
"id": "1570721150",
"name": "AndyPipe",
"screen_name": "daniandi2002"
},
{
"id": "180432443",
"name": "Andres Felipe",
"screen_name": "andypipe206"
},
{
"id": "236309993",
"name": "AndyPipes",
"screen_name": "xcellxx"
}
]
// use the twitter-text library to parse and validate a Tweet string | |
// updated for twitter-text 2.0 new semantics | |
// npm install twitter-text | |
// node ./validate-tweet.js | |
var twitter = require('twitter-text'); | |
var tweet = "123 @andypiper 123456 @mauropiano #winning $TWTR 8901789zz #gnip 0123456789... http://blog.rust-lang.org/2016/05/26/Rust-1.9.html more at http://localhost:3000/notes/595 @juandoming Dallas teachers improved student performance by 20% w/mobile video messaging. Would this be useful to you? stars.now wibble wobble testing #pipes"; | |
//var tweet = "@juandoming Dallas teachers improved student performance by 20% w/mobile video messaging. Would this be useful to you? stars.now" | |
//var tweet = "testing testing #blog https://andypiper.co.uk" | |
var parsed = twitter.parseTweet(tweet); | |
var chars = parsed.weightedLength; | |
var valid = parsed.valid; | |
var usernames = twitter.extractMentions(tweet); | |
var urls = twitter.extractUrls(tweet); | |
var hash = twitter.extractHashtags(tweet); | |
var cash = twitter.extractCashtags(tweet); | |
console.log("text: " + tweet + "\n"); | |
console.log("validity: " + valid + "\n"); | |
console.log("length: " + chars); | |
console.log("users: " + usernames); | |
console.log("links: " + urls); | |
console.log("hashtags: " + hash); | |
console.log("cashtags: " + cash); |
Hi Andy! Would you like to add an example of searching tweets in Full Archive, filtering with jq?
I am able to pipe my result into jq, but when I try to add filters I get the response "cannot index array with string". Unfortunately, I am using Windows command prompt. Still, I am able to use some of your examples, such as the first one on this page (exchanging ' for "), so I wonder if my problems actually have something to do with the Full Archive results being built up differently?
I might miss something obvious since I am a complete beginner.
Best regards,
Stella
I probably would not use
twurl
for this, as it is a command line tool intended more for scripting and interactive usage. Depending on your coding language of choice, I'd pick a relevant API library wrapper, and then use the GETfollowers/list
endpoint. Depending on the number of followers you have, you might need to tweak thecount
parameter up to the maximum (200), and implement paging. Finally, the filtering would be down to some custom coding based on the chosen fields from the returned user object.