-
-
Save fay59/8f719cd81967e0eb2234897491e051ec to your computer and use it in GitHub Desktop.
#!/bin/bash | |
# requires jq | |
# arg 1: iCloud web album URL | |
# arg 2: folder to download into (optional) | |
function curl_post_json { | |
curl -sH "Content-Type: application/json" -X POST -d "@-" "$@" | |
} | |
BASE_API_URL="https://p23-sharedstreams.icloud.com/$(echo $1 | cut -d# -f2)/sharedstreams" | |
pushd $2 > /dev/null | |
STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream") | |
CHECKSUMS=$(echo $STREAM | jq -r '.photos[] | [(.derivatives[] | {size: .fileSize | tonumber, value: .checksum})] | max_by(.size | tonumber).value') | |
echo $STREAM \ | |
| jq -c "{photoGuids: [.photos[].photoGuid]}" \ | |
| curl_post_json "$BASE_API_URL/webasseturls" \ | |
| jq -r '.items[] | "https://" + .url_location + .url_path' \ | |
| while read URL; do | |
for CHECKSUM in $CHECKSUMS; do | |
if echo $URL | grep $CHECKSUM > /dev/null; then | |
curl -sOJ $URL & | |
break | |
fi | |
done | |
done | |
popd > /dev/null | |
wait |
-
I downloaded this script and run it, but didn't know why it's not working for me:
https://www.icloud.com/photos/ -- is the URL of my browser logged into icloud photo;
./ -- this is the local directory in my MacBook:
$ ./icloud-album-download.sh https://www.icloud.com/photos/ ./
$ (return nothing back here, nothing downloaded, no error message) -
tried it in shell debug mode: still the same, even no debugging info showed:
$ sh +x icloud-album-download.sh https://www.icloud.com/photos/ ./
$ (again, return nothing back here, nothing downloaded, no error message)
Do you know what's was wrong here?
Hi @gwu888
Did you try the original gist or the one I commented ? Only the second was working for me.
I used this script for a friend, but I think you need public sharing for icloud links. Your session cookie isn't shared and so it cannot works with this url for sure
some of the error occur because jq is missing.
You will found jq
jq is a lightweight and flexible command-line JSON processor
An update on @vchatela's version.
I added in a few bits as I needed to download a large iCloud share.
Added user screen updates.
Added Code comments.
De-dup the Large File checksum list.
Moved the Large File checksum to an array so we can do a fast lookup.
Check the filesystem for the requested file so we are not spending time downloading a file to only throw away on save.
Unsilenced the download so we can see the progress.
#!/bin/bash
# requires jq
# arg 1: iCloud web album URL
# arg 2: folder to download into (optional)
clear
function curl_post_json {
curl -sH "Content-Type: application/json" -X POST -d "@-" "$@"
}
printf "Getting iCloud Stream\n"
BASE_API_URL="https://p23-sharedstreams.icloud.com/$(echo $1 | cut -d# -f2)/sharedstreams"
pushd $2 > /dev/null
STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream")
HOST=$(echo $STREAM | jq '.["X-Apple-MMe-Host"]' | cut -c 2- | rev | cut -c 2- | rev)
if [ "$HOST" ]; then
BASE_API_URL="https://$(echo $HOST)/$(echo $1 | cut -d# -f2)/sharedstreams"
STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream")
fi
printf "Grabbing Large File Checksums\n"
CHECKSUMS=$(echo $STREAM | jq -r '.photos[] | [(.derivatives[] | {size: .fileSize | tonumber, value: .checksum})] | max_by(.size | tonumber).value')
printf "Adding Checksums to Array\n"
for CHECKSUM in $CHECKSUMS; do
arrCHKSUM+=($CHECKSUM)
done
printf "Total Downloads: ${#arrCHKSUM[@]}\n"
# Dedup checksum to only include unique ids.
arrCHKSUM=($(printf "%s\n" "${arrCHKSUM[@]}" | sort -u))
printf "Unique Downloads: ${#arrCHKSUM[@]}\n"
printf "Streaming All Assets\n"
echo $STREAM \
| jq -c "{photoGuids: [.photos[].photoGuid]}" \
| curl_post_json "$BASE_API_URL/webasseturls" \
| jq -r '.items | to_entries[] | "https://" + .value.url_location + .value.url_path + "&" + .key' \
| while read URL; do
# Get this URL's checksum value, not all URL's will be downloaded as there are both the fill size AND the thumbnail link in the Assets stream.
LOCAL_CHECKSUM=$(echo "${URL##*&}")
# If the url's checksum exists in the large checksum array then proceed with the download steps.
if [[ " ${arrCHKSUM[*]} " =~ " ${LOCAL_CHECKSUM} " ]]; then
# Get the filename from the URL, first we delimit on the forward slashes grabbing index 6 where the filename starts.
# then we must delimit again on ? to remove all the URL parameters after the filename.
# Example: https://www.example.com/4/5/IMG_0828.JPG?o=param1&v=param2&z=param3....
FILE=$(echo $URL|cut -d "/" -f6 | cut -d "?" -f1)
# Don't download movies
if [[ "$FILE" == *.mp4* ]]; then
echo "Skipping Movie"
else
# Don't download files that already exist
if [[ -f "$FILE" ]]; then
printf "Skipping $FILE\n"
else
# Original curl -sOJ $URL -> s = silent : O = download to file : J = Save using uploaded filename -- this also skips files that already exist.
curl -OJ $URL
fi
fi
else
echo "Skipping Thumbnail"
fi
done
popd > /dev/null
wait
@dmschlab thank you a lot, I will use this to sync for magic mirror
how to download mp4 as well?
Update: figured it out
i copied the 'curl -OJ $URL' inside the mp4 'if'
if [[ "$FILE" == *.mp4* ]]; then
curl -OJ $URL
i would suggest :
pushd 2> /dev/null
to skip showing errors when path is not set
also if there are multiple files with same name, i would suggest to rename them by appending some index (or download timestamp with seconds) to the name
i made some slight changes to be able to download also mp4, as well as eventual files with same name. Also hides the pushd/popd warnings when no specific path is set:
#!/bin/bash
# requires jq
# arg 1: iCloud web album URL
# arg 2: folder to download into (optional)
clear
function curl_post_json {
curl -sH "Content-Type: application/json" -X POST -d "@-" "$@"
}
printf "Getting iCloud Stream\n"
BASE_API_URL="https://p23-sharedstreams.icloud.com/$(echo $1 | cut -d# -f2)/sharedstreams"
pushd $2 2> /dev/null
STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream")
HOST=$(echo $STREAM | jq '.["X-Apple-MMe-Host"]' | cut -c 2- | rev | cut -c 2- | rev)
if [ "$HOST" ]; then
BASE_API_URL="https://$(echo $HOST)/$(echo $1 | cut -d# -f2)/sharedstreams"
STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream")
fi
printf "Grabbing Large File Checksums\n"
CHECKSUMS=$(echo $STREAM | jq -r '.photos[] | [(.derivatives[] | {size: .fileSize | tonumber, value: .checksum})] | max_by(.size | tonumber).value')
printf "Adding Checksums to Array\n"
for CHECKSUM in $CHECKSUMS; do
arrCHKSUM+=($CHECKSUM)
done
printf "Total Downloads: ${#arrCHKSUM[@]}\n"
# Dedup checksum to only include unique ids.
arrCHKSUM=($(printf "%s\n" "${arrCHKSUM[@]}" | sort -u))
printf "Unique Downloads: ${#arrCHKSUM[@]}\n"
printf "Streaming All Assets\n"
echo $STREAM \
| jq -c "{photoGuids: [.photos[].photoGuid]}" \
| curl_post_json "$BASE_API_URL/webasseturls" \
| jq -r '.items | to_entries[] | "https://" + .value.url_location + .value.url_path + "&" + .key' \
| while read URL; do
# Get this URL's checksum value, not all URL's will be downloaded as there are both the fill size AND the thumbnail link in the Assets stream.
LOCAL_CHECKSUM=$(echo "${URL##*&}")
# If the url's checksum exists in the large checksum array then proceed with the download steps.
if [[ " ${arrCHKSUM[*]} " =~ " ${LOCAL_CHECKSUM} " ]]; then
# Get the filename from the URL, first we delimit on the forward slashes grabbing index 6 where the filename starts.
# then we must delimit again on ? to remove all the URL parameters after the filename.
# Example: https://www.example.com/4/5/IMG_0828.JPG?o=param1&v=param2&z=param3....
FILE=$(echo $URL|cut -d "/" -f6 | cut -d "?" -f1)
# Don't download movies
if [[ "$FILE" == *.mp4* ]]; then
echo "Downloading movie"
curl -OJ $URL
else
# Don't download files that already exist
if [[ -f "$FILE" ]]; then
printf "File $FILE already present. Renaming..\n"
TIMESTAMP=$(date +%s%N)
curl $URL -o "${TIMESTAMP}_${FILE}"
else
# Original curl -sOJ $URL -> s = silent : O = download to file : J = Save using uploaded filename -- this also skips files that already exist.
curl -OJ $URL
fi
fi
else
echo "Skipping Thumbnail"
fi
done
popd 2> /dev/null
wait
i made some slight changes to be able to download also mp4, as well as eventual files with same name. Also hides the pushd/popd warnings when no specific path is set:
#!/bin/bash # requires jq # arg 1: iCloud web album URL # arg 2: folder to download into (optional) clear function curl_post_json { curl -sH "Content-Type: application/json" -X POST -d "@-" "$@" } printf "Getting iCloud Stream\n" BASE_API_URL="https://p23-sharedstreams.icloud.com/$(echo $1 | cut -d# -f2)/sharedstreams" pushd $2 2> /dev/null STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream") HOST=$(echo $STREAM | jq '.["X-Apple-MMe-Host"]' | cut -c 2- | rev | cut -c 2- | rev) if [ "$HOST" ]; then BASE_API_URL="https://$(echo $HOST)/$(echo $1 | cut -d# -f2)/sharedstreams" STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream") fi printf "Grabbing Large File Checksums\n" CHECKSUMS=$(echo $STREAM | jq -r '.photos[] | [(.derivatives[] | {size: .fileSize | tonumber, value: .checksum})] | max_by(.size | tonumber).value') printf "Adding Checksums to Array\n" for CHECKSUM in $CHECKSUMS; do arrCHKSUM+=($CHECKSUM) done printf "Total Downloads: ${#arrCHKSUM[@]}\n" # Dedup checksum to only include unique ids. arrCHKSUM=($(printf "%s\n" "${arrCHKSUM[@]}" | sort -u)) printf "Unique Downloads: ${#arrCHKSUM[@]}\n" printf "Streaming All Assets\n" echo $STREAM \ | jq -c "{photoGuids: [.photos[].photoGuid]}" \ | curl_post_json "$BASE_API_URL/webasseturls" \ | jq -r '.items | to_entries[] | "https://" + .value.url_location + .value.url_path + "&" + .key' \ | while read URL; do # Get this URL's checksum value, not all URL's will be downloaded as there are both the fill size AND the thumbnail link in the Assets stream. LOCAL_CHECKSUM=$(echo "${URL##*&}") # If the url's checksum exists in the large checksum array then proceed with the download steps. if [[ " ${arrCHKSUM[*]} " =~ " ${LOCAL_CHECKSUM} " ]]; then # Get the filename from the URL, first we delimit on the forward slashes grabbing index 6 where the filename starts. # then we must delimit again on ? to remove all the URL parameters after the filename. # Example: https://www.example.com/4/5/IMG_0828.JPG?o=param1&v=param2&z=param3.... FILE=$(echo $URL|cut -d "/" -f6 | cut -d "?" -f1) # Don't download movies if [[ "$FILE" == *.mp4* ]]; then echo "Downloading movie" curl -OJ $URL else # Don't download files that already exist if [[ -f "$FILE" ]]; then printf "File $FILE already present. Renaming..\n" TIMESTAMP=$(date +%s%N) curl $URL -o "${TIMESTAMP}_${FILE}" else # Original curl -sOJ $URL -> s = silent : O = download to file : J = Save using uploaded filename -- this also skips files that already exist. curl -OJ $URL fi fi else echo "Skipping Thumbnail" fi done popd 2> /dev/null wait
works 100% - thank you
works great, thanks
any chance to modify for google photos shared albums?
Thanks @fusionneur now October 2023 and still working
i made some slight changes to be able to download also mp4, as well as eventual files with same name. Also hides the pushd/popd warnings when no specific path is set:
#!/bin/bash # requires jq # arg 1: iCloud web album URL # arg 2: folder to download into (optional) clear function curl_post_json { curl -sH "Content-Type: application/json" -X POST -d "@-" "$@" } printf "Getting iCloud Stream\n" BASE_API_URL="https://p23-sharedstreams.icloud.com/$(echo $1 | cut -d# -f2)/sharedstreams" pushd $2 2> /dev/null STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream") HOST=$(echo $STREAM | jq '.["X-Apple-MMe-Host"]' | cut -c 2- | rev | cut -c 2- | rev) if [ "$HOST" ]; then BASE_API_URL="https://$(echo $HOST)/$(echo $1 | cut -d# -f2)/sharedstreams" STREAM=$(echo '{"streamCtag":null}' | curl_post_json "$BASE_API_URL/webstream") fi printf "Grabbing Large File Checksums\n" CHECKSUMS=$(echo $STREAM | jq -r '.photos[] | [(.derivatives[] | {size: .fileSize | tonumber, value: .checksum})] | max_by(.size | tonumber).value') printf "Adding Checksums to Array\n" for CHECKSUM in $CHECKSUMS; do arrCHKSUM+=($CHECKSUM) done printf "Total Downloads: ${#arrCHKSUM[@]}\n" # Dedup checksum to only include unique ids. arrCHKSUM=($(printf "%s\n" "${arrCHKSUM[@]}" | sort -u)) printf "Unique Downloads: ${#arrCHKSUM[@]}\n" printf "Streaming All Assets\n" echo $STREAM \ | jq -c "{photoGuids: [.photos[].photoGuid]}" \ | curl_post_json "$BASE_API_URL/webasseturls" \ | jq -r '.items | to_entries[] | "https://" + .value.url_location + .value.url_path + "&" + .key' \ | while read URL; do # Get this URL's checksum value, not all URL's will be downloaded as there are both the fill size AND the thumbnail link in the Assets stream. LOCAL_CHECKSUM=$(echo "${URL##*&}") # If the url's checksum exists in the large checksum array then proceed with the download steps. if [[ " ${arrCHKSUM[*]} " =~ " ${LOCAL_CHECKSUM} " ]]; then # Get the filename from the URL, first we delimit on the forward slashes grabbing index 6 where the filename starts. # then we must delimit again on ? to remove all the URL parameters after the filename. # Example: https://www.example.com/4/5/IMG_0828.JPG?o=param1&v=param2&z=param3.... FILE=$(echo $URL|cut -d "/" -f6 | cut -d "?" -f1) # Don't download movies if [[ "$FILE" == *.mp4* ]]; then echo "Downloading movie" curl -OJ $URL else # Don't download files that already exist if [[ -f "$FILE" ]]; then printf "File $FILE already present. Renaming..\n" TIMESTAMP=$(date +%s%N) curl $URL -o "${TIMESTAMP}_${FILE}" else # Original curl -sOJ $URL -> s = silent : O = download to file : J = Save using uploaded filename -- this also skips files that already exist. curl -OJ $URL fi fi else echo "Skipping Thumbnail" fi done popd 2> /dev/null wait
Feb 9, 2024. Still working! Thank you very much guys!
Hi,
Can anyone explain how exactly to run this script?
I'm not a coder (never learned how to code) so I don't even know where to start to run this. Like, where do I paste the shared album link in the script. Am I supposed to edit the script before running it in Terminal?
p.s. I downloaded jq and python and Git because a tutorial told me to, but I still, don't know where to start with getting this script running.
Hi, Can anyone explain how exactly to run this script? I'm not a coder (never learned how to code) so I don't even know where to start to run this. Like, where do I paste the shared album link in the script. Am I supposed to edit the script before running it in Terminal?
p.s. I downloaded jq and python and Git because a tutorial told me to, but I still, don't know where to start with getting this script running.
hi. you don't need python, the code is a shell script (runs on linux (baremetal host, virtual machine, container or WSL) or linux related apps such as cygwin or git bash.
are you mainly using windows or linux?
a TLDR guide
- First, you need jq, and also you need to be able to run it from anywhere, by properly installing or adding the portable version to environment variables. Assuming you downloaded it and properly installed it
https://ioflood.com/blog/install-jq-command-linux/
2.1
if you are using windows, assuming you already have git downloaded from https://git-scm.com/download/win, you should have git bash in start menu, so you can run it and browser to the folder where you placed this script using cd
UPDATE 1 MAY 2024: git bash doesn't recognize rev command. Just use cygwin instead
eg.
cd /c/Users/<your_user>/Desktop
2.2
if you are using linux already, then just run the following in the terminal:
cd ~/Desktop
3
then make the script executable
chmod +x ./icloud-album-download.sh
4
then run it from current directory, by passing the URL as argument as described in the comments in the script (first lines)
./icloud-album-download.sh <URL>
# arg 1: iCloud web album URL
# arg 2: folder to download into (optional)
Would someone be able to help me out... I've used this app in the past, but having issues today when trying to run it again. Using Windows 11 with Git installed (jq as well)
./icloud-album-download.sh: line 18: rev: command not found
./icloud-album-download.sh: line 18: rev: command not found
Grabbing Large File Checksums
jq: error (at :1): Cannot iterate over null (null)
Adding Checksums to Array
Total Downloads: 0
Unique Downloads: 0
Streaming All Assets
jq: error (at :1): Cannot iterate over null (null)
Would someone be able to help me out... I've used this app in the past, but having issues today when trying to run it again. Using Windows 11 with Git installed (jq as well)
./icloud-album-download.sh: line 18: rev: command not found
./icloud-album-download.sh: line 18: rev: command not found
Grabbing Large File Checksums
jq: error (at :1): Cannot iterate over null (null)
Adding Checksums to Array
Total Downloads: 0
Unique Downloads: 0
Streaming All Assets
jq: error (at :1): Cannot iterate over null (null)
i updated the comment above. apparently rev command is not recognized by git bash. Just use cygwin instead. It's pretty easy to setup. Download the setup from https://www.cygwin.com/.
Thanks.
For anyone else who's technologically challenged, I used cd /cygdrive/<the path to your file>
, ie. cd /cygdrive/c/Users/<your_user>/Desktop
to navigate to where the file is located
Lifesaver! - I didn't want to download 500 images by hand.
Work perfectly on a MacOS;
installed jq with
brew install jq
then copied this script in an only-text document of TextEdit (command shift T to made a txt.only document and command shift w to made a full page view)
using cd and chmod to change executability from /Applications/Utility/Terminal.app
cd /path/in/the/folder/you/have/saved/the/script/from/textedit
chmod +x ./icloud-album-download.sh
then, in iPhoto copy the album link like https://www.icloud.com/sharedalbum/it-it/#NuMbErOfAlBuM
Uj947nXmRqV2nRaWshKtHzTvckUUpD yours worked perfectly - thank you!
I'm new to this but I found my way around it after some time
I used powershell to install chocolatey and then jq
I saved the latest script on my desktop with Notepadd++ on Unix format as 'icloud-album-download.sh'
I installed cygwin
I ran the following lines:
cd c:/Users/'my user'/Desktop
chmod +x ./icloud-album-download.sh
./icloud-album-download.sh 'shared icloud album link'
Seems like everything runs without errors but nothing downloads, I get this on cygwin command window:
Getting iCloud Stream
Grabbing Large File Checksums
Adding Checksums to Array
Total Downloads: 0
Unique Downloads: 0
Streaming All Assets
Quickly wrote a new Powershell script to download iCloud shared album (photos + videos):
https://gist.github.com/GodSaveEarth/9b8d574d0f5ab789aebdfdb2db59b8df
This is really cool. Do you know if there is a way to get the comments?