Skip to content

Instantly share code, notes, and snippets.

@gljeremy
Forked from tmcw/disconnect.rb
Created September 21, 2011 14:03
Show Gist options
  • Save gljeremy/1232102 to your computer and use it in GitHub Desktop.
Save gljeremy/1232102 to your computer and use it in GitHub Desktop.
bulk-export run data from Garmin Connect
#!/usr/bin/env ruby
## disconnect
# ./disconnect.rb -u yourusername [-o /your/path] [-p yourhttpproxyserver]
#
# This is a command-line utility for the bulk-downloading of run data from
# the connect.garmin.com web application, which has lackluster export
# capabilities.
#
# Using this code is a matter of your own relationship with Garmin Connect
# and their TOS. I can't imagine this being very destructive to their service,
# and it's just filling in a hole in their existing service.
#
# It's built against Garmin Connect as of September 22, 2011. It's a scraper:
# thus if Garmin changes, this **will break**.
#
# This script requires all of the utilities on the line below: install them
# with rubygems
%w{rubygems json fileutils mechanize choice highline/import}.map{|x| require x}
LOGIN_PAGE = "https://connect.garmin.com/signin"
ACTIVITIES_SEARCH = "http://connect.garmin.com/proxy/activity-search-service-1.0/json/activities?_dc=1220170621856&start=%d"
GPX_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.1/gpx/activity/%d?full=true"
KML_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.0/kml/activity/%d?full=true"
TCX_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.0/tcx/activity/%d?full=true"
Choice.options do
header ''
header 'Specific options:'
option :user, :required => true do
short '-u'
long '--user=USER'
desc 'connect.garmin.com username. Required'
end
option :dir do
short '-o'
long '--output-dir=OUTPUT'
desc 'the directory to save .tcx files'
default 'tcx'
end
option :proxy do
short '-p'
long '--http-proxy'
desc 'Use HTTP proxy for remote operations'
end
end
password = ask("Enter your password: " ) { |q| q.echo = "*" }
def login(agent, user, password)
agent.get(LOGIN_PAGE) do |page|
puts "Loaded login page."
login_form = page.form('login')
login_form['login:loginUsernameField'] = user
login_form['login:password'] = password
puts "Sent login information."
page = agent.submit(login_form, login_form.buttons.first)
raise "Login incorrect!" if page.title().match('Sign In')
puts "Login successful!"
return page
end
end
def download_run(agent, id)
print "."
# This downloads TCX files: you can swap out the constant, or add
# more lines that download the different kinds of exports. I prefer TCX,
# because despite being a 'private standard,' it includes all data,
# including heart rate data.
agent.get(TCX_EXPORT % (id).to_i).save_as(File.join(Choice[:dir], "%d.tcx" % id))
end
def activities(agent)
start = 0
while true
j = agent.get(ACTIVITIES_SEARCH % start)
search = JSON.parse(j.content)
# Got some JSON content, let's see if there are any activities left to process
if search['results']['activities']==nil then break end
runs = search['results']['activities'].map {|r|
# Get each activity id to insert into the download URL
r['activity']['activityId']
}.map {|id|
# Download a run.
download_run(agent, id)
start=start+1
}
puts "|"
end
end
agent = Mechanize.new
if Choice[:proxy] != nil then agent.set_proxy(Choice[:proxy], 80, nil, nil) end
# One needs to log in to get access to private runs. Mechanize will store
# the session data for the API call that cames next.
home_page = login(agent, Choice[:user], password)
FileUtils.mkdir_p(Choice[:dir]) if not File.directory?(Choice[:dir])
puts "Downloading runs..."
activities(agent)
@cloveras
Copy link

Worked great - thank you!

@cloveras
Copy link

Simple Perl script for renaming the TCX and sorting into directories: https://github.com/cloveras/Hacks/blob/master/rename-tcx.pl, hope it's useful.

@mfrimannm
Copy link

Cant get it to work... :( any suggestions? :)

@lancefisher
Copy link

Worked great! Thanks!

@gljeremy
Copy link
Author

@mfrimannm check your OpenSSL version and settings, or try something like this to get it to work:

http://stackoverflow.com/a/8568915

@ChrisHammond
Copy link

I ran into the same problem with Windows, thanks to the stackoverflow post I added the line after agent=Mech......

agent = Mechanize.new
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE

from there the script ran correctly.

@mahony
Copy link

mahony commented Apr 18, 2012

Brilliant! It's ridiculous (although not surprising) that Garmin chose to exclude a bulk export feature. Excellent work! Thank you.

Copy link

ghost commented May 1, 2012

Hey, guys.
Could you help me with the following problem.

Loaded login page.
Sent login information.
Login successful!
Downloading runs...
/var/lib/gems/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:1094:in auto_io': closed stream (IOError) from /var/lib/gems/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:401:incontent_encoding_gunzip'
from /var/lib/gems/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:763:in response_content_encoding' from /var/lib/gems/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:261:infetch'
from /var/lib/gems/1.9.1/gems/mechanize-2.4/lib/mechanize.rb:407:in get' from ./disconnect.rb:81:inactivities'
from ./disconnect.rb:109:in `

'

Thank you

@thesloth2
Copy link

I have no programming knowlege - Ruby or otherwise, but I'd really like to get this script to work.
Please could someone assist the newbie.

I've installed Ruby 1.9.3 & Ruby Gems 1.8.24 on Windows. When I run the script I get the following error:
C:\move\disconnect>ruby disconnect.rb
C:/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require': cannot load such file -- mechanize (LoadError) from C:/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:i nrequire'
from disconnect.rb:20:in block in <main>' from disconnect.rb:20:inmap'
from disconnect.rb:20:in `

'

I tried the script on OSX & Ubuntu Linux and got a similar message.

Thanks in advance

@ChrisHammond
Copy link

@thesloth2 you'll need to download the machanize libraries, and when you run it again you'll get another error for another library you will have to DL as well.I honestly don't recall commands to call though as it was a few weeks ago now that I did it

@thesloth2
Copy link

ChrisHammond many thanks!
I needed to run the following & then apply the above mentioned OpenSSL fix:
gem install nokogiri
gem install mechanize
gem install choice
gem install highline
gem install import

I don't understand the magic behind this, but do see all of my .tcx files downloading. Very happy.

@mikehester
Copy link

Hi and I am on a mac OS 10.6 i believe. When I download this I get the errors for library not found. When I try to perform the gem install of certain things I get an error about not being able to find the ruby.h header file and says that it did not install the package. I am not that familiar with such things and any help would be appreciated. In my case the script does not start - just aborts.

Thanks!

@novonaar
Copy link

Works after adding the following line in Windows:
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE

Thanks!!

@sdbaylis
Copy link

First experience with Ruby / Gems etc. Got it all installed, setup, dependencies loaded etc. (I think), however when I run nothing happens. So in the command window I'm typing:

c:\local\temp> ruby disconnect.rb myusername mydownloadfolder

and nothing happens. Does anyone have any pointers for where to start troubleshooting?
Many thanks.

@mikehester
Copy link

mikehester commented Jun 21, 2012 via email

@sdbaylis
Copy link

Hi Mike, thanks for the quick response! I tried both those ideas and the same thing happens either way - 1 to 2 second pause, then back to:

Usage: disconnect.rb [-uop]

Specific options:

etc etc.

No error messages though.

@sdbaylis
Copy link

Ken - thanks, I was passing the inputs incorrectly. I now get an SSL error so will tackle that one separately!

@sdbaylis
Copy link

Just to update that last comment: Chris Hammonds comment from two months ago suggesting to add:

agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE

Did the trick. Thanks all.

@thomasmoelhave
Copy link

I was having some problems connection timeouts (I have a lot of workouts on Garmin Connect) and tweaked this version a tiny bit to use a longer timeout. Get it here if you have similar issues: https://gist.github.com/3253706

@nickbroon
Copy link

It’s a real shame the tcx files exported from Garmin Connect do not contain the names and descriptions that you can give Activities on that site. It appears the tcx file could contain these as the format has space for activity names and Notes attribute. And still to be determined if Strava would import the activity name and the Notes section from the file even if they were there.
Could this script be enhanced to grab the name and description for each activity from the Garmin Connect website and inject them into the tcx file generated?

@daemonza
Copy link

Getting this problem, anyone know how to fix it?

ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:29:in `initialize': getaddrinfo: nodename nor servname provided, or not known   (SocketError)
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:29:in `open'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:29:in `block in connect'
from /Users/wernergillmer/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
from /Users/wernergillmer/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:29:in `connect'
from /Users/wernergillmer/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from /Users/wernergillmer/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:750:in `start'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent.rb:511:in `connection_for'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent.rb:806:in `request'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
from /Users/wernergillmer/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize.rb:407:in `get'
from ./disconnect.rb:55:in `login'
from ./disconnect.rb:103:in `<main>'

@daemonza
Copy link

Regarding my above problem, this one works - https://gist.github.com/3253706

@daemonza
Copy link

Noticed strava got a 25 file limit upload in batches. Wrote this script to make it a bit easier to upload to Stava after using, the above disconnect.rb script.

https://gist.github.com/3878849

@gkrathwohl
Copy link

I wasted a lot of time trying to make something similar to this before I found this, so thanks! However, I'm interested in using this to find trails in my area, not just download my own tracks. There's a tool for exploring like this on garmin connect, but it only lets you view one at a time, and I want to download all of the results to view them simultaneously on google earth. How can I use this to download the activity numbers of activities based on a lat and long?

/proxy/activity-search-service-1.0/json/activities?_dc=1220170621856&start=%d

I think it has something to do with the number after the dc, but I couldn't figure it out by looking at the source code. Any help?

@holman
Copy link

holman commented Feb 18, 2013

❤️

@gmcmillan
Copy link

thanks for making this! worked perfectly.

@hvdosten
Copy link

Thanks a lot for this code. I tried to do a bulk download of all my activities from connect.gamin, but I get this error message:

vdO:downloads hvdosten$ ruby disconnect.rb –user=hvdosten –output-dir=garmin
Enter your password: ******
Loaded login page.
disconnect.rb:58:in block in login': undefined method[]=’ for nil:NilClass (NoMethodError)
from /Library/Ruby/Gems/2.0.0/gems/mechanize-2.7.3/lib/mechanize.rb:442:in get’ from disconnect.rb:55:inlogin’
from disconnect.rb:103:in `’
vdO:downloads hvdosten$

I would be very happy if there is anybody out there who could give me a hint how to overcome this problem. Thanks a lot in advance!!
Harald

@NRGfxIT
Copy link

NRGfxIT commented May 16, 2015

Most grateful for all the work people have put into this.

As a none programmer I have spent and afternoon getting to the point where the script will run. Unfortunately I come up against a similar error as Harald above:

C:\Sites>ruby disconnect.rb -u MyUserName -o c:/temp
Enter your password: *******
Loaded login page.
disconnect.rb:115:in block in login': undefined method[]=' for nil:NilClass (NoMethodError)
from C:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/mechanize-2.7.3/lib/mechanize.rb:442:in get' from disconnect.rb:109:inlogin'
from disconnect.rb:213:in `

'

I am at a loss as to were to go from here, any help would be most welcomed.

@robrohan
Copy link

robrohan commented Aug 8, 2015

This script won't work anymore as the new login page seems to require javascript to render the login form and mechanize appears not to run javascript.

@jsimpson
Copy link

@robrohan i have a fork that works if you're still looking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment