Google Analytics Real Time Visitors Widget for Dashing with OAuth2 Authentication

README.md

Description

A Dashing widget for displaying the number of current visitors (in real time) to your website, as reported by Google Analytics.

This widget is a fork of https://gist.github.com/mtowers/5986576

Dependencies

google-api-ruby-client

Add it to dashing's gemfile:

gem 'google-api-client'

and run bundle install.

Usage

To use this widget, you'll first need to set up a Google API project and attach it to the Google Analytics profile you wish to monitor, as well as request access to the Google Analytics Real Time API.

1. Request acces to the Google Analytics Real Time API

1. Go to http://goo.gl/7ZPlMI and fill the form

2. Create and download a new private key for Google API access.

1. Go to https://code.google.com/apis/console
2. Click 'Create Project'
3. Enable 'Analytics API' service and accept both TOS's
4. Click 'API Access' in the left-hand nav menu
5. Click 'Create an OAuth 2.0 Client ID'
6. Enter a product name (e.g. Dashing Widget) - logo and url are optional
7. Click 'Next'
8. Under Application Type, select 'Service Account'
9. Click 'Create Client ID'
10. Click 'Download private key' NOTE: This will be your only opportunity to download this key.
11. Note the password for your new private key ('notasecret')
12. Close the download key dialog
13. Find the details for the service account you just created and copy it's email address which will look something like this: [email protected] - you'll need it in your ruby code later

3. Attach your Google API service account to your Google Analytics profile

Note: you will need to be an administrator of the Google Analytics profile

1. Log in to your Google Analytics account: http://www.google.com/analytics/
2. Click 'Admin' in the upper-right corner
3. Select the account containing the profile you wish to use
4. Select the property containing the profile you wish to use
5. Select the profile you wish to use
6. Click the 'Users' tab
7. Click '+ New User'
8. Enter the email address you copied from step 13 above
9. Click 'Add User'

4. Locate the ID for your Google Analytics profile

1. On your Google Analytics profile page, click the 'Profile Settings' tab
2. Under 'General Information' copy your Profile ID (e.g. 654321) - you'll need it in your ruby code later

5. Start coding (finally)

1. Copy the visitor_count.rb file in to your dashing jobs\ folder.
2. Update the service_account_email, key_file, key_secret and profileID variables

    service_account_email = '[YOUR SERVICE ACCOUTN EMAIL]' # Email of service account
    key_file = 'path/to/your/keyfile.p12' # File containing your private key
    key_secret = 'notasecret' # Password to unlock private key
    profileID = '[YOUR PROFILE ID]' # Analytics profile ID.

3. Add the widget HTML to your dashboard

    <li data-row="1" data-col="1" data-sizex="1" data-sizey="1"> 
        <div data-id="visitor_count_real_time" data-view="Graph" data-title="Current Visitors"></div> 
    </li>

Notes

If you want to modify this plugin to pull other data from Google Analytics, be sure to check out the Google Analytics Query Explorer.

visitor_count_real_time.rb

require 'google/api_client'
require 'date'

# Update these to match your own apps credentials
service_account_email = '[YOUR SERVICE ACCOUNT EMAIL]' # Email of service account
key_file = 'path/to/your/keyfile.p12' # File containing your private key
key_secret = 'notasecret' # Password to unlock private key
profileID = '[YOUR PROFILE ID]' # Analytics profile ID.

# Get the Google API client
client = Google::APIClient.new(
  :application_name => '[YOUR APPLICATION NAME]', 
  :application_version => '0.01'
)

visitors = []

# Load your credentials for the service account
key = Google::APIClient::KeyUtils.load_from_pkcs12(key_file, key_secret)
client.authorization = Signet::OAuth2::Client.new(
  :token_credential_uri => 'https://accounts.google.com/o/oauth2/token',
  :audience => 'https://accounts.google.com/o/oauth2/token',
  :scope => 'https://www.googleapis.com/auth/analytics.readonly',
  :issuer => service_account_email,
  :signing_key => key)

# Start the scheduler
SCHEDULER.every '5s', :first_in => 0 do

  # Request a token for our service account
  client.authorization.fetch_access_token!

  # Get the analytics API
  analytics = client.discovered_api('analytics','v3')

  # Execute the query
  visitCount = client.execute(:api_method => analytics.data.realtime.get, :parameters => { 
    'ids' => "ga:" + profile_id,
    'metrics' => "ga:activeVisitors",
  })

  visitors << { x: Time.now.to_i, y: visits.to_i }

  # Update the dashboard
  send_event('visitor_count_real_time', points: visitors)
end

This looks like this might an issue with the fact that Google is currently considering this realtime analytics to be invitation only Beta. Womp womp. I'm getting the same thing:

scheduler caught exception:
Authorization failed.  Server message:
{
  "error" : "invalid_grant"
}
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/signet-0.5.0/lib/signet/oauth_2/client.rb:883:in `fetch_access_token'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/signet-0.5.0/lib/signet/oauth_2/client.rb:898:in `fetch_access_token!'
/home/ubuntu/sweet_dashboard_project/jobs/visitor_count_real_time.rb:29:in `block in <top (required)>'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'

The same problem as acheloos-m ;/

I am getting this message after it has been running a while: {"error":"server_error","error_description":"JWT has expired."}. Shouldn't the fetch_access_token method be getting a new token every time this runs?

Works greate except there's no graph visible for us. Any hints? The embedded SVG shows only the coordinate numbers...

Okay, Update after 3 weeks of usage:

• Graph renders after a day
• Dashboard is getting slower and slower every day as all realtime informations are stored and calculated in the widget
• Everything else is awesome :)

Do you have an explanation of why it takes a day to render ? I'm having some issues with the axe Y data and I think it may be because of that.

Was there any update on the "invalid_grant" situation? I just finished getting it all setup and am seeing the same error mentioned above.

Also getting "invalid_grant" any response on this? Also:
"copy the visitor_count.rb file in to your dashing jobs\ folder." - There is no file with that name included in the gist? Should that not be visitor_count_real_time.rb? Also, the first comment claims to be the updated version, is that official??

I'm having trouble getting dashing to start after adding this widget. I get the following error:

/var/lib/gems/1.9.1/gems/backports-3.6.3/lib/backports/std_lib.rb:9:in `require': cannot load such file -- google/api_client (LoadError)

I ran gem install google-api-client then bundle install. If I do gem list, I can see google-api-client listed, but if I do bundle update, I don't see it there.

I'm a ruby noob, so I'm not really sure how to fix this.

The dashboard is getting slower after continued usage, because old values are never deleted.
I would suggest adding a variable at the top of the file:

max_amount = 100

as well as adding after this line

visitors << { x: Time.now.to_i, y: response.data.rows }

the following code:

if visitors.size > max_amount
    visitors.shift
end

So the first added element is removed.

I'm still getting 'invalid_grant' response errors. Anyone know how to fix this?

For those not being granted access:

The Real Time Reporting API is currently available as a developer preview in limited beta. If you're interested in signing up, request access to the beta.

Extracted from the Real Time Reporting API overview

This plugin works for me, but the problem I have is that it stops reporting the number of active users after 5 minutes. Any ideas how to fix this ?

What is wrong? How to solve?

MINGW64 /c/Teste
$ ruby -v
ruby 2.3.3p222 (2016-11-21 revision 56859) [i386-mingw32]

MINGW64 /c/Teste
$ gem environment
RubyGems Environment:
  - RUBYGEMS VERSION: 2.6.11
  - RUBY VERSION: 2.3.3 (2016-11-21 patchlevel 222) [i386-mingw32]
  - INSTALLATION DIRECTORY: C:/Ruby23/lib/ruby/gems/2.3.0
  - USER INSTALLATION DIRECTORY: C:/Users/marcelovilela.mg/.gem/ruby/2.3.0
  - RUBY EXECUTABLE: C:/Ruby23/bin/ruby.exe
  - EXECUTABLE DIRECTORY: C:/Ruby23/bin
  - SPEC CACHE DIRECTORY: C:/Users/marcelovilela.mg/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: C:/ProgramData
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-mingw32
  - GEM PATHS:
     - C:/Ruby23/lib/ruby/gems/2.3.0
     - C:/Users/marcelovilela.mg/.gem/ruby/2.3.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - C:\Users\marcelovilela.mg\bin
     - C:\Program Files\Git\mingw64\bin
     - C:\Program Files\Git\usr\local\bin
     - C:\Program Files\Git\usr\bin
     - C:\Program Files\Git\usr\bin
     - C:\Program Files\Git\mingw64\bin
     - C:\Program Files\Git\usr\bin
     - C:\Users\marcelovilela.mg\bin
     - C:\Ruby23\bin
     - C:\ProgramData\Oracle\Java\javapath
     - C:\Program Files\Common Files\Microsoft Shared\Windows Live
     - C:\Windows\system32
     - C:\Windows
     - C:\Windows\System32\Wbem
     - C:\Windows\System32\WindowsPowerShell\v1.0
     - C:\Program Files\Intel\DMIX
     - C:\Program Files\TortoiseSVN\bin
     - C:\python26
     - C:\python26\scripts
     - C:\cheetah-2.4.4\bin
     - C:\Program Files\nodejs
     - C:\Program Files (x86)\Skype\Phone
     - C:\Program Files\Git\cmd
     - C:\Users\marcelovilela.mg\AppData\Roaming\npm
     - C:\Python27\Scripts
     - C:\Python27
     - C:\Program Files\Git\usr\bin\vendor_perl
     - C:\Program Files\Git\usr\bin\core_perl

MINGW64 /c/Teste
$ gem list google-api-client -d
google-api-client (0.11.1)
    Authors: Steven Bazyl, Tim Emiola, Sergio Gomes, Bob Aman
    Homepage: https://github.com/google/google-api-ruby-client
    License: Apache 2.0
    Installed at: C:/Ruby23/lib/ruby/gems/2.3.0

    Client for accessing Google APIs

MINGW64 /c/Teste
$ ruby visitor_count.rb
C:/Ruby23/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- google/api_client (LoadError)
        from C:/Ruby23/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from visitor_count.rb:1:in `<main>'

We recently implemented this widget on our dashboard and applied some modifications.

We added the ability to set the timezone (activesupport) as our server is set to UTC.

We are storing 12 hours of results at 1 per minute instead of 1 per second (max 720) in memcached (dalli) so if the service is restarted it retains the graph data.

Add to smashings gem file:

gem 'activesupport'
gem 'dalli'

and run bundle install

require 'google/api_client'
require 'date'
require 'active_support/all'
require 'dalli'

Time.zone = '[YOUR TIMEZONE]'

# Update these to match your own apps credentials
service_account_email = '[YOUR SERVICE ACCOUNT EMAIL]' # Email of service account
key_file = 'path/to/your/keyfile.p12' # File containing your private key
key_secret = 'notasecret' # Password to unlock private key
profile_id = '[YOUR PROFILE ID]' # Analytics profile ID.

# Get the Google API client
client = Google::APIClient.new(:application_name => '[YOUR APPLICATION NAME]', 
  :application_version => '0.01')

# Load your credentials for the service account
key = Google::APIClient::KeyUtils.load_from_pkcs12(key_file, key_secret)
client.authorization = Signet::OAuth2::Client.new(
  :token_credential_uri => 'https://accounts.google.com/o/oauth2/token',
  :audience => 'https://accounts.google.com/o/oauth2/token',
  :scope => 'https://www.googleapis.com/auth/analytics.readonly',
  :issuer => service_account_email,
  :signing_key => key)

visitors = []

# Memcached initialization
options = { namespace: "dashboard", compress: true }
dc = Dalli::Client.new('localhost:11211', options)

# Query visitors data from memcached on start
visitors = dc.get('visitor_count') if visitors.empty?

# Initialize the graph data
send_event('visitor_count_real_time', points: visitors)

# Total number of values to store
max_amount = 720
  
# Start the scheduler
SCHEDULER.every '1m', :first_in => 0 do
  # Request a token for our service account
  client.authorization.fetch_access_token!

  # Get the analytics API
  analytics = client.discovered_api('analytics','v3')

  # Execute the query
  response = client.execute(:api_method => analytics.data.realtime.get, :parameters => {
    'ids' => "ga:" + profile_id,
    'metrics' => "rt:activeUsers",
  })

  # Get the current visit count from the query
  visit_count = response.data.rows[0][0].to_i
  
  visitors << { x: Time.now.to_i, y: visit_count }
  
  # Remove the oldest value if total number is more than allowed
  if visitors.size > max_amount
    visitors.shift
  end
  
  # Store visitors data in memcached
  dc.set('visitor_count', visitors)

  # Update the dashboard
  send_event('visitor_count_real_time', points: visitors)
end

I've done an update which does a few things:

• Instead of oauth I created a service account in google auth (Note the private key is base64 encoded which I did to avoid some encoding issues when passing it through environment variables we use Docker).
• I've set the timer to run every five seconds and store 24 hours of data
• I've set the query to group by dimensions as we're planning to add a view that shows the most popular pages

require 'base64'
require 'date'
require 'google/apis/analytics_v3'

# Decode the private key
ENV['GOOGLE_PRIVATE_KEY'] = Base64.strict_decode64(ENV['GOOGLE_PRIVATE_KEY']).gsub('\n', "\n")

scopes = %w(https://www.googleapis.com/auth/analytics.readonly)
authorization = Google::Auth::ServiceAccountCredentials.make_creds(scope: scopes)
Google::Apis::RequestOptions.default.authorization = authorization

# Schedule the token refresh
SCHEDULER.every '30m', first_in: 1800 do
  # Refresh the token for our service account
  authorization.fetch_access_token!
end

# Setup services and variables
Analytics = Google::Apis::AnalyticsV3
analytics = Analytics::AnalyticsService.new

dimensions = %w(rt:pagePath)
metrics = %w(rt:activeUsers)
sort = %w(rt:pagePath)

visitors = []
max_amount = 8640

# Schedule the data updating
SCHEDULER.every '10s', first_in: 0 do
  # Get the analytics API
  response = analytics.get_realtime_data("ga:#{ENV['GOOGLE_ANALYTICS_PROFILE_ID']}", 
    metrics.join(','), 
    dimensions: dimensions.join(','),
    sort: sort.join(','))

  visitors << { 
    x: Time.now.to_i,
    y: response.totals_for_all_results['rt:activeUsers'].to_i
  }
  visitors.shift if visitors.size > max_amount

  # Update the dashboard
  send_event('visitor_count_real_time', points: visitors)
end