This script collects domains from your Chrome history for the default profile and checks to see if they are being proxied through Cloudflare by checking presence of a header.

check_cloudflare.rb

# Except sqlite3 all the libraries are standard that should be present with a Ruby installation.
# If you don't have sqlite3 installed. Use `gem install sqlite3`

require 'fileutils'
require 'sqlite3'
require 'uri'
require 'net/http'
require 'set'
require 'thread'

chrome_history_location = "#{ENV['HOME']}/Library/Application\ Support/Google/Chrome/Default/History"
temp_location = "/tmp/Chrome_history"
FileUtils.cp(chrome_history_location, temp_location)

sqlite_db = SQLite3::Database.new temp_location
chrome_history = sqlite_db.execute('SELECT DISTINCT(url) FROM urls;').flatten; nil

FileUtils.rm([temp_location])

domain_set = Set.new
cloudflare_set = Set.new
query_uris = Array.new

chrome_history.each do |url|
  host = URI.parse(url).host rescue nil
  query_uris += [URI::HTTPS.build({host: host}), URI::HTTP.build({host: host})] if !domain_set.include?(host) && !host.nil?
  domain_set.add(host)
end; nil

uri_mutex, set_mutex, read_mutex = Mutex.new, Mutex.new, Mutex.new
(1..16).map do
  Thread.new(query_uris, cloudflare_set) do |query_uris, cloudflare_set|
    while !(uri = uri_mutex.synchronize { query_uris.pop }).nil?
      cf_header_present = !Net::HTTP.get_response(uri)['cf-ray'].nil? rescue nil
      read_mutex.synchronize{ print("#{query_uris.length} remaining\r") }
      set_mutex.synchronize { cloudflare_set.add(uri.host) } if cf_header_present
    end
  end
end.each(&:join); nil

p cloudflare_set.to_a.sort

@vrischmann I changed p to puts when I copied the script from local to gist. Will change it. Thanks!

if you got cannot load such file -- sqlite3 (LoadError)
you'll need to gem install sqlite3

I've added multiple-profile support here: https://gist.github.com/lpetre/77a33d295671409382f8b0608420eca0

chrome_history_location on ubuntu:

chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"

ruby seems to have a bug with hostnames with an underscore in them; I had to add url.gsub! /_/, '%5F' before line 19 to make this run

Can you please make it so that it writes all the domains in to a text file? The windows cmd doesn't let me scroll all the way up so I can't see all the domains.

@bxkx Try using output redirection

ruby script > file

That did it, thanks!

nice work. i wonder if there is any way to look at the cached pages and see if the page had been re-written by cloudflare - i somehow doubt it, but would be nice to know, as i believe only those sites are at risk.

@bxkx - you can increase the cmd buffer or pipe to a file, eg, ruby script.rb > output.txt

It doesn't seem to work if there's an underscore in the host:

/usr/lib64/ruby/2.2.0/uri/generic.rb:593:in `check_host': bad component(expected host component): fate_extella.wicurio.com (URI::InvalidComponentError)
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:634:in `host='
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:668:in `hostname='
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:187:in `initialize'
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:134:in `new'
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:134:in `build'
        from /usr/lib64/ruby/2.2.0/uri/http.rb:62:in `build'
        from tmp.rb:26:in `block in <main>'
        from tmp.rb:24:in `each'
        from tmp.rb:24:in `<main>'

(Ruby 2.2)

@lbeltrame You can try doing what @llimllib suggested here: https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064#gistcomment-2010236

@lbeltrame Happened to me as well with a different URL. I think its the _ in the url. I just searched for the problem domain in my chrome history, deleted it, and then the script ran fine.

For those who simply want to check their history or whatever tables are in your chrome history database, I created a small shell script to generate those files.

https://github.com/darrentu/convert-db-to-csv

On Linux, I had to switch the path to this:

chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"

See Chrome/Chromium User Data Directories