Last active
May 27, 2017 21:57
-
-
Save kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064 to your computer and use it in GitHub Desktop.
This script collects domains from your Chrome history for the default profile and checks to see if they are being proxied through Cloudflare by checking presence of a header.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Except sqlite3 all the libraries are standard that should be present with a Ruby installation. | |
# If you don't have sqlite3 installed. Use `gem install sqlite3` | |
require 'fileutils' | |
require 'sqlite3' | |
require 'uri' | |
require 'net/http' | |
require 'set' | |
require 'thread' | |
chrome_history_location = "#{ENV['HOME']}/Library/Application\ Support/Google/Chrome/Default/History" | |
temp_location = "/tmp/Chrome_history" | |
FileUtils.cp(chrome_history_location, temp_location) | |
sqlite_db = SQLite3::Database.new temp_location | |
chrome_history = sqlite_db.execute('SELECT DISTINCT(url) FROM urls;').flatten; nil | |
FileUtils.rm([temp_location]) | |
domain_set = Set.new | |
cloudflare_set = Set.new | |
query_uris = Array.new | |
chrome_history.each do |url| | |
host = URI.parse(url).host rescue nil | |
query_uris += [URI::HTTPS.build({host: host}), URI::HTTP.build({host: host})] if !domain_set.include?(host) && !host.nil? | |
domain_set.add(host) | |
end; nil | |
uri_mutex, set_mutex, read_mutex = Mutex.new, Mutex.new, Mutex.new | |
(1..16).map do | |
Thread.new(query_uris, cloudflare_set) do |query_uris, cloudflare_set| | |
while !(uri = uri_mutex.synchronize { query_uris.pop }).nil? | |
cf_header_present = !Net::HTTP.get_response(uri)['cf-ray'].nil? rescue nil | |
read_mutex.synchronize{ print("#{query_uris.length} remaining\r") } | |
set_mutex.synchronize { cloudflare_set.add(uri.host) } if cf_header_present | |
end | |
end | |
end.each(&:join); nil | |
p cloudflare_set.to_a.sort |
For those who simply want to check their history or whatever tables are in your chrome history database, I created a small shell script to generate those files.
On Linux, I had to switch the path to this:
chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@lbeltrame Happened to me as well with a different URL. I think its the
_
in the url. I just searched for the problem domain in my chrome history, deleted it, and then the script ran fine.