Skip to content

Instantly share code, notes, and snippets.

@natzir

natzir/crux-from-google-serp.ipynb

Last active July 2, 2023 09:54
Show Gist options
  • Save natzir/23ce7400be681f43743b3dc04c3cd2c5 to your computer and use it in GitHub Desktop.
Save natzir/23ce7400be681f43743b3dc04c3cd2c5 to your computer and use it in GitHub Desktop.
Download ZIP
crux-from-google-serp.ipynb
Display the source blob
Display the rendered blob
Raw
crux-from-google-serp.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyPoaWq19kkCUAGMUNvw4XkI",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/natzir/23ce7400be681f43743b3dc04c3cd2c5/crux-from-google-serp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Get the CrUX Metrics from Google Search Results\n",
"\n",
"\n",
"---\n",
"\n",
"**Author:** Natzir Turrado, Technical SEO / Data Scientist\n",
"<br>**Twitter:** [@natzir9](https://twitter.com/natzir9)\n",
"\n",
"---\n",
"\n",
"This Colab notebook allows you to scrape the Google SERP using SerpApi and extract the Chrome User Experience Report (CrUX) metrics of each result. Best of all, it's completely free!\n",
"\n",
"Before you can use this document, please make sure to **create a copy** of it in your Google Drive (File > Save a copy in Drive).\n",
"\n",
"\n",
"To start, you need two free API keys: SerpApi and CrUX. Get them from these links:\n",
"- [SerpApi](https://serpapi.com/) (100 searches per month with the Free Plan)\n",
"- [CrUX Api](https://console.cloud.google.com/marketplace/product/google/chromeuxreport.googleapis.com) (150 queries per minute per Google Cloud project)\n",
"---\n",
"\n",
"\n",
"\n"
],
"metadata": {
"id": "1mmZhBOPE8Vn"
}
},
{
"cell_type": "code",
"source": [
"#@markdown # Step 1: 'Play' this cell to install the the required libraries\n",
"#@markdown ---\n",
"\n",
"!pip install google-search-results\n",
"!apt install nodejs\n",
"!npm install crux-api\n",
"!npm install [email protected]"
],
"metadata": {
"id": "090N8rDXEJmC",
"cellView": "form"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@markdown # Step 2: Creating a node.js script for retrieving CrUX data\n",
"#@markdown In this step, a Node.js script will be created to retrieve CrUX data. This data captures user experience metrics from actual browsers all around the world.\n",
"#@markdown\n",
"#@markdown If CrUX data for a particular URL is unavailable, the script will fetch the origin data from that URL. Please note that due to criteria such as public discoverability and significant user traffic, not all urls are included in its dataset.\n",
"#@markdown\n",
"#@markdown This step must be performed independently because a single Colab cell cannot support both JavaScript and Python scripts concurrently.\n",
"#@markdown\n",
"#@markdown ### Continue by running this cell.\n",
"#@markdown ---\n",
"\n",
"%%writefile script.mjs\n",
"const fs = require('fs');\n",
"const createQueryRecord = require('crux-api').createQueryRecord;\n",
"const nodeFetch = require('node-fetch');\n",
"const results = JSON.parse(fs.readFileSync('dict_results.json', 'utf8')).organic_results;\n",
"const urls = results.map(result => ({ url: result.link }));\n",
"let device = JSON.parse(fs.readFileSync('device.json', 'utf8')).device;\n",
"\n",
"if(device === 'mobile') {\n",
" device = 'phone';\n",
"}\n",
"\n",
"const queryRecord = createQueryRecord({ key: JSON.parse(fs.readFileSync('crux_key.json', 'utf8')).key, fetch: nodeFetch });\n",
"\n",
"async function getData(url) {\n",
" let res = await queryRecord({\n",
" url: url.url,\n",
" formFactor: device\n",
" });\n",
"\n",
" let dataType = 'URL';\n",
"\n",
" if (!res || Object.keys(res).length === 0) {\n",
" res = await queryRecord({\n",
" origin: url.url,\n",
" formFactor: device\n",
" });\n",
" dataType = 'Origin';\n",
" }\n",
"\n",
" return {\n",
" url: url.url,\n",
" dataType: dataType,\n",
" response: res\n",
" };\n",
"}\n",
"\n",
"async function main() {\n",
" const results = [];\n",
"\n",
" for (const url of urls) {\n",
" const data = await getData(url);\n",
" results.push(data);\n",
" }\n",
"\n",
" fs.writeFileSync('result.json', JSON.stringify(results, null, 2));\n",
"}\n",
"\n",
"main();"
],
"metadata": {
"id": "MSmVRITp3Tfx",
"cellView": "form"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!node script.mjs\n",
"\n",
"#@markdown # Step 3: Extract CrUX data from Google search\n",
"#@markdown Set up the parameters for the Google SerpApi and CrUX API calls to fetch the top organic results and retrieve the corresponding speed data metrics.\n",
"#@markdown\n",
"#@markdown **API keys**\n",
"\n",
"CRUX_KEY = '' #@param {type: \"string\"}\n",
"SERPAPI_KEY = '' #@param {type: \"string\"}\n",
"\n",
"#@markdown **Query parameters**\n",
"\n",
"QUERY = 'smart tv' #@param {type: \"string\"}\n",
"DEVICE = 'mobile' #@param [\"mobile\", \"desktop\", \"tablet\"]\n",
"GOOGLE_DOMAIN = 'United Kingdom' #@param [\"Algérie\",\"American Samoa\",\"Angola\",\"Anguilla\",\"Antigua and Barbuda\",\"Argentina\",\"Australia\",\"Azərbaycan\",\"België\",\"Belize\",\"Bénin\",\"Bolivia\",\"Bosna i Hercegovina\",\"Botswana\",\"Brasil\",\"British Virgin Islands\",\"Brunei\",\"Burkina Faso\",\"Burundi\",\"Cameroun\",\"Canada\",\"Centrafrique\",\"Česká republika\",\"Chile\",\"Colombia\",\"Cook Islands\",\"Costa Rica\",\"Côte d’Ivoire\",\"Crna Gora\",\"Cuba\",\"Danmark\",\"Deutschland\",\"Djibouti\",\"Dominica\",\"Ecuador\",\"Eesti\",\"El Salvador\",\"España\",\"Fiji\",\"France\",\"Gabon\",\"Ghana\",\"Gibraltar\",\"Grønland\",\"Guadeloupe\",\"Guatemala\",\"Guernsey\",\"Guyana\",\"Haïti\",\"Honduras\",\"Hrvatska\",\"India\",\"Indonesia\",\"International\",\"Ireland\",\"Ísland\",\"Isle of Man\",\"Italia\",\"Jamaica\",\"Jersey\",\"Kenya\",\"Kiribati\",\"Latvija\",\"Lesotho\",\"Liechtenstein\",\"Lietuvos\",\"Luxemburg\",\"Madagasikara\",\"Magyarország\",\"Malawi\",\"Malaysia\",\"Maldives\",\"Mali\",\"Malta\",\"Maroc\",\"Mauritius\",\"México\",\"Micronesia\",\"Moçambique\",\"Moldova\",\"Montserrat\",\"Namibia\",\"Nauru\",\"Nederland\",\"New Zealand\",\"Nicaragua\",\"Niger\",\"Nigeria\",\"Niue\",\"Norfolk Island\",\"Norge\",\"O’zbekiston\",\"Österreich\",\"Pakistan\",\"Panamá\",\"Paraguay\",\"Perú\",\"Pilipinas\",\"Pitcairn Islands\",\"Polska\",\"Portugal\",\"Puerto Rico\",\"Rep. Dem. du Congo\",\"Rep. du Congo\",\"República Dominicana\",\"România\",\"Rwanda\",\"Saint Helena\",\"Saint Vincent and the Grenadines\",\"Samoa\",\"San Marino\",\"São Tomé e Príncipe\",\"Schweiz\",\"Sénégal\",\"Sesel\",\"Sierra Leone\",\"Singapore\",\"Slovenija\",\"Slovensko\",\"Solomon Islands\",\"Soomaaliya\",\"South Africa\",\"Sri Lanka\",\"Suomi\",\"Sverige\",\"Tajikistan\",\"Tanzania\",\"Tchad\",\"The Bahamas\",\"The Gambia\",\"Timor-Leste\",\"Togo\",\"Tokelau\",\"Tonga\",\"Trinidad and Tobago\",\"Türkiye\",\"Türkmenistan\",\"Uganda\",\"United Kingdom\",\"Uruguay\",\"Vanuatu\",\"Venezuela\",\"Việt Nam\",\"Virgin Islands\",\"Zambia\",\"Zimbabwe\",\"Ελλάδα\",\"Беларусь\",\"България\",\"Кыргызстан\",\"Қазақстан\",\"Македонија\",\"Монгол улс\",\"Россия\",\"Србија\",\"Україна\",\"საქართველო\",\"Հայաստան\",\"ישראל\",\"افغانستان\",\"الأراضي الفلسطينية\",\"الأردن\",\"الامارات العربية المتحدة\",\"البحرين\",\"السعودية\",\"العراق\",\"الكويت\",\"تونس\",\"عُمان\",\"قطر\",\"لبنان\",\"ليبيــا\",\"مصر\",\"ኢትዮጵያ\",\"नेपाल\",\"বাংলাদেশ\",\"ประเทศไทย\",\"ລາວ\",\"ព្រះរាជាណាចក្រកម្ពុជា\",\"한국\",\"中国\",\"台灣\",\"日本\",\"香港\"]\n",
"LOCATION = 'United Kingdom' #@param [\"Afghanistan\",\"Albania\",\"Algeria\",\"American Samoa\",\"Andorra\",\"Angola\",\"Anguilla\",\"Antarctica\",\"Antigua and Barbuda\",\"Argentina\",\"Armenia\",\"Aruba\",\"Australia\",\"Austria\",\"Azerbaijan\",\"Bahamas\",\"Bahrain\",\"Bangladesh\",\"Barbados\",\"Belarus\",\"Belgium\",\"Belize\",\"Benin\",\"Bermuda\",\"Bhutan\",\"Bolivia\",\"Bosnia and Herzegovina\",\"Botswana\",\"Bouvet Island\",\"Brazil\",\"British Indian Ocean Territory\",\"Brunei Darussalam\",\"Bulgaria\",\"Burkina Faso\",\"Burundi\",\"Cambodia\",\"Cameroon\",\"Canada\",\"Cape Verde\",\"Cayman Islands\",\"Central African Republic\",\"Chad\",\"Chile\",\"China\",\"Christmas Island\",\"Cocos (Keeling) Islands\",\"Colombia\",\"Comoros\",\"Congo\",\"Congo - Democratic Republic\",\"Cook Islands\",\"Costa Rica\",\"Cote d'Ivoire\",\"Croatia\",\"Cyprus\",\"Czech Republic\",\"Denmark\",\"Djibouti\",\"Dominica\",\"Dominican Republic\",\"East Timor\",\"Ecuador\",\"Egypt\",\"El Salvador\",\"Equatorial Guinea\",\"Eritrea\",\"Estonia\",\"Ethiopia\",\"Falkland Islands (Malvinas)\",\"Faroe Islands\",\"Fiji\",\"Finland\",\"France\",\"French Guiana\",\"French Polynesia\",\"French Southern Territories\",\"Gabon\",\"Gambia\",\"Georgia\",\"Germany\",\"Ghana\",\"Gibraltar\",\"Greece\",\"Greenland\",\"Grenada\",\"Guadeloupe\",\"Guam\",\"Guatemala\",\"Guinea\",\"Guinea - Bissau\",\"Guyana\",\"Haiti\",\"Heard and McDonald Islands\",\"Honduras\",\"Hong Kong\",\"Hungary\",\"Iceland\",\"India\",\"Indonesia\",\"Iraq\",\"Ireland\",\"Israel\",\"Italy\",\"Jamaica\",\"Japan\",\"Jordan\",\"Kazakhstan\",\"Kenya\",\"Kiribati\",\"Kuwait\",\"Kyrgyzstan\",\"Lao People's Democratic Republic\",\"Latvia\",\"Lebanon\",\"Lesotho\",\"Liberia\",\"Libya\",\"Liechtenstein\",\"Lithuania\",\"Luxembourg\",\"Macau\",\"Macedonia\",\"Madagascar\",\"Malawi\",\"Malaysia\",\"Maldives\",\"Mali\",\"Malta\",\"Marshall Islands\",\"Martinique\",\"Mauritania\",\"Mauritius\",\"Mayotte\",\"Mexico\",\"Micronesia\",\"Moldova\",\"Monaco\",\"Mongolia\",\"Montserrat\",\"Morocco\",\"Mozambique\",\"Namibia\",\"Nauru\",\"Nepal\",\"Netherlands\",\"Netherlands Antilles\",\"New Caledonia\",\"New Zealand\",\"Nicaragua\",\"Niger\",\"Nigeria\",\"Niue\",\"Norfolk Island\",\"Northern Mariana Islands\",\"Norway\",\"Oman\",\"Pakistan\",\"Palau\",\"Palestinian Territory\",\"Panama\",\"Papua New Guinea\",\"Paraguay\",\"Peru\",\"Philippines\",\"Pitcairn\",\"Poland\",\"Portugal\",\"Puerto Rico\",\"Qatar\",\"Reunion\",\"Romania\",\"Russian Federation\",\"Rwanda\",\"Saint Kitts and Nevis\",\"Saint Lucia\",\"Saint Vincent and the Grenadines\",\"Samoa\",\"San Marino\",\"Sao Tome and Principe\",\"Saudi Arabia\",\"Senegal\",\"Serbia and Montenegro\",\"Seychelles\",\"Sierra Leone\",\"Singapore\",\"Slovakia\",\"Slovenia\",\"Solomon Islands\",\"Somalia\",\"South Africa\",\"South Georgia and The South Sandwich Islands\",\"South Korea\",\"Spain\",\"Sri Lanka\",\"St. Helena\",\"St. Pierre and Miquelon\",\"Suriname\",\"Svalbard and Jan Mayen Islands\",\"Swaziland\",\"Sweden\",\"Switzerland\",\"Taiwan\",\"Tajikistan\",\"Tanzania\",\"Thailand\",\"Togo\",\"Tokelau\",\"Tonga\",\"Trinidad and Tobago\",\"Tunisia\",\"Turkey\",\"Turkmenistan\",\"Turks and Caicos Islands\",\"Tuvalu\",\"Uganda\",\"Ukraine\",\"United Arab Emirates\",\"United Kingdom\",\"United States\",\"United States Minor Outlying Islands\",\"Uruguay\",\"Uzbekistan\",\"Vanuatu\",\"Vatican\",\"Venezuela\",\"Viet Nam\",\"Virgin Islands (British)\",\"Virgin Islands (U.S.)\",\"Wallis and Futuna Islands\",\"Western Sahara\",\"Yemen\",\"Zambia\",\"Zimbabwe\"]\n",
"LANGUAGE = 'English' #@param [\"Afrikaans\",\"Albanian\",\"Amharic\",\"Arabic\",\"Armenian\",\"Azerbaijani\",\"Basque\",\"Belarusian\",\"Bengali\",\"Bihari\",\"Bosnian\",\"Breton\",\"Bulgarian\",\"Cambodian\",\"Catalan\",\"Chinese (Simplified)\",\"Chinese (Traditional)\",\"Corsican\",\"Croatian\",\"Croatian\",\"Czech\",\"Danish\",\"Dutch\",\"English\",\"Esperanto\",\"Estonian\",\"Faroese\",\"Filipino\",\"Finnish\",\"French\",\"Frisian\",\"Galician\",\"Georgian\",\"German\",\"Greek\",\"Guarani\",\"Gujarati\",\"Hausa\",\"Hebrew\",\"Hindi\",\"Hungarian\",\"Icelandic\",\"Indonesian\",\"Interlingua\",\"Irish\",\"Italian\",\"Japanese\",\"Javanese\",\"Kannada\",\"Kazakh\",\"Kinyarwanda\",\"Kirundi\",\"Korean\",\"Kurdish\",\"Kyrgyz\",\"Laothian\",\"Latin\",\"Latvian\",\"Lingala\",\"Lithuanian\",\"Macedonian\",\"Malagasy\",\"Malay\",\"Malayalam\",\"Maltese\",\"Maori\",\"Marathi\",\"Moldavian\",\"Mongolian\",\"Montenegrin\",\"Nepali\",\"Norwegian\",\"Norwegian (Nynorsk)\",\"Occitan\",\"Oriya\",\"Oromo\",\"Pashto\",\"Persian\",\"Polish\",\"Portuguese (Brazil)\",\"Portuguese (Portugal)\",\"Punjabi\",\"Quechua\",\"Romanian\",\"Romansh\",\"Russian\",\"Scots Gaelic\",\"Serbian\",\"Sesotho\",\"Shona\",\"Sindhi\",\"Sinhalese\",\"Slovak\",\"Slovenian\",\"Somali\",\"Spanish\",\"Sundanese\",\"Swahili\",\"Swedish\",\"Tajik\",\"Tamil\",\"Tatar\",\"Telugu\",\"Thai\",\"Tigrinya\",\"Tonga\",\"Turkish\",\"Turkmen\",\"Twi\",\"Uighur\",\"Ukrainian\",\"Urdu\",\"Uzbek\",\"Vietnamese\",\"Welsh\",\"Xhosa\",\"Yiddish\",\"Yoruba\",\"Zulu\"]\n",
"\n",
"google = {\"Algérie\":\"google.dz\",\"American Samoa\":\"google.as\",\"Angola\":\"google.co.ao\",\"Anguilla\":\"google.com.ai\",\"Antigua and Barbuda\":\"google.com.ag\",\"Argentina\":\"google.com.ar\",\"Australia\":\"google.com.au\",\"Azərbaycan\":\"google.az\",\"België\":\"google.be\",\"Belize\":\"google.com.bz\",\"Bénin\":\"google.bj\",\"Bolivia\":\"google.com.bo\",\"Bosna i Hercegovina\":\"google.ba\",\"Botswana\":\"google.co.bw\",\"Brasil\":\"google.com.br\",\"British Virgin Islands\":\"google.vg\",\"Brunei\":\"google.com.bn\",\"Burkina Faso\":\"google.bf\",\"Burundi\":\"google.bi\",\"Cameroun\":\"google.cm\",\"Canada\":\"google.ca\",\"Centrafrique\":\"google.cf\",\"Česká republika\":\"google.cz\",\"Chile\":\"google.cl\",\"Colombia\":\"google.com.co\",\"Cook Islands\":\"google.co.ck\",\"Costa Rica\":\"google.co.cr\",\"Côte d’Ivoire\":\"google.ci\",\"Crna Gora\":\"google.me\",\"Cuba\":\"google.com.cu\",\"Danmark\":\"google.dk\",\"Deutschland\":\"google.de\",\"Djibouti\":\"google.dj\",\"Dominica\":\"google.dm\",\"Ecuador\":\"google.com.ec\",\"Eesti\":\"google.ee\",\"El Salvador\":\"google.com.sv\",\"España\":\"google.es\",\"Fiji\":\"google.com.fj\",\"France\":\"google.fr\",\"Gabon\":\"google.ga\",\"Ghana\":\"google.com.gh\",\"Gibraltar\":\"google.com.gi\",\"Grønland\":\"google.gl\",\"Guadeloupe\":\"google.gp\",\"Guatemala\":\"google.com.gt\",\"Guernsey\":\"google.gg\",\"Guyana\":\"google.gy\",\"Haïti\":\"google.ht\",\"Honduras\":\"google.hn\",\"Hrvatska\":\"google.hr\",\"India\":\"google.co.in\",\"Indonesia\":\"google.co.id\",\"International\":\"google.com\",\"Ireland\":\"google.ie\",\"Ísland\":\"google.is\",\"Isle of Man\":\"google.im\",\"Italia\":\"google.it\",\"Jamaica\":\"google.com.jm\",\"Jersey\":\"google.je\",\"Kenya\":\"google.co.ke\",\"Kiribati\":\"google.ki\",\"Latvija\":\"google.lv\",\"Lesotho\":\"google.co.ls\",\"Liechtenstein\":\"google.li\",\"Lietuvos\":\"google.lt\",\"Luxemburg\":\"google.lu\",\"Madagasikara\":\"google.mg\",\"Magyarország\":\"google.hu\",\"Malawi\":\"google.mw\",\"Malaysia\":\"google.com.my\",\"Maldives\":\"google.mv\",\"Mali\":\"google.ml\",\"Malta\":\"google.com.mt\",\"Maroc\":\"google.co.ma\",\"Mauritius\":\"google.mu\",\"México\":\"google.com.mx\",\"Micronesia\":\"google.fm\",\"Moçambique\":\"google.co.mz\",\"Moldova\":\"google.md\",\"Montserrat\":\"google.ms\",\"Namibia\":\"google.com.na\",\"Nauru\":\"google.nr\",\"Nederland\":\"google.nl\",\"New Zealand\":\"google.co.nz\",\"Nicaragua\":\"google.com.ni\",\"Niger\":\"google.ne\",\"Nigeria\":\"google.com.ng\",\"Niue\":\"google.nu\",\"Norfolk Island\":\"google.com.nf\",\"Norge\":\"google.no\",\"O’zbekiston\":\"google.co.uz\",\"Österreich\":\"google.at\",\"Pakistan\":\"google.com.pk\",\"Panamá\":\"google.com.pa\",\"Paraguay\":\"google.com.py\",\"Perú\":\"google.com.pe\",\"Pilipinas\":\"google.com.ph\",\"Pitcairn Islands\":\"google.pn\",\"Polska\":\"google.pl\",\"Portugal\":\"google.pt\",\"Puerto Rico\":\"google.com.pr\",\"Rep. Dem. du Congo\":\"google.cd\",\"Rep. du Congo\":\"google.cg\",\"República Dominicana\":\"google.com.do\",\"România\":\"google.ro\",\"Rwanda\":\"google.rw\",\"Saint Helena\":\"google.sh\",\"Saint Vincent and the Grenadines\":\"google.com.vc\",\"Samoa\":\"google.ws\",\"San Marino\":\"google.sm\",\"São Tomé e Príncipe\":\"google.st\",\"Schweiz\":\"google.ch\",\"Sénégal\":\"google.sn\",\"Sesel\":\"google.sc\",\"Sierra Leone\":\"google.com.sl\",\"Singapore\":\"google.com.sg\",\"Slovenija\":\"google.si\",\"Slovensko\":\"google.sk\",\"Solomon Islands\":\"google.com.sb\",\"Soomaaliya\":\"google.so\",\"South Africa\":\"google.co.za\",\"Sri Lanka\":\"google.lk\",\"Suomi\":\"google.fi\",\"Sverige\":\"google.se\",\"Tajikistan\":\"google.com.tj\",\"Tanzania\":\"google.co.tz\",\"Tchad\":\"google.td\",\"The Bahamas\":\"google.bs\",\"The Gambia\":\"google.gm\",\"Timor-Leste\":\"google.tl\",\"Togo\":\"google.tg\",\"Tokelau\":\"google.tk\",\"Tonga\":\"google.to\",\"Trinidad and Tobago\":\"google.tt\",\"Türkiye\":\"google.com.tr\",\"Türkmenistan\":\"google.tm\",\"Uganda\":\"google.co.ug\",\"United Kingdom\":\"google.co.uk\",\"Uruguay\":\"google.com.uy\",\"Vanuatu\":\"google.vu\",\"Venezuela\":\"google.co.ve\",\"Việt Nam\":\"google.com.vn\",\"Virgin Islands\":\"google.co.vi\",\"Zambia\":\"google.co.zm\",\"Zimbabwe\":\"google.co.zw\",\"Ελλάδα\":\"google.gr\",\"Беларусь\":\"google.by\",\"България\":\"google.bg\",\"Кыргызстан\":\"google.kg\",\"Қазақстан\":\"google.kz\",\"Македонија\":\"google.mk\",\"Монгол улс\":\"google.mn\",\"Россия\":\"google.ru\",\"Србија\":\"google.rs\",\"Україна\":\"google.com.ua\",\"საქართველო\":\"google.ge\",\"Հայաստան\":\"google.am\",\"ישראל\":\"google.co.il\",\"افغانستان\":\"google.com.af\",\"الأراضي الفلسطينية\":\"google.ps\",\"الأردن\":\"google.jo\",\"الامارات العربية المتحدة\":\"google.ae\",\"البحرين\":\"google.com.bh\",\"السعودية\":\"google.com.sa\",\"العراق\":\"google.iq\",\"الكويت\":\"google.com.kw\",\"تونس\":\"google.tn\",\"عُمان\":\"google.com.om\",\"قطر\":\"google.com.qa\",\"لبنان\":\"google.com.lb\",\"ليبيــا\":\"google.com.ly\",\"مصر\":\"google.com.eg\",\"ኢትዮጵያ\":\"google.com.et\",\"नेपाल\":\"google.com.np\",\"বাংলাদেশ\":\"google.com.bd\",\"ประเทศไทย\":\"google.co.th\",\"ລາວ\":\"google.la\",\"ព្រះរាជាណាចក្រកម្ពុជា\":\"google.com.kh\",\"한국\":\"google.co.kr\",\"中国\":\"google.com.hk\",\"台灣\":\"google.com.tw\",\"日本\":\"google.co.jp\",\"香港\":\"google.com.hk\"}\n",
"location = {\"Afghanistan\":\"AF\",\"Albania\":\"AL\",\"Algeria\":\"DZ\",\"American Samoa\":\"AS\",\"Andorra\":\"AD\",\"Angola\":\"AO\",\"Anguilla\":\"AI\",\"Antarctica\":\"AQ\",\"Antigua and Barbuda\":\"AG\",\"Argentina\":\"AR\",\"Armenia\":\"AM\",\"Aruba\":\"AW\",\"Australia\":\"AU\",\"Austria\":\"AT\",\"Azerbaijan\":\"AZ\",\"Bahamas\":\"BS\",\"Bahrain\":\"BH\",\"Bangladesh\":\"BD\",\"Barbados\":\"BB\",\"Belarus\":\"BY\",\"Belgium\":\"BE\",\"Belize\":\"BZ\",\"Benin\":\"BJ\",\"Bermuda\":\"BM\",\"Bhutan\":\"BT\",\"Bolivia\":\"BO\",\"Bosnia and Herzegovina\":\"BA\",\"Botswana\":\"BW\",\"Bouvet Island\":\"BV\",\"Brazil\":\"BR\",\"British Indian Ocean Territory\":\"IO\",\"Brunei Darussalam\":\"BN\",\"Bulgaria\":\"BG\",\"Burkina Faso\":\"BF\",\"Burundi\":\"BI\",\"Cambodia\":\"KH\",\"Cameroon\":\"CM\",\"Canada\":\"CA\",\"Cape Verde\":\"CV\",\"Cayman Islands\":\"KY\",\"Central African Republic\":\"CF\",\"Chad\":\"TD\",\"Chile\":\"CL\",\"China\":\"CN\",\"Christmas Island\":\"CX\",\"Cocos (Keeling) Islands\":\"CC\",\"Colombia\":\"CO\",\"Comoros\":\"KM\",\"Congo\":\"CG\",\"Congo - Democratic Republic\":\"CD\",\"Cook Islands\":\"CK\",\"Costa Rica\":\"CR\",\"Cote d'Ivoire\":\"CI\",\"Croatia\":\"HR\",\"Cyprus\":\"CY\",\"Czech Republic\":\"CZ\",\"Denmark\":\"DK\",\"Djibouti\":\"DJ\",\"Dominica\":\"DM\",\"Dominican Republic\":\"DO\",\"East Timor\":\"TL\",\"Ecuador\":\"EC\",\"Egypt\":\"EG\",\"El Salvador\":\"SV\",\"Equatorial Guinea\":\"GQ\",\"Eritrea\":\"ER\",\"Estonia\":\"EE\",\"Ethiopia\":\"ET\",\"Falkland Islands (Malvinas)\":\"FK\",\"Faroe Islands\":\"FO\",\"Fiji\":\"FJ\",\"Finland\":\"FI\",\"France\":\"FR\",\"French Guiana\":\"GF\",\"French Polynesia\":\"PF\",\"French Southern Territories\":\"TF\",\"Gabon\":\"GA\",\"Gambia\":\"GM\",\"Georgia\":\"GE\",\"Germany\":\"DE\",\"Ghana\":\"GH\",\"Gibraltar\":\"GI\",\"Greece\":\"GR\",\"Greenland\":\"GL\",\"Grenada\":\"GD\",\"Guadeloupe\":\"GP\",\"Guam\":\"GU\",\"Guatemala\":\"GT\",\"Guinea\":\"GN\",\"Guinea - Bissau\":\"GW\",\"Guyana\":\"GY\",\"Haiti\":\"HT\",\"Heard and McDonald Islands\":\"HM\",\"Honduras\":\"HN\",\"Hong Kong\":\"HK\",\"Hungary\":\"HU\",\"Iceland\":\"IS\",\"India\":\"IN\",\"Indonesia\":\"ID\",\"Iraq\":\"IQ\",\"Ireland\":\"IE\",\"Israel\":\"IL\",\"Italy\":\"IT\",\"Jamaica\":\"JM\",\"Japan\":\"JP\",\"Jordan\":\"JO\",\"Kazakhstan\":\"KZ\",\"Kenya\":\"KE\",\"Kiribati\":\"KI\",\"Kuwait\":\"KW\",\"Kyrgyzstan\":\"KG\",\"Lao People's Democratic Republic\":\"LA\",\"Latvia\":\"LV\",\"Lebanon\":\"LB\",\"Lesotho\":\"LS\",\"Liberia\":\"LR\",\"Libya\":\"LY\",\"Liechtenstein\":\"LI\",\"Lithuania\":\"LT\",\"Luxembourg\":\"LU\",\"Macau\":\"MO\",\"Macedonia\":\"MK\",\"Madagascar\":\"MG\",\"Malawi\":\"MW\",\"Malaysia\":\"MY\",\"Maldives\":\"MV\",\"Mali\":\"ML\",\"Malta\":\"MT\",\"Marshall Islands\":\"MH\",\"Martinique\":\"MQ\",\"Mauritania\":\"MR\",\"Mauritius\":\"MU\",\"Mayotte\":\"YT\",\"Mexico\":\"MX\",\"Micronesia\":\"FM\",\"Moldova\":\"MD\",\"Monaco\":\"MC\",\"Mongolia\":\"MN\",\"Montserrat\":\"MS\",\"Morocco\":\"MA\",\"Mozambique\":\"MZ\",\"Namibia\":\"NA\",\"Nauru\":\"NR\",\"Nepal\":\"NP\",\"Netherlands\":\"NL\",\"Netherlands Antilles\":\"AN\",\"New Caledonia\":\"NC\",\"New Zealand\":\"NZ\",\"Nicaragua\":\"NI\",\"Niger\":\"NE\",\"Nigeria\":\"NG\",\"Niue\":\"NU\",\"Norfolk Island\":\"NF\",\"Northern Mariana Islands\":\"MP\",\"Norway\":\"NO\",\"Oman\":\"OM\",\"Pakistan\":\"PK\",\"Palau\":\"PW\",\"Palestinian Territory\":\"PS\",\"Panama\":\"PA\",\"Papua New Guinea\":\"PG\",\"Paraguay\":\"PY\",\"Peru\":\"PE\",\"Philippines\":\"PH\",\"Pitcairn\":\"PN\",\"Poland\":\"PL\",\"Portugal\":\"PT\",\"Puerto Rico\":\"PR\",\"Qatar\":\"QA\",\"Reunion\":\"RE\",\"Romania\":\"RO\",\"Russian Federation\":\"RU\",\"Rwanda\":\"RW\",\"Saint Kitts and Nevis\":\"KN\",\"Saint Lucia\":\"LC\",\"Saint Vincent and the Grenadines\":\"VC\",\"Samoa\":\"WS\",\"San Marino\":\"SM\",\"Sao Tome and Principe\":\"ST\",\"Saudi Arabia\":\"SA\",\"Senegal\":\"SN\",\"Serbia and Montenegro\":\"CS\",\"Seychelles\":\"SC\",\"Sierra Leone\":\"SL\",\"Singapore\":\"SG\",\"Slovakia\":\"SK\",\"Slovenia\":\"SI\",\"Solomon Islands\":\"SB\",\"Somalia\":\"SO\",\"South Africa\":\"ZA\",\"South Georgia and The South Sandwich Islands\":\"GS\",\"South Korea\":\"KR\",\"Spain\":\"ES\",\"Sri Lanka\":\"LK\",\"St. Helena\":\"SH\",\"St. Pierre and Miquelon\":\"PM\",\"Suriname\":\"SR\",\"Svalbard and Jan Mayen Islands\":\"SJ\",\"Swaziland\":\"SZ\",\"Sweden\":\"SE\",\"Switzerland\":\"CH\",\"Taiwan\":\"TW\",\"Tajikistan\":\"TJ\",\"Tanzania\":\"TZ\",\"Thailand\":\"TH\",\"Togo\":\"TG\",\"Tokelau\":\"TK\",\"Tonga\":\"TO\",\"Trinidad and Tobago\":\"TT\",\"Tunisia\":\"TN\",\"Turkey\":\"TR\",\"Turkmenistan\":\"TM\",\"Turks and Caicos Islands\":\"TC\",\"Tuvalu\":\"TV\",\"Uganda\":\"UG\",\"Ukraine\":\"UA\",\"United Arab Emirates\":\"AE\",\"United Kingdom\":\"GB\",\"United States\":\"US\",\"United States Minor Outlying Islands\":\"UM\",\"Uruguay\":\"UY\",\"Uzbekistan\":\"UZ\",\"Vanuatu\":\"VU\",\"Vatican\":\"VA\",\"Venezuela\":\"VE\",\"Viet Nam\":\"VN\",\"Virgin Islands (British)\":\"VG\",\"Virgin Islands (U.S.)\":\"VI\",\"Wallis and Futuna Islands\":\"WF\",\"Western Sahara\":\"EH\",\"Yemen\":\"YE\",\"Zambia\":\"ZM\",\"Zimbabwe\":\"ZW\"}\n",
"language = {\"Afrikaans\":\"af\",\"Albanian\":\"sq\",\"Amharic\":\"am\",\"Arabic\":\"ar\",\"Armenian\":\"hy\",\"Azerbaijani\":\"az\",\"Basque\":\"eu\",\"Belarusian\":\"be\",\"Bengali\":\"bn\",\"Bihari\":\"bh\",\"Bosnian\":\"bs\",\"Breton\":\"br\",\"Bulgarian\":\"bg\",\"Cambodian\":\"km\",\"Catalan\":\"ca\",\"Chinese (Simplified)\":\"zh-CN\",\"Chinese (Traditional)\":\"zh-TW\",\"Corsican\":\"co\",\"Croatian\":\"hr\",\"Croatian\":\"sh\",\"Czech\":\"cs\",\"Danish\":\"da\",\"Dutch\":\"nl\",\"English\":\"en\",\"Esperanto\":\"eo\",\"Estonian\":\"et\",\"Faroese\":\"fo\",\"Filipino\":\"tl\",\"Finnish\":\"fi\",\"French\":\"fr\",\"Frisian\":\"fy\",\"Galician\":\"gl\",\"Georgian\":\"ka\",\"German\":\"de\",\"Greek\":\"el\",\"Guarani\":\"gn\",\"Gujarati\":\"gu\",\"Hausa\":\"ha\",\"Hebrew\":\"iw\",\"Hindi\":\"hi\",\"Hungarian\":\"hu\",\"Icelandic\":\"is\",\"Indonesian\":\"id\",\"Interlingua\":\"ia\",\"Irish\":\"ga\",\"Italian\":\"it\",\"Japanese\":\"ja\",\"Javanese\":\"jw\",\"Kannada\":\"kn\",\"Kazakh\":\"kk\",\"Kinyarwanda\":\"rw\",\"Kirundi\":\"rn\",\"Korean\":\"ko\",\"Kurdish\":\"ku\",\"Kyrgyz\":\"ky\",\"Laothian\":\"lo\",\"Latin\":\"la\",\"Latvian\":\"lv\",\"Lingala\":\"ln\",\"Lithuanian\":\"lt\",\"Macedonian\":\"mk\",\"Malagasy\":\"mg\",\"Malay\":\"ms\",\"Malayalam\":\"ml\",\"Maltese\":\"mt\",\"Maori\":\"mi\",\"Marathi\":\"mr\",\"Moldavian\":\"mo\",\"Mongolian\":\"mn\",\"Montenegrin\":\"sr-ME\",\"Nepali\":\"ne\",\"Norwegian\":\"no\",\"Norwegian (Nynorsk)\":\"nn\",\"Occitan\":\"oc\",\"Oriya\":\"or\",\"Oromo\":\"om\",\"Pashto\":\"ps\",\"Persian\":\"fa\",\"Polish\":\"pl\",\"Portuguese (Brazil)\":\"pt-BR\",\"Portuguese (Portugal)\":\"pt-PT\",\"Punjabi\":\"pa\",\"Quechua\":\"qu\",\"Romanian\":\"ro\",\"Romansh\":\"rm\",\"Russian\":\"ru\",\"Scots Gaelic\":\"gd\",\"Serbian\":\"sr\",\"Sesotho\":\"st\",\"Shona\":\"sn\",\"Sindhi\":\"sd\",\"Sinhalese\":\"si\",\"Slovak\":\"sk\",\"Slovenian\":\"sl\",\"Somali\":\"so\",\"Spanish\":\"es\",\"Sundanese\":\"su\",\"Swahili\":\"sw\",\"Swedish\":\"sv\",\"Tajik\":\"tg\",\"Tamil\":\"ta\",\"Tatar\":\"tt\",\"Telugu\":\"te\",\"Thai\":\"th\",\"Tigrinya\":\"ti\",\"Tonga\":\"to\",\"Turkish\":\"tr\",\"Turkmen\":\"tk\",\"Twi\":\"tw\",\"Uighur\":\"ug\",\"Ukrainian\":\"uk\",\"Urdu\":\"ur\",\"Uzbek\":\"uz\",\"Vietnamese\":\"vi\",\"Welsh\":\"cy\",\"Xhosa\":\"xh\",\"Yiddish\":\"yi\",\"Yoruba\":\"yo\",\"Zulu\":\"zu\"}\n",
"\n",
"from serpapi import GoogleSearch\n",
"import json\n",
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"params = {\n",
" \"q\": QUERY,\n",
" \"api_key\": SERPAPI_KEY,\n",
" \"engine\": \"google\",\n",
" \"google_domain\": google[GOOGLE_DOMAIN],\n",
" \"gl\": location[LOCATION],\n",
" \"hl\": language[LANGUAGE],\n",
" \"device\": DEVICE\n",
"}\n",
"\n",
"search = GoogleSearch(params)\n",
"dict_results = search.get_dict()\n",
"\n",
"organic_results = dict_results[\"organic_results\"]\n",
"\n",
"with open('dict_results.json', 'w') as file:\n",
" json.dump(dict_results, file)\n",
"\n",
"with open('device.json', 'w') as file:\n",
" json.dump({\"device\": params[\"device\"]}, file)\n",
"\n",
"with open('crux_key.json', 'w') as file:\n",
" json.dump({'key': CRUX_KEY}, file)\n",
"\n",
"data = []\n",
"\n",
"for result in organic_results:\n",
" position = result.get(\"position\", None)\n",
" url = result.get(\"link\", None)\n",
" title = result.get(\"title\", None)\n",
" description = result.get(\"snippet\", None)\n",
"\n",
" data.append([position, url, title, description])\n",
"\n",
"df = pd.DataFrame(data, columns=[\"Position\", \"URL\", \"Title\", \"Description\"])\n",
"\n",
"with open('result.json') as file:\n",
" data = json.load(file)\n",
"\n",
"table = []\n",
"\n",
"key_metrics = [\n",
" 'cumulative_layout_shift',\n",
" 'first_contentful_paint',\n",
" 'first_input_delay',\n",
" 'interaction_to_next_paint',\n",
" 'largest_contentful_paint',\n",
" 'experimental_time_to_first_byte'\n",
"]\n",
"\n",
"for element in data:\n",
" url = element['url']\n",
" dataType = element['dataType'] if 'dataType' in element else None\n",
" register = {'url': url, 'dataType': dataType}\n",
"\n",
"\n",
" if 'response' in element and element['response'] is not None:\n",
" record = element['response']['record']\n",
" key = record['key']\n",
" metrics = record['metrics']\n",
"\n",
" for metric in key_metrics:\n",
" if metric in metrics:\n",
" metric_data = metrics[metric]\n",
" histogram = metric_data['histogram']\n",
" percentiles = metric_data['percentiles']\n",
" p75 = percentiles['p75']\n",
"\n",
" for i, data in enumerate(histogram):\n",
" register[f'{metric}_{i}_start'] = data.get('start')\n",
" register[f'{metric}_{i}_end'] = data.get('end')\n",
" register[f'{metric}_{i}_density'] = data.get('density')\n",
"\n",
" register[f'{metric}'] = p75\n",
"\n",
" end_1 = histogram[0]['end']\n",
" end_2 = histogram[1]['end']\n",
" if end_1 is not None and float(p75) < float(end_1):\n",
" register[f'{metric}_status'] = 'Good'\n",
" elif end_2 is not None and float(p75) > float(end_2):\n",
" register[f'{metric}_status'] = 'Bad'\n",
" else:\n",
" register[f'{metric}_status'] = 'Need improvement'\n",
" else:\n",
" for i in range(2):\n",
" register[f'{metric}_{i}_start'] = None\n",
" register[f'{metric}_{i}_end'] = None\n",
" register[f'{metric}_{i}_density'] = None\n",
" register[f'{metric}'] = None\n",
" register[f'{metric}_status'] = None\n",
" else:\n",
" for metric in key_metrics:\n",
" for i in range(2):\n",
" register[f'{metric}_{i}_start'] = None\n",
" register[f'{metric}_{i}_end'] = None\n",
" register[f'{metric}_{i}_density'] = None\n",
" register[f'{metric}'] = None\n",
" register[f'{metric}_status'] = None\n",
"\n",
" table.append(register)\n",
"\n",
"df_crux = pd.DataFrame(table)\n",
"numeric_cols = df_crux.select_dtypes(include=[np.number]).columns\n",
"df_crux[numeric_cols] = df_crux[numeric_cols].where(pd.notnull(df_crux[numeric_cols]), None)\n",
"\n",
"df2 = df_crux.copy()\n",
"columns_to_remove = [col for col in df2.columns if 'start' in col or 'end' in col or 'density' in col]\n",
"df2.drop(columns=columns_to_remove, inplace=True)\n",
"df2 = df2.fillna(\"-\")\n",
"\n",
"for metric in key_metrics:\n",
" df2[metric] = df2[metric].astype(str) + \" (\" + df2[metric + '_status'] + \")\"\n",
"\n",
"status_columns = [metric + '_status' for metric in key_metrics]\n",
"df2.drop(columns=status_columns, inplace=True)\n",
"\n",
"merged_df = df.merge(df2, left_on='URL', right_on='url')\n",
"merged_df.drop('url', axis=1, inplace=True)\n",
"\n",
"merged_df = merged_df.rename(columns={\n",
" \"cumulative_layout_shift\": \"CLS\",\n",
" \"first_contentful_paint\": \"FCP\",\n",
" \"first_input_delay\": \"FID\",\n",
" \"interaction_to_next_paint\": \"INP\",\n",
" \"largest_contentful_paint\": \"LCP\",\n",
" \"experimental_interaction_to_next_paint\": \"EINP\",\n",
" \"experimental_time_to_first_byte\": \"TTFB\",\n",
"})\n",
"\n",
"def color_based_on_value(value):\n",
" if 'Good' in value:\n",
" color = 'green'\n",
" elif 'Need improvement' in value:\n",
" color = 'orange'\n",
" elif 'Bad' in value:\n",
" color = 'red'\n",
" else:\n",
" color = 'black'\n",
" return 'color: %s' % color\n",
"\n",
"styled_df = merged_df.style.applymap(color_based_on_value, subset=['CLS', 'FCP', 'FID', 'INP', 'LCP', 'TTFB'])\n",
"styled_df\n",
"\n",
"#@markdown ## After setting the parameters, click the 'Play' button\n",
"#@markdown ---"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 488
},
"id": "KcSNcXzKEvVZ",
"outputId": "0d313146-5faf-4bec-9799-6cfd9a8c6ace",
"cellView": "form"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<pandas.io.formats.style.Styler at 0x7f06ec1697e0>"
],
"text/html": [
"<style type=\"text/css\">\n",
"#T_ff94f_row0_col5, #T_ff94f_row0_col6, #T_ff94f_row0_col9, #T_ff94f_row1_col6, #T_ff94f_row1_col7, #T_ff94f_row1_col9, #T_ff94f_row4_col8, #T_ff94f_row4_col9, #T_ff94f_row5_col8, #T_ff94f_row6_col5, #T_ff94f_row7_col6, #T_ff94f_row7_col8, #T_ff94f_row7_col10, #T_ff94f_row8_col6, #T_ff94f_row8_col9, #T_ff94f_row8_col10 {\n",
" color: orange;\n",
"}\n",
"#T_ff94f_row0_col7, #T_ff94f_row1_col5, #T_ff94f_row1_col10, #T_ff94f_row2_col5, #T_ff94f_row2_col6, #T_ff94f_row2_col7, #T_ff94f_row2_col8, #T_ff94f_row2_col9, #T_ff94f_row2_col10, #T_ff94f_row3_col5, #T_ff94f_row3_col6, #T_ff94f_row3_col7, #T_ff94f_row3_col9, #T_ff94f_row3_col10, #T_ff94f_row4_col5, #T_ff94f_row4_col6, #T_ff94f_row4_col7, #T_ff94f_row4_col10, #T_ff94f_row5_col5, #T_ff94f_row5_col6, #T_ff94f_row5_col7, #T_ff94f_row5_col9, #T_ff94f_row5_col10, #T_ff94f_row6_col6, #T_ff94f_row6_col7, #T_ff94f_row6_col9, #T_ff94f_row6_col10, #T_ff94f_row7_col5, #T_ff94f_row7_col7, #T_ff94f_row7_col9, #T_ff94f_row8_col5, #T_ff94f_row8_col7 {\n",
" color: green;\n",
"}\n",
"#T_ff94f_row0_col8, #T_ff94f_row0_col10, #T_ff94f_row1_col8, #T_ff94f_row3_col8, #T_ff94f_row6_col8, #T_ff94f_row8_col8 {\n",
" color: red;\n",
"}\n",
"</style>\n",
"<table id=\"T_ff94f\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
" <th id=\"T_ff94f_level0_col0\" class=\"col_heading level0 col0\" >Position</th>\n",
" <th id=\"T_ff94f_level0_col1\" class=\"col_heading level0 col1\" >URL</th>\n",
" <th id=\"T_ff94f_level0_col2\" class=\"col_heading level0 col2\" >Title</th>\n",
" <th id=\"T_ff94f_level0_col3\" class=\"col_heading level0 col3\" >Description</th>\n",
" <th id=\"T_ff94f_level0_col4\" class=\"col_heading level0 col4\" >dataType</th>\n",
" <th id=\"T_ff94f_level0_col5\" class=\"col_heading level0 col5\" >CLS</th>\n",
" <th id=\"T_ff94f_level0_col6\" class=\"col_heading level0 col6\" >FCP</th>\n",
" <th id=\"T_ff94f_level0_col7\" class=\"col_heading level0 col7\" >FID</th>\n",
" <th id=\"T_ff94f_level0_col8\" class=\"col_heading level0 col8\" >INP</th>\n",
" <th id=\"T_ff94f_level0_col9\" class=\"col_heading level0 col9\" >LCP</th>\n",
" <th id=\"T_ff94f_level0_col10\" class=\"col_heading level0 col10\" >TTFB</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
" <td id=\"T_ff94f_row0_col0\" class=\"data row0 col0\" >1</td>\n",
" <td id=\"T_ff94f_row0_col1\" class=\"data row0 col1\" >https://www.currys.co.uk/tv-and-audio/televisions/tvs/smart-tvs</td>\n",
" <td id=\"T_ff94f_row0_col2\" class=\"data row0 col2\" >Smart TVs - Cheap Smart TV Deals</td>\n",
" <td id=\"T_ff94f_row0_col3\" class=\"data row0 col3\" >Find the perfect Smart TVs for you in our full TVs range here at Currys. Shop online for delivery or order & collect.</td>\n",
" <td id=\"T_ff94f_row0_col4\" class=\"data row0 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row0_col5\" class=\"data row0 col5\" >0.25 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row0_col6\" class=\"data row0 col6\" >2653 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row0_col7\" class=\"data row0 col7\" >69 (Good)</td>\n",
" <td id=\"T_ff94f_row0_col8\" class=\"data row0 col8\" >556 (Bad)</td>\n",
" <td id=\"T_ff94f_row0_col9\" class=\"data row0 col9\" >3936 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row0_col10\" class=\"data row0 col10\" >2008 (Bad)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
" <td id=\"T_ff94f_row1_col0\" class=\"data row1 col0\" >2</td>\n",
" <td id=\"T_ff94f_row1_col1\" class=\"data row1 col1\" >https://www.argos.co.uk/sd/smart-tv/</td>\n",
" <td id=\"T_ff94f_row1_col2\" class=\"data row1 col2\" >Results for smart tv</td>\n",
" <td id=\"T_ff94f_row1_col3\" class=\"data row1 col3\" >Get set for smart tv at Argos. Same Day delivery 7 days a week, or fast store collection.</td>\n",
" <td id=\"T_ff94f_row1_col4\" class=\"data row1 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row1_col5\" class=\"data row1 col5\" >0.06 (Good)</td>\n",
" <td id=\"T_ff94f_row1_col6\" class=\"data row1 col6\" >1875 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row1_col7\" class=\"data row1 col7\" >103 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row1_col8\" class=\"data row1 col8\" >890 (Bad)</td>\n",
" <td id=\"T_ff94f_row1_col9\" class=\"data row1 col9\" >2834 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row1_col10\" class=\"data row1 col10\" >613 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
" <td id=\"T_ff94f_row2_col0\" class=\"data row2 col0\" >3</td>\n",
" <td id=\"T_ff94f_row2_col1\" class=\"data row2 col1\" >https://www.amazon.co.uk/smart-tv/s?k=smart+tv</td>\n",
" <td id=\"T_ff94f_row2_col2\" class=\"data row2 col2\" >Amazon.co.uk: Smart Tv</td>\n",
" <td id=\"T_ff94f_row2_col3\" class=\"data row2 col3\" >Hisense 50A6BGTUK 4K UHD Smart TV, with Dolby Vision HDR, DTS Virtual X, Youtube, Netflix, Disney +, Freeview Play and Alexa ...</td>\n",
" <td id=\"T_ff94f_row2_col4\" class=\"data row2 col4\" >Origin</td>\n",
" <td id=\"T_ff94f_row2_col5\" class=\"data row2 col5\" >0.08 (Good)</td>\n",
" <td id=\"T_ff94f_row2_col6\" class=\"data row2 col6\" >1031 (Good)</td>\n",
" <td id=\"T_ff94f_row2_col7\" class=\"data row2 col7\" >23 (Good)</td>\n",
" <td id=\"T_ff94f_row2_col8\" class=\"data row2 col8\" >187 (Good)</td>\n",
" <td id=\"T_ff94f_row2_col9\" class=\"data row2 col9\" >1686 (Good)</td>\n",
" <td id=\"T_ff94f_row2_col10\" class=\"data row2 col10\" >552 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
" <td id=\"T_ff94f_row3_col0\" class=\"data row3 col0\" >4</td>\n",
" <td id=\"T_ff94f_row3_col1\" class=\"data row3 col1\" >https://ao.com/tv-and-entertainment/tvs</td>\n",
" <td id=\"T_ff94f_row3_col2\" class=\"data row3 col2\" >TV Deals on 4K, 8k, 0LED & QLED Televisions</td>\n",
" <td id=\"T_ff94f_row3_col3\" class=\"data row3 col3\" >TV deals 2022 plus price match promise on a wide range of 4K Ultra HD TVs, Smart TVs, QLED & OLED televisions. Next day delivery, 7 days a week.</td>\n",
" <td id=\"T_ff94f_row3_col4\" class=\"data row3 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row3_col5\" class=\"data row3 col5\" >0.09 (Good)</td>\n",
" <td id=\"T_ff94f_row3_col6\" class=\"data row3 col6\" >1222 (Good)</td>\n",
" <td id=\"T_ff94f_row3_col7\" class=\"data row3 col7\" >22 (Good)</td>\n",
" <td id=\"T_ff94f_row3_col8\" class=\"data row3 col8\" >517 (Bad)</td>\n",
" <td id=\"T_ff94f_row3_col9\" class=\"data row3 col9\" >1915 (Good)</td>\n",
" <td id=\"T_ff94f_row3_col10\" class=\"data row3 col10\" >385 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
" <td id=\"T_ff94f_row4_col0\" class=\"data row4 col0\" >5</td>\n",
" <td id=\"T_ff94f_row4_col1\" class=\"data row4 col1\" >https://www.appliancesdirect.co.uk/ct/tv-and-home-entertainment/tvs/smart</td>\n",
" <td id=\"T_ff94f_row4_col2\" class=\"data row4 col2\" >Cheap Smart TVs | Best Smart TV Deals at Appliances Direct</td>\n",
" <td id=\"T_ff94f_row4_col3\" class=\"data row4 col3\" >Shop the latest Smart TVs in a range of screen sizes. 4K, 8K, OLED, QLED & android TVs are available.</td>\n",
" <td id=\"T_ff94f_row4_col4\" class=\"data row4 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row4_col5\" class=\"data row4 col5\" >0.01 (Good)</td>\n",
" <td id=\"T_ff94f_row4_col6\" class=\"data row4 col6\" >1746 (Good)</td>\n",
" <td id=\"T_ff94f_row4_col7\" class=\"data row4 col7\" >21 (Good)</td>\n",
" <td id=\"T_ff94f_row4_col8\" class=\"data row4 col8\" >340 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row4_col9\" class=\"data row4 col9\" >3756 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row4_col10\" class=\"data row4 col10\" >262 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
" <td id=\"T_ff94f_row5_col0\" class=\"data row5 col0\" >6</td>\n",
" <td id=\"T_ff94f_row5_col1\" class=\"data row5 col1\" >https://www.johnlewis.com/browse/electricals/televisions/view-all-tvs/smart/_/N-6srfZlugk</td>\n",
" <td id=\"T_ff94f_row5_col2\" class=\"data row5 col2\" >Smart TVs | John Lewis & Partners</td>\n",
" <td id=\"T_ff94f_row5_col3\" class=\"data row5 col3\" >Browse through our collection of smart TVs to find the best options for your viewing needs. Our range includes large smart TVs ...</td>\n",
" <td id=\"T_ff94f_row5_col4\" class=\"data row5 col4\" >Origin</td>\n",
" <td id=\"T_ff94f_row5_col5\" class=\"data row5 col5\" >0.00 (Good)</td>\n",
" <td id=\"T_ff94f_row5_col6\" class=\"data row5 col6\" >1430 (Good)</td>\n",
" <td id=\"T_ff94f_row5_col7\" class=\"data row5 col7\" >42 (Good)</td>\n",
" <td id=\"T_ff94f_row5_col8\" class=\"data row5 col8\" >363 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row5_col9\" class=\"data row5 col9\" >1851 (Good)</td>\n",
" <td id=\"T_ff94f_row5_col10\" class=\"data row5 col10\" >746 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
" <td id=\"T_ff94f_row6_col0\" class=\"data row6 col0\" >7</td>\n",
" <td id=\"T_ff94f_row6_col1\" class=\"data row6 col1\" >https://www.pricerunner.com/sp/32-inch-smart-tv.html</td>\n",
" <td id=\"T_ff94f_row6_col2\" class=\"data row6 col2\" >32 inch smart tv</td>\n",
" <td id=\"T_ff94f_row6_col3\" class=\"data row6 col3\" >Find the lowest prices on 32 inch smart tv ✓ See 1000+ top models in UK shops ✓ Discover deals online now.</td>\n",
" <td id=\"T_ff94f_row6_col4\" class=\"data row6 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row6_col5\" class=\"data row6 col5\" >0.10 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row6_col6\" class=\"data row6 col6\" >1178 (Good)</td>\n",
" <td id=\"T_ff94f_row6_col7\" class=\"data row6 col7\" >21 (Good)</td>\n",
" <td id=\"T_ff94f_row6_col8\" class=\"data row6 col8\" >791 (Bad)</td>\n",
" <td id=\"T_ff94f_row6_col9\" class=\"data row6 col9\" >1291 (Good)</td>\n",
" <td id=\"T_ff94f_row6_col10\" class=\"data row6 col10\" >528 (Good)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
" <td id=\"T_ff94f_row7_col0\" class=\"data row7 col0\" >8</td>\n",
" <td id=\"T_ff94f_row7_col1\" class=\"data row7 col1\" >https://www.very.co.uk/electricals/televisions/smart-tvs/e/b/22125,4740.end</td>\n",
" <td id=\"T_ff94f_row7_col2\" class=\"data row7 col2\" >Smart TVs | Smart Television Deals | Very.co.uk</td>\n",
" <td id=\"T_ff94f_row7_col3\" class=\"data row7 col3\" >Discover leading brand Smart TVs at Very.co.uk. ... Samsung UE43CU8000, 43 inch, Crystal, 4K Ultra HD, Smart TV · £599 £429 ( ...</td>\n",
" <td id=\"T_ff94f_row7_col4\" class=\"data row7 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row7_col5\" class=\"data row7 col5\" >0.08 (Good)</td>\n",
" <td id=\"T_ff94f_row7_col6\" class=\"data row7 col6\" >1803 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row7_col7\" class=\"data row7 col7\" >27 (Good)</td>\n",
" <td id=\"T_ff94f_row7_col8\" class=\"data row7 col8\" >363 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row7_col9\" class=\"data row7 col9\" >2477 (Good)</td>\n",
" <td id=\"T_ff94f_row7_col10\" class=\"data row7 col10\" >1009 (Need improvement)</td>\n",
" </tr>\n",
" <tr>\n",
" <th id=\"T_ff94f_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
" <td id=\"T_ff94f_row8_col0\" class=\"data row8 col0\" >9</td>\n",
" <td id=\"T_ff94f_row8_col1\" class=\"data row8 col1\" >https://www.which.co.uk/reviews/televisions/article/what-is-smart-tv-aNeqk6F0RAAn</td>\n",
" <td id=\"T_ff94f_row8_col2\" class=\"data row8 col2\" >What Is Smart TV? Best Smart TVs For 2023 - Which?</td>\n",
" <td id=\"T_ff94f_row8_col3\" class=\"data row8 col3\" >A smart TV is an internet-connected television that offers a range of online features, such as on-demand content from ...</td>\n",
" <td id=\"T_ff94f_row8_col4\" class=\"data row8 col4\" >URL</td>\n",
" <td id=\"T_ff94f_row8_col5\" class=\"data row8 col5\" >0.00 (Good)</td>\n",
" <td id=\"T_ff94f_row8_col6\" class=\"data row8 col6\" >2551 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row8_col7\" class=\"data row8 col7\" >38 (Good)</td>\n",
" <td id=\"T_ff94f_row8_col8\" class=\"data row8 col8\" >717 (Bad)</td>\n",
" <td id=\"T_ff94f_row8_col9\" class=\"data row8 col9\" >2906 (Need improvement)</td>\n",
" <td id=\"T_ff94f_row8_col10\" class=\"data row8 col10\" >1648 (Need improvement)</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
]
},
"metadata": {},
"execution_count": 6
}
]
},
{
"cell_type": "code",
"source": [
"#@markdown # Step 4: Press 'Play' to download the CSV data (optional)\n",
"\n",
"from google.colab import files\n",
"\n",
"filename = f'CrUX-from-Google-SERP-{QUERY}.csv'\n",
"merged_df.to_csv(filename)\n",
"files.download(filename)\n",
"\n",
"#@markdown ---"
],
"metadata": {
"cellView": "form",
"id": "ahsj9GiyZHTq"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"If you want to modify the code, here are the documentation for both APIs:\n",
"- [SerpApi Documentation](https://serpapi.com/search-api)\n",
"- [SerpApi Python Library](https://github.com/serpapi/google-search-results-python)\n",
"- [CrUX API Documentation](https://developer.chrome.com/docs/crux/api/)\n",
"\n",
"---\n",
"\n",
"**Author:** Natzir Turrado, Technical SEO / Data Scientist\n",
"<br>**Twitter:** [@natzir9](https://twitter.com/natzir9)\n",
"\n",
"---"
],
"metadata": {
"id": "XriYaj_5anzg"
}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment