Skip to content

Instantly share code, notes, and snippets.

@jamesramsay

jamesramsay/.gitignore

Last active October 8, 2015 09:11
Show Gist options
  • Save jamesramsay/98a7bd2e86b901698bc7 to your computer and use it in GitHub Desktop.
Save jamesramsay/98a7bd2e86b901698bc7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
.gitignore
*.har
*.json
*.sh
Raw
README.md

Advertising analysis

Process:

  1. Collect HAR recording for mobile with and without blocking
  2. Process HAR to JSON
  3. Merge advertising and editorial JSON files to get combined classification
  4. Process JSON to TSV

Process HAR

<"mobile.nytimes.com.20151008.advertising.har" ./har-to-json.sh > mobile.nytimes.com.20151008.advertising.json
<"mobile.nytimes.com.20151008.editorial.har" ./har-to-json.sh > mobile.nytimes.com.20151008.editorial.json

Merge JSON

ksdiff --merge --output mobile.nytimes.com.20151008.json mobile.nytimes.com.20151008.editorial.json mobile.nytimes.com.20151008.advertising.json

Process to TSV

<"mobile.nytimes.com.20151008.json" ./array-of-objects-to-tsv.sh > mobile.nytimes.com.20151008.tsv
Raw
mobile.nytimes.com.20151008.tsv
date host classification mime size compressed url
2015-10-08 mobile.nytimes.com editorial text/html 59391 15498 http://mobile.nytimes.com/
2015-10-08 pagead2.googlesyndication.com advertising image/gif 0 -428 http://pagead2.googlesyndication.com/activeview?id=osdtos&avi=B-f00ZjEWVojzLLGSlAKjy7HQBgAAAAAQATgByAEJwAIC4AIA4AQBoAYfwhMDEIAB&cid=5Ggs5y4t4UCi0r6Icw2B3RZeduk&ti=1&adk=4072990747&p=8,28,58,348&tos=6038,0,0,0,0&mtos=6038,6038,6038,6038,6038&rs=3&ht=0&tfs=3629&tls=9667&fp=correlator%3D3616326616612864%26eid%3D108809080%252C108809029%252C108809050%26iu%3D%252F29390238%252FNYT%252Fhomepage%252Fsectionfront%26oid%3D3%26url%3Dhttp%253A%252F%252Fmobile.nytimes.com%252F&afp=%26output%3Djson_html%26impl%3Dfif%26dt%3D1444295014395%26adx%3D28%26ady%3D8%26ifi%3D2%26flash%3D0&tmo=2&tme=2&tdl=3&tcl=4&abd=0-0-15&r=u&bs=375,627&bos=375,627&ps=375,9807&ss=375,667&tt=6039&pt=3628&deb=1-1-1-4-15-8&tvt=6039&uc=15&tgt=IMG&cl=1
2015-10-08 www.googletagservices.com editorial text/javascript 2900 1296 http://www.googletagservices.com/tag/js/gpt.js
2015-10-08 c.amazon-adsystem.com advertising application/x-javascript 15033 4756 http://c.amazon-adsystem.com/aax2/amzn_ads.js
2015-10-08 graphics8.nytimes.com editorial image/jpeg 52524 52524 http://graphics8.nytimes.com/images/2015/10/08/world/08kunduz/08kunduz-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 64259 64259 http://graphics8.nytimes.com/images/2015/10/07/world/08syriamigrants-web2/08syriamigrants-web2-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 51186 51186 http://graphics8.nytimes.com/images/2015/10/08/opinion/08krastev/08krastev-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 62657 62657 http://graphics8.nytimes.com/images/2015/10/08/opinion/08thu3web/08thu3web-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 6251 6251 http://graphics8.nytimes.com/images/2015/10/08/opinion/08thurs1/08thurs1-thumbLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 3727 3727 http://graphics8.nytimes.com/images/2015/10/07/us/07firstdraft-clinton2/07firstdraft-clinton2-thumbLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 5291 5291 http://graphics8.nytimes.com/images/2015/10/08/us/08cong-web/08cong-web-thumbLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 106960 106960 http://graphics8.nytimes.com/images/2015/10/08/business/08HANDMADEdress/08HANDMADEdress-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 56096 56096 http://graphics8.nytimes.com/images/2015/10/07/world/09DUNEDOO-WEB-1/09DUNEDOO-WEB-1-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 12156 12156 http://graphics8.nytimes.com/images/2015/10/07/us/ROSEBURG-hp/ROSEBURG-hp-thumbLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 72356 72356 http://graphics8.nytimes.com/images/2015/10/08/sports/PIRATES-web/PIRATES-web-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 89660 89660 http://graphics8.nytimes.com/images/2015/10/08/fashion/08PHOTOBOOTH1/08PHOTOBOOTH1-articleLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/png 4735 4735 http://graphics8.nytimes.com/images/2015/08/04/nytnow/mini-crossword-promo-image-2/mini-crossword-promo-image-2-thumbLarge-v3.png
2015-10-08 graphics8.nytimes.com editorial image/jpeg 32061 32061 http://graphics8.nytimes.com/images/2015/10/08/arts/08BOOKMARRA/08BOOKMARRA-thumbLarge.jpg
2015-10-08 graphics8.nytimes.com editorial image/jpeg 51852 51852 http://graphics8.nytimes.com/images/2015/10/11/arts/11DELTORO1/11DELTORO1-articleLarge.jpg
2015-10-08 wt.o.nytimes.com advertising image/gif 67 67 http://wt.o.nytimes.com/dcse9n3rf00000s5j4umphv6p_1d1j/njs.gif?dcsuri=/mobile/MWRnoJavaScriptTest&WT.js=No&WT.tv=10.2.55&WT.cg_n=Homepage&WT.cg_s=
2015-10-08 partner.googleadservices.com editorial text/javascript 149523 49912 http://partner.googleadservices.com/gpt/pubads_impl_73.js
2015-10-08 tpc.googlesyndication.com advertising text/html 4049 1877 http://tpc.googlesyndication.com/safeframe/1-0-2/html/container.html
2015-10-08 www.nytimes.com editorial application/json 21633 21654 http://www.nytimes.com/svc/recommendations/v3/personalized.json?access_key=8162C64B-A092-480E-A4A6-FB7B7E00C10D&loc=mobileweb
2015-10-08 mobile.nytimes.com editorial application/json 98 98 http://mobile.nytimes.com/svc/web-products/userinfo-v2.json?lt=1444033672860
2015-10-08 meter-svc.nytimes.com advertising application/x-javascript 357 357 http://meter-svc.nytimes.com/meter.js?url=http://mobile.nytimes.com/&referer=http://mobile.nytimes.com/international/&callback=jQuery21406889387394767255_1444295027724&_=1444295027725
2015-10-08 static01.nyt.com editorial application/x-javascript 70725 21522 http://static01.nyt.com/bi/js/tagx/tagx.js
2015-10-08 wt.o.nytimes.com advertising image/gif 67 67 http://wt.o.nytimes.com/dcsga5emk0000000gky4w6o4m_7t4c/njs.gif?dcsuri=/mobile/nojavascript&WT.js=No&WT.tv=10.2.55&WT.z_gpt=NoJavaScript&WT.cg_n=Homepage&WT.cg_s=
2015-10-08 aax.amazon-adsystem.com advertising text/javascript 8 8 http://aax.amazon-adsystem.com/e/dtb/bid?src=3030&u=http%3A%2F%2Fmobile.nytimes.com%2F&cb=9090069
2015-10-08 js.moatads.com advertising application/x-javascript 128253 41745 http://js.moatads.com/nytdfp348531439194/moatad.js
2015-10-08 pagead2.googlesyndication.com advertising text/javascript 59382 22670 http://pagead2.googlesyndication.com/pagead/osd.js
2015-10-08 wt.o.nytimes.com advertising image/gif 67 67 http://wt.o.nytimes.com/dcsga5emk0000000gky4w6o4m_7t4c/dcs.gif?&dcsdat=1444295028173&dcssip=mobile.nytimes.com&dcsuri=/&dcsref=http://mobile.nytimes.com/international/&WT.tz=11&WT.bh=20&WT.ul=en-US&WT.cd=24&WT.sr=375x667&WT.jo=Yes&WT.ti=The%20New%20York%20Times&WT.js=Yes&WT.jv=1.5&WT.ct=unknown&WT.bs=375x627&WT.fv=Not%20enabled&WT.slv=Not%20enabled&WT.tv=10.2.55&WT.dl=0&WT.ssl=0&WT.es=mobile.nytimes.com/&WT.cg_n=Homepage&WT.z_rcgn=Homepage&WT.z_gpt=Homepage&WT.z_nyts=0M.LX22FN2TMnDXrmvxADeHDYdR7KkGqm/deFz9JchiAJYOVLIKocjgsV.Ynx4rkFI&WT.z_nytd=null&WT.z_rmid=007f0101321a5612345e0002&WT.z.gsg=0&WT.z_ref=mobile.nytimes.com&WT.dcsvid=0&WT.ce=2&WT.vt_f_tlh=1444295014&WT.vtvs=1444295014775&WT.vtid=c4144aac-ab43-465a-bfc2-34bb43dafe2d&WT.co_f=c4144aac-ab43-465a-bfc2-34bb43dafe2d
2015-10-08 tagx.nytimes.com advertising text/javascript 33701 33755 http://tagx.nytimes.com/?url=http%3A%2F%2Fmobile.nytimes.com%2F&assetUrl=http%3A%2F%2Fwww.nytimes.com%2F&referrer=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F&CG=Homepage&SCG=&sourceApp=mobileWeb
2015-10-08 s.tagsrvcs.com advertising text/plain 0 0 http://s.tagsrvcs.com/2/818492/analytics.js?pp=mobile.nytimes.com&sn=Homepage&c1=http%3A%2F%2Fmobile.nytimes.com%2F&ui=9e23c8100fb790a493ea3bcb279040c4&dt=8184921433871988867000
2015-10-08 st.dynamicyield.com advertising text/javascript 6175 1764 http://st.dynamicyield.com/ast?sec=8765260
2015-10-08 graphics8.nytimes.com advertising application/x-javascript 26416 6486 http://graphics8.nytimes.com/bi/js/analytics/EventTracker.js
2015-10-08 www.google-analytics.com advertising text/javascript 26303 10948 http://www.google-analytics.com/analytics.js
2015-10-08 b.scorecardresearch.com advertising application/x-javascript 0 20 http://b.scorecardresearch.com/c2/3005403/cs.js
2015-10-08 graphics8.nytimes.com advertising application/x-javascript 649 466 http://graphics8.nytimes.com/bi/js/analytics/comscore.js
2015-10-08 static.chartbeat.com advertising application/x-javascript 27906 11030 http://static.chartbeat.com/js/chartbeat.js
2015-10-08 dc8xl0ndzn2cb.cloudfront.net advertising application/javascript 1990 1990 http://dc8xl0ndzn2cb.cloudfront.net/js/nytimes/v1/keywee.js
2015-10-08 static.dynamicyield.com advertising application/javascript 115335 40370 http://static.dynamicyield.com/scripts/12007/dyjq-min.js?v=12092
2015-10-08 static.dynamicyield.com advertising application/javascript 88560 26268 http://static.dynamicyield.com/scripts/12007/dy-min.js?v=12092
2015-10-08 et.nytimes.com advertising application/javascript 113 130 http://et.nytimes.com/?subject=page&url=http%3A%2F%2Fmobile.nytimes.com%2F&assetUrl=http%3A%2F%2Fwww.nytimes.com%2F&referrer=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F&clientTimeStamp=1444295029224&totalTime=0&sourceApp=mobileWeb&callback=NYTD.EventTracker.cb0&pageMetaData=%7B%22PT%22%3A%22Homepage%22%2C%22CG%22%3A%22Homepage%22%2C%22SCG%22%3A%22%22%2C%22byl%22%3A%22%22%2C%22tom%22%3A%22%22%2C%22hdl%22%3A%22%22%2C%22ptime%22%3A%22%22%2C%22cre%22%3A%22%22%2C%22articleid%22%3A%22%22%2C%22channels%22%3A%22%22%2C%22CN%22%3A%22%22%2C%22CT%22%3A%22%22%7D&additionalClientData=%7B%22ul%22%3A%22en-US%22%2C%22js%22%3A%22Yes%22%2C%22jv%22%3A%221.5%22%2C%22fi%22%3A%22No%22%2C%22tz%22%3A11%2C%22bh%22%3A20%2C%22cd%22%3A24%2C%22sr%22%3A%22375x667%22%2C%22bs%22%3A%22375x627%22%2C%22jo%22%3A%22Yes%22%2C%22or%22%3A%22port%22%7D
2015-10-08 graphics8.nytimes.com advertising application/x-javascript 29 49 http://graphics8.nytimes.com/bi/js/analytics/show_ads.js
2015-10-08 www.google-analytics.com advertising image/gif 35 35 http://www.google-analytics.com/collect?v=1&_v=j39&a=493056336&t=pageview&_s=1&dl=http%3A%2F%2Fwww.nytimes.com%2F&ul=en-us&de=UTF-8&dt=The%20New%20York%20Times&sd=24-bit&sr=375x667&vp=375x627&je=1&_u=QACAAAABI~&jid=&cid=261821889.1444033632&tid=UA-58630905-2&cd1=http%3A%2F%2Fmobile.nytimes.com%2F&cd4=Homepage&cd10=mobile.nytimes.com&cd13=mobile.nytimes.com&cd14=null&cd15=owned&cd16=internal&cd17=100000003965243&cd18=null&cd19=null&cd21=sectionfront&cd23=Homepage&cd25=null&cd42=mobileWeb&cd60=null&cd62=edition%7CUS&cd63=9e23c8100fb790a493ea3bcb279040c4&cd64=0&cd65=anon&cd2=http%3A%2F%2Fmobile.nytimes.com%2F&cd6=null&cd7=null&cd8=null&cd9=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F&cd12=null&cd43=null&cd44=null&cd45=null&cd46=null&cd61=0&cd66=819&cd67=null&cd68=null&cg1=homepage&cg2=null&cg3=Homepage&cg4=null&cd3=null&cd5=0&cd11=null&cd20=null&cd22=2015-10-08%2016%3A00&cd24=null&cd38=null&cd39=null&cd40=null&cd129=20&cd130=0012%7C1%260033%7C1%260036%7C-2%260051%7C1%260052%7C1%260061%7C-2%260063%7C0%260064%7C1%260066%7C1%260067%7C-2%260069%7C0&cd133=3323e74974c4a7d3ae3d0f1200ba18ac&cd50=null&cd26=null&cd27=null&cd28=null&cd29=null&cd30=2015-10-08-04&cd48=null&cd31=null&cd34=null&cd36=null&cd37=null&cd101=null&cd121=null&cd52=null&cd53=null&z=461951954
2015-10-08 b.scorecardresearch.com advertising text/plain 0 0 http://b.scorecardresearch.com/b?c1=2&c2=3005403&ns__t=1444295029246&ns_c=UTF-8&c8=The%20New%20York%20Times&c7=http%3A%2F%2Fmobile.nytimes.com%2F&c9=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F
2015-10-08 dc8xl0ndzn2cb.cloudfront.net advertising application/x-javascript 66426 66426 http://dc8xl0ndzn2cb.cloudfront.net/sp.js
2015-10-08 connect.facebook.net advertising application/x-javascript 3049 1560 https://connect.facebook.net/en_US/fbds.js
2015-10-08 pnytimes.chartbeat.net advertising image/gif 43 43 http://pnytimes.chartbeat.net/ping?h=nytimes.com&p=nytimes.com%2F&u=DoRzS3BW0DShu3MVW&d=mobile.nytimes.com&g=16698&g0=Homepage%2C%2C&n=0&f=3000d&c=0&x=0&m=0&y=9807&o=375&w=627&j=45&R=1&W=0&I=0&E=0&e=0&v=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F&t=DmzoBC3MAxADpLhTZC5ib4wBY8CT4&V=66&z=t%3DmmKjVNxu0cCX_NjDD0WbnYCnadBh%26E%3D0%26x%3D0%26c%3D0.14%26y%3D9807%26w%3D627&i=The%20New%20York%20Times&tz=-660&sn=1&_
2015-10-08 et.nytimes.com advertising application/javascript 113 130 http://et.nytimes.com/?subject=adx-ab-allocation&testName=MobileWeb_WhatsNew&module=homepage&treatment=0&nyt.wp.ab=kIT75S9uT1Uwi3qzTLH%2Bog&pgType=Homepage&sourceApp=mobileWeb&instant=1&callback=NYTD.EventTracker.cb1
2015-10-08 pagead2.googlesyndication.com advertising image/gif 42 42 http://pagead2.googlesyndication.com/activeview?id=osdim&avi=BQU3JczEWVrvzO4yw-QOFx4mADgAAAAAQATgByAEJwAIC4AIA4AQBoAYfwhMDEIAB&cid=5Gjq5OZEj18Ek1yq1359vitYT58&ti=1&adk=4072990747&p=8,28,58,348&tos=1011,0,0,0,0&mtos=1011,1011,1011,1011,1011&rs=3&ht=0&tfs=159&tls=1170&fp=correlator%3D584417000030208%26eid%3D108809080%252C108809049%26iu%3D%252F29390238%252FNYT%252Fhomepage%252Fsectionfront%26oid%3D3%26url%3Dhttp%253A%252F%252Fmobile.nytimes.com%252F&afp=%26output%3Djson_html%26impl%3Dfif%26dt%3D1444295027856%26adx%3D28%26ady%3D8%26ifi%3D2%26flash%3D0&tmo=2&tme=159&tdl=69&tcl=69&abd=1-0-4&r=u&bs=375,627&bos=375,627&ps=375,9807&ss=375,667&tt=1171&pt=-1&deb=1-0-1-2-5--1&tvt=1012&uc=4&tgt=IMG&cl=1
2015-10-08 pixel.keywee.co advertising image/gif 43 43 http://pixel.keywee.co/i?e=pv&url=http%3A%2F%2Fmobile.nytimes.com%2F&page=The%20New%20York%20Times&refr=http%3A%2F%2Fmobile.nytimes.com%2Finternational%2F&tv=js-2.4.3&tna=cf&aid=18&p=web&tz=Australia%2FSydney&lang=en-US&cs=UTF-8&f_java=1&res=375x667&cd=24&cookie=1&eid=0119f4f1-26ad-4416-ad73-b15b8c7f5084&dtm=1444295030360&vp=375x627&ds=375x9807&vid=4&duid=64a76d599a28d644&fp=3517054369
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment