Dan Andreescu milimetric
milimetric
/ .gitignore
Last active
December 15, 2015 12:49
Which skin do Wikipedia Editors use? Only look at editors with 5 or more edits over the past 30 days.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| *swp |
milimetric
/ get_udp2log_ports.py
Last active
December 15, 2015 14:09
— forked from ottomata/get_udp2log_ports.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def get_udp2log_ports(): | |
| """Returns the listen ports of running udp2log processes""" | |
| pattern = "/usr/bin/udp2log" | |
| return [get_p(cmd) for cmd in [get_cmd(pid) for pid in iter_pids()] if has_p(pattern, cmd)] | |
| def has_p(pattern, cmd): | |
| return pattern in cmd[0] and '-p' in cmd | |
| def get_p(cmd): | |
| return int(cmd[cmd.index('-p') + 1]) |
milimetric
/ python deduplicate list
Created
April 26, 2013 16:22
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def deduplicate(list_of_objects, key_function): | |
| uniques = dict() | |
| for o in list_of_objects: | |
| key = key_function(o) | |
| if not key in uniques: | |
| uniques[key] = o | |
| return uniques.values() |
milimetric
/ umapi.parallel.test
Last active
December 17, 2015 10:09
quick and dirty script to test parallelism on umapi
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # fill in u and p to the proper usernames and passwords | |
| username=u | |
| password=p | |
| htuser=u | |
| htpass=p | |
| curl --data "username=$username&password=$password" https://$htuser:[email protected]/login -c ~/umapi.session | |
| for cohort in test e2_aft5_cta4 e3_ob2b_gettingstarted_page-impression e3_ob4b_gettingstarted-addlinks_page-impression e3_ob4b_gettingstarted-clarify_page-impression e3_ob4b_gettingstarted-copyedit_page-impression | |
| do |
milimetric
/ limn.instances.to.save
Last active
December 17, 2015 10:59
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /srv/debugging.wmflabs.org/ | |
| /srv/dev-reportcard.wmflabs.org/ | |
| /srv/ee-dashboard.wmflabs.org/ | |
| /srv/gerrit-stats.wmflabs.org/ | |
| /srv/gp.wmflabs.org/ | |
| /srv/mobile-reportcard-dev.wmflabs.org/ | |
| /srv/mobile-reportcard.wmflabs.org/ | |
| /srv/test-reportcard.wmflabs.org/ |
milimetric
/ zero_investigation.pig
Last active
December 18, 2015 01:59
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| REGISTER 'kraken-pig-0.0.2-SNAPSHOT.jar' | |
| REGISTER 'kraken-generic-0.0.2-SNAPSHOT-jar-with-dependencies.jar' | |
| REGISTER 'geoip-1.2.5.jar' | |
| IMPORT 'include/load_webrequest.pig'; | |
| SET default_parallel 2; | |
| DEFINE TO_HOUR org.wikimedia.analytics.kraken.pig.ConvertDateFormat('yyyy-MM-dd\'T\'HH:mm:ss', 'yyyy-MM-dd_HH'); | |
| DEFINE EXTRACT org.apache.pig.builtin.REGEX_EXTRACT_ALL(); | |
| DEFINE ZERO org.wikimedia.analytics.kraken.pig.Zero(); | |
| LOG_FIELDS = LOAD_WEBREQUEST('/wmf/raw/webrequest/webrequest-wikipedia-mobile/dt=2013-05-01*'); | |
| LOG_FIELDS = FILTER LOG_FIELDS BY (x_cs != '-'); |
milimetric
/ gist:6438290
Created
September 4, 2013 15:05
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| self.create_test_cohort( | |
| editor_count=4, | |
| revisions_per_editor=3, | |
| revision_timestamps=[ | |
| [ | |
| datetime(2012, 12, 31, 23, 0, 0), | |
| datetime(2013, 1, 1, 0, 30, 0), | |
| datetime(2013, 1, 1, 1, 0, 0), | |
| ], | |
| [ |
milimetric
/ Missing Sequence Numbers in Hive
Last active
January 11, 2023 16:43
Two ways to find missing sequence numbers in huge Hive tables. First way - gets the left and right boundaries of each run of missing sequences. Second way - gets the count of boundaries, which if greater than 2 signifies missing sequences. The second way doesn't tell you which sequences are missing or how many are missing, but runs faster.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* Common setup, two variants follow | |
| */ | |
| use test; | |
| set tablename=webrequest_esams0; | |
| add jar /home/otto/hive-serdes-1.0-SNAPSHOT.jar; | |
| add jar /usr/lib/hive/lib/hive-contrib-0.10.0-cdh4.3.1.jar; | |
| create temporary function rowSequence AS 'org.apache.hadoop.hive.contrib.udf.UDFRowSequence'; |
milimetric
/ import_one_hour.sh
Last active
January 7, 2020 15:35
Imports one hour of Domasz's pageview data into a partitioned Hive table. It takes roughly 12 seconds to import one hour of data, btw. This means roughly 8 days to import all 7 years of data.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # | |
| # This script does the following: | |
| # 0. reads four arguments from the CLI, in order, as YEAR, MONTH, DAY, HOUR | |
| # 1. downloads the specified hour worth of data from http://dumps.wikimedia.org/other/pagecounts-raw/ | |
| # 2. extracts the data into hdfs | |
| # 3. creates a partition on a hive table pointing to this data | |
| # | |
| print_help() { |
milimetric
/ aggregate_daily.hql
Created
October 4, 2013 19:57
Hive script to create an internal table and insert hourly data aggregated at the daily level.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| DROP TABLE IF EXISTS milimetric_pagecounts_daily; | |
| CREATE TABLE IF NOT EXISTS milimetric_pagecounts_daily( | |
| project string, | |
| page string, | |
| views int, | |
| bytes int, | |
| year int, | |
| month int, | |
| day int | |
| ) |
OlderNewer