Jonas Jarutis jarutis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2015-07-17 04:45:53,122 INFO [main] (notebook.kernel.pfork.BetterFork$) - Remote process starting | |
| 2015-07-17 04:45:53,723 INFO [Remote-akka.actor.default-dispatcher-4] (akka.event.slf4j.Slf4jLogger) - Slf4jLogger started | |
| 2015-07-17 04:45:53,774 INFO [Remote-akka.actor.default-dispatcher-4] (Remoting) - Starting remoting | |
| 2015-07-17 04:45:53,953 INFO [Remote-akka.actor.default-dispatcher-3] (Remoting) - Remoting started; listening on addresses :[akka.tcp://[email protected]:46791] | |
| 2015-07-17 04:45:53,954 INFO [Remote-akka.actor.default-dispatcher-3] (Remoting) - Remoting now listens on addresses: [akka.tcp://[email protected]:46791] | |
| 2015-07-17 04:46:14,040 INFO [Remote-akka.actor.default-dispatcher-2] (notebook.client.ReplCalculator) - ReplCalculator preStart | |
| 2015-07-17 04:46:14,042 INFO [Remote-akka.actor.default-dispatcher-2] (notebook.client.ReplCalculator) - INIT SCRIPT: dummy | |
| 2015-07-17 04:46:14,451 INFO [Remote-akka.actor.default-dispatcher-4] (org.apache.spark.SecurityManager) - Changing view acls |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from 'batch_views.items i' | |
| period_field 'SUBSTRING(i.created_at, 1, 10)' | |
| period_type Report::DatePeriod | |
| dimensions portal: 'i.country', | |
| platform: 'i.app_id', | |
| price_group: 'i.price_groups', | |
| detailed_price_group: 'i.price_groups_detailed', | |
| paid_organic: 'u.paid_organic', |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/spark-shell --master yarn-client --num-executors 50 --driver-memory 7g --executor-memory 7g --executor-cores 1 --jars /home/jjarutis/ini/joda-time-2.7/joda-time-2.7.jar,/home/jjarutis/ini/joda-time-2.7/joda-convert-1.7.jar,/home/jjarutis/math_2.10-0.1.0.jar | |
| import org.joda.time.{LocalDate, Days} | |
| import org.apache.spark.sql.DataFrame | |
| import org.apache.spark.sql.Row | |
| import org.apache.spark.rdd.RDD | |
| import vinted.math.{RolingSum, TotalSum} | |
| // Helpers | |
| val context = new org.apache.spark.sql.SQLContext(sc) |
jarutis
/ join.scala
Created
January 19, 2015 19:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import com.twitter.scalding.typed._ | |
| case class NextEvent(id: String, time: Long) | |
| val events1 = List( | |
| NextEvent("user1", 1421280000498L), | |
| NextEvent("user1", 1421280000769L), | |
| NextEvent("user2", 1421280000819L) | |
| ) |
jarutis
/ install_packages.r
Last active
January 18, 2016 06:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| install.packages('rJava', type='source') | |
| # Setup RMysql to use percona server | |
| Sys.setenv(PKG_CPPFLAGS = "-I/usr/local/Cellar/percona-server/5.6.22-72.0/include/mysql/") | |
| Sys.setenv(PKG_LIBS = "-L/usr/local/Cellar/percona-server/5.6.22-72.0/lib -lperconaserverclient") | |
| install.packages('RMySQL', type='source') | |
| # Setup connection to hadoop | |
| install.packages('RImpala') | |
| install.packages('RJDBC') |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "metadata": { | |
| "kernelspec": { | |
| "codemirror_mode": { | |
| "name": "python", | |
| "singleOperators": {}, | |
| "version": 2 | |
| }, | |
| "display_name": "IPython (Python 2)", | |
| "language": "python", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #include <iostream> | |
| #include <math.h> | |
| #include <impala_udf/uda-test-harness.h> | |
| #include "uda-dist.h" | |
| using namespace impala; | |
| using namespace impala_udf; | |
| using namespace std; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #include <iostream> | |
| #include <math.h> | |
| #include <impala_udf/uda-test-harness.h> | |
| #include "uda-dist.h" | |
| using namespace impala; | |
| using namespace impala_udf; | |
| using namespace std; |