Robert Metzger rmetzger
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| for commit in $(git rev-list master) | |
| do | |
| git checkout $commit | |
| DATE=`git log | sed '3q;d'` | |
| AUTHORS=`git log --format='%aN' | sort -u | wc -l` | |
| echo "$DATE,$AUTHORS" >> graph.csv | |
| done |
rmetzger
/ gist:674dd277c66f5c1486a6
Created
April 16, 2015 07:54
Test Apache Kafka. I'm planning to merge this to Flink at a later point
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @Test | |
| public void testKafkaWithoutFlink() { | |
| // start consumer: | |
| new Thread(new Runnable() { | |
| @Override | |
| public void run() { | |
| LOG.info("Starting consumer"); | |
| // consume from "testtopic" |
rmetzger
/ gist:a218beca4b0442f3c1f3
Last active
August 29, 2015 14:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java b/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java | |
| index 1f39694..cadb938 100644 | |
| --- a/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java | |
| +++ b/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java | |
| @@ -20,10 +20,12 @@ package org.apache.flink.addons.hbase.example; | |
| import org.apache.flink.api.common.functions.FlatMapFunction; | |
| import org.apache.flink.api.common.functions.MapFunction; | |
| +import org.apache.flink.api.common.functions.RichMapFunction; | |
| import org.apache.flink.api.java.DataSet; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public static void main(String[] args) throws Exception { | |
| // set up the execution environment | |
| final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); | |
| DataSet<Tuple2<Integer, String>> one = env.fromElements(new Tuple2<Integer, String>(1, "one"), | |
| new Tuple2<Integer, String>(2, "two"), new Tuple2<Integer, String>(3, "three") | |
| ); | |
| DataSet<Tuple1<Integer>> two = env.fromElements(new Tuple1<Integer>(1)); | |
| DataSet<Integer> three = one.coGroup(two).where(0).equalTo(0).with(new CoGroupFunction<Tuple2<Integer, String>, Tuple1<Integer>, Integer>() { | |
| @Override |
rmetzger
/ websitetop.patch
Created
January 11, 2015 19:54
Because svn doesn't support local branches, I have to put my wip on flink.apache.org here
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Index: index.html | |
| =================================================================== | |
| --- index.html (revision 1650799) | |
| +++ index.html (working copy) | |
| @@ -13,6 +13,79 @@ | |
| <link href="css/main/bootstrap-theme.min.css" rel="stylesheet"> | |
| <link href="css/main/main.css" rel="stylesheet"> | |
| <link rel="stylesheet" href="css/syntax.css"> | |
| + <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" ></script> | |
| + <script src="js/main/jquery.mobile.events.min.js"></script> |
rmetzger
/ gist:600bea0361736c235eb1
Created
December 19, 2014 14:20
If you have this error, read the comment below ....
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 14:16:45,207 DEBUG org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl entered state INITED | |
| 14:16:45,215 WARN org.apache.hadoop.security.token.Token - Cannot find class for token kind YARN_AM_RM_TOKEN | |
| 14:16:45,215 DEBUG org.apache.hadoop.security.SecurityUtil - Acquired token Kind: YARN_AM_RM_TOKEN, Service: 127.0.0.1:8030, Ident: 00 00 01 4a 62 b1 45 25 00 00 00 01 00 00 00 01 | |
| 14:16:45,216 DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:robert (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:63) | |
| 14:16:45,217 DEBUG org.apache.hadoop.yarn.ipc.YarnRPC - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC | |
| 14:16:45,217 DEBUG org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.Applicatio |
rmetzger
/ gist:3f8e472e06756447eb92
Created
December 11, 2014 13:02
Run Apache Flink on Hortonworks HDP 2.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Prepare HDFS | |
| whoami # in my case: robert | |
| sudo su hdfs | |
| hadoop fs -mkdir /user/robert | |
| hadoop fs -chmod 777 /user/robert | |
| export HADOOP_CONF_PATH=/etc/hadoop/conf/ | |
| wget http://apache.lauf-forum.at/incubator/flink/flink-0.7.0-incubating/flink-0.7.0-incubating-bin-hadoop2-yarn.tgz | |
| tar xf flink-0.7.0-incubating-bin-hadoop2-yarn.tgz | |
| cd flink-yarn-0.7.0-incubating/ |
rmetzger
/ Janino test
Created
October 4, 2014 14:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /** | |
| * Licensed to the Apache Software Foundation (ASF) under one | |
| * or more contributor license agreements. See the NOTICE file | |
| * distributed with this work for additional information | |
| * regarding copyright ownership. The ASF licenses this file | |
| * to you under the Apache License, Version 2.0 (the | |
| * "License"); you may not use this file except in compliance | |
| * with the License. You may obtain a copy of the License at | |
| * | |
| * http://www.apache.org/licenses/LICENSE-2.0 |
rmetzger
/ gist:cf4d2eba940483d704a2
Created
June 20, 2014 18:52
Spark ASF-GitHub-Integration merge tool output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./merge_pull_request.sh | |
| Which pull request would you like to merge? (e.g. 34): 30 | |
| === Pull Request #30 === | |
| title Added Version and Revision of Stratosphere in Webinterface | |
| source JonathanH5/wi_version | |
| target master | |
| url https://api.github.com/repos/apache/incubator-flink/pulls/30 | |
| Proceed with merging pull request #30? (y/n): y |
rmetzger
/ gist:1a9ed4080eedb4e0c8f1
Created
May 15, 2014 13:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 13:41:45,088 INFO eu.stratosphere.nephele.jobmanager.JobManager - Creating initial execution graph from job graph Large Test Plan | |
| 13:41:45,635 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input vertex DataSource(File hdfs:///user/vagrant/stratosphere-tests/testjob-in/orders.avro) generated 2 input splits | |
| 13:41:45,695 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input vertex DataSource(region) generated 8 input splits | |
| 13:41:45,738 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input vertex DataSource(customer) generated 8 input splits | |
| 13:41:45,771 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input vertex DataSource(lineitem) generated 8 input splits | |
| 13:41:45,828 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input vertex DataSource(orders) generated 8 input splits | |
| 13:41:45,897 INFO eu.stratosphere.nephele.executiongraph.ExecutionGraph - Job input |