Travis Hegner travishegner
travishegner
/ VectorSum.scala
Created
April 22, 2019 19:16
User Defined Aggregate Function: Vector Sum
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.trilliumstaffing.hadoop.tools.udaf | |
| import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction} | |
| import org.apache.spark.sql.types.{DataType, StructField, StructType} | |
| import org.apache.spark.ml.linalg.SQLDataTypes.VectorType | |
| import org.apache.spark.ml.linalg.{Vector, Vectors} | |
| import breeze.linalg.{Vector => BV} | |
| import org.apache.spark.sql.Row | |
| class VectorSum extends UserDefinedAggregateFunction { |
travishegner
/ HBase.scala
Created
June 15, 2018 13:22
HBase scala object for writing and deleting from spark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.trilliumstaffing.hadoop.tools | |
| import java.nio.ByteBuffer | |
| import org.apache.hadoop.hbase.{HBaseConfiguration, TableName} | |
| import org.apache.hadoop.hbase.client._ | |
| import org.apache.spark.sql.Dataset | |
| import org.apache.spark.ml.linalg.{Vector, Vectors} | |
| import org.apache.hadoop.hbase.protobuf.ProtobufUtil | |
| import org.apache.hadoop.hbase.util.{Base64, Bytes} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "id":"/test", | |
| "instances":1, | |
| "cpus":1, | |
| "mem":1024, | |
| "container":{ | |
| "type":"DOCKER", | |
| "docker":{ | |
| "network":"BRIDGE", | |
| "image":"ubuntu:14.04", |
travishegner
/ stderr
Created
February 3, 2016 13:44
mesos-docker-executor logs -- successful launch
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| I0202 08:18:27.692706 7254 docker.cpp:614] Running docker -H unix:///var/run/docker.sock run --cpu-shares 102 --memory 536870912 -e MARATHON_APP_VERSION=2016-01-29T16:42:17.101Z -e HOST=node3.example.com -e MARATHON_APP_RESOURCE_CPUS=0.1 -e MARATHON_APP_DOCKER_IMAGE=registry.example.com/irc:e536abc322f21cf946cb9b9d0b86c1cfb2068dac -e PORT_10001=31847 -e MESOS_TASK_ID=irc.72170899-c9af-11e5-b589-02427fa617b6 -e PORT=31847 -e MARATHON_APP_RESOURCE_MEM=512.0 -e PORTS=31847 -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/irc -e PORT0=31847 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-853e5e6e-d937-41c6-913d-48cc0a0270e3-S0.1eddc1e5-f53b-40dd-9355-e9c79c5a4044 -v /tmp/mesos/slaves/853e5e6e-d937-41c6-913d-48cc0a0270e3-S0/frameworks/3b552a3b-c098-40d6-af33-ed59ec2cd54f-0000/executors/irc.72170899-c9af-11e5-b589-02427fa617b6/runs/1eddc1e5-f53b-40dd-9355-e9c79c5a4044:/mnt/mesos/sandbox --net bridge --net=vx_0x10009 --name mesos-853e5e6e-d937-41c6-913d-48cc0a0270e3-S0 |
travishegner
/ haproxy2apache
Last active
May 6, 2022 15:44
A quick bash one liner to convert haproxy logs to something like apache logs so that logstalgia can read them
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tail -f log | stdbuf -o0 awk '{print $9" "$6" - - "$7" "$18" "$19" "$20" "$11" "$12}' | sed -u "s/:[0-9]\{4,5\}//g" | sed -u "s/\(:[0-6][0-9]\)\.[0-9]\{3\}/\\1 -400/g" | logstalgia --sync --paddle-mode vhost |