dqtweb

Migrate from Facebook scribe to Apache Flume (Part II)

In last article we talked about how to setup flume and write files HDFS. This article, we begin to change flume to write file in scribe like style category. Multiplexing Way?

The first thought is using source multiplex to distribute log to different destination. Flume distribute log events by event header. So we google to find out which field in header is referring to scribe header.

https://apache.googlesource.com/flume/+/d66bf94b1dd059bc7e4b1ff332be59a280498077/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java

category in header will refer to scribe category. So we try to use multiplexing source:

dqtweb

#! /usr/bin/env python

import redis
import random
import pylibmc
import sys

r = redis.Redis(host = 'localhost', port = 6389)
mc = pylibmc.Client(['localhost:11222'])

dqtweb

import json
from pyspark.sql.types import *

# Define the schema
schema = StructType(
    [StructField("name", StringType(), True), StructField("age", IntegerType(), True)]
)

# Write the schema
with open("schema.json", "w") as f:

dqtweb

/**
 * IE 5.5+, Firefox, Opera, Chrome, Safari XHR object
 * 
 * @param string url
 * @param object callback
 * @param mixed data
 * @param null x
 */
function ajax(url, callback, data, x) {
	try {

dqtweb

SELECT
    type
  , count(*)
  , count(DISTINCT u)
  , count(CASE WHEN plat=1 THEN u ELSE NULL END)
  , count(DISTINCT CASE WHEN plat=1 THEN u ELSE NULL END)
  , count(CASE WHEN (type=2 OR type=6) THEN u ELSE NULL END)
  , count(DISTINCT CASE WHEN (type=2 OR type=6) THEN u ELSE NULL END)
FROM
    t

dqtweb

// example, we want to sort list of objects based on an object's attribute. ex: "age"
var peopleList = [{"name": "Laura", "age":5}, {"name": "Peppa", "age": 40}, {"name": "Josh", "age":22}];
var comparision = function(a, b) { return a.age > b.age ? 1 : -1; };

dqtweb

# so that you can open with excel

def addUTF8Bom(filename):
    f = codecs.open(filename, 'r', 'utf-8')
    content = f.read()
    f.close()
    f2 = codecs.open(filename, 'w', 'utf-8')
    f2.write(u'\ufeff')
    f2.write(content)
    f2.close()

dqtweb

[Unit]
Description=Jupyter Notebook

[Service]
Type=simple
ExecStart=/home/roeyd/Notebook/.env/bin/jupyter lab --port 9090
WorkingDirectory=/home/roeyd/Notebook

[Install]
WantedBy=default.target

dqtweb

const io = require('socket.io-client');

const socket = io('http://localhost:3000', {
  transportOptions: {
    polling: {
      extraHeaders: {
        'Authorization': 'Bearer abc',
      },
    },
  },

dqtweb

Moved to git-repository: https://github.com/denji/awesome-http-benchmark

Tools

Located in alphabetical order (not prefer)

• ab – slow and single threaded, written in C
• apib – most of the features of ApacheBench (ab), also designed as a more modern replacement, written in C
• autocannon – fast HTTP/1.1 benchmarking tool written in Node.js
• baloo – Expressive end-to-end HTTP API testing made easy, written in Go (golang)