KentVN dqtweb

Migrate from Facebook scribe to Apache Flume (Part II)

In last article we talked about how to setup flume and write files HDFS. This article, we begin to change flume to write file in scribe like style category. Multiplexing Way?

The first thought is using source multiplex to distribute log to different destination. Flume distribute log events by event header. So we google to find out which field in header is referring to scribe header.

https://apache.googlesource.com/flume/+/d66bf94b1dd059bc7e4b1ff332be59a280498077/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java

category in header will refer to scribe category. So we try to use multiplexing source:

Migrate from Facebook scribe to Apache Flume (Part I)

Reason

We use scribe as our logging server for a long time. At first, everything works fine. Easy to config, easy to manage. As data grows everyday, single scribe server can’t handle that. We have to migrate some category to second log server and attach a big disk. As data is keep growing, we want a big data storage for that instead of local disk. So we decide to use Scribe with HDFS plugin. It is as tough as we first compile scribe from source. Finally we complied the scribed with hdfs support. But after a short period usage, we find a bug that haven’t solved by facebook. (the project is deprecated several years ago). The bug cause scribe can’t write to hdfs if it accidently killed by SIG 9. So we start to test flume and find out ways to migrate.

Configure Flume

Flume is easy to deploy because it is written in Java. Install java and download jar package, we’ve done all the jobs

	/**
	* IE 5.5+, Firefox, Opera, Chrome, Safari XHR object
	*
	* @param string url
	* @param object callback
	* @param mixed data
	* @param null x
	*/
	function ajax(url, callback, data, x) {
	try {

	import json
	from pyspark.sql.types import *

	# Define the schema
	schema = StructType(
	[StructField("name", StringType(), True), StructField("age", IntegerType(), True)]
	)

	# Write the schema
	with open("schema.json", "w") as f:

	#! /usr/bin/env python

	import redis
	import random
	import pylibmc
	import sys

	r = redis.Redis(host = 'localhost', port = 6389)
	mc = pylibmc.Client(['localhost:11222'])

	# Redis Cheatsheet
	# All the commands you need to know


	redis-server /path/redis.conf # start redis with the related configuration file
	redis-cli # opens a redis prompt


	# Strings.

	def valid_date_type(arg_date_str):
	"""custom argparse date type for user dates values given from the command line"""
	try:
	return datetime.datetime.strptime(arg_date_str, "%Y-%m-%d")
	except ValueError:
	msg = "Given Date ({0}) not valid! Expected format, YYYY-MM-DD!".format(arg_date_str)
	raise argparse.ArgumentTypeError(msg)

	def valid_datetime_type(arg_datetime_str):
	"""custom argparse type for user datetime values given from the command line"""

	#!/usr/bin/python
	import subprocess
	import os
	import sys

	#res = subprocess.Popen(['ls','-al','/ahome'],stdout=subprocess.PIPE,stderr=subprocess.PIPE);
	#output,error = res.communicate()

	#if res.returncode:
	# #raise Exception(error)

	from datetime import datetime
	import time

	#-------------------------------------------------
	# conversions to strings
	#-------------------------------------------------
	# datetime object to string
	dt_obj = datetime(2008, 11, 10, 17, 53, 59)
	date_str = dt_obj.strftime("%Y-%m-%d %H:%M:%S")
	print date_str