elprup

Migrate from Facebook scribe to Apache Flume (Part II)

In last article we talked about how to setup flume and write files HDFS. This article, we begin to change flume to write file in scribe like style category. Multiplexing Way?

The first thought is using source multiplex to distribute log to different destination. Flume distribute log events by event header. So we google to find out which field in header is referring to scribe header.

https://apache.googlesource.com/flume/+/d66bf94b1dd059bc7e4b1ff332be59a280498077/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java

category in header will refer to scribe category. So we try to use multiplexing source:

elprup

替换上你的Email，密码，域名ID，记录ID等参数，就可以运行了。 会在后台一直运行，每隔30秒检查一遍IP，如果修改了就更新IP。

获得domain_id可以用curl curl -k https://dnsapi.cn/Domain.List -d "login_email=xxx&login_password=xxx"

获得record_id类似 curl -k https://dnsapi.cn/Record.List -d "login_email=xxx&login_password=xxx&domain_id=xxx"

elprup

#coding=utf8

import urllib
import urllib2
import cookielib
import base64
import re, sys, json
import binascii
import rsa

elprup

"""
Simple example for streamcallback usage in Tornado.
"""
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import tornado.httpclient

from tornado.options import define, options

elprup

import os
from Crypto.Cipher import AES
import pyaes
import base64

def _create_secret_key(size):
    randlist = map(lambda xx: (hex(ord(xx))[2:]), os.urandom(size))
    return (''.join(randlist))[0:16]

def _aes_encrypt2(text, sec_key):