qxj / example.sh

Last active October 20, 2015 07:05

tail log files and publish text stream to remote kafka server.

	#!/bin/bash

	log_agent.py publish --file '/home/work/log/weblog/web/pp-stats_.log' --file '/home/work/log/weblog/donatello/web_.log' --status ~/log_agent.status --throttling 1000 --monitor 10.161.19.223:12121

qxj / lr.py

Created April 26, 2015 12:21

Python logistic regression (with L2 regularization)

	#!/usr/bin/env python
	# -- coding: utf-8; tab-width: 4; --
	# @(#) lr.py
	# http://blog.smellthedata.com/2009/06/python-logistic-regression-with-l2.html
	#

	from scipy.optimize.optimize import fmin_cg, fmin_bfgs, fmin
	import numpy as np

	def sigmoid(x):

qxj / lda_gibbs.py

Last active August 29, 2015 14:19 — forked from mblondel/lda_gibbs.py

	"""
	(C) Mathieu Blondel - 2010
	License: BSD 3 clause

	Implementation of the collapsed Gibbs sampler for
	Latent Dirichlet Allocation, as described in

	Finding scientifc topics (Griffiths and Steyvers)
	"""

qxj / kafka.md

Last active August 29, 2015 14:20 — forked from ashrithr/kafka.md

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

Producers send messages to brokers
Consumers read messages from brokers
Messages are sent to a topic

qxj / crontab.sh

Created May 16, 2015 02:04

Collect *.cron in the directory, and then APPEND to the original crontab

	#!/usr/bin/env bash
	# @(#) crontab.sh Time-stamp: <Julian Qian 2015-05-15 18:14:28>
	# Copyright 2015 Julian Qian
	# Author: Julian Qian <[email protected]>
	# Version: $Id: crontab.sh,v 0.1 2015-05-14 10:53:03 jqian Exp $
	#

	# Collect *.cron in the directory, and then APPEND to the original crontab

	# TODO fix potential confliction when more than one crontab.sh instances are running concurrently.

qxj / hadoop_avro_job.sh

Created June 9, 2015 07:46

If input files are serialized with avro, unserialize them by org.apache.avro.mapred.AvroAsTextInputFormat in hadoop streaming.

	#!/usr/bin/env bash
	# @(#) norm.sh Time-stamp: <Julian Qian 2015-06-09 15:35:35>
	# Copyright 2015 Julian Qian
	# Author: Julian Qian <[email protected]>
	# Version: $Id: norm.sh,v 0.1 2015-06-08 18:03:30 jqian Exp $
	#

	day=$(date +%Y%m%d -d yesterday)
	input=/user/hive/warehouse/query_log/ds=$day/hr=00

qxj / MultiLayerExperiment.java

Created August 17, 2015 08:25

分层实验的示例代码。原文见这里：http://blog.sina.com.cn/s/blog_e59371cc0102vopg.html ，但是其中代码格式错乱了，帮忙整理了一下。

	// Overlapping Experiment Demo
	import java.security.MessageDigest;
	import java.security.NoSuchAlgorithmException;
	import java.security.SecureRandom;
	import java.util.LinkedList;
	import java.util.List;

	public class MultiLayerExperiment {
	private static String byteArrayToHex(byte[] byteArray) {
	char[] hexDigits = {'0', '1', '2', '3', '4', '5', '6', '7',

qxj / websvr_http.py

Last active October 19, 2017 06:49

A simple web server with rotating file logger

	#!/usr/bin/env python

	"""A simple web server with rotating file logger
	"""

	import logging
	import logging.handlers
	import BaseHTTPServer

qxj / repositories

Created January 3, 2016 08:53

sbt configuration: ~/.sbt/repositories

	[repositories]
	local

	#repox-maven: http://127.0.0.1:8078/
	#repox-ivy: http://127.0.0.1:8078/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

	oschina: http://maven.oschina.net/content/groups/public/
	oschina-ivy:http://maven.oschina.net/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

	sbt-releases-repo: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

qxj / nn_demo.py

Last active January 10, 2016 12:36

A bare bones neural network implementation to describe the inner workings of backpropagation. https://iamtrask.github.io/2015/07/12/basic-python-network/

	#!/usr/bin/env python

	import numpy as np
	X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
	y = np.array([[0,1,1,0]]).T
	alpha,hidden_dim = (0.5,4)
	np.random.seed(1)
	# randomly initialize our weights with mean 0
	synapse_0 = 2*np.random.random((3,hidden_dim)) - 1
	synapse_1 = 2*np.random.random((hidden_dim,1)) - 1

Julian Qian qxj

Introduction to Kafka