imweijh

Send docker logs to ELK through gelf log driver

There's so many way to send logs to an elk... logspout, filebeat, journalbeat, etc.

But docker has a gelf log driver and logstash a gelf input. So here we are.

Here is a docker-compose to test a full elk with a container sending logs via gelf.

imweijh

#!/bin/bash

if [ $# -lt 2 ]; then
    echo "usage:$0 dev output_dir [iodepth]"
    echo "example 1: Testing the whole block device. Attention: That will destory the filesystem on the target block device"
    echo "./run_fio.sh /dev/sdb fio_test"
    echo ""
    echo "example 2: Testing a file, but not destory filesystem. Suppose the target device mount on /data"
    echo "fallocate -l 1G /data/test.dat"
    echo "./run_fio.sh /data/test.dat fio_test"

imweijh

input {
  file {
	path => "d:/temp/2020052109.tzt"
        start_position => "beginning"
	sincedb_path => "NUL"
	
	codec => multiline {
          pattern => "^%{TIME}"
          negate => true
          what => previous

imweijh

用法：安装anaconda
python szfdc.py > hr4.txt

一次一栋，抓其他的要自己改改 urlz 链接

imweijh

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

count = 1
url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49916"
#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49818"
#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49633"
#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49574"
html0 = urlopen(url0)

imweijh

Set up geospatial scientific Python with Miniconda on Windows

Anaconda is an excellent, simple way to get Python up and running on your computer. But, it includes a lot of packages you'll never use but consume gigs and gigs of hard drive space. Instead, you can just install miniconda and then choose the individual packages you need. The steps below explain how to do this to set up a Python environment for geospatial data science. These steps are Windows-specific, but the same process works on Mac or Linux (just don't download the wheels from Gohlke - conda/pip install them directly). If you're having trouble, here are more detailed instructions on getting geopandas and geospatial Python up and running.

Install Miniconda

• Download the Miniconda for Python 3.5 installer
• Install miniconda to C:\Anaconda and set it as the system's default Python

imweijh

input { 
  generator {
    message => "2020-02-10T15:14:15.003Z"
    count => 1
  }
}

filter {

  ruby {

imweijh

# this is an alternate compact function implementation which also removes keys with empty string values
# writing the tests for this is left as an excercise to the reader :D

def compact(h)
  h.inject({}) do |result, (k, v)|
    if v.is_a?(Hash)
      result[k] = compact(v)
    elsif v.is_a?(String)
      result[k] = v unless v.empty?
    elsif !v.nil?

imweijh

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

imweijh

input {
	beats {
		port => 5301
	}
}

filter {
	if [fields][type] == "monthly-indexed.r2web.bsd.alljob" {
		mutate {
            gsub => ["message","/"," "]