weijh imweijh

Set up geospatial scientific Python with Miniconda on Windows

Anaconda is an excellent, simple way to get Python up and running on your computer. But, it includes a lot of packages you'll never use but consume gigs and gigs of hard drive space. Instead, you can just install miniconda and then choose the individual packages you need. The steps below explain how to do this to set up a Python environment for geospatial data science. These steps are Windows-specific, but the same process works on Mac or Linux (just don't download the wheels from Gohlke - conda/pip install them directly). If you're having trouble, here are more detailed instructions on getting geopandas and geospatial Python up and running.

Install Miniconda

Download the Miniconda for Python 3.5 installer
Install miniconda to C:\Anaconda and set it as the system's default Python

Send docker logs to ELK through gelf log driver

There's so many way to send logs to an elk... logspout, filebeat, journalbeat, etc.

But docker has a gelf log driver and logstash a gelf input. So here we are.

Here is a docker-compose to test a full elk with a container sending logs via gelf.

IPAM and CMDB software

IPAM

HaCi http://haci.larsux.de/ - 2015-03
- IPAM only, v4/v6, multiple POPs, uses templates, space visualization
GestioIP https://www.gestioip.net/ - 2016-02

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	# this is an alternate compact function implementation which also removes keys with empty string values
	# writing the tests for this is left as an excercise to the reader :D

	def compact(h)
	h.inject({}) do \|result, (k, v)\|
	if v.is_a?(Hash)
	result[k] = compact(v)
	elsif v.is_a?(String)
	result[k] = v unless v.empty?
	elsif !v.nil?

	input {
	generator {
	message => "2020-02-10T15:14:15.003Z"
	count => 1
	}
	}

	filter {

	ruby {

	from urllib.request import urlopen
	from bs4 import BeautifulSoup
	import re

	count = 1
	url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49916"
	#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49818"
	#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49633"
	#url0 = "http://zjj.sz.gov.cn/ris/bol/szfdc/projectdetail.aspx?id=49574"
	html0 = urlopen(url0)

	input {
	file {
	path => "d:/temp/2020052109.tzt"
	start_position => "beginning"
	sincedb_path => "NUL"

	codec => multiline {
	pattern => "^%{TIME}"
	negate => true
	what => previous

	用法：安装anaconda
	python szfdc.py > hr4.txt

	一次一栋，抓其他的要自己改改 urlz 链接

	#!/bin/bash

	if [ $# -lt 2 ]; then
	echo "usage:$0 dev output_dir [iodepth]"
	echo "example 1: Testing the whole block device. Attention: That will destory the filesystem on the target block device"
	echo "./run_fio.sh /dev/sdb fio_test"
	echo ""
	echo "example 2: Testing a file, but not destory filesystem. Suppose the target device mount on /data"
	echo "fallocate -l 1G /data/test.dat"
	echo "./run_fio.sh /data/test.dat fio_test"