David Neubauer ddneat

Scaling Web Scraper

Session at SoCraTes Linz, 29. October 2016 https://twitter.com/SoCraTesAT

Keep in Mind

Just because I'm able to... doesn't mean I'm allowed to.
We should care about the law.
DANGER: Copyright
Keep in Mind - German Version: https://de.wikipedia.org/wiki/Datenbankwerk

CSS Naming Convention (BEM)

COMPONENTS

A page module that has a certain purpose and is a wrapper for it’s children, in example a modal or a slider can be a component. Components use a name:

.component {}
.component-name {}

Server: Ubuntu 10.04 / 64 bit, User: root

update packages

apt-get upgrade
apt-get update

=================================================================

curl multiple files

curl -O -v --cookie "insert_cookie" https://images-url.com/?id=[0-10]

lorempixel with custom name

for i in {1..10}; do wget -O $i.jpg http://lorempixel.com/512/512; done;

download from lorempixel

for i in {1..100}; do wget http://lorempixel.com/512/512; done

	function isInside(p, [a, b]) {
	return (b[0] - a[0]) * (p[1] - a[1]) > (b[1] - a[1]) * (p[0] - a[0]);
	}

	function getEdges(polygon) {
	let edges = [];

	for (let i = 0; i < polygon.length; i++) {
	let edge = [polygon[(i + polygon.length - 1) % polygon.length], polygon[i]];

	Latency Comparison Numbers
	--------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	" Options
	set shiftwidth=2
	set softtabstop=2
	set tabstop=2
	set expandtab
	set number
	set ignorecase
	set hlsearch
	set noeb vb t_vb=
	set wildmenu

	// if you would like to use the custom easing you need to use following dependency
	// http://gsgd.co.uk/sandbox/jquery/easing/

	$('.scroll').on('click', 'a', function(event){
	event.preventDefault();
	var target = $(this).attr('href');
	$('html, body').animate({
	scrollTop: $(target).offset().top - $('header').height()
	}, 700, 'easeInOutCubic' );
	});

	download and unzip manymes-chrome-extension.zip

	open chrome://extensions/

	activate developer mode (checkbox)

	press load unpacked extensions...

	choose the unpacked manymes-crhome-extension folder

	# Flatten a transparent image with a white background:
	convert -flatten img1.png img1-white.png

	# Make an image transparent
	convert -transparent '#FFFFFF' nontransparent.gif transparent.png

	# convert an image into tiles
	convert -size 3200x3200 tile:single_tile.png final.png

	# making a montage from a collection of images