A guide to the basic concepts of Airflow and how to implement data engineering workflows in production
By Mike Metzger, Data Engineer Consultant @ Flexible Creations
An introduction to the components of Apache Airflow and why to use them
# pip install docxcompose | |
# Combines all word files under docs folder | |
import os | |
from pathlib import Path | |
from docxcompose.composer import Composer | |
from docx import Document as Document_compose | |
A guide to the basic concepts of Airflow and how to implement data engineering workflows in production
By Mike Metzger, Data Engineer Consultant @ Flexible Creations
An introduction to the components of Apache Airflow and why to use them
#/usr/bin/bash | |
# strings to look for in our file | |
# Note: you could just parse the whole file. But if you put in a bad password your IP | |
# could end up on the bad guy list | |
declare -a badstrings=("Failed password for invalid user" | |
"input_userauth_request: invalid user" | |
"pam_unix(sshd:auth): check pass; user unknown" | |
"input_userauth_request: invalid user" | |
"does not map back to the address" | |
"pam_unix(sshd:auth): authentication failure" |
# deb cdrom:[Ubuntu 19.10 _Eoan Ermine_ - Release amd64 (20191017)]/ eoan main restricted | |
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to | |
# newer versions of the distribution. | |
deb http://id.archive.ubuntu.com/ubuntu/ eoan main restricted | |
# deb-src http://id.archive.ubuntu.com/ubuntu/ eoan main restricted | |
## Major bug fix updates produced after the final release of the | |
## distribution. | |
deb http://id.archive.ubuntu.com/ubuntu/ eoan-updates main restricted |
# ------------------------------------------------ | |
# Config files are located in /etc/wireguard/wg0 | |
# ------------------------------------------------ | |
# ---------- Server Config ---------- | |
[Interface] | |
Address = 10.10.0.1/24 # IPV4 CIDR | |
Address = fd86:ea04:1111::1/64 # IPV6 CIDR | |
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Add forwarding when VPN is started | |
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE # Remove forwarding when VPN is shutdown |
' https://medium.com/@bhupathy/install-terminator-on-windows-with-wsl-2826591d2156 | |
set shell = CreateObject("Wscript.Shell") | |
xServerProcessName = "vcxsrv.exe" | |
RunXserverProcess( xServerProcessName ) | |
RunTerminator() | |
KillXserverProcess( xServerProcessName ) |
Preface: | |
I have no idea if any of this is the "right" way of doing it. This is just how I got my unit working. | |
Good luck. | |
WARNING: Do not use a V1 Waveshare display module with a stock pwnagotchi! The pwnagotchi expects a V2 module, and may irreparably damage a V1 module! | |
A V1 display module *can* work, but you must modify `waveshare.py` and `display.py` beforehand. For the time being, this is left as an exercise for the reader. | |
If you have already burnt a V1 display, try disconnecting it from the Pi and leaving it overnight. This may rejuvenate the display somewhat, but it will likely still display signs of damage. | |
Installation: | |
1. `dd` Raspbian Lite to an SD card. |
####### installation of self-signed certificate into CentOS7 (RHEL) ####### | |
mkdir -p /etc/pki/ca-trust/source/anchors | |
cp /path/to/self_signed_cert.crt /etc/pki/ca-trust/source/anchors/self_signed_cert.crt | |
chmod +x /etc/pki/ca-trust/source/anchors/self_signed_cert.crt | |
sudo update-ca-trust | |
####### installation of self-signed certificate into Ubuntu (Debian) ####### | |
mkdir -p /usr/local/share/ca-certificates/ | |
cp /path/to/self_signed_cert.crt /usr/local/share/ca-certificates/self_signed_cert.crt | |
chmod +x /usr/local/share/ca-certificates/self_signed_cert.crt |
#!/usr/bin/env bash | |
### Bash Environment Setup | |
# http://redsymbol.net/articles/unofficial-bash-strict-mode/ | |
# https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html | |
# set -o xtrace | |
set -o errexit | |
set -o errtrace | |
set -o nounset | |
set -o pipefail |
Andy Thomason is a Senior Programmer at Genomics PLC. He has been witing graphics systems, games and compilers since the '70s and specialises in code performance.