A guide to the basic concepts of Airflow and how to implement data engineering workflows in production
By Mike Metzger, Data Engineer Consultant @ Flexible Creations
An introduction to the components of Apache Airflow and why to use them
| # pip install docxcompose | |
| # Combines all word files under docs folder | |
| import os | |
| from pathlib import Path | |
| from docxcompose.composer import Composer | |
| from docx import Document as Document_compose | |
A guide to the basic concepts of Airflow and how to implement data engineering workflows in production
By Mike Metzger, Data Engineer Consultant @ Flexible Creations
An introduction to the components of Apache Airflow and why to use them
| #/usr/bin/bash | |
| # strings to look for in our file | |
| # Note: you could just parse the whole file. But if you put in a bad password your IP | |
| # could end up on the bad guy list | |
| declare -a badstrings=("Failed password for invalid user" | |
| "input_userauth_request: invalid user" | |
| "pam_unix(sshd:auth): check pass; user unknown" | |
| "input_userauth_request: invalid user" | |
| "does not map back to the address" | |
| "pam_unix(sshd:auth): authentication failure" |
| # deb cdrom:[Ubuntu 19.10 _Eoan Ermine_ - Release amd64 (20191017)]/ eoan main restricted | |
| # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to | |
| # newer versions of the distribution. | |
| deb http://id.archive.ubuntu.com/ubuntu/ eoan main restricted | |
| # deb-src http://id.archive.ubuntu.com/ubuntu/ eoan main restricted | |
| ## Major bug fix updates produced after the final release of the | |
| ## distribution. | |
| deb http://id.archive.ubuntu.com/ubuntu/ eoan-updates main restricted |
| # ------------------------------------------------ | |
| # Config files are located in /etc/wireguard/wg0 | |
| # ------------------------------------------------ | |
| # ---------- Server Config ---------- | |
| [Interface] | |
| Address = 10.10.0.1/24 # IPV4 CIDR | |
| Address = fd86:ea04:1111::1/64 # IPV6 CIDR | |
| PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Add forwarding when VPN is started | |
| PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE # Remove forwarding when VPN is shutdown |
| ' https://medium.com/@bhupathy/install-terminator-on-windows-with-wsl-2826591d2156 | |
| set shell = CreateObject("Wscript.Shell") | |
| xServerProcessName = "vcxsrv.exe" | |
| RunXserverProcess( xServerProcessName ) | |
| RunTerminator() | |
| KillXserverProcess( xServerProcessName ) |
| Preface: | |
| I have no idea if any of this is the "right" way of doing it. This is just how I got my unit working. | |
| Good luck. | |
| WARNING: Do not use a V1 Waveshare display module with a stock pwnagotchi! The pwnagotchi expects a V2 module, and may irreparably damage a V1 module! | |
| A V1 display module *can* work, but you must modify `waveshare.py` and `display.py` beforehand. For the time being, this is left as an exercise for the reader. | |
| If you have already burnt a V1 display, try disconnecting it from the Pi and leaving it overnight. This may rejuvenate the display somewhat, but it will likely still display signs of damage. | |
| Installation: | |
| 1. `dd` Raspbian Lite to an SD card. |
| ####### installation of self-signed certificate into CentOS7 (RHEL) ####### | |
| mkdir -p /etc/pki/ca-trust/source/anchors | |
| cp /path/to/self_signed_cert.crt /etc/pki/ca-trust/source/anchors/self_signed_cert.crt | |
| chmod +x /etc/pki/ca-trust/source/anchors/self_signed_cert.crt | |
| sudo update-ca-trust | |
| ####### installation of self-signed certificate into Ubuntu (Debian) ####### | |
| mkdir -p /usr/local/share/ca-certificates/ | |
| cp /path/to/self_signed_cert.crt /usr/local/share/ca-certificates/self_signed_cert.crt | |
| chmod +x /usr/local/share/ca-certificates/self_signed_cert.crt |
| #!/usr/bin/env bash | |
| ### Bash Environment Setup | |
| # http://redsymbol.net/articles/unofficial-bash-strict-mode/ | |
| # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html | |
| # set -o xtrace | |
| set -o errexit | |
| set -o errtrace | |
| set -o nounset | |
| set -o pipefail |
Andy Thomason is a Senior Programmer at Genomics PLC. He has been witing graphics systems, games and compilers since the '70s and specialises in code performance.