plembo

Read a large data file with pandas

Below is some working code I used to read a large tab-delimited data file. The file was over 3 Gb uncompressed and couldn't be loaded on a laptop with 8 GB RAM.

There are a lot of different ways to handle insufficient memory problems in pandas. In this case I used the built-in chunksize method to first load the data in chunks, and then iterated over them before concatenating into a single dataframe.

import pandas as pd
. . .
cols = [...]

plembo

Ubiquiti Edgerouter firewall rules for IOT networks

Just going to present several variations on a theme here, tested with a Ubiquiti EdgeRouter 4 in my home lab.

Several resources were consulted in the process of creating these firewall rules, cited below under "Resources".

In these examples, the "default" or "management" VLAN1 is VLAN1, on 192.168.1.0/24. A separate VLAN8 was created for IOT devices on 192.168.8.0/24, along with its own DHCP service on the router.

The minimum requirements here are to have the IOT devices on VLAN8 network get an address from the VLAN8 DHCP server and access the Internet through the VLAN's gateway (192.168.8.1), allow managment network access to the

plembo

Create a firewall group on an EdgeRouter

When you have more than one network to assign to a firewall rule it can sometimes be more efficient to create a firewall group and then name that in the rule. It also can make the rule easier to read. Here are the commands to create a firewall group (of type "network") called LAN_NETWORKS on an EdgeRouter:

set firewall group network-group LAN_NETWORKS description 'RFC1918 Private Networks'
set firewall group network-group LAN_NETWORKS network 192.168.0.0/16
set firewall group network-group LAN_NETWORKS network 172.16.0.0/12
set firewall group network-group LAN_NETWORKS network 10.0.0.0/8

plembo

Honeywell VisionPro 8000 on an IoT VLAN

Honeywell wifi thermostats have a bit of a reputation when it comes to playing nice with even the simplest home network. It only gets worse on slightly more sophisticated environments. But I wanted to put all my IOT devices on their own VLAN for security, so it was time to face the angry bear.

Note that many hours were wasted due to my own ignorance, and a remarkable lack of practical examples in product documentation. Sometimes I thought that the manufacturers really didn't want anyone to succeed at this. Having said that, it's clear from the many responses to questions in customer forums that isn't actually the case. It's just the operation of the old adage, "Those who can, can't teach".

plembo

Manually updating the SSL certificates on the Ubiquiti EdgeRouter 4 using the CLI

Always know how to do... whatever using the cli.

1. Obtain the server key, server cert and intermediate cert in PEM format. I use letsencrypt.org, and so name my intermediate cert "letsencrypt-chain.crt". The system doesn't care what you use as a filename extension. I usually use ".key" for keys, and ".crt" for certs. In the case of the combo cert I used ".pem" just because...

2. Combine the server key and cert into a single file (in that order):

$ cat server-key.pem server-cert.pem > _.example.com-combo.pem

You can name things anything you want so long as you make sure the lighttpd uses those names

plembo

Bulk convert a directory of files with ffmpeg

A very simple bash one-liner. Note that the input and output formats can be anything that ffmpeg supports. In my case the input was a pile of (very old) short QuickTime videos of the kids that I wanted to play with more modern software.

Files that had been named something like "20031201.mov" before conversion, were renamed to "20031201.mp4".

$ for i in *.mov; do ffmpeg -i "$i" "${i%.*}.mp4"; done

Many people think that duct tape is the single most important invention of all time.

plembo

Want to configure UPnP on EdgeRouter-4?

Don't do it!

Introduction

As a former sysadmin that once helped ride herd over around 1,000 servers, of which around 10% were Internet-facing, I've never been a fan of autoconfiguation when it comes to punching holes through the firewall. I've seen the tripwire logs, and it's sobering.

The Problem

So enabling UPnP on my EdgeRouter-4 wasn't something I had on the roadmap, but when I ran into difficulty getting DLNA to work on the default VLAN, I thought it was worth looking into (never mind that, as I later learned, having UPnP turned on wasn't going make a bit of difference on my local network: another reason that mastering theory should always come before practice).

plembo

KVM name resolver secrets

Just a few notes on configuring the built in DNS name-resolver for KVM (libvirtd).

Not really secrets, because everything below is documented on the libvirt wiki under Network XML Format.

Adding a search domain

If you want to pass a search domain to virtual hosts that get their network config automatically with DHCP, you can add a domain tag to each virtual network config, like the "default" network.

plembo

Reserving DHCP addresses in KVM

In using KVM (libvirtd) as part of my virtual "lab" environment at home, I often have at least a few guests using DHCP whose addresses have been reserved. Setting this up is simple, once you've read the documentation.

This configuation is done at the network level and found in the /etc/libvirt/qemu/networks/[networkname].xml file.

Here's a sample (from the doc) from the <ip> section for the configuration:

plembo

Changing a KVM (libvirtd) network configuration

The documentation recommends you do this using virsh net update, but you'll need to read it very carefully to figure out exactly how -- because they provide you with only a single example to work with.

Only the following virtual network components can be changed using net-update:

ip-dhcp-host
ip-dhcp-range (add/delete only, no modify)
forward-interface (add/delete only)