gordonmurray

Firn 0.7.1 — SciFact nprobes sweep

Follow-up to the baseline reproduction at https://gist.github.com/gordonmurray/97c10e02081fd2acf50283d5c53347ec.

Same dataset (BEIR SciFact), same model (lightonai/LateOn), same namespace, same IVF_PQ index. The only thing being varied is the query-time nprobes value.

Files

gordonmurray

Firn 0.7.1 — PyLate / LateOn SciFact reproduction

Results and infra recipe for reproducing an earlier PyLate-on-Firn BEIR SciFact run on local hardware.

Files

• firn-beir-scifact.md — the writeup. Quality, latency, cache metrics, follow-ups.
• docker-compose.override.yml — drops into a firnflow checkout

gordonmurray

Using Skeema to create or alter database tables

In this example we will use Skeema to create and alter some tables in a MariaDB database.

Skeema is a very useful tool that works with mySQL or Mariadb databases to help you to compare your development database files to a production database. Skeema will generate any alter statements that are required and apply those changes to your production database.

Skeema is a free and open source tool developed using Go and is available on Github at https://github.com/skeema/skeema

If you have very large tables in your production database, performing alters to the tables can take a lot of time to complete fully and may cause problems for your users if there are users actively using your software while you change the database.

gordonmurray

#!/usr/bin/env bash

set -xe

REGION="eu-west-1"

# list RDS instances to a file
aws rds --region ${REGION} describe-db-instances | jq -r '.DBInstances[].DBInstanceIdentifier' > rds.log

# loop over the rds instances in the file

gordonmurray

# Simple script to list all RDS instances and then disable Automatic Minor Version Upgrades 

# list RDS instances to a file
aws rds --region eu-west-1 describe-db-instances | jq -r '.DBInstances[].DBInstanceIdentifier' > rds.log

# loop over the items in the file, disable auto minor version upgrades
while read rds; do
    aws rds modify-db-instance --region eu-west-1 --db-instance-identifier ${rds} --no-auto-minor-version-upgrade
    sleep 5
done <rds.log

gordonmurray

# Moved to a Repo here instead and added some additional files
https://github.com/gordonmurray/ecs_cluster_using_aws_cli

gordonmurray

#!/usr/bin/env bash 

# The following assumes you have an AWS account with the AWS CLI installed locally
# It will ask which VPC and Subnets to use

# Show the commands being executed
set -ex

# AWS region
REGION="eu-west-1"

gordonmurray

Convert Caddy Server certificates to LetsEncrypt certificates to be used by Nginx

Caddy

When using Caddy Server, it stores certificates in ~/.caddy/acme/acme-v01.api.letsencrypt.org/sites/{your domain name}/

3 files are stored in the folder called:

• {yourdomain}.crt
• {yourdomain}.json
• {yourdomain}.key

gordonmurray

# prerequisites for docker
apt-get update
apt-get -y install apt-transport-https
apt-get -y install ca-certificates
apt-get -y install curl
apt-get -y install software-properties-common

# docker repos
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - \
    && echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" >> /etc/apt/sources.list.d/additional-repositories.list \

gordonmurray

# list your aws elasticache parameter groups
aws elasticache describe-cache-parameter-groups --profile=xxxxx --region=us-east-1

# list the parameters currently set in a particular aws elasticache parameter groups
aws elasticache describe-cache-parameters --profile=xxxxx --region=us-east-1 --cache-parameter-group-name=xxxxx > xxxxx.json