barbietunnie

Running MongoDB locally on Apple Silicion

Manually running as a background process

If your deployment does not use TLS connections

mongod --config /opt/homebrew/etc/mongod.conf --fork

# You can optionally specify the db path

barbietunnie

How to Install Postgres 16 on Ubuntu

To install PostgreSQL 16 on Ubuntu, follow these steps:

Step 1: Update the System

Before installing PostgreSQL, ensure that your system is up-to-date.

sudo apt update

barbietunnie

Get human-friendly file size of a remote file

To determine the size of a file at a remote URL without downloading it, you can use an HTTP HEAD request. This type of request retrieves only the HTTP headers, which often include the Content-Length header that indicates the file size in bytes.

curl -sI https://datasets-documentation.s3.eu-west-3.amazonaws.com/uk-house-prices/postgres/uk_prices.sql.tar.gz | awk '/Content-Length/ { 
    size=$2; 
 split("B KB MB GB TB", unit); 

barbietunnie

How to read Parquet files with DuckDB

• Create a new DuckDB in-memory connection using DBeaver, specifying the path as :memory:

• Create a new script to run your query with, using the connection created

  SELECT * FROM parquet_schema('/path/to/parquet/file.parquet');
  
  SELECT * FROM read_parquet('/path/to/parquet/file.parquet');
  

barbietunnie

How to resolve the error "Unable to install or update any libraries ModuleNotFoundError: No module named 'apt_pkg'" on Ubuntu

The issue can be resolved by uninstalling and installing the python3-apt package

sudo apt remove python3-apt

sudo apt install python3-apt

barbietunnie

Install mysqlclient with pip on Ubuntu 22

Without installing pkg-config before attempting to install mysqlclient on Ubuntu 22 fails and subsequently breaks apt-pkg.

To install it without issues, run the following commands:

sudo apt install pkg-config
pip install mysqlclient

barbietunnie

Redshift Notes

1. Find dependent objects for a table or view in Postgresql or Redshift

SELECT dependent_ns.nspname as dependent_schema
, dependent_view.relname as dependent_view 
, source_ns.nspname as source_schema
, source_table.relname as source_table
, pg_attribute.attname as column_name

barbietunnie

Granting privileges in Postgres

Steps

Below are some common ways to grant access to a PostgreSQL user:

1. Grant CONNECT to the database:

GRANT CONNECT ON DATABASE database_name TO username;

barbietunnie

Grouping results and removing unwanted ones

Here we want to scrape product name, price and rating from ebay product pages:

url = 'https://www.ebay.com/itm/Sony-PlayStation-4-PS4-Pro-1TB-4K-Console-Black/203084236670' 

wanted_list = ['Sony PlayStation 4 PS4 Pro 1TB 4K Console - Black', 'US $349.99', '4.8'] 

scraper.build(url, wanted_list)

barbietunnie

FastAPI and Uvicorn Logging

When running FastAPI app, all the logs in console are from Uvicorn and they do not have timestamp and other useful information. As Uvicorn applies python logging module, we can override Uvicorn logging formatter by applying a new logging configuration.

Meanwhile, it's able to unify the your endpoints logging with the Uvicorn logging by configuring all of them in the config file log_conf.yaml.

Before overriding:

uvicorn main:app --reload