Skip to content

Instantly share code, notes, and snippets.

View dsouzajude's full-sized avatar

Jude DSouza dsouzajude

View GitHub Profile
@dsouzajude
dsouzajude / postgres-dump-and-import-schema-without-indexes.sh
Last active October 17, 2024 13:06
Dump and import schema without indexes
# Read more about pg_dump, pre-data, post-data sections and the Fc flags at:
# https://www.postgresql.org/docs/13/app-pgdump.html
# -- Dump operations from Source Database --------------------------------------------
# Dumps the schema but excludes all the constraints and indexes (i.e pre-data section)
pg_dump -h $SOURCE_ENDPOINT --section=pre-data -U postgres -d $DB_NAME --file=schema-pre-data.dump.sql
# Dump the schema for the indices and constraints as well as we will need it later
# to apply it to the tables after the destination database is in sync.
@dsouzajude
dsouzajude / postgres-sanitize-and-import-users-to-rds.sh
Last active October 22, 2022 16:07
Sanitize and import users to RDS
# --- Dump roles from the Source Database ------------------------------
# Dump all the roles
pg_dumpall -h $SOURCE_ENDPOINT --roles-only -U postgres > roles.org.dump.sql
# Extract only the role creation / alter SQL commands
awk '/CREATE/' roles.org.dump.sql > roles.dump.sql
# Sanitize superuser and replication roles
sed -i -e's/NOSUPERUSER//g; s/SUPERUSER//g; s/NOREPLICATION//g; s/REPLICATION//g' roles.dump.sql
@dsouzajude
dsouzajude / create-db-replication-user.sql
Created October 21, 2022 15:51
Create a database replication user
-- Create replication user
CREATE USER rep_user WITH replication PASSWORD <PASSWORD>;
-- Grant SELECT and USAGE
GRANT SELECT ON ALL TABLES IN SCHEMA <SCHEMA> TO rep_user;
GRANT USAGE ON SCHEMA <SCHEMA> TO rep_user;
@dsouzajude
dsouzajude / check-tables-with-no-pk-or-unique-constraint.sql
Created October 21, 2022 15:11
Check for tables in PostgreSQL with no primary key or unique constriant
SELECT tbl.table_schema,
tbl.table_name
FROM information_schema.tables tbl
WHERE table_type = 'BASE TABLE'
AND table_schema NOT IN ('pg_catalog', 'information_schema')
AND table_name not like 'awsdms%'
AND NOT EXISTS (SELECT 1
FROM information_schema.key_column_usage kcu
WHERE kcu.table_name = tbl.table_name
AND kcu.table_schema = tbl.table_schema)
@dsouzajude
dsouzajude / consul-service-config.json
Last active September 26, 2020 12:06
Sample Consul Service config with sidecar proxy definition to export prometheus metrics
{
"service": {
"id": "0DD4A483-AFD6-4866-ABF9-9FC5A50A2AE7",
"name": "serviceA",
"port": 8080,
"address": "10.0.1.151",
"connect": {
"sidecar_service": {
"name": "serviceA-proxy",
"proxy": {
@dsouzajude
dsouzajude / consul-service-config.json
Last active September 26, 2020 11:21
Sample Consul Service config with sidecar proxy definition
{
"service": {
"name": "serviceA",
"port": 8080,
"connect": { "sidecar_service": {} }
}
}
@dsouzajude
dsouzajude / ecs-container-definitions-consul-connect-mesh.json
Last active September 4, 2020 13:52
Sample ECS Container Definitions for a service to be integrated into the Consul Connect Mesh
[
{
"name": "${service_name}",
"image": "${service_image}",
"essential": true,
"portMappings": [{"containerPort": 8080}],
"healthCheck": {
"command": ["CMD-SHELL","curl -f http://localhost:8080/health || exit 1"],
"interval": 15,
"retries": 3
@dsouzajude
dsouzajude / proxy-defaults.hcl
Last active November 25, 2021 04:54
Consul Connect Proxy Defaults with Custom Public Listener Configuration that is configured to do HTTP Tracing
Kind = "proxy-defaults"
Name = "global"
Config {
local_connect_timeout_ms = 1000
handshake_timeout_ms = 10000
protocol = "http"
bind_address = "0.0.0.0"
bind_port = 21000
envoy_stats_flush_interval = "60s"
@dsouzajude
dsouzajude / proxy-defaults.hcl
Last active September 3, 2020 19:28
Consul Connect Proxy Defaults with Custom Public Listener Configuration that is configured with Access Logging
Kind = "proxy-defaults"
Name = "global"
Config {
local_connect_timeout_ms = 1000
handshake_timeout_ms = 10000
protocol = "http"
bind_address = "0.0.0.0"
bind_port = 21000
envoy_stats_flush_interval = "60s"
@dsouzajude
dsouzajude / prometheus-consul-envoy.yml
Created September 3, 2020 11:12
Prometheus Config with Consul SD
global:
scrape_interval: 10s
external_labels:
Environment: sandbox
Region: eu-west-1
Source: prometheus
scrape_configs:
- job_name: consul-services
metrics_path: "/metrics"