philerooski

"""Incrementally backfills COPY_FILES_TASK / COPY_CHANGES_TASK / COPY_SENT_MESSAGES_TASK's
RDS_SNAPSHOTS_STAGE backlog one snapshot_date at a time (SNOW-580).

These tasks time out trying to COPY INTO their entire backlog (150K+ files,
1TB+) in a single statement, even on COMPUTE_MEDIUM. This script drains the
backlog day by day instead: each COPY INTO is scoped to one snapshot_date via
the PATTERN clause, which keeps each run's file count/volume small enough to
finish well inside the warehouse statement timeout. It's safe to re-run --
COPY INTO skips files it has already loaded -- so a partial or interrupted run
can just be re-invoked to pick up where it left off.

philerooski

"""
Load RDS snapshot data from S3 via Snowflake external stage into tables.

This script supports two modes:
1. Bootstrap mode (--bootstrap-stack): Creates a new schema, external stage,
   file format, and grants privileges before loading data.
2. Manual mode: Loads data into an existing schema with pre-configured stage.

The script dynamically discovers all data types from the S3 stage URL,
creates tables using INFER_SCHEMA from Parquet files, loads the data,

philerooski

"""
Verify that all grants introduced in PR #317 (SNOW-460) are applied.

Checks:
  1. Roles exist (new roles added in roles.sql)
  2. Schemas exist (schema init migrations)
  3. Schema ownership (ownership_grants migrations)
  4. Database-level privileges (USAGE, MONITOR on SAGE)
  5. Schema-level privileges (USAGE, MONITOR on SAGE.<schema>)
  6. Role-to-role grants (analyst roles → DATA_ENGINEER, admin roles → SAGE_ADMIN, etc.)

philerooski

USE ROLE USERADMIN;

-- Verification script for DEFAULT_SECONDARY_ROLES test runs.
-- Run this before and after test scripts to compare distribution changes.

-- ============================================================================
-- SECTION 1: SUMMARY COUNT BY DEFAULT_SECONDARY_ROLES BUCKET
-- Captures counts of users in ALL_ENABLED, EMPTY ([]), NULL, and OTHER buckets.
-- ============================================================================
SELECT 'SECTION 1: SUMMARY COUNT BY DSR BUCKET' AS report_section;

philerooski

USE ROLE USERADMIN;

-- Rollback script for test_set_default_secondary_roles_by_analyst_rules.sql.
-- Resets DEFAULT_SECONDARY_ROLES to empty for all users except SNOWFLAKE.
EXECUTE IMMEDIATE $$
DECLARE
    updated_users ARRAY DEFAULT ARRAY_CONSTRUCT();
    username STRING;
    dsr_value STRING;
    user_cursor CURSOR FOR

philerooski

USE ROLE USERADMIN;

-- Copied from admin/users.sql lines 165-254 for isolated testing.
-- Set DEFAULT_SECONDARY_ROLES based on user type and role access.
-- A user is treated as an analyst if ALL of the following are true:
--   1) user type is not SERVICE
--   2) user name does not contain 'service' (case-insensitive)
--   3) user is not granted any of these roles:
--      DATA_ENGINEER, ACCOUNTADMIN, SYSADMIN, SECURITYADMIN, USERADMIN
-- Analysts get DEFAULT_SECONDARY_ROLES=('ALL').

philerooski

"""
Verify snapshot data load integrity.

This script checks:
1. LOAD_LOG table for any errors or anomalies
2. Compares tables between PROD_576 and PROD_568 schemas
3. Validates record counts (PROD_576 should have more records)

Usage:
    python verify_snapshot_load.py --database SYNAPSE_RDS_SNAPSHOT \\

philerooski

#!/usr/bin/env python3
"""
Download all form data from a specific form group to local directory.
"""

import argparse
import sys
import json
import tempfile
import shutil

philerooski

"""
Load RDS snapshot data from S3 via Snowflake external stage into tables.

This script supports two modes:
1. Bootstrap mode (--bootstrap-stack): Creates a new schema, external stage,
   file format, and grants privileges before loading data.
2. Manual mode: Loads data into an existing schema with pre-configured stage.

The script dynamically discovers all data types from the S3 stage URL,
creates tables using INFER_SCHEMA from Parquet files, loads the data,

philerooski

"""
Analyze errors from the LOAD_LOG table.

This script queries the LOAD_LOG table for failed operations,
categorizes errors by type, and groups data types by error category.

This is a complementary script to https://gist.github.com/philerooski/a740b25f066f1ad205344637160aa969
"""

import snowflake.connector