Skip to content

Instantly share code, notes, and snippets.

@luisdelatorre012
luisdelatorre012 / kafka_consumer.py
Last active August 27, 2024 16:37
Kafka Consumer
import json
import signal
import asyncio
from confluent_kafka import Consumer, KafkaError, Message
from fastavro import reader
import io
from sqlalchemy.ext.asyncio import create_async_engine, AsyncEngine
from sqlalchemy import Table, Column, Integer, TIMESTAMP, MetaData
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.exc import SQLAlchemyError
SELECT
ServiceProviderAccountNumber,
EventRecordedTime,
ServiceProviderLocation.value AS ServiceProviderLocation
FROM
test.ServiceProviderAccountChangedEventRaw AS d
CROSS APPLY
OPENJSON(d.message)
WITH (
ServiceProviderAccountNumber varchar(20) '$.data.id',
@luisdelatorre012
luisdelatorre012 / consecutive_months.dax
Last active August 12, 2024 16:32
consecutive months
ConsecutiveMonthsWithSale =
VAR RefDate = MAX('Date'[Date])
VAR RefMonth = EOMONTH(RefDate, 0)
VAR AllMonths =
CALCULATETABLE(
DISTINCT(
ADDCOLUMNS(
VALUES('Date'[Date]),
"MonthEnd", EOMONTH('Date'[Date], 0)
)
@luisdelatorre012
luisdelatorre012 / fake_customer_data.py
Last active August 8, 2024 12:45
fake customer data
import polars as pl
from faker import Faker
import random
# Initialize the Faker instance
fake = Faker()
# Define the number of unique customers and the total number of records
num_customers = 100
num_records = 1000
@luisdelatorre012
luisdelatorre012 / unnest_example_bigquery.sql
Created July 25, 2024 16:41
unnest example big query
SELECT
text_column,
element AS filtered_element
FROM
your_table,
UNNEST(
ARRAY(
SELECT
element
FROM (
@luisdelatorre012
luisdelatorre012 / example_pyproject.toml
Created July 22, 2024 23:20
Example pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "your_package_name" # Replace with the actual package name
version = "0.1.9"
description = "Package description"
authors = [
{ name = "author name", email = "[email protected]" },
@luisdelatorre012
luisdelatorre012 / pre_commit_hooks.md
Last active July 12, 2024 01:01
How to set up pre-commit hooks

Check this out: https://github.com/astral-sh/ruff-pre-commit

You:

walk me through setting up pre-commit hooks. I'm using pycharm professional.

ChatGPT:

Setting up pre-commit hooks in PyCharm Professional is a great way to ensure your code meets certain standards before committing it to your version control system. Here’s a step-by-step guide to get you started:

@luisdelatorre012
luisdelatorre012 / entra_id.py
Last active September 9, 2024 11:58
How to use Microsoft Entra ID for authentication
from dynaconf import Dynaconf
import msal
import httpx
class TokenAcquisitionError(Exception):
"""Custom exception for token acquisition errors."""
pass
def get_access_token():
config = Dynaconf(

You:

I'm using ansible to automate deployment onto a machine that will need to pull from a git repo. should each machine get a unique git ssh key?

ChatGPT:

When using Ansible to automate deployment and manage access to a Git repository, it's important to consider the security implications and the management overhead. Here are the key points to consider:

Using Unique Git SSH Keys

@luisdelatorre012
luisdelatorre012 / job_scheduler.py
Last active June 18, 2024 11:55
Job scheduler with no overlap
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from ortools.sat.python import cp_model
# Initialize the random generator
rng = np.random.default_rng(seed=42) # For reproducibility
# Generate random data for 10 jobs
num_jobs = 10