Vitor Cabral vcabral19

Software Data Engineer with a background in bringing real-time analytics applications into production

vcabral19 / date_utils_for_spark_etl.py

Created May 13, 2022 12:56

some useful date functions (python + pyspark)

	from datetime import date, datetime

	def _convert_date_string_to_epoch(date_string: str) -> Union[int, float]:
	date_iso = date.fromisoformat(date_string)
	return datetime.fromordinal(date_iso.toordinal()).timestamp()


	def _convert_epoch_to_date_string(epoch_time: int) -> str:
	date_ts = datetime.fromtimestamp(epoch_time)
	return date_ts.strftime("%Y-%m-%d")

vcabral19 / sync_kafka_prod.py

Last active July 4, 2021 19:56

using dataclass with faker (to generate random values) I've played with producing messages to kafka in a sync fashion

	from dataclasses import dataclass, field
	import json
	import random

	from confluent_kafka import Consumer, Producer
	from confluent_kafka.admin import AdminClient, NewTopic
	from faker import Faker


	faker = Faker()

vcabral19 / async_kafka_prod_cons.py

Created July 1, 2021 20:58

playing with async python and kafka

	import asyncio

	from confluent_kafka import Consumer, Producer
	from confluent_kafka.admin import AdminClient, NewTopic


	BROKER_URL = "PLAINTEXT://localhost:9092"
	TOPIC_NAME = "my-first-python-topic"

vcabral19 / EnvVarTest.java

Last active July 1, 2021 20:59

playing with variables in java

	package com.myproject;

	import java.io.IOException;
	import java.io.InputStream;
	import java.util.Properties;

	public class EnvVarTest {

	public final Properties properties;