Wittline’s gists

Wittline / all_values.yaml

Created July 9, 2022 00:38

All values found

Wittline / data_types.yaml

Created July 9, 2022 00:37

All possible data types and their percentages

Wittline / inference_result.yaml

Last active July 9, 2022 00:28

Example of Inference result

Wittline / data.csv

Created July 9, 2022 00:02

Example of data set in csv format

Wittline / test_livy.ipynb

Last active June 13, 2022 09:53

test_livy.ipynb

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

Wittline / docker-compose.yaml

Created June 12, 2022 22:44

docker-compose. with Apache Spark Cluster, PostgreSQL and Apache Livy server

	version: '3'

	services:
	spark-master:
	image: docker.io/bitnami/spark:2
	environment:
	- SPARK_MODE=master
	- SPARK_RPC_AUTHENTICATION_ENABLED=no
	- SPARK_RPC_ENCRYPTION_ENABLED=no
	- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no

Wittline / Dockerfile

Created June 12, 2022 22:37

Dockerfile for an Apache Livy server

	FROM docker.io/bitnami/spark:2

	USER root
	ENV LIVY_HOME /opt/bitnami/livy
	WORKDIR /opt/bitnami/

	RUN install_packages unzip \
	&& curl "https://downloads.apache.org/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip" -O \
	&& unzip "apache-livy-0.7.1-incubating-bin" \
	&& rm -rf "apache-livy-0.7.1-incubating-bin.zip" \

Wittline / Dockerfile

Created June 5, 2022 16:37

	FROM python:3.8

	WORKDIR /ingestion

	COPY requirements.txt .

	COPY wait-for-it.sh /usr/wait-for-it.sh

	RUN pip install -r requirements.txt

Wittline / main.py

Created June 5, 2022 16:28

FastAPI Request

	from typing import Optional
	from fastapi import FastAPI
	from fastapi.responses import JSONResponse
	import uvicorn
	import pandas as pd
	import psycopg2
	import db_engine as dbe
	import os
	from pydantic import BaseModel
	import configparser

Wittline / db_engine.py

Created June 5, 2022 16:22

Class Responsible of handle the connection with PostgreSQL

	import psycopg2
	import pandas as pd
	from sql_queries import create_table_queries, drop_table_queries, fill_table_queries, create_constraints


	def create_connection(params):
	"""
	create a new connection with the postgreSQL
	database and return the cur and conn object
	:param params: connection string

Ramses Alexander Coraspe Valdez Wittline