taiwotman

Describe a complex data modeling challenge you faced in a previous project. How did you approach the problem, and what factors influenced your decisions regarding the data model?

A complex data modelling challenge I faced was designing a data model for a multinational e-commerce company that wanted to analyze customer behaviour across multiple countries.
The complexity in this scenario arises from the need to handle diverse data sources, different types of data (structured, semi-structured, and unstructured), and the requirement to support multi-language data. Additionally, I was required to implement the data model to be scalable to handle the increasing volume of data as the company grows.

My first approach was understanding the business requirements and the type of analysis that the company wants to perform.  Second, I  explored the available data sources to understand the data’s structure, quality, and content. Third, based on the requirements and the nature of the data, as a data engineer, I chose a hy

taiwotman

docker compose up run

taiwotman

$ docker build -t <image-name> .

taiwotman

FROM python:latest

WORKDIR app
COPY . /app

RUN python3 -m pip install -r requirements.txt

taiwotman

psycopg2==2.9.3
pytest== 7.1.2

taiwotman

version: "3.9"
services:
    postgres:
      image: postgres:10.5
      restart: always
      environment:
        - POSTGRES_USER=<username>
        - POSTGRES_PASSWORD=<password>
      ports:
        - '5433:5432'

taiwotman

## Key Docker Commands

docker build --pull --no-cache --tag=<image-name> .

docker-compose up

docker-compose down

taiwotman

taiwotman

#Transformation
df = df.withColumn("latitude", col("latitude").cast(DoubleType()))\
       .withColumn("longitude", col("longitude").cast(DoubleType()))

df.printSchema()

df.show(10)


#More Insight

taiwotman

from pyspark.sql import SparkSession
from pyspark.sql.types import DoubleType
from pyspark.sql.functions import col

jar_path = 'rds_jar_driver.jar'
spark = SparkSession \
    .builder \
    .appName("AWS REDSHIFT PYSPARK APP") \
    .config("spark.jars", jar_path)\
    .config('spark.driver.extraClassPath', jar_path) \