j-thepac

Principles

• Loose Coupling
• High Cohesion
• Change is Local
• It is Easy to Remove

Smells

• Rigidity ( A -> B -> C . something hardcoded in C )
• Fragility
• Immobility

j-thepac

DP 900

Azure Data Fundamentals: Explore core data concepts

• Type: structured, semi-structured, or unstructured.
• Data stores : File stores ,Databases
• file Formats :
  • Delimited text files
  • JavaScript Object Notation (JSON)
  • Extensible Markup Language (XML)
• Optimized File Format

j-thepac

Kubernetes

Problem with VM:

• Virtual machines comes with unwanted apps and services Eg: Browser
• All process / Threads were not is user control .
• Memory allocation was not in user control.
• Syncing Virtual machines is not easy
• Lot of configuration was needed
• To delete and reconfigure was a nightmare.

Kubernetes

• configure, automate, and manage applications using containers Image

j-thepac

Synapse

Note :

• For JSON file make sure all new lines , blank spaces are removed.
• Dataflow , Notebooks etc - names has to start with String.
• blob = flat namespace , Gen2 = directories
• Azure Active Directory (Entra ID) : Used to Create 1 Functional ID (secret ) called Serive Principle across all applications to communicate with each other
• SP ( serivce Principle) : Managed by User ie., renew etc.,-
• Managed identity (service principal) : managed by Azure

j-thepac

DOCKER

Env : https://labs.play-with-docker.com/

• Docker = Container and Images
• Docker Can be executed as
  • command line
  • dockerfile
  • docker-compose.yml

Image

Snapshot of the application containing all the libraries , supporting files etc.,

j-thepac

Kafka

kafkaSample

Docker YML

## docker-compose.yml (start kafka Broker and Zookeeper)

    #docker-compose up -d #detach mode
	version: '3'
	services:
 zookeeper:

j-thepac

System

• Sop (std operation procedure)
• Rca (root cause analysis)
• Modular
• Scalable
• Scheme agnostic (No SqL) : No Schema
• Model
• Decouple

j-thepac

ETL

BigData:

• OLTP (online Transactional processing)
• OLAP(online analytical processing)

Data processing systems

• OLTP (OnLine Transaction Processing) : is used for managing current day to day data information.
  • ACID
• Atomicity(entire transaction happens at once or nothing happens)

j-thepac

from fastapi import FastAPI,Request
from pydantic import BaseModel
import uvicorn
from smslottery import validate_bet
from fastapi.middleware.cors import CORSMiddleware

origins = ["*"]
class Data(BaseModel):
    game:str
    text:str

j-thepac

Important Resources

Open Applications

• OSX - open is a universal opener in OSX
• Linux - has xdg-open (open bashrc > add "alias open = xdg-open" > save file >source ./bashrc)
• Windows - use start

Online Virtual terminals