nmagee

prefect

Install

In a new virtual environment:

pip install prefect requests sympy

Server

nmagee

networks:
  redpanda_network:
    driver: bridge
volumes:
  redpanda-0: null
  redpanda-1: null
  redpanda-2: null
services:
  redpanda-0:
    command:

nmagee

Kubernetes Cron Jobs

To process multiple jobs at scale in an organized and manageable way, organizations might choose from a variety of tools.

One powerful tool to run and scheduled jobs is Kubernetes, an open source platform first created by Google. Kubernetes (or K8S) orchestrates dozens or even hundreds (or thousands) of containerized jobs using a "declared state" model - which is to say that developers describe the state of their application and the cluster makes it happen.

In this lab you will create and submit a CronJob to run in the UVA Kubernetes cluster.

Instructions

nmagee

DS2002 Lab 3 - Data Cleaning Scripts

Sample Data

• Sample data - https://s3.amazonaws.com/ds2002-resources/labs/lab3-bundle.tar.gz - tar-zipped TSV
• Stock Data - https://s3.amazonaws.com/ds2002-resources/labs/stock_data.tsv - TSV
• Flight Log - https://s3.amazonaws.com/ds2002-resources/labs/flights.csv - CSV

1. Write a bash script to fetch and decompress a remote bundle

nmagee

bash Scripting Exercises

1. Download the files below. Then try running the textstats.sh script with each of them.

2. Play wordle using the script below. Can you figure out

   • How to print the secret word before you start guessing? and
   • Where bash is getting the list of 5-letter words?

3. Using the fileinfo.sh script below, complete the script to display basic information about any file passed as an argument to the script. You will have to complete line 8 to begin, as well as use echo to display values to the user. BONUS: add another variable that calculates how many lines the file contains. Display the 4 file attributes.

nmagee

from airflow import Dataset
from airflow.decorators import (
    dag,
    task,
) 
from airflow.models import Variable
from pendulum import datetime, duration
import requests
from pymongo import MongoClient, errors
from bson.json_util import dumps

nmagee

S3 Sample Data

Fetch dummy data folders and files

curl https://s3.amazonaws.com/ds2002-resources/data/bundle.tar.gz > \
  bundle.tar.gz && \
  tar -xzvf bundle.tar.gz && \
  rm bundle.tar.gz

nmagee

import boto3
from botocore.exceptions import ClientError

def get_secret(secret_name,region_name):

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name

nmagee

#!/bin/bash

mkfs -t xfs /dev/xvdh
mkdir /mnt/volume2
mount -t xfs /dev/xvdh /mnt/volume2
echo "/dev/xvdh /mnt/volume2 xfs defaults  0 0" > /etc/fstab

nmagee

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  AWS CloudFormation Sample Template EIP_With_Association: This template
  creates an Amazon EC2 instance and an Elastic IP Address. You will be
  billed for the AWS resources used if you create a stack from this template.
Parameters:
  InstanceType:
    Description: Web Server EC2 instance type
    Type: String
    Default: t2.small