nmagee

DS2002 Lab 3 - Data Cleaning Scripts

Sample Data

• Sample data - https://s3.amazonaws.com/ds2002-resources/labs/lab3-bundle.tar.gz - tar-zipped TSV
• Stock Data - https://s3.amazonaws.com/ds2002-resources/labs/stock_data.tsv - TSV
• Flight Log - https://s3.amazonaws.com/ds2002-resources/labs/flights.csv - CSV

1. Write a bash script to fetch and decompress a remote bundle

nmagee

bash Scripting Exercises

1. Download the files below. Then try running the textstats.sh script with each of them.

2. Play wordle using the script below. Can you figure out

   • How to print the secret word before you start guessing? and
   • Where bash is getting the list of 5-letter words?

3. Using the fileinfo.sh script below, complete the script to display basic information about any file passed as an argument to the script. You will have to complete line 8 to begin, as well as use echo to display values to the user. BONUS: add another variable that calculates how many lines the file contains. Display the 4 file attributes.

nmagee

from airflow import Dataset
from airflow.decorators import (
    dag,
    task,
) 
from airflow.models import Variable
from pendulum import datetime, duration
import requests
from pymongo import MongoClient, errors
from bson.json_util import dumps

nmagee

S3 Sample Data

Fetch dummy data folders and files

curl https://s3.amazonaws.com/ds2002-resources/data/bundle.tar.gz > \
  bundle.tar.gz && \
  tar -xzvf bundle.tar.gz && \
  rm bundle.tar.gz

nmagee

import boto3
from botocore.exceptions import ClientError

def get_secret(secret_name,region_name):

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name

nmagee

#!/bin/bash

mkfs -t xfs /dev/xvdh
mkdir /mnt/volume2
mount -t xfs /dev/xvdh /mnt/volume2
echo "/dev/xvdh /mnt/volume2 xfs defaults  0 0" > /etc/fstab

nmagee

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  AWS CloudFormation Sample Template EIP_With_Association: This template
  creates an Amazon EC2 instance and an Elastic IP Address. You will be
  billed for the AWS resources used if you create a stack from this template.
Parameters:
  InstanceType:
    Description: Web Server EC2 instance type
    Type: String
    Default: t2.small

nmagee

#cloud-config
repo_update: true
repo_upgrade: all

packages:
 - docker
 - jq
 - htop
 
runcmd:

nmagee

name: Remote Dispatch Action

on: [ repository_dispatch, workflow_dispatch ]

jobs:
  update-version:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Event Information

nmagee

name: Container Build CICD

on:
  push:
    branches:
    - 'main'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: org/container-name