Saketha Ramanjam sakethramanujam

⚠️ This post assumes that you have a basic understanding of the Airflow Web UI! ⚠️

Connecting a Local Database on Apache Airflow

What's the problem? 🤔

Once the airflow environment is setup, dags are written and the webserver is launched the ui would be something like this So, here's my sample workflow dag that I've written. I've backfilled it for testing purposes and so it does display the task status and other metrics.

Raspberry Pi InfluxDB: The solution for IoT Data storage

Raspberry Pi is costeffect linux computer very commonly used for IoT home automation projects.

Here are the 3 problems with conventional databases for IoT data store applications:

Too much or complex configuration
Unable to expire data / set retentional policies
Not tailor made of Time Series Data

	# !usr/bin/env python3
	from bs4 import BeautifulSoup
	import sys
	import pandas as pd
	import argparse


	def args():
	parser = argparse.ArgumentParser()
	parser.add_argument('-f','--filename',help='path to input', type=str)

	#! /usr/bin/env python3
	import platform
	import os
	import subprocess
	import requests
	import zipfile
	import io
	import re

	import pandas as pd
	import urllib
	from bs4 import BeautifulSoup
	import argparse


	def _parse():
	parser = argparse.ArgumentParser()
	parser.add_argument('-html', type=str, help='path to html file')
	parser.add_argument('-f', '--filename', type=str, help='name to save the output')

	import asyncio
	import json
	import urllib.request
	from pymongo import MongoClient


	def connect(database_name: str, collection_name: str) -> object:
	try:
	conn = MongoClient()
	db = conn.snapi

	#!/usr/bin/python3
	from airflow.models import DAG
	from airflow.operators.bash_operator import BashOperator
	from datetime import datetime, timedelta

	# let's setup arguments for our dag

	my_dag_id = "my_first_dag"

	default_args = {

	from airflow import DAG
	from airflow.operators.bash_operator import BashOperator
	from airflow.operators.python_operator import PythonOperator
	from summary import read_file,daily,monthly
	import datetime as dt

	default_args={
	'owner': 'airflow',
	'start_date': dt.datetime(2019, 5, 31, 10, 00, 00),
	'concurrency':1,