vitorbaptista

gov.br.txt:
  python google_search.py | grep http | tee -a gov.br.txt

vitorbaptista

Dominando o Fluxo de Trabalho com Dados - Limpeza de Dados - OpenRefine

Para exercitar o conhecimento adquirido na aula, tente transformar os dados do arquivo caged.csv no formato Tidy Data. Os principais desafios desse arquivo são:

1. Como adicionar uma coluna "UF"?
2. Como adicionar uma coluna "Ano"?
3. Como apagar as linhas irrelevantes, como nome de colunas repetidas, linhas em branco, e "Total"?

Você pode achar útil:

vitorbaptista

country,mobility_area,date,value
LY,residential,2020-03-29,21
LY,residential,2020-03-28,17
LY,residential,2020-03-27,14
LY,residential,2020-03-26,20
LY,residential,2020-03-25,19
LY,residential,2020-03-24,12
LY,residential,2020-03-23,13
LY,residential,2020-03-22,15
LY,residential,2020-03-21,11

vitorbaptista

core_members_ip_locations.tsv:
	csvcut -c6 core_members.csv | \
		grep -v ip_address | \
		xargs -L 1 ip2geotools -d dbipcity -f csv-tab

vitorbaptista

vitorbaptista

# Created by https://www.gitignore.io/api/python

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

vitorbaptista

Dados foram extraídos de http://www.camara.sp.gov.br/wp-content/uploads/salariosabertos/HTML_ativos_2017_09/todos.html (sem seguir os links).

vitorbaptista

.PHONY: clean

all: reimbursements.csv

clean:
	rm -f reimbursements.csv reimbursements.json

reimbursements.csv: reimbursements.json
	@echo 'issue_date,term_id,term,applicant_id,congressperson_id,congressperson_name,congressperson_document,party,state,subquota_id,subquota_description,subquota_group_id,subquota_group_description,supplier,cnpj_cpf,passenger,leg_of_the_trip,document_type,document_number,installment,batch_number,document_id,document_value,remark_value,total_net_value,total_reimbursement_value,probability,receipt.url,receipt.fetched,last_update,available_in_latest_dataset,suspicions.irregular_companies_classifier,suspicions.meal_price_outlier,suspicions.election_expenses,suspicions.invalid_cnpj_cpf,suspicions.over_monthly_subquota_limit,suspicions.suspicious_traveled_speed_day,suspicions.meal_price_outlier,suspicions.irregular_companies_classifier' > $@
	jq -r '.results[] | [.issue_date, .term_id, .term, .applicant_id, .congressperson_id, .congressperson_name, .congressperson_document, .party, .s

vitorbaptista

from datetime import datetime
import airflow.models
from airflow.operators.latest_only_operator import LatestOnlyOperator
import utils.helpers as helpers

args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2017, 4, 1),
    'retries': 1,

vitorbaptista

Keybase proof

I hereby claim:

• I am vitorbaptista on github.
• I am vitorbaptista (https://keybase.io/vitorbaptista) on keybase.
• I have a public key ASCzHuiG5mXAuFZmosmpdpi4kLA8YpficoW3o3cmzrpYiAo

To claim this, I am signing this object: