saryeHaddadi

UPDATE: I have baked the ideas in this file inside a Python CLI tool called pyds-cli. Please find it here: https://github.com/ericmjl/pyds-cli

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

Code is clean if it can be understood easily – by everyone on the team. Clean code can be read and enhanced by a developer other than its original author. With understandability comes readability, changeability, extensibility and maintainability.

General rules

Follow standard conventions.
Keep it simple stupid. Simpler is always better. Reduce complexity as much as possible.
Boy scout rule. Leave the campground cleaner than you found it.
Always find root cause. Always look for the root cause of a problem.

Design rules

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

	import pyodbc
	import pandas as pd

	df = pd.read_csv('myfile.csv')
	MY_TABLE = 'some_tbl'

	conn = pyodbc.connect(driver='{ODBC Driver 17 for SQL Server}',
	server='MYSERVER',
	database='MYDB',
	uid='MYUSER', pwd='MYPASSWORD')

	CIDR Total Number Network Description:
	Notation: of addresses: Mask:
	--------------------------------------------------------------------
	/0 4,294,967,296 0.0.0.0 All Addresses
	/1 2,147,483,684 128.0.0.0 128 /8 networks
	/2 1,073,741,824 192.0.0.0 64 /8 networks
	/3 536,870,912 224.0.0.0 32 /8 networks
	/4 268,435,456 240.0.0.0 16 /8 networks
	/5 134,217,728 248.0.0.0 8 /8 networks
	/6 67,108,864 252.0.0.0 4 /8 networks

	# Source: https://stackoverflow.com/a/43317244
	$path = ".\aws-ec2-key.pem"
	# Reset to remove explict permissions
	icacls.exe $path /reset
	# Give current user explicit read-permission
	icacls.exe $path /GRANT:R "$($env:USERNAME):(R)"
	# Disable inheritance and remove inherited permissions
	icacls.exe $path /inheritance:r

	syntax on
	filetype plugin indent on

	"Get the 2-space YAML as the default when hit carriage return after the colon
	autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab

	set is hlsearch ai ic scs
	nnoremap <esc><esc> :nohls<cr>

	"https://vim.fandom.com/wiki/Moving_lines_up_or_down