mahynski

tl;dr

This is a guide to setting up git repos for python code that take advantage of as many automatic code linting, testing, and continuous improvement tools (CI) as I have found useful. There are certainly more. I work in a linux-based environment, and the following was tested on Ubuntu 18.04 LTS assuming you have admin privileges.

Repository configuration

Most repos are built as packages and therefore should take advantage of python's use of __init__.py files to indicate structure. This makes writing documentation, tests, and use of the code itself (importing it externally) rational and readable. Python includes a nice discussion about this here with examples on how to configure your code as a package. To be brief, here is the basic layout by example:

my_repository/

mahynski

tl;dr

An initial disclaimer: "Premature optimization is the root of all evil!" Never optimize your code before your profile it. If you determine your code is too slow, and learn why after profiling (where it is spending time), you can begin optimization.

This is very brief summary of Cython and how to use it to quickly accelerate pure python code. Cython is much more adaptable than what I discuss below; I have chosen to focus primarily on how you can use it to accelerate existing code, usually written during the course of prototyping, with minimal modification. This is my primary use case for Cython, and it allows you to use your code in production for research much more easily than trying to develop optimized C/C++ from scratch.

In fact, if you are writing scientific code and working with numerical computations, numba often allows to accelerate code even more easily. I recommend you check that out first. Cython is the ne

mahynski

tl;dr

Numba is a "just in time" (jit) compiler for python. It is designed to optimize floating point operations and loops which makes it ideal for scientific code. The simplest way to use it is to decorate you function as shown below, which instructs Numba to compile the function into fast machine code the first time it is called.

from numba import jit

@jit(nopython=True)
def dotproduct(v1, v2):
	isum = 0.0

mahynski

tl;dr

Modern computational software and hardware have made it relatively easy to process data and train machine learning models; however, implementing these models requires trust of the end user, which means explainable AI methods, such as SHAP, need to be leveraged. In the realm of scientific research another common problem is that data science/analysis is often performed by one (or a team of) individual(s), but the audience or collaborators are those with more detailed scientific knowledge of a problem and less expertise on the data science end. In order to collaborate and/or present the results to a more general audience, an interactive visualization tool is needed. This enables scientists and engineers to absorb a model, perform 'what-if" analyses to test its limits, and extract scientific knowledge. To that end, several dashboard tools have been developed. This tutorial focuses on setting up ExplainerDashboard

mahynski

tl;dr

hyperopt-sklearn is a python package that serves as a wrapper around hyperopt that extends its functionality to scikit-learn that can perform automatic hyperparameter tuning and other pipeline optimization. It is based on hyperopt and was originally presented in Komer B., Bergstra J., and Eliasmith C. "Hyperopt-Sklearn: automatic hyperparameter configuration for Scikit-learn" Proc. SciPy 2014. This is an AutoML framework though it is not quite as powerful as others like TPOT or auto-sklearn; however, it can be more transparent and easier to extract optimized models from. The primary advantage that I find with this tool is its simplicity and how ea

mahynski

tl;dr

This is an opinionated guide to using Jupyter notebooks, and it is certainly not the first. Google even has a manifesto on the subject. Depending on the application, clearly tastes will differ. I generally do python prototyping to develop scientific code and workflows using Jupyter. This means that functions and classes are usually initially spun up and then, when basically functional, converted to scripts or built into existing libraries for proper version control, unittesting, and issue tracking. There is ipytest for testing inside the notebook, but I recommend outsourcing to a repo using pre-commit, for example, so this can be tracked with git. So the notebook is primarily for (1) initial testing and sandbox-level development, or (2) for scripting these toge

mahynski

tl;dr

Google colab is essentially a web-based Jupyter notebook. It links to GitHub and/or a Google Drive account allowing you to save your work in the cloud and run on google's servers. It also gives you free access to GPU and TPU resources. Google provides a number of introductions to Colab, but if you are already familiar with python and Jupyter (and markdown) you are ready to go.

One purpose of this guide is to be a < 5 minute quick-start to get going so you can use and write (python) code, do data science, machine learning, etc. on-the-go or from whatever remote location you please.

Colab also lets you work together (as the name suggests) with other, the same as any other google document, for example.

In contrast, if you don't work remotely (from your resources) you may prefer to configure things locally. Working locally alo has many other benefits, and is still

mahynski

mahynski

Data Science

• Jupyter Notebook
• Google Colab
• Data Carpentry
• Pandas
• Pandas Cookbook
• Pandas Tutorials
• PostgreSQL
• Bokeh

mahynski

Table of Contents

1. 2D Materials from Programmable Matter
2. Using Symmetry to Count Configurations