ML_Notes_Sklearn_PyTorch.md

Machine Learning Notes

ML Concepts

• Dealing with Missing Values: For dealing with missing values in some columns of training data, one approach is to just drop the columns having missing values. But this is not recommended because the column could have important information. Instead a better approach is to fill in missing values with the column's average using sklearn.impute.SimpleImputer.

• Dealing with Categorical Data: Categorical data are like Enums - they have one of a fixed set of values (eg. male, female, other). 2 approaches:

  • Ordinal - convert to 1 numerical column having values like $0, 1, 2, ..$
  • One-Hot Encoding (eg. with pandas pd.get_dummies()) - convert to mutiple columns (one for each value in fixed set), each having value 1 or 0 to indicate whether the value is present or missing. It's impractical to do on large (> 15) no. of columns. It's usually slightly better than Ordinal encoding.

• A Random Forest model (a collection of many decision trees) randomly selects observations/rows and specific features/variables to build multiple decision trees from and then averages the results. After a large number of trees are built using this method, each tree "votes" or chooses the class, and the class receiving the most votes by a simple majority is the "winner" or predicted class. Source: random forests vs decision trees. Example: sklearn.ensemble.RandomForestClassifier(n_estimators=100, max_depth=5, random_state=1)

• Probabilistic Classification Predict the probability of output classes (ie we know true probabilities, not just true labels as in usual classification). For example, this (unsolved) problem asks us to predict probability (b/w 0 to 1) of whether a customer continues his account with the bank or closes it. Models are evaluated using Area under ROC Curve, which plots True Positive Rate against False Positive Rate at each probability threshold.

• Sampling (only in train datat to handle imbalanced class counts) using pandas df.sample(count): under-sample majority class with replace=False (default), over-sample minority class with replace=True. A simple binary-class example:

# Training data has 600 Normal (0), 60 Fault (1) rows -- undersample or oversample on training data not test data

# oversample minority class (Fault)
Xtrain_over = pd.concat([Xtrain[ytrain == 0], Xtrain[ytrain == 1].sample(600, replace=True)]) 
ytrain_over = pd.Series([0]*600 + [1]*600)

# undersample majority class (Normal)
Xtrain_under = pd.concat([Xtrain[ytrain == 0].sample(60), Xtrain[ytrain == 1]])
ytrain_under = pd.Series([[0]*60 + [1]*60])

Sklearn

Confusion Matrix (pretty heatmap colored display)

Classification Metrics:

from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, average_precision_score, recall_score, f1_score, balanced_accuracy_score

# Per-class precision, recall, f1 etc. and also overall accuracy, weighted accuracy
print(classification_report(ytest, ypred))

# Overall / average metrics for all test data (ytest is true labels, ypred is model predictions)
print(f'Accuracy: {accuracy_score(ytest, ypred) :.2%}')
print(f'Avg Precision: {average_precision_score(ytest, ypred) :.2%}')
print(f'Avg Recall: {recall_score(ytest, ypred) :.2%}')
print(f'F1 Score: {f1_score(ytest, ypred) :.2%}')
print(f'Balanced Accuracy: {balanced_accuracy_score(ytest, ypred) :.2%}')

Regression example with decision tree:

from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error

model = DecisionTreeRegressor()
model.fit(train_X, train_y)
ypredict = model.predict(test_X)
mean_absolute_error(test_y, ypredict)

PyTorch

• Loss nn.CrossEntropy() expects raw predicted logits and internally handles softmax, so model's output layer should NOT have softmax activation!! But this is NOT true for Binary Cross Entropy (it doesn't implicitly do Sigmoid so Sigmoid is required after output linear layer)