This script demonstrates how to use the `find_best_split_seed` function from the `find_robust_seed` package to identify the most stable random seed for train-test splits. The goal is to ensure that the training and test datasets maintain the statistical properties of the original dataset.

fish_market_demo.py

# %%
"""
This script demonstrates how to use the `find_best_split_seed` function from the `find_robust_seed` package
implemented in the https://github.com/lucazav/Robust-Multi-Objective-Optimization-for-Train-Test-Splits repository
to identify the most stable random seed for train-test splits. The goal is to ensure that the training 
and test datasets maintain the statistical properties of the original dataset.

Key Features:
- Loads and cleans a dataset (Fish Market) using PyJanitor for column name normalization.
- Uses `find_best_split_seed` to evaluate multiple random seeds and find the most robust split.
- Supports any scikit-learn splitter (e.g., `ShuffleSplit`, `StratifiedShuffleSplit`, `GroupShuffleSplit`).
- Evaluates the best split with `compare_split_distributions` to verify statistical consistency.

"""

import numpy as np
import pandas as pd
import janitor

from datetime import datetime
from sklearn.model_selection import ShuffleSplit
from find_robust_seed import find_best_split_seed, compare_split_distributions

# %%
# Example usage with different splitters
if __name__ == "__main__":
    # Load data from an Excel file and perform initial cleaning.
    data = pd.read_csv(r'C:\Data\Fish Market\Fish.csv')
    data_cleaned = data.clean_names()  # PyJanitor normalizes column names, removing spaces/special characters.

    # Separate features (X) and target (y).
    target_col_name = 'weight'
    X = data_cleaned.drop(target_col_name, axis=1)
    y = data_cleaned[[target_col_name]]

    # Define categorical and numerical columns.
    categorical_columns = ['species']

    # Call the main function to find the best seed.
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    results_file = f'best_split_seed_results_{timestamp}.json'

    # Create a splitter (ShuffleSplit in this example)
    splitter = ShuffleSplit(n_splits=5, test_size=0.2)

    res = find_best_split_seed(
        X=X,
        y=y,
        splitter=splitter,  # Pass the splitter object
        categorical_columns=categorical_columns,
        numerical_columns=None,
        n_samples=200,
        weights={'numerical': 1.0, 'spearman': 1.0, 'cramers': 1.0, 'numcat': 1.0},
        n_cv_splits=5,
        splitter=None,
        groups=None,
        objective_functions=None,
        verbose=True,
        save_results_file=results_file,
        random_search_seed=4245
    )

# %%
    # Compare the selected train-test split against the original dataset to verify statistical similarity.
    compare_df = compare_split_distributions(
        best_seed=res['best_seed'], X=X, y=y, categorical_columns=categorical_columns
    )
    
    print(compare_df)