Skip to content

Instantly share code, notes, and snippets.

@ahmedmoustafa

ahmedmoustafa/dsci2410-spring2026-project.ipynb

Created May 6, 2026 07:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ahmedmoustafa/6b4cc9377dd9ff4833282183aede195c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ahmedmoustafa/6b4cc9377dd9ff4833282183aede195c to your computer and use it in GitHub Desktop.
Download ZIP
dsci2410-spring2026-project.ipynb
Display the source blob
Display the rendered blob
Raw
dsci2410-spring2026-project.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/ahmedmoustafa/6b4cc9377dd9ff4833282183aede195c/dsci2410-spring2026-project.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6hmCaMVuI0Ty"
},
"source": [
"# DSCI 2410 Final Project\n",
"\n",
"| Field | Value |\n",
"| --- | --- |\n",
"| Student ID | _replace with your ID_ |\n",
"| First name | _replace with your first name_ |\n",
"| Date | _today's date_ |\n",
"\n",
"Welcome. This notebook contains the final project for the course. It walks you through six short lessons that together teach the core skills of an introductory data analysis. Every lesson opens with a short motivation, then shows you a working demonstration, then asks you to make one small change, and finally asks you to write one or two sentences of interpretation that refer to numbers from your own subset of the data.\n",
"\n",
"A few things to keep in mind before you start.\n",
"\n",
"- The project is designed to take roughly three hours of focused work.\n",
"- Every student receives a different random subset of the dataset, so your numbers will not match your classmates'. Your interpretations should refer to *your* numbers.\n",
"- You may use AI assistants (with disclosure in the final cell), but the interpretation prompts ask you to look at your own output and explain what you see, which AI cannot do for you reliably.\n",
"\n",
"Before submitting, run *Runtime > Restart and run all* to confirm the notebook executes from a fresh kernel without errors.\n",
"\n",
"---"
],
"id": "6hmCaMVuI0Ty"
},
{
"cell_type": "markdown",
"metadata": {
"id": "xMDgnLWxI0T0"
},
"source": [
"## Lesson 1: Setup and Warm-up (10 points)\n",
"\n",
"Every analysis begins with loading data and verifying that what you got is what you expected. This first lesson does exactly that, and adds a short warm-up where you find out how many students in your unique subset share your first name. The warm-up is a small exercise in filtering rows in pandas, which is the foundation of nearly every analysis that follows.\n",
"\n",
"### 1.1 Load your personalized subset\n",
"\n",
"Set `MY_STUDENT_ID` to your actual student ID (as a string, in quotes), then run the cell. The helper function below uses your ID to select a unique 1,500-row subset from the 5,000-row dataset. **Do not modify the helper functions.**"
],
"id": "xMDgnLWxI0T0"
},
{
"cell_type": "code",
"metadata": {
"id": "4Aef4Y40I0T0"
},
"source": [
"import hashlib\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"# Plot styling for the rest of the notebook\n",
"sns.set_theme(style=\"whitegrid\", context=\"notebook\")\n",
"plt.rcParams[\"figure.figsize\"] = (8, 5)\n",
"\n",
"DATA_URL = \"https://raw.githubusercontent.com/ahmedmoustafa/datasets/main/student_lifestyle/student_lifestyle.csv\"\n",
"full_data = pd.read_csv(DATA_URL)\n",
"\n",
"\n",
"def get_my_dataset(student_id: str, n: int = 1500) -> pd.DataFrame:\n",
" \"\"\"Return a deterministic 1,500-row subset for the given student ID.\"\"\"\n",
" seed = int(hashlib.sha256(student_id.encode()).hexdigest(), 16) % (2**32)\n",
" return full_data.sample(n=n, random_state=seed).reset_index(drop=True)\n",
"\n",
"\n",
"# >>>> REPLACE WITH YOUR ACTUAL STUDENT ID <<<<\n",
"MY_STUDENT_ID = \"REPLACE_WITH_YOUR_STUDENT_ID\"\n",
"\n",
"data = get_my_dataset(MY_STUDENT_ID)\n",
"\n",
"# Reproducibility check (the grader will verify these values)\n",
"print(f\"Student ID: {MY_STUDENT_ID}\")\n",
"print(f\"Subset shape: {data.shape}\")\n",
"print(f\"First 3 student_ids in subset: {data['student_id'].head(3).tolist()}\")\n",
"print(f\"Mean GPA in subset: {data['gpa'].mean():.4f}\")\n"
],
"outputs": [],
"execution_count": null,
"id": "4Aef4Y40I0T0"
},
{
"cell_type": "markdown",
"metadata": {
"id": "0LkfMl53I0T1"
},
"source": [
"**What you should see.** The output should report `Subset shape: (1500, 25)`, three student identifiers that begin with the letter `S`, and a mean GPA somewhere between roughly 2.9 and 3.4. If the output is missing or the shape is wrong, your `MY_STUDENT_ID` is not set correctly.\n",
"\n",
"### 1.2 Warm-up: count your namesakes\n",
"\n",
"Replace `MY_FIRST_NAME` with your own first name (as a string, in quotes), then run the cell. The cell counts how many students in your subset share your first name and identifies the most common first name in your subset.\n",
"\n",
"> **Note on spelling.** The dataset includes multiple romanizations for some names (for example, *Mohamed* and *Mohammed*; *Youssef*, *Yousef*, *Yossef*, and *Yussuf*; *Mariam* and *Maryam*). If your exact spelling is missing, look at the distribution printed below and pick the variant you want to count. State your choice clearly in the interpretation."
],
"id": "0LkfMl53I0T1"
},
{
"cell_type": "code",
"metadata": {
"id": "2v9hKZeMI0T1"
},
"source": [
"# >>>> REPLACE WITH YOUR FIRST NAME <<<<\n",
"MY_FIRST_NAME = \"REPLACE_WITH_YOUR_FIRST_NAME\"\n",
"\n",
"shared = (data[\"first_name\"] == MY_FIRST_NAME).sum()\n",
"total = len(data)\n",
"print(f\"Students sharing your first name: {shared} ({shared/total:.2%})\")\n",
"\n",
"counts = data[\"first_name\"].value_counts()\n",
"print(f\"Most common first name in your subset: {counts.index[0]} ({counts.iloc[0]} students)\")\n",
"\n",
"# Top 15 first names in the subset (for reference)\n",
"counts.head(15)\n"
],
"outputs": [],
"execution_count": null,
"id": "2v9hKZeMI0T1"
},
{
"cell_type": "markdown",
"metadata": {
"id": "CNzgjNeDI0T1"
},
"source": [
"### 1.3 Your interpretation\n",
"\n",
"Replace each italicized line with your own short answer.\n",
"\n",
"**Q1.1.** How many students in your subset share your first name, and what percentage of the subset is that?\n",
"_Your answer._\n",
"\n",
"**Q1.2.** Which first name is the most common in your subset, and how many students have it?\n",
"_Your answer._\n",
"\n",
"**Q1.3.** In one sentence, compare the frequency of your first name to that of the most common name. (Examples: \"My name is more common than the most common name\", \"My name is far less common, with about a fifth as many students\", and so on.)\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "CNzgjNeDI0T1"
},
{
"cell_type": "markdown",
"metadata": {
"id": "8HhCcG2PI0T1"
},
"source": [
"## Lesson 2: Inspecting Your Data (15 points)\n",
"\n",
"Real datasets are rarely clean. Before any meaningful analysis, you have to inspect what you have, find the problems, and decide what to do about them. The dataset for this project intentionally contains a small number of imperfections. Your task in this lesson is to find them.\n",
"\n",
"### 2.1 Shape, types, and a glance at the first rows\n",
"\n",
"The first thing to do with any dataset is to ask: how big is it, what are the columns, and what do the first few rows look like?"
],
"id": "8HhCcG2PI0T1"
},
{
"cell_type": "code",
"metadata": {
"id": "CiCHvPczI0T1"
},
"source": [
"print(\"Shape (rows, columns):\", data.shape)\n",
"print()\n",
"print(\"Data types of each column:\")\n",
"print(data.dtypes)\n",
"data.head()\n"
],
"outputs": [],
"execution_count": null,
"id": "CiCHvPczI0T1"
},
{
"cell_type": "markdown",
"metadata": {
"id": "F93BkkKNI0T2"
},
"source": [
"**What you should notice.** The subset has 1,500 rows and 25 columns. Some columns hold numbers (integers and floats), some hold strings (categorical data and the survey date), and one is the unique student identifier."
],
"id": "F93BkkKNI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "tG2zj_-cI0T2"
},
"source": [
"### 2.2 Numeric summary\n",
"\n",
"The `.describe()` method provides a quick statistical summary of every numeric column. We transpose it (`.T`) so each variable appears on a row, which is easier to read."
],
"id": "tG2zj_-cI0T2"
},
{
"cell_type": "code",
"metadata": {
"id": "N-XdsolKI0T2"
},
"source": [
"data.describe().T\n"
],
"outputs": [],
"execution_count": null,
"id": "N-XdsolKI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "T4s2ta7jI0T2"
},
"source": [
"**Reading the output.** Look at the `min` and `max` columns. Most variables have plausible ranges (for example, `sleep_hours` ought to be roughly between three and twelve), but at least one numeric column has an impossible value at the minimum or maximum. Note which one for your interpretation below."
],
"id": "T4s2ta7jI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "wYaUzbAuI0T2"
},
"source": [
"### 2.3 Missing values\n",
"\n",
"Missingness is one of the most common quality issues in real data. The cell below counts the number of missing values in each column."
],
"id": "wYaUzbAuI0T2"
},
{
"cell_type": "code",
"metadata": {
"id": "WNZs3R-vI0T2"
},
"source": [
"missing = pd.DataFrame({\n",
" \"n_missing\": data.isnull().sum(),\n",
" \"pct_missing\": (data.isnull().sum() / len(data) * 100).round(2),\n",
"})\n",
"missing[missing[\"n_missing\"] > 0].sort_values(\"n_missing\", ascending=False)\n"
],
"outputs": [],
"execution_count": null,
"id": "WNZs3R-vI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "MxItWYc9I0T2"
},
"source": [
"**What you should notice.** Missingness in `sleep_hours` is small (about three percent) and is best understood as ordinary survey non-response. The missingness in the prior-term GPA columns (`gpa_term1`, `gpa_term2`, `gpa_term3`) is much larger and is *structural*: a first-year student does not have a GPA from four terms ago because that term has not happened yet. The cell below makes the structural pattern explicit."
],
"id": "MxItWYc9I0T2"
},
{
"cell_type": "code",
"metadata": {
"id": "8RpLznuuI0T2"
},
"source": [
"# Cross-tabulate which prior-term columns are missing by year of study\n",
"for col in [\"gpa_term1\", \"gpa_term2\", \"gpa_term3\", \"gpa_term4\"]:\n",
" counts = data.groupby(\"year_of_study\")[col].apply(lambda x: x.isnull().mean()).round(3)\n",
" print(f\"{col} -- fraction missing by year_of_study:\")\n",
" print(counts.to_string())\n",
" print()\n"
],
"outputs": [],
"execution_count": null,
"id": "8RpLznuuI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "d5VL4GN6I0T2"
},
"source": [
"### 2.4 Inconsistent text formatting\n",
"\n",
"Categorical columns sometimes contain the same value written in different ways. The cell below lists the unique values of `major`."
],
"id": "d5VL4GN6I0T2"
},
{
"cell_type": "code",
"metadata": {
"id": "CpjH0cP1I0T2"
},
"source": [
"data[\"major\"].value_counts()\n"
],
"outputs": [],
"execution_count": null,
"id": "CpjH0cP1I0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "mNxTujQxI0T2"
},
"source": [
"**Notice in particular.** Most rows record the major in title case (`Engineering`, `Computer Science`, etc.), but a small fraction use lowercase (`engineering`, `computer science`, etc.). Any analysis that groups by major should normalize the column first, for example with `data[\"major\"].str.title()`."
],
"id": "mNxTujQxI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "6d367hJ2I0T2"
},
"source": [
"### 2.5 Invalid values\n",
"\n",
"Sometimes a column contains values that are not impossible at the type level (for example, an integer column may accept any integer) but are impossible in the real world. The cell below looks at `coffee_cups`, where negative values cannot be physically meaningful."
],
"id": "6d367hJ2I0T2"
},
{
"cell_type": "code",
"metadata": {
"id": "4XYNOssGI0T2"
},
"source": [
"data[data[\"coffee_cups\"] < 0]\n"
],
"outputs": [],
"execution_count": null,
"id": "4XYNOssGI0T2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "Rb3Uw0M8I0T3"
},
"source": [
"### 2.6 Your interpretation\n",
"\n",
"**Q2.1.** What percentage of rows in your subset have a missing value in `sleep_hours`?\n",
"_Your answer._\n",
"\n",
"**Q2.2.** Look at the cross-tabulation in 2.3. Why are the prior-term GPA columns missing for some students? Explain in one sentence.\n",
"_Your answer._\n",
"\n",
"**Q2.3.** How many distinct text spellings of `major` did you find, including the lowercase variants?\n",
"_Your answer._\n",
"\n",
"**Q2.4.** How many rows in your subset have a negative `coffee_cups` value?\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "Rb3Uw0M8I0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZGUfa2W8I0T3"
},
"source": [
"## Lesson 3: Looking at One Variable at a Time (15 points)\n",
"\n",
"Before comparing one variable to another, look at each variable in isolation. The shape of a distribution (its center, spread, symmetry, and any outliers) tells you what the typical case looks like and what the unusual cases look like. This lesson practices three of the most common single-variable plots.\n",
"\n",
"### 3.1 Histogram of GPA"
],
"id": "ZGUfa2W8I0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "W_8iazISI0T3"
},
"source": [
"fig, ax = plt.subplots()\n",
"sns.histplot(data[\"gpa\"], bins=30, kde=True, ax=ax)\n",
"ax.set_title(\"Distribution of GPA in your subset\")\n",
"ax.set_xlabel(\"GPA\")\n",
"ax.set_ylabel(\"Number of students\")\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "W_8iazISI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "v-yqK5fsI0T3"
},
"source": [
"**What you should notice.** The smooth curve is a kernel density estimate, which is a smoothed version of the histogram. Look at where the bulk of the distribution sits, whether it is symmetric or skewed, and whether you can see any small bumps or extreme values."
],
"id": "v-yqK5fsI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "1xklzT4YI0T3"
},
"source": [
"### 3.2 Box plot of `study_hours`\n",
"\n",
"A box plot summarizes a distribution with five numbers: the minimum, the first quartile ($Q_1$), the median ($Q_2$), the third quartile ($Q_3$), and the maximum. Points beyond the whiskers are flagged as potential outliers."
],
"id": "1xklzT4YI0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "SuFFXK_6I0T3"
},
"source": [
"fig, ax = plt.subplots(figsize=(8, 3))\n",
"sns.boxplot(x=data[\"study_hours\"], ax=ax)\n",
"ax.set_title(\"Distribution of weekly study hours\")\n",
"ax.set_xlabel(\"Study hours per week\")\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "SuFFXK_6I0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "frcv_EwgI0T3"
},
"source": [
"**A few things to look for.** The vast majority of students study somewhere in a moderate range, but you should see a small number of points far to the right of the rest. These are the implausibly high values you encountered in Lesson 2."
],
"id": "frcv_EwgI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "8SC_05lvI0T3"
},
"source": [
"### 3.3 Count plot of a categorical variable\n",
"\n",
"For categorical variables, a count plot (a bar chart of frequencies) is the analogue of a histogram."
],
"id": "8SC_05lvI0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "_Vy75M6JI0T3"
},
"source": [
"fig, ax = plt.subplots()\n",
"sns.countplot(\n",
" data=data,\n",
" x=\"home_region\",\n",
" order=data[\"home_region\"].value_counts().index,\n",
" ax=ax,\n",
")\n",
"ax.set_title(\"Number of students by home region\")\n",
"ax.set_xlabel(\"Home region\")\n",
"ax.set_ylabel(\"Number of students\")\n",
"plt.xticks(rotation=20)\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "_Vy75M6JI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "3ECa-IGaI0T3"
},
"source": [
"### 3.4 Your task\n",
"\n",
"Pick **one** continuous variable from the dataset that has not been plotted in this lesson (for example, `sleep_hours`, `social_media_min`, `monthly_spending`, or `coffee_cups`) and produce a histogram of it in the cell below. Set a descriptive title and axis labels."
],
"id": "3ECa-IGaI0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "D-v6vPKyI0T3"
},
"source": [
"# Replace 'your_variable_here' with the name of the variable you chose.\n",
"your_variable_here = \"sleep_hours\" # change this\n",
"\n",
"fig, ax = plt.subplots()\n",
"sns.histplot(data[your_variable_here].dropna(), bins=30, kde=True, ax=ax)\n",
"ax.set_title(f\"Distribution of {your_variable_here} in your subset\")\n",
"ax.set_xlabel(your_variable_here)\n",
"ax.set_ylabel(\"Number of students\")\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "D-v6vPKyI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "CDGyjyNHI0T3"
},
"source": [
"### 3.5 Your interpretation\n",
"\n",
"**Q3.1.** What is the mean GPA in your subset, and is the median similar to the mean? (If the two are very different, the distribution is skewed.)\n",
"_Your answer._\n",
"\n",
"**Q3.2.** Approximately how many points lie to the right of the whisker in the `study_hours` box plot? What do those points represent?\n",
"_Your answer._\n",
"\n",
"**Q3.3.** For the variable you plotted in 3.4, describe the shape of the distribution in one sentence (where the bulk of the values sit, whether it is skewed, and any outliers).\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "CDGyjyNHI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "-gqCd722I0T3"
},
"source": [
"## Lesson 4: Comparing Groups (20 points)\n",
"\n",
"Many research questions are comparisons. Do men and women differ on a given outcome? Do commuters report more stress than non-commuters? This lesson covers two things in turn. First, you will visualize a group comparison and read off the group summary statistics by eye. Second, you will run a two-sample t-test, which is the standard tool for asking whether the difference you see between two groups is large enough to be considered statistically significant.\n",
"\n",
"### 4.1 GPA by major\n",
"\n",
"A box plot is a natural way to compare a continuous variable across categorical groups."
],
"id": "-gqCd722I0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "IDLVb96kI0T3"
},
"source": [
"# Normalize the major column to title case to merge the lowercase variants\n",
"data[\"major_clean\"] = data[\"major\"].str.title()\n",
"\n",
"fig, ax = plt.subplots()\n",
"sns.boxplot(data=data, x=\"major_clean\", y=\"gpa\", ax=ax)\n",
"ax.set_title(\"GPA by major\")\n",
"ax.set_xlabel(\"Major\")\n",
"ax.set_ylabel(\"GPA\")\n",
"plt.xticks(rotation=20)\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "IDLVb96kI0T3"
},
{
"cell_type": "code",
"metadata": {
"id": "B1AyZNBRI0T3"
},
"source": [
"# Group means, sorted from highest to lowest\n",
"data.groupby(\"major_clean\")[\"gpa\"].mean().round(3).sort_values(ascending=False)\n"
],
"outputs": [],
"execution_count": null,
"id": "B1AyZNBRI0T3"
},
{
"cell_type": "markdown",
"metadata": {
"id": "h2avJc5QI0T4"
},
"source": [
"**What you should notice.** Some majors have visibly higher mean GPA than others. The boxes also differ in spread: a wider box indicates more variability within that major.\n",
"\n",
"### 4.2 A first hypothesis test\n",
"\n",
"A two-sample t-test asks whether the mean of an outcome differs between two groups. The Welch variant (used here) does not assume that the two groups have equal variances.\n",
"\n",
"**Hypotheses.** Let $\\mu_1$ and $\\mu_2$ denote the population means of group 1 and group 2 respectively.\n",
"\n",
"$H_0: \\mu_1 = \\mu_2 \\quad \\text{(the two groups have equal means)}$\n",
"\n",
"$H_a: \\mu_1 \\neq \\mu_2 \\quad \\text{(the two groups have different means)}$\n",
"\n",
"**Significance level:** $\\alpha = 0.05$.\n",
"\n",
"**Test statistic.**\n",
"\n",
"$$t = \\frac{\\bar{x}_1 - \\bar{x}_2}{\\sqrt{\\dfrac{s_1^2}{n_1} + \\dfrac{s_2^2}{n_2}}}$$\n",
"\n",
"where $\\bar{x}_i$, $s_i$, and $n_i$ are the sample mean, sample standard deviation, and sample size of group $i$ respectively.\n",
"\n",
"**Decision rule.** If the resulting $p$-value is less than $\\alpha$, reject $H_0$ and conclude that the two means differ. Otherwise, fail to reject $H_0$.\n",
"\n",
"**Effect size (Cohen's $d$).** Even when a difference is statistically significant, you should report how large it is. Cohen's $d$ expresses the difference in standard-deviation units:\n",
"\n",
"$$d = \\frac{\\bar{x}_1 - \\bar{x}_2}{s_p}, \\qquad s_p = \\sqrt{\\frac{s_1^2 + s_2^2}{2}}$$\n",
"\n",
"By convention, $|d| \\approx 0.2$ is small, $|d| \\approx 0.5$ is medium, and $|d| \\approx 0.8$ is large."
],
"id": "h2avJc5QI0T4"
},
{
"cell_type": "markdown",
"metadata": {
"id": "eTVTbWM8I0T4"
},
"source": [
"### 4.3 Test commuters vs. non-commuters on stress level\n",
"\n",
"The cell below runs the test described above on the question of whether commuter students report different stress levels than non-commuters."
],
"id": "eTVTbWM8I0T4"
},
{
"cell_type": "code",
"metadata": {
"id": "wzfYtOkgI0T4"
},
"source": [
"from scipy.stats import ttest_ind\n",
"\n",
"group_a = data[data[\"commuter\"] == \"Yes\"][\"stress_level\"]\n",
"group_b = data[data[\"commuter\"] == \"No\"][\"stress_level\"]\n",
"\n",
"t_stat, p_value = ttest_ind(group_a, group_b, nan_policy=\"omit\", equal_var=False)\n",
"\n",
"pooled_std = np.sqrt((group_a.var(ddof=1) + group_b.var(ddof=1)) / 2)\n",
"cohens_d = (group_a.mean() - group_b.mean()) / pooled_std\n",
"\n",
"print(f\"Mean stress, commuters = {group_a.mean():.3f} (n = {len(group_a)})\")\n",
"print(f\"Mean stress, non-commuters = {group_b.mean():.3f} (n = {len(group_b)})\")\n",
"print(f\"t = {t_stat:.3f}, p = {p_value:.4f}, Cohen's d = {cohens_d:.3f}\")\n"
],
"outputs": [],
"execution_count": null,
"id": "wzfYtOkgI0T4"
},
{
"cell_type": "markdown",
"metadata": {
"id": "FXT-vhAeI0T-"
},
"source": [
"### 4.4 Your task\n",
"\n",
"The cell below tests the same two groups (commuters vs. non-commuters), but on `gpa` instead of `stress_level`. Run it as is, and write your interpretation below."
],
"id": "FXT-vhAeI0T-"
},
{
"cell_type": "code",
"metadata": {
"id": "2MdTxbDNI0T-"
},
"source": [
"group_a_gpa = data[data[\"commuter\"] == \"Yes\"][\"gpa\"]\n",
"group_b_gpa = data[data[\"commuter\"] == \"No\"][\"gpa\"]\n",
"\n",
"t_stat_gpa, p_value_gpa = ttest_ind(group_a_gpa, group_b_gpa, nan_policy=\"omit\", equal_var=False)\n",
"\n",
"pooled_std_gpa = np.sqrt((group_a_gpa.var(ddof=1) + group_b_gpa.var(ddof=1)) / 2)\n",
"cohens_d_gpa = (group_a_gpa.mean() - group_b_gpa.mean()) / pooled_std_gpa\n",
"\n",
"print(f\"Mean GPA, commuters = {group_a_gpa.mean():.3f} (n = {len(group_a_gpa)})\")\n",
"print(f\"Mean GPA, non-commuters = {group_b_gpa.mean():.3f} (n = {len(group_b_gpa)})\")\n",
"print(f\"t = {t_stat_gpa:.3f}, p = {p_value_gpa:.4f}, Cohen's d = {cohens_d_gpa:.3f}\")\n"
],
"outputs": [],
"execution_count": null,
"id": "2MdTxbDNI0T-"
},
{
"cell_type": "markdown",
"metadata": {
"id": "2aZqDqyPI0T-"
},
"source": [
"### 4.5 Your interpretation\n",
"\n",
"**Q4.1.** Which major has the highest mean GPA in your subset? Which has the lowest?\n",
"_Your answer._\n",
"\n",
"**Q4.2.** For the test in 4.3 (commuters vs. non-commuters on stress level), state the $p$-value and your conclusion (reject $H_0$ or fail to reject) at $\\alpha = 0.05$.\n",
"_Your answer._\n",
"\n",
"**Q4.3.** For the test in 4.4 (commuters vs. non-commuters on GPA), state the $p$-value and your conclusion. Is the effect size large, medium, or small according to Cohen's conventions?\n",
"_Your answer._\n",
"\n",
"**Q4.4.** Even when a result is statistically significant, the effect size matters for whether the difference is *practically* meaningful. In one sentence, explain why a tiny effect size can still produce a small $p$-value if the sample size is large enough.\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "2aZqDqyPI0T-"
},
{
"cell_type": "markdown",
"metadata": {
"id": "msp0zd7yI0T-"
},
"source": [
"## Lesson 5: Relationships Between Variables (15 points)\n",
"\n",
"Two continuous variables can be related linearly. The Pearson correlation coefficient $r$ summarizes how strong and in what direction. It ranges from $-1$ (perfect negative linear association) through $0$ (no linear association) to $+1$ (perfect positive linear association). This lesson asks you to compute one correlation in detail, visualize it as a scatter plot, and then survey the full correlation matrix of every continuous variable in your subset to see which pairs are most strongly related.\n",
"\n",
"### 5.1 Scatter plot of study hours vs. GPA"
],
"id": "msp0zd7yI0T-"
},
{
"cell_type": "code",
"metadata": {
"id": "XU4jGD3pI0T-"
},
"source": [
"fig, ax = plt.subplots()\n",
"sns.scatterplot(data=data, x=\"study_hours\", y=\"gpa\", alpha=0.4, ax=ax)\n",
"ax.set_title(\"Weekly study hours and GPA\")\n",
"ax.set_xlabel(\"Weekly study hours\")\n",
"ax.set_ylabel(\"GPA\")\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "XU4jGD3pI0T-"
},
{
"cell_type": "markdown",
"metadata": {
"id": "gfI4csZ9I0T-"
},
"source": [
"**What this tells you.** Each point is one student. Look at whether the cloud of points slopes upward (positive association), slopes downward (negative association), or has no visible trend (no linear association)."
],
"id": "gfI4csZ9I0T-"
},
{
"cell_type": "markdown",
"metadata": {
"id": "KseWm4u_I0T-"
},
"source": [
"### 5.2 Pearson correlation with significance test\n",
"\n",
"The Pearson correlation coefficient is computed as\n",
"\n",
"$$r = \\frac{\\sum_{i=1}^{n} (x_i - \\bar{x})(y_i - \\bar{y})}\n",
" {\\sqrt{\\sum_{i=1}^{n} (x_i - \\bar{x})^2}\n",
" \\sqrt{\\sum_{i=1}^{n} (y_i - \\bar{y})^2}}$$\n",
"\n",
"The associated significance test has\n",
"\n",
"$$H_0: \\rho = 0 \\quad \\text{(no linear association in the population)}$$\n",
"\n",
"$$H_a: \\rho \\neq 0 \\quad \\text{(some linear association in the population)}$$\n",
"\n",
"where $\\rho$ is the population correlation coefficient. The squared correlation $R^2 = r^2$ is the proportion of variance in one variable that is linearly associated with the other. Conventional benchmarks for the strength of $|r|$ are $|r| < 0.3$ weak, $0.3 \\leq |r| < 0.7$ moderate, and $|r| \\geq 0.7$ strong."
],
"id": "KseWm4u_I0T-"
},
{
"cell_type": "code",
"metadata": {
"id": "7fob72cBI0T_"
},
"source": [
"from scipy.stats import pearsonr\n",
"\n",
"pair = data[[\"study_hours\", \"gpa\"]].dropna()\n",
"r, p_value = pearsonr(pair[\"study_hours\"], pair[\"gpa\"])\n",
"print(f\"n = {len(pair)}, r = {r:.3f}, R-squared = {r**2:.3f}, p = {p_value:.4f}\")\n"
],
"outputs": [],
"execution_count": null,
"id": "7fob72cBI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "TcpzpAblI0T_"
},
"source": [
"### 5.3 The full correlation matrix\n",
"\n",
"The cell below computes the Pearson correlation between every pair of continuous variables in your subset and visualizes the matrix as a heatmap."
],
"id": "TcpzpAblI0T_"
},
{
"cell_type": "code",
"metadata": {
"id": "ZG1J_Zv4I0T_"
},
"source": [
"continuous_vars = [\n",
" \"sleep_hours\", \"study_hours\", \"exercise_min\", \"social_media_min\",\n",
" \"coffee_cups\", \"stress_level\", \"monthly_spending\", \"gpa\",\n",
"]\n",
"corr = data[continuous_vars].corr()\n",
"\n",
"fig, ax = plt.subplots(figsize=(7, 6))\n",
"sns.heatmap(\n",
" corr, annot=True, fmt=\".2f\", cmap=\"vlag\",\n",
" center=0, vmin=-1, vmax=1, square=True, ax=ax,\n",
")\n",
"ax.set_title(\"Pearson correlations among continuous variables\")\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "ZG1J_Zv4I0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "ATG9XcKtI0T_"
},
"source": [
"**What you should notice.** Each cell shows the correlation between the row variable and the column variable. The diagonal is always one (a variable with itself). Strong red cells indicate strong positive correlations; strong blue cells indicate strong negative correlations.\n",
"\n",
"### 5.4 Your task\n",
"\n",
"Look at the row of the heatmap that corresponds to `gpa`. Identify the variable other than `gpa` itself that has the strongest correlation (in absolute value) with `gpa`."
],
"id": "ATG9XcKtI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "pzW_a_fzI0T_"
},
"source": [
"### 5.5 Your interpretation\n",
"\n",
"**Q5.1.** State the value of $r$ between `study_hours` and `gpa` in your subset and the corresponding $R^2$. Is the correlation weak, moderate, or strong by the conventional benchmarks?\n",
"_Your answer._\n",
"\n",
"**Q5.2.** Which continuous variable is the most strongly correlated (in absolute value) with `gpa` in your subset? State the variable name and the value of $r$.\n",
"_Your answer._\n",
"\n",
"**Q5.3.** Which continuous variable is the most weakly correlated with `gpa` in your subset? Why might that variable have a weak relationship with academic performance?\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "pzW_a_fzI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "IjCbHz5tI0T_"
},
"source": [
"## Lesson 6: Time Trends and Final Reflection (25 points)\n",
"\n",
"Data that has a time dimension can reveal patterns that summary statistics alone do not. This final lesson plots how self-reported stress changes across the academic year and asks you to identify the peak periods. The lesson closes with a short reflection on what you have learned about your unique subset across the six lessons.\n",
"\n",
"### 6.1 Mean stress level by survey month\n",
"\n",
"Each row in the dataset includes a `survey_date`, the date on which the student completed the survey. Aggregating by month lets you see how the average response changes over time."
],
"id": "IjCbHz5tI0T_"
},
{
"cell_type": "code",
"metadata": {
"id": "pm8G4TetI0T_"
},
"source": [
"data_time = data.copy()\n",
"data_time[\"month\"] = pd.to_datetime(data_time[\"survey_date\"]).dt.to_period(\"M\").astype(str)\n",
"monthly_stress = data_time.groupby(\"month\")[\"stress_level\"].mean()\n",
"\n",
"fig, ax = plt.subplots()\n",
"monthly_stress.plot(marker=\"o\", ax=ax)\n",
"ax.set_title(\"Mean self-reported stress by survey month\")\n",
"ax.set_xlabel(\"Survey month\")\n",
"ax.set_ylabel(\"Mean stress level (1 to 10)\")\n",
"plt.xticks(rotation=30)\n",
"plt.tight_layout()\n",
"plt.show()\n",
"\n",
"print(\"Monthly mean stress:\")\n",
"print(monthly_stress.round(2).to_string())\n"
],
"outputs": [],
"execution_count": null,
"id": "pm8G4TetI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "3UIpa4XNI0T_"
},
"source": [
"**Reading the trend.** Stress is not constant throughout the year. Look at which months have the highest mean stress and ask yourself what is happening academically in those months.\n",
"\n",
"### 6.2 Your task\n",
"\n",
"Modify the cell below to plot the **mean GPA** by month instead of the mean stress level. Use the same `data_time` dataframe."
],
"id": "3UIpa4XNI0T_"
},
{
"cell_type": "code",
"metadata": {
"id": "p1zISE5CI0T_"
},
"source": [
"# Replace 'stress_level' with the variable you want to plot\n",
"variable_to_plot = \"stress_level\" # change this to \"gpa\"\n",
"\n",
"monthly_var = data_time.groupby(\"month\")[variable_to_plot].mean()\n",
"\n",
"fig, ax = plt.subplots()\n",
"monthly_var.plot(marker=\"o\", ax=ax)\n",
"ax.set_title(f\"Mean {variable_to_plot} by survey month\")\n",
"ax.set_xlabel(\"Survey month\")\n",
"ax.set_ylabel(f\"Mean {variable_to_plot}\")\n",
"plt.xticks(rotation=30)\n",
"plt.tight_layout()\n",
"plt.show()\n"
],
"outputs": [],
"execution_count": null,
"id": "p1zISE5CI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "GG0vMjLtI0T_"
},
"source": [
"### 6.3 Your interpretation of the time trends\n",
"\n",
"**Q6.1.** Which two months show the highest mean stress in your subset? Propose a brief explanation grounded in the academic calendar.\n",
"_Your answer._\n",
"\n",
"**Q6.2.** Looking at the mean GPA by month, does the pattern resemble the stress pattern, look opposite to it, or look unrelated? Describe what you see in one sentence.\n",
"_Your answer._\n",
"\n",
"### 6.4 Final reflection\n",
"\n",
"This is the longest written response in the project. Aim for **three short paragraphs**, roughly two hundred words in total. Write in your own voice rather than in a generic textbook style.\n",
"\n",
"**Q6.3 (one paragraph).** Of all the numbers you produced in this project, which single result surprised you the most? Quote the specific number, state what you expected, and propose at least one hypothesis (drawing on other variables in your subset) for why the result came out the way it did.\n",
"_Your answer._\n",
"\n",
"**Q6.4 (one paragraph).** Pick any two findings from across the six lessons (a correlation, a group comparison, a time trend, a quality issue, anything) and propose a single contextual story that links them. Then propose at least one alternative explanation, such as a confounding variable in the dataset that could produce the same pattern.\n",
"_Your answer._\n",
"\n",
"**Q6.5 (one paragraph).** If you had another week to keep working with this dataset, what would you investigate next, and why?\n",
"_Your answer._\n",
"\n",
"---"
],
"id": "GG0vMjLtI0T_"
},
{
"cell_type": "markdown",
"metadata": {
"id": "MxE0Dbh5I0T_"
},
"source": [
"## References and AI Disclosure\n",
"\n",
"_List the external resources you used while completing this project: documentation, Stack Overflow URLs, articles, AI assistants, and so on._\n",
"\n",
"_If you used AI assistance for any part of this project, state the tool you used, the lessons it helped with, and approximately what fraction of the work it produced. Undisclosed AI use is treated as plagiarism._"
],
"id": "MxE0Dbh5I0T_"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.x"
},
"colab": {
"provenance": [],
"toc_visible": true,
"include_colab_link": true
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment