Skip to content

Instantly share code, notes, and snippets.

@pb111

pb111/Data Cleaning with Python and Pandas.ipynb

Created February 5, 2019 14:41
Show Gist options
  • Save pb111/e6a4b98eaf586905c439a83407b95e5d to your computer and use it in GitHub Desktop.
Save pb111/e6a4b98eaf586905c439a83407b95e5d to your computer and use it in GitHub Desktop.
Download ZIP
Data Cleaning with Python and Pandas
Display the source blob
Display the rendered blob
Raw
Data Cleaning with Python and Pandas.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Cleaning with Python and Pandas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this project, I discuss various useful techniques to clean a messy dataset with Python, NumPy and Pandas."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This project is divided into various sections which are listed below:-"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents:-"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1.\tIntroduction to Python data cleaning\n",
"\n",
"2.\tTidy data\n",
"\n",
"3.\tSigns of an untidy dataset\n",
"\n",
"4.\tPython data cleansing – prerequisites\n",
"\n",
"5.\tImport the required Python libraries\n",
"\n",
"6.\tThe source dataset\n",
"\n",
"7.\tExploratory data analysis (EDA)\n",
"\n",
"8.\tVisual exploratory data analysis (Visual EDA)\n",
"\n",
"9.\tFindings of EDA and Visual EDA\n",
"\n",
"10.\tSplit the ‘age_sex’ column into two separate columns\n",
"\n",
"11.\tReorder the column labels\n",
"\n",
"12.\tDealing with negative numerical values\n",
"\n",
"13.\tDealing with outliers\n",
"\n",
"14.\tDealing with missing numerical values\n",
"\n",
"15.\tCheck with ASSERT statement\n",
"\n",
"16.\tReshaping the data into tidy data format\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Introduction to Python data cleaning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Whenever we have to work with a real world dataset, the first problem that we face is to clean it. The real world dataset never comes clean. It consists lot of discrepancies in the dataset. So, we have to clean the dataset for further processing. \n",
"\n",
"Cleaning data is the process of preparing the dataset for analysis. It is very important because the accuracy of machine learning or data mining models are affected because of poor quality of data. \n",
"\n",
"So, data scientists spend a large amount of their time cleaning the dataset and transform them into a format with which they can work with. In fact, data scientists spend 80% of their time cleaning the data.\n",
"\n",
"A very common scenario is that the dataset contains missing values coded as `NaN`. Also, the missing values are coded in different ways. The dataset may contain negative or invalid values. It may contain outliers. It may be in the untidy format. All of these are examples of a messy dataset. \n",
"\n",
"In this project, I present several useful ways to handle these discrepancies in the dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Tidy data format\n",
"\n",
"\n",
"Data comes in a wide variety of shapes and formats. Hadley Wickham, the Chief Scientist at RStudio, write a paper about tidy data in 2014 that formalizes the shape of the data. So, it gives us a goal when formatting the data. \n",
"\n",
"\n",
"He states in his paper that –\n",
"\n",
"**\"Tidy data provides a standard way to organize data values within a dataset.\"**\n",
"\n",
"\n",
"There are three principles of tidy data. These are as follows:-\n",
"\n",
"**•\tColumns represent separate variables.**\n",
"\n",
"**•\tRows represent individual observations.**\n",
"\n",
"**•\tObservational units form tables.**\n",
"\n",
"\n",
"Tidy data makes it easier to fix common data problems. So, we need to transform the untidy dataset into tidy data. \n",
"\n",
"Before we look into the details of cleaning the dataset, we have to understand what constitutes an untidy data. We need to diagnose our data and find common signs of a messy dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Signs of an untidy dataset\n",
"\n",
"\n",
"We have to take a closer look to find common signs of a messy dataset. These common signs are as follows:-\n",
"\n",
"\n",
"•\t**Missing numerical data**\n",
"\n",
"Missing numerical data needs to be identified and addressed. Either they need to be deleted or replaced with a suitable test statistic.\n",
"\n",
"\n",
"•\t**Untidy data**\n",
"\n",
"Untidy dataset can contain multiple problems. They prevent us from transforming the messy dataset into a clean dataset that is suitable for analysis.\n",
"\n",
"\n",
"•\t**Unexpected data values**\n",
"\n",
"Mismatched data types of a column and data values can cause potential problems. They need to be investigated and solved.\n",
"\n",
"\n",
"•\t**Inconsistent column names**\n",
"\n",
"Column names contain inconsistent capitalizations and bad characters. They need to be addressed properly.\n",
"\n",
"\n",
"•\t**Outliers**\n",
"\n",
"Outliers need to be detected. They pose potential problems needs to be investigated and removed.\n",
"\n",
"\n",
"•\t**Duplicate rows and columns**\n",
"\n",
"Duplicate rows and columns make data redundant. They can bias an analysis. Hence, they needs to be found and dropped.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Python data cleaning - prerequisites\n",
"\n",
"\n",
"We need three Python libraries for the data cleansing process – NumPy, Pandas and Matplotlib.\n",
"\n",
"\n",
"•\t**NumPy** – NumPy is the fundamental Python library for scientific computing. It adds support for large and multi-dimensional arrays and matrices. It also supports large collection of high-level mathematical functions to operate on these arrays.\n",
"\n",
"\n",
"•\t**Pandas** - Pandas is a software library for Python programming language which provide tools for data manipulation and analysis tasks. It will enable us to manipulate numerical tables and time series using data structures and operations.\n",
"\n",
"\n",
"•\t**Matplotlib** - Matplotlib is the core data visualization library of Python programming language. It provides an object-oriented API for embedding plots into applications.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Import the required Python libraries\n",
"\n",
"\n",
"We have seen that we need two Python libraries – NumPy and Python for the data cleansing process. We need to import these libraries before we actually start using them. We can import them with their usual shorthand notation as follows:-\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# import the Python libraries\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. The source dataset\n",
"\n",
"\n",
"For this project, I have created a fictitious dataset. The dataset consists of details of my facebook friends.\n",
"\n",
"The dataset can be imported as follows:-\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"data = \"C:/cleaning_data/friends.txt\"\n",
"\n",
"df = pd.read_csv(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Exploratory data analysis\n",
"\n",
"\n",
"Now, it is time to understand the data. We should diagnose the data for any discrepancies by doing exploratory data analysis.\n",
"We should proceed as follows:-\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.shape attribute\n",
"\n",
"\n",
"We can check the dimensions of the data with **df.shape** attribute.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(10, 10)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"We can see that our data have 10 rows and 10 columns."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.head() and df.tail() methods\n",
"\n",
"\n",
"We can view the top five and bottom five rows of the dataset with **df.head()** and **df.tail()** methods respectively.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age_sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37_M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>37</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38_M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>xx</td>\n",
" <td>38</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35_M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>?</td>\n",
" <td>35</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40_M</td>\n",
" <td>A</td>\n",
" <td>0</td>\n",
" <td>55.6</td>\n",
" <td>40</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39_M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>39</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age_sex section height(cm) weight(kg) age spend_A \\\n",
"0 Rahul Pandey 37_M B 175.4 85.6 37 1500.0 \n",
"1 Rakesh Kumar 38_M B NaN xx 38 NaN \n",
"2 Nitesh Mondol 35_M B 165.1 ? 35 500.0 \n",
"3 Arvind Dwivedi 40_M A 0 55.6 40 300.0 \n",
"4 Nikhil Shikharwar 39_M B 160.8 160.0 39 200.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300 \n",
"1 1000.0 NaN \n",
"2 -100.0 200 \n",
"3 300.0 NaN \n",
"4 NaN 300 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age_sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36_M</td>\n",
" <td>A</td>\n",
" <td>xx</td>\n",
" <td>-60</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35_M</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>35</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34_M</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>34</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>xx</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36_M</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38_M</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>38</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age_sex section height(cm) weight(kg) age spend_A \\\n",
"5 Ashutosh Merothiya 36_M A xx -60 36 100.0 \n",
"6 Shubhro Das 35_M C 170.3 75.5 35 NaN \n",
"7 Suvendu Das 34_M C 155.1 56.8 34 1000.0 \n",
"8 Swadesh Kumar 36_M C 154.2 60.6 36 100.0 \n",
"9 Arun Vishwakarma 38_M A 162.1 58.7 38 600.0 \n",
"\n",
" spend_B spend_C \n",
"5 400.0 NaN \n",
"6 500.0 500 \n",
"7 100.0 xx \n",
"8 200.0 300 \n",
"9 200.0 200 "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"We can see that there are lot of discrepancies in the dataset. \n",
"\n",
"\n",
"For example, the age and sex columns are combined together with an underscore. There should be two separate columns of age and sex. \n",
"\n",
"\n",
"The height and weight columns contain missing values. Some values are coded as \"xx\", \"?\", \"0\" and negative values. \n",
"They are all invalid values as height and weight must be positive real numbers.\n",
"\n",
"\n",
"The three columns spend_A, spend_B and spend_C denote spending at three supermarkets A,B and C. These columns must contain positive real numbers. The missing values in these columns denote nothing spend in that market. The negative value and the \n",
"value coded as \"xx\" should be addressed properly."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.info() method\n",
"\n",
"We can get a concise summary of the dataset with **df.info()** method. This method prints information about a DataFrame including the index dtype and column dtypes, non-null values and memory usage.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 10 entries, 0 to 9\n",
"Data columns (total 10 columns):\n",
"fname 10 non-null object\n",
"lname 10 non-null object\n",
"age_sex 10 non-null object\n",
"section 10 non-null object\n",
"height(cm) 9 non-null object\n",
"weight(kg) 10 non-null object\n",
"age 10 non-null int64\n",
"spend_A 8 non-null float64\n",
"spend_B 9 non-null float64\n",
"spend_C 7 non-null object\n",
"dtypes: float64(2), int64(1), object(7)\n",
"memory usage: 880.0+ bytes\n"
]
}
],
"source": [
"df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"We can see that this method prints information of all columns. Several columns contain missing values. We have seen that columns contain missing and invalid values are coded differently. So, we need to explore this issue further.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.dtypes attribute\n",
"\n",
"We can check the data types of each column in the dataframe with **df.dtypes** attribute. The above command returns \n",
"the data type of each column.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"fname object\n",
"lname object\n",
"age_sex object\n",
"section object\n",
"height(cm) object\n",
"weight(kg) object\n",
"age int64\n",
"spend_A float64\n",
"spend_B float64\n",
"spend_C object\n",
"dtype: object"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.dtypes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also, we can see that data types of height(cm) and weight(kg) columns are object data type. Again this is not true. The columns height(cm) and weight(kg) must contain positive real numbers. So, their data type must be float64.\n",
"\n",
"Also, spend_A, spend_B and spend_C columns must contain numeric values. We can see that the data type of spend_A and spend_B columns are float64. But the data type of column spend_C is object. So, we need to convert its data type to float64."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Handling invalid values\n",
"\n",
"There is an invalid value coded as \"xx\" in the height(cm) column. We can coerce invalid values to \"NaN\" using the errors keyword as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"df[\"height(cm)\"] = pd.to_numeric(df[\"height(cm)\"], errors='coerce')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similary, there is an invalid value coded as \"xx\" in the weight(kg) column. Again, we can coerce it using the errors keyword\n",
"as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"df[\"weight(kg)\"] = pd.to_numeric(df[\"weight(kg)\"], errors='coerce')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similary, there is an invalid value coded as \"xx\" in the spend_C column. Again, we can coerce it using the errors keyword\n",
"as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"df[\"spend_C\"] = pd.to_numeric(df[\"spend_C\"], errors='coerce')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check the data type again with df.dtypes"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"fname object\n",
"lname object\n",
"age_sex object\n",
"section object\n",
"height(cm) float64\n",
"weight(kg) float64\n",
"age int64\n",
"spend_A float64\n",
"spend_B float64\n",
"spend_C float64\n",
"dtype: object"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.dtypes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"Now, we can see that all the columns have appropriate data types. The columns height(cm) and weight(kg) have float64 data type.\n",
"The columns spend_A, spend_B and spend_C have float64 data type."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age_sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37_M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>37</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38_M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>38</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35_M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>35</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40_M</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>40</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39_M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>39</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age_sex section height(cm) weight(kg) age spend_A \\\n",
"0 Rahul Pandey 37_M B 175.4 85.6 37 1500.0 \n",
"1 Rakesh Kumar 38_M B NaN NaN 38 NaN \n",
"2 Nitesh Mondol 35_M B 165.1 NaN 35 500.0 \n",
"3 Arvind Dwivedi 40_M A 0.0 55.6 40 300.0 \n",
"4 Nikhil Shikharwar 39_M B 160.8 160.0 39 200.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 -100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age_sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36_M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>-60.0</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35_M</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>35</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34_M</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>34</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36_M</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38_M</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>38</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age_sex section height(cm) weight(kg) age \\\n",
"5 Ashutosh Merothiya 36_M A NaN -60.0 36 \n",
"6 Shubhro Das 35_M C 170.3 75.5 35 \n",
"7 Suvendu Das 34_M C 155.1 56.8 34 \n",
"8 Swadesh Kumar 36_M C 154.2 60.6 36 \n",
"9 Arun Vishwakarma 38_M A 162.1 58.7 38 \n",
"\n",
" spend_A spend_B spend_C \n",
"5 100.0 400.0 NaN \n",
"6 NaN 500.0 500.0 \n",
"7 1000.0 100.0 NaN \n",
"8 100.0 200.0 300.0 \n",
"9 600.0 200.0 200.0 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"We can see that all the invalid values(the values coded as \"xx\") and missing values are now coded as \"NaN\". The use of the \n",
"keyword errors='coerce' enable us to convert all the invalid values into NaN."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.describe() method\n",
"\n",
"We can view the summary statistics of numerical columns with **df.describe()** method. It enable us to detect outliers in the data which require further investigation.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>8.0000</td>\n",
" <td>8.000000</td>\n",
" <td>10.000000</td>\n",
" <td>8.000000</td>\n",
" <td>9.000000</td>\n",
" <td>6.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>142.8750</td>\n",
" <td>61.600000</td>\n",
" <td>36.800000</td>\n",
" <td>537.500000</td>\n",
" <td>311.111111</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>58.1695</td>\n",
" <td>60.126985</td>\n",
" <td>1.932184</td>\n",
" <td>492.624169</td>\n",
" <td>310.017921</td>\n",
" <td>109.544512</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>0.0000</td>\n",
" <td>-60.000000</td>\n",
" <td>34.000000</td>\n",
" <td>100.000000</td>\n",
" <td>-100.000000</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>154.8750</td>\n",
" <td>56.500000</td>\n",
" <td>35.250000</td>\n",
" <td>175.000000</td>\n",
" <td>200.000000</td>\n",
" <td>225.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>161.4500</td>\n",
" <td>59.650000</td>\n",
" <td>36.500000</td>\n",
" <td>400.000000</td>\n",
" <td>200.000000</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>166.4000</td>\n",
" <td>78.025000</td>\n",
" <td>38.000000</td>\n",
" <td>700.000000</td>\n",
" <td>400.000000</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>175.4000</td>\n",
" <td>160.000000</td>\n",
" <td>40.000000</td>\n",
" <td>1500.000000</td>\n",
" <td>1000.000000</td>\n",
" <td>500.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" height(cm) weight(kg) age spend_A spend_B spend_C\n",
"count 8.0000 8.000000 10.000000 8.000000 9.000000 6.000000\n",
"mean 142.8750 61.600000 36.800000 537.500000 311.111111 300.000000\n",
"std 58.1695 60.126985 1.932184 492.624169 310.017921 109.544512\n",
"min 0.0000 -60.000000 34.000000 100.000000 -100.000000 200.000000\n",
"25% 154.8750 56.500000 35.250000 175.000000 200.000000 225.000000\n",
"50% 161.4500 59.650000 36.500000 400.000000 200.000000 300.000000\n",
"75% 166.4000 78.025000 38.000000 700.000000 400.000000 300.000000\n",
"max 175.4000 160.000000 40.000000 1500.000000 1000.000000 500.000000"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"We can see that there are discrepancies in height(cm) and weight(kg) columns.\n",
"\n",
"The minimum value of height(cm) is 0. It is not possible because height cannot be 0.\n",
"\n",
"The minimum and maximum values of weight(kg) are -60 and 160. Weight cannot be negative and weight cannot be as high as 160. \n",
"So, both are invalid values.\n",
"\n",
"They are **outliers** and need to be properly addressed."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### df.columns attribute\n",
"\n",
"We can get the column labels of the dataframe with **df.columns** attribute.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Index(['fname', 'lname', 'age_sex', 'section', 'height(cm)', 'weight(kg)',\n",
" 'age', 'spend_A', 'spend_B', 'spend_C'],\n",
" dtype='object')"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visual Exploratory Data Analysis\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we should conduct data visualization to find discrepancies in the data. Data visualization is a great way to find \n",
"errors in the data and detect outliers. They help us to detect patterns in the data.\n",
"\n",
"\n",
"We can use various types of plots for data visualization purpose. These plots are listed below:-\n",
"\n",
"\n",
"•\t**Bar plot**\n",
"\n",
"•\t**Histograms**\n",
"\n",
"•\t**Box plot**\n",
"\n",
"•\t**Scatter plot**\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Bar plot\n",
"\n",
"A bar plot is a plot that presents data with rectangular bars with lengths proportional to the values that they represent.\n",
"\n",
"We can plot a bar plot of the age column as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAEJtJREFUeJzt3X2sZHV9x/H3h11UECpQrnRlXdciKlplsberkWoU1G7RKjY2FY3dNLSrrUSsphVtUjGpDSZa2j+MzSrgNlF8QA2WWpXyoKUPwAUWdnFRFKmui3CNoqINduHbP+aQXOm9zNw7D3f4+X4lk3vOb87M78Od5TPnnjkzk6pCkvTwd8BqB5AkjYaFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWrE2klOduSRR9bGjRsnOaUkPexdd91136uqmX7bTbTQN27cyNzc3CSnlKSHvST/Pch2HnKRpEZY6JLUCAtdkhphoUtSIyx0SWrEwIWeZE2SG5Jc0q0/McnVSW5N8vEkjxhfTElSP8vZQz8T2LNg/T3AuVV1LPAD4PRRBpMkLc9AhZ5kPfBS4EPdeoCTgIu6TXYAp44joCRpMIO+sejvgL8ADu3Wfxm4u6r2d+t7gaMXu2GSbcA2gA0bNqw8qVbFnqceN/R9HHfLnv4bSRpa3z30JC8D7qqq6xYOL7Lpot82XVXbq2q2qmZnZvq+c1WStEKD7KGfCLw8ySnAo4BforfHfliStd1e+npg3/hiSpL66buHXlVvr6r1VbUReDVweVW9FrgCeFW32Vbg4rGllCT1Ncx56G8D3pLk6/SOqZ83mkiSpJVY1qctVtWVwJXd8m3A5tFHkiStxEQ/Pldaife/4fKh7+ON/3DSCJJI0823/ktSIyx0SWqEhS5JjbDQJakRvii6mLMfM4L7+OHw9yFp6v3KFTuHuv13X7hpREncQ5ekZljoktQIC12SGmGhS1IjpupF0Y1n/fPQ93H7OS8dQRJJevhxD12SGmGhS1IjLHRJaoSFLkmNmKoXRaVp9r7ff9lQt3/rxy8ZURIBXHb5MUPfx8knfWMESaaHe+iS1Ii+hZ7kUUmuSXJjkpuTvKsb/3CSbybZ2V1G94EEkqRlG+SQy73ASVV1T5IDgauS/Et33Z9X1UXjiydJGlTfQq+qAu7pVg/sLjXOUJKk5RvoRdEka4DrgCcB76+qq5P8CfDuJH8FXAacVVX3LnLbbcA2gA0bNowseOueseMZQ9/Hrq27RpBE02TvWf829H2sP+d5I0iiaTTQi6JVdV9VbQLWA5uT/BrwduCpwG8ARwBvW+K226tqtqpmZ2ZmRhRbkvRgyzrLparuBq4EtlTVHdVzL3ABsHkM+SRJAxrkLJeZJId1ywcBLwJuSbKuGwtwKrB7nEElSQ9tkGPo64Ad3XH0A4BPVNUlSS5PMgME2Am8YYw5JUl9DHKWy03ACYuMnzSWRJKm3tlnn72qt9fifKeoJDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNWKQ7xR9VJJrktyY5OYk7+rGn5jk6iS3Jvl4kkeMP64kaSmD7KHfC5xUVccDm4AtSZ4DvAc4t6qOBX4AnD6+mJKkfvoWevXc060e2F0KOAm4qBvfAZw6loSSpIEMdAw9yZokO4G7gEuBbwB3V9X+bpO9wNFL3HZbkrkkc/Pz86PILElaxECFXlX3VdUmYD2wGThusc2WuO32qpqtqtmZmZmVJ5UkPaRlneVSVXcDVwLPAQ5Lsra7aj2wb7TRJEnLMchZLjNJDuuWDwJeBOwBrgBe1W22Fbh4XCElSf2t7b8J64AdSdbQewL4RFVdkuQrwMeS/DVwA3DeGHNKkvroW+hVdRNwwiLjt9E7ni5JmgK+U1SSGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaMch3ij4+yRVJ9iS5OcmZ3fjZSb6TZGd3OWX8cSVJSxnkO0X3A2+tquuTHApcl+TS7rpzq+q944snSRrUIN8pegdwR7f84yR7gKPHHUyStDzLOoaeZCO9L4y+uhs6I8lNSc5PcvgSt9mWZC7J3Pz8/FBhJUlLG7jQkxwCfAp4c1X9CPgAcAywid4e/PsWu11Vba+q2aqanZmZGUFkSdJiBir0JAfSK/OPVNWnAarqzqq6r6ruBz4IbB5fTElSP4Oc5RLgPGBPVf3tgvF1CzZ7JbB79PEkSYMa5CyXE4HXAbuS7OzG3gGclmQTUMDtwOvHklCSNJBBznK5CsgiV31u9HEkSSvlO0UlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpEYN8p+jjk1yRZE+Sm5Oc2Y0fkeTSJLd2Pw8ff1xJ0lIG2UPfD7y1qo4DngO8McnTgLOAy6rqWOCybl2StEr6FnpV3VFV13fLPwb2AEcDrwB2dJvtAE4dV0hJUn/LOoaeZCNwAnA1cFRV3QG90gceu8RttiWZSzI3Pz8/XFpJ0pIGLvQkhwCfAt5cVT8a9HZVtb2qZqtqdmZmZiUZJUkDGKjQkxxIr8w/UlWf7obvTLKuu34dcNd4IkqSBjHIWS4BzgP2VNXfLrjqs8DWbnkrcPHo40mSBrV2gG1OBF4H7Eqysxt7B3AO8IkkpwPfAn5vPBElSYPoW+hVdRWQJa4+ebRxJEkr5TtFJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNGOQr6M5PcleS3QvGzk7ynSQ7u8sp440pSepnkD30DwNbFhk/t6o2dZfPjTaWJGm5+hZ6VX0Z+P4EskiShjDMMfQzktzUHZI5fGSJJEkrstJC/wBwDLAJuAN431IbJtmWZC7J3Pz8/AqnkyT1s6JCr6o7q+q+qrof+CCw+SG23V5Vs1U1OzMzs9KckqQ+VlToSdYtWH0lsHupbSVJk7G23wZJLgReAByZZC/wTuAFSTYBBdwOvH6MGSVJA+hb6FV12iLD540hiyRpCL5TVJIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhrRt9CTnJ/kriS7F4wdkeTSJLd2Pw8fb0xJUj+D7KF/GNjyoLGzgMuq6ljgsm5dkrSK+hZ6VX0Z+P6Dhl8B7OiWdwCnjjiXJGmZVnoM/aiqugOg+/nY0UWSJK3E2F8UTbItyVySufn5+XFPJ0m/sFZa6HcmWQfQ/bxrqQ2rantVzVbV7MzMzAqnkyT1s9JC/yywtVveClw8mjiSpJUa5LTFC4H/BJ6SZG+S04FzgBcnuRV4cbcuSVpFa/ttUFWnLXHVySPOIkkagu8UlaRGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEb0/caih5LkduDHwH3A/qqaHUUoSdLyDVXonRdW1fdGcD+SpCF4yEWSGjFsoRfwxSTXJdk2ikCSpJUZ9pDLiVW1L8ljgUuT3FJVX164QVf02wA2bNgw5HSSpKUMtYdeVfu6n3cBnwE2L7LN9qqararZmZmZYaaTJD2EFRd6kkcnOfSBZeAlwO5RBZMkLc8wh1yOAj6T5IH7+WhVfX4kqSRJy7biQq+q24DjR5hFkjQET1uUpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRgxV6Em2JPlqkq8nOWtUoSRJyzfMl0SvAd4P/DbwNOC0JE8bVTBJ0vIMs4e+Gfh6Vd1WVT8DPga8YjSxJEnLlapa2Q2TVwFbquqPuvXXAc+uqjMetN02YFu3+hTgqyuPC8CRwPeGvI9hTUMGmI4c05ABpiPHNGSA6cgxDRlgOnKMIsMTqmqm30Zrh5ggi4z9v2eHqtoObB9inp+fNJmrqtlR3d/DNcO05JiGDNOSYxoyTEuOacgwLTkmmWGYQy57gccvWF8P7BsujiRppYYp9GuBY5M8MckjgFcDnx1NLEnScq34kEtV7U9yBvAFYA1wflXdPLJkSxvZ4ZshTEMGmI4c05ABpiPHNGSA6cgxDRlgOnJMLMOKXxSVJE0X3ykqSY2w0CWpERa6JDVimPPQxy7JU+m9+/Roeue47wM+W1V7VjXYL7Akm4Gqqmu7j3rYAtxSVZ9bxUz/WFV/sFrza/UtONNuX1X9a5LXAM8F9gDbq+p/VzXghEzti6JJ3gacRu8jBfZ2w+vpPWgfq6pzVivbauie3I4Grq6qexaMb6mqz08owzvpfXbPWuBS4NnAlcCLgC9U1bsnkOHBp8YGeCFwOUBVvXzcGRaT5DfpfRzG7qr64oTmfDawp6p+lOQg4CzgWcBXgL+pqh9OKMebgM9U1bcnMd8SGT5C79/lwcDdwCHAp4GT6fXc1glmOQZ4Jb336ewHbgUunMTjMc2F/jXg6Q9+Zu2eiW+uqmNXJ9nPZfnDqrpgAvO8CXgjvb2NTcCZVXVxd931VfWscWfo5trVzf9I4LvA+gVlcnVVPXMCGa6nV1gfovdXW4AL6T3RU1VfGneGLsc1VbW5W/5jeo/PZ4CXAP80iR2OJDcDx3enEG8HfgpcRK/Ejq+q3x13hi7HD4GfAN+g91h8sqrmJzH3ggw3VdUzk6wFvgM8rqruSxLgxkn82+xyvAn4HeBLwCnATuAH9Ar+T6vqyrEGqKqpvAC30Pv8ggePPwH46mrn67J8a0Lz7AIO6ZY3AnP0Sh3ghgn+996w2HK3vnNCGQ4A/ozeXwiburHbVuGxX/i7uBaY6ZYfDeyaUIY9C5avX43H44HfRfe4vAQ4D5gHPg9sBQ6dUIbdwCOAw4EfA0d0449a+HuaQI5dwJpu+WDgym55wyT+X53mY+hvBi5LcivwwJ9yG4AnAWcseasRS3LTUlcBR00oxprqDrNU1e1JXgBclOQJLP6ZOuPysyQHV9VPgV9/YDDJY4D7JxGgqu4Hzk3yye7nnazOa0EHJDmcXpGluj3SqvpJkv0TyrB7wV+JNyaZraq5JE8GJnnMuLrH5YvAF5McSO/Q3GnAe4G+Hyo1AufR2wlcA/wl8MkktwHPoXfYdpLWAvfR+0v2UICq+lb3exmrqT3kApDkAHrHJY+mV1x7gWur6r4JZrgT+C16fzb93FXAf1TV4yaQ4XLgLVW1c8HYWuB84LVVtWbcGbo5H1lV9y4yfiSwrqp2TSLHg+Z+KXBiVb1jwvPeTu9JLPQO/Ty3qr6b5BDgqqraNIEMjwH+HngevU/zexa9nZ9vA2+qqhvHnaHLcUNVnbDEdQdV1f9MKMfjAKpqX5LD6L22862qumYS83cZzgROB/4LeD7wnqq6IMkM8Kmqev5Y55/mQp8GSc4DLqiqqxa57qNV9ZoJZFgP7K+q7y5y3YlV9e/jzqDBJDkYOKqqvjnBOQ8FfpXenuHeqrpzUnN38z+5qr42yTmnWZKnA8fRe4H8lonObaFLUht8Y5EkNcJCl6RGWOiS1AgLXZIa8X/ciUQroSXqdgAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df['age'].plot('bar')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"The bar plot shows that there are 10 counts of the age label. So, there are no missing values in the age column."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Histograms\n",
"\n",
"\n",
"We use histograms for plotting continuous data counts. A histogram is a representation of the distribution of data.\n",
"\n",
"In this case, we use histograms for plotting distribution of data values of height(cm) and weight(kg) columns.\n",
"\n",
"We can draw a histogram as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADztJREFUeJzt3X2MZXV9x/H3x10UUSq1jEqEdcBYWtKobEdjgtpIrUUUqDa1GNua1rptqqnENnV9iPJPE2nrQ01Nca1EwGdULPWhim2VmFRwFxdZXRDUteJSdtW0i0pF4Ns/7pl2WGZmz8zOmXsvv/crmcy5Z8+958NvLp8587vnnpuqQpJ0//eAcQeQJK0PC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUiI3jDrDQscceW7Ozs+OOIUlTY8eOHd+rqpk+205U4c/OzrJ9+/Zxx5CkqZHk2323dUpHkhph4UtSIyx8SWqEhS9JjbDwJakRg56lk2QPcDtwN3BXVc0NuT9J0tLW47TMZ1TV99ZhP5KkZTilI0mNGLrwC/hMkh1Jtgy8L0nSMoae0jmtqvYmeQRwZZIbquqqhRt0vwi2AGzatGngOJK0tNmtnxjLfve88Tnrsp9Bj/Cram/3fR9wOfDkRbbZVlVzVTU3M9PrchCSpFUYrPCTPCTJ0fPLwLOAXUPtT5K0vCGndB4JXJ5kfj/vq6p/HnB/kqRlDFb4VfVN4AlDPb4kaWU8LVOSGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUiMELP8mGJF9O8vGh9yVJWtp6HOG/Ati9DvuRJC1j0MJPcjzwHOAfhtyPJOnQhj7CfyvwF8A9A+9HknQIgxV+kucC+6pqxyG225Jke5Lt+/fvHyqOJDVvyCP804Czk+wBPgCcnuQ9B29UVduqaq6q5mZmZgaMI0ltG6zwq+rVVXV8Vc0C5wL/WlW/M9T+JEnL8zx8SWrExvXYSVV9DvjceuxLkrQ4j/AlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUiF6Fn+SXhg4iSRpW3yP8C5Nck+RPkhwzaCJJ0iB6FX5VPRV4EXACsD3J+5L82qDJJElrqvccflXdBLwOeBXwK8DbktyQ5PlDhZMkrZ2+c/iPT/IWYDdwOnBWVf1it/yWAfNJktbIxp7b/R3wTuA1VXXH/Mqq2pvkdYvdIcmRwFXAg7r9fLiq3nCYeSVJq9S38M8E7qiquwGSPAA4sqp+XFWXLnGfnwCnV9UPkxwBfCHJp6rqi4cfW5K0Un3n8D8LPHjB7aO6dUuqkR92N4/ovmrFCSVJa6Jv4R+5oLzplo861J2SbEiyE9gHXFlVV68upiTpcPUt/B8l2Tx/I8kvA3cssz0AVXV3VT0ROB548mJv4EqyJcn2JNv379/fN7ckaYX6zuGfB1yWZG93+zjgt/vupKr+K8nngDOAXQf92zZgG8Dc3JxTPpI0kF6FX1VfSvILwMlAgBuq6qfL3SfJDPDTruwfDDwTuOBwA0uSVqfvET7Ak4DZ7j6nJqGqLllm++OAi5NsYDR19KGq+viqk0qSDkuvwk9yKfBYYCdwd7e6gCULv6q+Apx6uAElSWuj7xH+HHBKVTnHLklTqu9ZOruARw0ZRJI0rL5H+McCX0tyDaN30AJQVWcPkkqStOb6Fv75Q4aQJA2v72mZn0/yGOBxVfXZJEcBG4aNJklaS30vj/xS4MPAO7pVjwY+NlQoSdLa6/ui7cuA04AD8H8fhvKIoUJJktZe38L/SVXdOX8jyUa88qUkTZW+hf/5JK8BHtx9lu1lwD8NF0uStNb6Fv5WYD9wPfBHwCcZfb6tJGlK9D1L5x5GH3H4zmHjSJKG0vdaOt9ikTn7qjppzRNJkgaxkmvpzDsS+C3g4WsfR5I0lF5z+FX1/QVf362qtwKnD5xNkrSG+k7pbF5w8wGMjviPHiSRJGkQfad03rRg+S5gD/CCNU8jSRpM37N0njF0EEnSsPpO6bxyuX+vqjevTRxJ0lBWcpbOk4AruttnAVcB3xkilCRp7a3kA1A2V9XtAEnOBy6rqj8cKpgkaW31vbTCJuDOBbfvBGbXPI0kaTB9j/AvBa5Jcjmjd9w+D7hksFSSpDXX9yydv0zyKeBp3arfr6ovDxdLkrTW+k7pABwFHKiqvwVuSXLiQJkkSQPo+xGHbwBeBby6W3UE8J6hQkmS1l7fI/znAWcDPwKoqr14aQVJmip9C//Oqiq6SyQnechwkSRJQ+hb+B9K8g7gmCQvBT6LH4YiSVOl71k6f9N9lu0B4GTg9VV15aDJJElr6pCFn2QD8OmqeiZgyUvSlDrklE5V3Q38OMnD1iGPJGkgfd9p+z/A9UmupDtTB6Cq/nSQVJKkNde38D/RffWW5ARGl194FHAPsK1705YkaQyWLfwkm6rqP6rq4lU89l3An1XVtUmOBnYkubKqvraqpJKkw3KoOfyPzS8k+chKHriqbq2qa7vl24HdwKNXnFCStCYOVfhZsHzSaneSZBY4Fbh6tY8hSTo8h5rDryWWe0vyUOAjwHlVdWCRf98CbAHYtGnTanYh6X5kduuKXi7UChzqCP8JSQ4kuR14fLd8IMntSe5T3gdLcgSjsn9vVX10sW2qaltVzVXV3MzMzMr/CyRJvSx7hF9VG1b7wEkCvAvY7YecS9L4reR6+Ct1GvC7wOlJdnZfZw64P0nSMvqeh79iVfUF7v2iryRpjIY8wpckTRALX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktSIwQo/yUVJ9iXZNdQ+JEn9DXmE/27gjAEfX5K0AoMVflVdBfxgqMeXJK3MxnEHSLIF2AKwadOmVT/O7NZPrFWkFdnzxueMZb+StFJjf9G2qrZV1VxVzc3MzIw7jiTdb4298CVJ68PCl6RGDHla5vuBfwdOTnJLkpcMtS9J0qEN9qJtVb1wqMeWJK2cUzqS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRlj4ktQIC1+SGmHhS1IjLHxJaoSFL0mNsPAlqREWviQ1wsKXpEZY+JLUCAtfkhph4UtSIyx8SWqEhS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiMsfElqhIUvSY2w8CWpERa+JDXCwpekRgxa+EnOSHJjkpuTbB1yX5Kk5Q1W+Ek2AG8Hng2cArwwySlD7U+StLwhj/CfDNxcVd+sqjuBDwDnDLg/SdIyhiz8RwPfWXD7lm6dJGkMNg742FlkXd1no2QLsKW7+cMkN65yf8cC31vlfVctF6zqbmPJehimKa9ZhzNNeacqay44rKyP6bvhkIV/C3DCgtvHA3sP3qiqtgHbDndnSbZX1dzhPs56mKasMF15zTqcacpr1sUNOaXzJeBxSU5M8kDgXOCKAfcnSVrGYEf4VXVXkpcDnwY2ABdV1VeH2p8kaXlDTulQVZ8EPjnkPhY47GmhdTRNWWG68pp1ONOU16yLSNV9XkeVJN0PeWkFSWrE1Bf+pF++IckJSf4tye4kX03yim79+Um+m2Rn93XmuLMCJNmT5Pou0/Zu3cOTXJnkpu77z05AzpMXjN3OJAeSnDdJ45rkoiT7kuxasG7RsczI27rn8VeSbJ6ArH+d5IYuz+VJjunWzya5Y8EYX7ieWZfJu+TPPsmru7G9McmvT0DWDy7IuSfJzm79sGNbVVP7xejF4G8AJwEPBK4DThl3roMyHgds7paPBr7O6FIT5wN/Pu58i+TdAxx70Lq/ArZ2y1uBC8adc5HnwX8yOh95YsYVeDqwGdh1qLEEzgQ+xej9K08Brp6ArM8CNnbLFyzIOrtwuwka20V/9t3/b9cBDwJO7DpjwzizHvTvbwJevx5jO+1H+BN/+YaqurWqru2Wbwd2M33vOD4HuLhbvhj4jTFmWcyvAt+oqm+PO8hCVXUV8IODVi81lucAl9TIF4Fjkhy3PkkXz1pVn6mqu7qbX2T0XpqJsMTYLuUc4ANV9ZOq+hZwM6PuWBfLZU0S4AXA+9cjy7QX/lRdviHJLHAqcHW36uXdn8sXTcI0SaeAzyTZ0b0LGuCRVXUrjH6BAY8YW7rFncu9/4eZxHGdt9RYTvpz+Q8Y/QUy78QkX07y+SRPG1eoRSz2s5/ksX0acFtV3bRg3WBjO+2F3+vyDZMgyUOBjwDnVdUB4O+BxwJPBG5l9GfdJDitqjYzusrpy5I8fdyBltO9qe9s4LJu1aSO66FM7HM5yWuBu4D3dqtuBTZV1anAK4H3JfmZceVbYKmf/cSOLfBC7n2wMujYTnvh97p8w7glOYJR2b+3qj4KUFW3VdXdVXUP8E7W8U/M5VTV3u77PuByRrlum59e6L7vG1/C+3g2cG1V3QaTO64LLDWWE/lcTvJi4LnAi6qbZO6mRr7fLe9gNCf+8+NLObLMz35Sx3Yj8Hzgg/Prhh7baS/8ib98QzdH9y5gd1W9ecH6hfOzzwN2HXzf9ZbkIUmOnl9m9KLdLkZj+uJusxcD/ziehIu61xHSJI7rQZYayyuA3+vO1nkK8N/zUz/jkuQM4FXA2VX14wXrZzL6vAuSnAQ8DvjmeFL+v2V+9lcA5yZ5UJITGeW9Zr3zLeKZwA1Vdcv8isHHdr1eqR7qi9HZDV9n9JvwtePOs0i+pzL68/ErwM7u60zgUuD6bv0VwHETkPUkRmczXAd8dX48gZ8D/gW4qfv+8HFn7XIdBXwfeNiCdRMzrox+Ed0K/JTRUeZLlhpLRtMOb++ex9cDcxOQ9WZGc9/zz9sLu21/s3t+XAdcC5w1IWO75M8eeG03tjcCzx531m79u4E/PmjbQcfWd9pKUiOmfUpHktSThS9JjbDwJakRFr4kNcLCl6RGWPiS1AgLX5IaYeFLUiP+F63QPlJxDSYoAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df['height(cm)'].plot('hist')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"The above histogram shows that there is a data value between 0 and 25. When we take a closer look , we find that there is a value of 0.0 in the height(cm) column. It is not possible as height cannot be 0. So, we need to take care of that."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD8CAYAAACYebj1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAEqxJREFUeJzt3X2QXXddx/H3hyRQQIYqWSWTJmyVjloYaMvaqeJDKVVLwUa0aBgVygBRph1kdEZaxinYGWfAEaoIUoPtNFSBliIaIIitiMgftN2WUPoAQwYqDe3Q2ELTUGhN+frHPTkuN3f3njycvcnu+zVzJ+fhd+753jM397O/85iqQpIkgMdNugBJ0pHDUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVJr5aQLOFCrV6+u6enpSZchSUeVm2+++X+qampcu6MuFKanp5mdnZ10GZJ0VEny313auftIktQyFCRJLUNBktQyFCRJLUNBktTqPRSSrEjy+SQfGzHvCUmuTrIjyQ1JpvuuR5I0v8XoKfwhcOc8814NfKuqnglcCrxtEeqRJM2j11BIchzwYuDv52myAdjSDF8LvDBJ+qxJkjS/vnsKfwX8CfD9eeavBe4GqKq9wIPA03quSZI0j96uaE7yEuC+qro5yenzNRsxrUa81yZgE8D69esPW43S4TZ94ccnst673vriiaxXS0+fPYXnA+ckuQv4IHBGkn8YarMTWAeQZCXwVOCB4Teqqs1VNVNVM1NTY2/dIUk6SL2FQlVdVFXHVdU0sBH4VFX97lCzrcArm+Fzmzb79RQkSYtj0W+Il+QSYLaqtgKXA1cl2cGgh7BxseuRJP2/RQmFqvo08Olm+OI5078HvGwxapAkjecVzZKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKkVm+hkOSYJDcm+UKS25P82Yg25yXZlWR783pNX/VIksbr83GcjwBnVNWeJKuAzyb5RFV9bqjd1VV1QY91SJI66i0UqqqAPc3oquZVfa1PknToej2mkGRFku3AfcB1VXXDiGa/meTWJNcmWddnPZKkhfUaClX1WFWdBBwHnJrk2UNNPgpMV9VzgOuBLaPeJ8mmJLNJZnft2tVnyZK0rC3K2UdV9W3g08BZQ9Pvr6pHmtH3As+bZ/nNVTVTVTNTU1O91ipJy1mfZx9NJTm2GX4icCbwpaE2a+aMngPc2Vc9kqTx+jz7aA2wJckKBuFzTVV9LMklwGxVbQVen+QcYC/wAHBej/VIksbo8+yjW4GTR0y/eM7wRcBFfdUgSTowXtEsSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWr1+YzmY5LcmOQLSW5P8mcj2jwhydVJdiS5Icl0X/VIksbrs6fwCHBGVT0XOAk4K8lpQ21eDXyrqp4JXAq8rcd6JElj9BYKNbCnGV3VvGqo2QZgSzN8LfDCJOmrJknSwno9ppBkRZLtwH3AdVV1w1CTtcDdAFW1F3gQeNqI99mUZDbJ7K5du/osWZKWtV5Doaoeq6qTgOOAU5M8e6jJqF7BcG+CqtpcVTNVNTM1NdVHqZIkFunso6r6NvBp4KyhWTuBdQBJVgJPBR5YjJokSfvr8+yjqSTHNsNPBM4EvjTUbCvwymb4XOBTVbVfT0GStDhW9vjea4AtSVYwCJ9rqupjSS4BZqtqK3A5cFWSHQx6CBt7rEeSNEZvoVBVtwInj5h+8Zzh7wEv66sGSdKB8YpmSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVKrUygkefaBvnGSdUn+I8mdSW5P8ocj2pye5MEk25vXxaPeS5K0OLo+jvOyJI8HrgTeX1Xf7rDMXuCPq+qWJE8Bbk5yXVXdMdTuv6rqJd1LliT1pVNPoap+HvgdYB0wm+T9SX55zDL3VtUtzfBDwJ3A2kOsV5LUo87HFKrqK8CfAm8Efgl4Z5IvJfmNccsmmQZOBm4YMftnk3whySeSPGue5TclmU0yu2vXrq4lS5IOUNdjCs9JcimDv/bPAH6tqn66Gb50zLI/BHwYeENV7R6afQvwjKp6LvA3wD+Peo+q2lxVM1U1MzU11aVkSdJB6NpTeBeDH/DnVtX5c3YL3cOg9zBSklUMAuEfq+qfhudX1e6q2tMMbwNWJVl9gJ9BknSYdD3QfDbw3ap6DCDJ44Bjqurhqrpq1AJJAlwO3FlV75inzdOBb1ZVJTmVQUjdf6AfQpJ0eHQNheuBM4E9zfiTgH8Dfm6BZZ4P/B7wxSTbm2lvAtYDVNVlwLnA65LsBb4LbKyqOqBPIEk6bLqGwjH7dvMAVNWeJE9aaIGq+iyQMW3exWDXlCTpCND1mMJ3kpyybyTJ8xj8ZS9JWkK69hTeAHwoyT3N+Brgt/spSZI0KZ1CoapuSvJTwE8y2CX0par6314rkyQtuq49BYCfAaabZU5OQlW9r5eqJEkT0SkUklwF/ASwHXismVyAoSBJS0jXnsIMcKKni0rS0tb17KPbgKf3WYgkafK69hRWA3ckuRF4ZN/Eqjqnl6okSRPRNRTe0mcRkqQjQ9dTUv8zyTOAE6rq+uZq5hX9liZJWmxdb539WuBa4O+aSWuZ5zbXkqSjV9cDzeczuMHdbmgfuPOjfRUlSZqMrqHwSFU9um8kyUoG1ylIkpaQrqHwn0neBDyxeTbzh4CP9leWJGkSuobChcAu4IvA7wPbWOCJa5Kko1PXs4++D7y3eUmSlqiu9z76GiOOIVTVjx/2iiRJE3Mg9z7a5xjgZcCPLLRAknUMbpj3dOD7wOaq+uuhNgH+msEzoB8GzquqWzrWJEk6zDodU6iq++e8vlFVfwWcMWaxvcAfV9VPA6cB5yc5cajNi4ATmtcm4D0HVr4k6XDquvvolDmjj2PQc3jKQstU1b3Avc3wQ0nuZHDR2x1zmm0A3tfcffVzSY5NsqZZVpK0yLruPnr7nOG9wF3Ab3VdSZJp4GTghqFZa4G754zvbKb9QCgk2cSgJ8H69eu7rlZaNqYv/PhE1nvXW188kfWqP13PPnrBwa4gyQ8BHwbeUFW7h2ePWt2I9W8GNgPMzMx40Zwk9aTr7qM/Wmh+Vb1jnuVWMQiEf6yqfxrRZCewbs74ccA9XWqSJB1+XS9emwFex2DXzlrgD4ATGRxXGHlsoTmz6HLgzvlCA9gKvCIDpwEPejxBkibnQB6yc0pVPQSQ5C3Ah6rqNQss83zg94AvJtneTHsTsB6gqi5jcGX02cAOBqekvupAP4Ak6fDpGgrrgUfnjD8KTC+0QFV9ltHHDOa2KQZ3YJUkHQG6hsJVwI1JPsLgQPBLGVyYJklaQrqeffTnST4B/EIz6VVV9fn+ypIkTULXA80ATwJ2N7eq2Jnk+J5qkiRNSNfHcb4ZeCNwUTNpFfAPfRUlSZqMrj2FlwLnAN8BqKp7GHObC0nS0adrKDzanClUAEme3F9JkqRJ6RoK1yT5O+DYJK8FrscH7kjSktP17KO/bJ7NvBv4SeDiqrqu18okSYtubCgkWQF8sqrOBAwCSVrCxu4+qqrHgIeTPHUR6pEkTVDXK5q/x+AeRtfRnIEEUFWv76UqSdJEdA2FjzcvSdIStmAoJFlfVV+vqi2LVZAkaXLGHVP4530DST7ccy2SpAkbFwpzb339430WIkmavHGhUPMMS5KWoHGh8Nwku5M8BDynGd6d5KEkuxdaMMkVSe5Lcts8809P8mCS7c3r4oP9EJKkw2PBA81VteIQ3vtK4F0s/DCe/6qqlxzCOiRJh9GBPE/hgFTVZ4AH+np/SdLh11sodPSzSb6Q5BNJnjXhWiRp2et68VofbgGeUVV7kpzN4PTXE0Y1TLIJ2ASwfv36xatQkpaZifUUqmp3Ve1phrcBq5Ksnqft5qqaqaqZqampRa1TkpaTiYVCkqcnSTN8alPL/ZOqR5LU4+6jJB8ATgdWJ9kJvJnBs52pqsuAc4HXJdkLfBfY2DzdTZI0Ib2FQlW9fMz8dzE4ZVWSdISY9NlHkqQjiKEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKklqEgSWoZCpKkVm+hkOSKJPcluW2e+UnyziQ7ktya5JS+apEkddNnT+FK4KwF5r8IOKF5bQLe02MtkqQOeguFqvoM8MACTTYA76uBzwHHJlnTVz2SpPEmeUxhLXD3nPGdzTRJ0oSsnOC6M2JajWyYbGKwi4n169cf9AqnL/z4QS97qO5664sntm5Jh89S/x2ZZE9hJ7BuzvhxwD2jGlbV5qqaqaqZqampRSlOkpajSYbCVuAVzVlIpwEPVtW9E6xHkpa93nYfJfkAcDqwOslO4M3AKoCqugzYBpwN7AAeBl7VVy2SpG56C4WqevmY+QWc39f6JUkHziuaJUktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEmtXkMhyVlJvpxkR5ILR8w/L8muJNub12v6rEeStLA+n9G8Ang38MvATuCmJFur6o6hpldX1QV91SFJ6q7PnsKpwI6q+mpVPQp8ENjQ4/okSYeoz1BYC9w9Z3xnM23Ybya5Ncm1Sdb1WI8kaYw+QyEjptXQ+EeB6ap6DnA9sGXkGyWbkswmmd21a9dhLlOStE+fobATmPuX/3HAPXMbVNX9VfVIM/pe4Hmj3qiqNlfVTFXNTE1N9VKsJKnfULgJOCHJ8UkeD2wEts5tkGTNnNFzgDt7rEeSNEZvZx9V1d4kFwCfBFYAV1TV7UkuAWaraivw+iTnAHuBB4Dz+qpHkjReb6EAUFXbgG1D0y6eM3wRcFGfNUiSuvOKZklSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSq9dQSHJWki8n2ZHkwhHzn5Dk6mb+DUmm+6xHkrSw3kIhyQrg3cCLgBOBlyc5cajZq4FvVdUzgUuBt/VVjyRpvD57CqcCO6rqq1X1KPBBYMNQmw3Almb4WuCFSdJjTZKkBfQZCmuBu+eM72ymjWxTVXuBB4Gn9ViTJGkBK3t871F/8ddBtCHJJmBTM7onyZcPsbYDtRr4n0N5gyy9HWOHvE2WqGW1XTp+r5fVNunooLbJIf6OPKNLoz5DYSewbs74ccA987TZmWQl8FTggeE3qqrNwOae6hwryWxVzUxq/Ucit8lobpf9uU32dyRvkz53H90EnJDk+CSPBzYCW4fabAVe2QyfC3yqqvbrKUiSFkdvPYWq2pvkAuCTwArgiqq6PcklwGxVbQUuB65KsoNBD2FjX/VIksbrc/cRVbUN2DY07eI5w98DXtZnDYfJxHZdHcHcJqO5XfbnNtnfEbtN4t4aSdI+3uZCktQyFBaQ5C1JvpFke/M6e868i5rbc3w5ya9Oss7FNu72JctFkruSfLH5bsw2034kyXVJvtL8+8OTrrNPSa5Icl+S2+ZMG7kNMvDO5ntza5JTJld5v+bZLkfF74mhMN6lVXVS89oG0NyuYyPwLOAs4G+b23oseR1vX7KcvKD5buw7vfBC4N+r6gTg35vxpexKBv8H5ppvG7wIOKF5bQLes0g1TsKV7L9d4Cj4PTEUDs4G4INV9UhVfQ3YweC2HstBl9uXLGdzb92yBfj1CdbSu6r6DPtfWzTfNtgAvK8GPgccm2TN4lS6uObZLvM5on5PDIXxLmi6ulfM2RXQ5RYeS9Vy/uzDCvi3JDc3V90D/FhV3QvQ/PujE6tucubbBn53joLfk2UfCkmuT3LbiNcGBt3bnwBOAu4F3r5vsRFvtVxO41rOn33Y86vqFAa7Rc5P8ouTLugIt9y/O0fF70mv1ykcDarqzC7tkrwX+Fgz2uUWHkvVcv7sP6Cq7mn+vS/JRxh0+b+ZZE1V3dvsGrlvokVOxnzbYFl/d6rqm/uGj+Tfk2XfU1jI0P7OlwL7ziTYCmxsHhJ0PIMDZzcudn0T0uX2JUtekicnecq+YeBXGHw/5t665ZXAv0ymwomabxtsBV7RnIV0GvDgvt1My8HR8nuy7HsKY/xFkpMYdOXuAn4foLldxzXAHcBe4PyqemxiVS6i+W5fMuGyJuHHgI80j/9YCby/qv41yU3ANUleDXydo+OK/YOW5APA6cDqJDuBNwNvZfQ22AaczeBA6sPAqxa94EUyz3Y5/Wj4PfGKZklSy91HkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJav0fxP6Nga2FSBwAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df['weight(kg)'].plot('hist')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"Similar analysis of histogram of weight(kg) column shows that there is a negative value of -60 and a very high value of 160 in \n",
"the weight(kg) column. Both are invalid values. Weight cannot be negative and also weight cannot be so high. So, we need to take\n",
"care of these invalid values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Box plot\n",
"\n",
"\n",
"We can visualize basic summary statistics with box plot. Box plot let us to detect outliers in the data. They help us to find minimum and maximum values. They present 25th, 50th, 75th percentiles. 50th percentile value is the median value.\n",
"\n",
"We can draw a boxplot as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAENFJREFUeJzt3X+MZWV9x/H3p4taf5D6A51QxS42aFfWstbR2lrNUKwFNUXbqGyMgmy6kqixqaZFt6m2dhPjj5o0/iBrFsFEV20RNUoRQuaKTbV1V1dcXKmAqCsbUFFgkKK7fvvHnGmuw8D8OPcyOw/vV3Jzz3nOc57zhRw+c3juufekqpAktevXVrsASdJ4GfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxh212gUAHHPMMbV+/frVLkNa0B133MFDH/rQ1S5Dups9e/b8qKoevVi/IyLo169fz+7du1e7DGlBg8GAqamp1S5Dupsk311KP6duJKlxBr0kNc6gl6TGGfSS1DiDXpIat2jQJzk/yc1J9g21fTzJ3u51Q5K9Xfv6JHcObTtvnMVLkha3lNsrLwDeC3x4rqGqXja3nOTdwK1D/a+rqk2jKlCS1M+iQV9VVyZZv9C2JAFeCvzxaMuSJI1K3y9MPRu4qaq+PdR2fJKvAbcBf1dVX+x5DGlkZq9Nxs9nMetI0jfoNwO7htYPAo+vqh8neRrwqSQnVtVt83dMshXYCjAxMcFgMOhZirS46enpZe9z1qV3cMGpy/sJBM9nHUlWHPRJjgL+HHjaXFtV3QXc1S3vSXId8ETgbr9vUFU7gB0Ak5OT5VfMdcS69HP+BILWtD63Vz4X+FZVHZhrSPLoJOu65ScAJwDX9ytRktTHUm6v3AV8CXhSkgNJtnSbzuBXp20AngNcleTrwL8B51TVLaMsWJK0PEu562bzPbSftUDbRcBF/cuSJI2K34yVpMYZ9JLUOINekhpn0EtS446IRwlKK3XSP1zGrXf+YuzHWX/u58Y6/m88+AF8/S3PG+sxdP9l0GtNu/XOX3DD218w1mPcF8+MHfcfEt2/OXUjSY0z6CWpcQa9JDXOoJekxvlhrNa0ozecy1MuPHf8B7pwvMMfvQFgvB8q6/7LoNeadvv+t3vXjbQIp24kqXEGvSQ1zqCXpMYZ9JLUOINekhrnXTda8+6TO1YuHf+PmknjYtBrTVvurZVJxlTJr6qq++Q40lIs5eHg5ye5Ocm+oba3JvlBkr3d6/lD296U5Nok1yT503EVLq1EVS37NT09vex9pCPJUuboLwBOXaD9PVW1qXtdApDkycAZwIndPu9Psm5UxUqSlm/RoK+qK4Fbljje6cDHququqvoOcC3wjB71SZJ66nPXzWuTXNVN7Tyia3ss8P2hPge6NknSKlnph7EfAN4GVPf+buBsYKFPuhacsEyyFdgKMDExwWAwWGEp0njNzMx4fmpNW1HQV9VNc8tJPgh8tls9ABw31PVxwI33MMYOYAfA5ORkjftHo6SVui9+1EwapxVN3SQ5dmj1xcDcHTmfAc5I8qAkxwMnAP/dr0RJUh+LXtEn2QVMAcckOQC8BZhKsonZaZkbgFcDVNXVST4BfBM4BLymqg6Pp3RJ0lIsGvRVtXmB5p330n87sL1PUZKk0fG3biSpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGLRr0Sc5PcnOSfUNt70zyrSRXJbk4ycO79vVJ7kyyt3udN87iJUmLW8oV/QXAqfPaLgc2VtXvAv8DvGlo23VVtal7nTOaMiVJK7Vo0FfVlcAt89ouq6pD3eqXgceNoTZJ0giMYo7+bODfh9aPT/K1JF9I8uwRjC9J6uGoPjsn2QYcAj7SNR0EHl9VP07yNOBTSU6sqtsW2HcrsBVgYmKCwWDQpxRpbGZmZjw/taatOOiTnAm8EDilqgqgqu4C7uqW9yS5DngisHv+/lW1A9gBMDk5WVNTUystRRqrwWCA56fWshVN3SQ5Ffhb4M+q6mdD7Y9Osq5bfgJwAnD9KAqVJK3Molf0SXYBU8AxSQ4Ab2H2LpsHAZcnAfhyd4fNc4B/THIIOAycU1W3LDiwJOk+sWjQV9XmBZp33kPfi4CL+hYlSRodvxkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGLSnok5yf5OYk+4baHpnk8iTf7t4f0bUnyb8kuTbJVUl+b1zFS5IWt9Qr+guAU+e1nQtcUVUnAFd06wCnASd0r63AB/qXKUlaqSUFfVVdCdwyr/l04MJu+ULgRUPtH65ZXwYenuTYURQrSVq+o3rsO1FVBwGq6mCSx3TtjwW+P9TvQNd2cHjnJFuZveJnYmKCwWDQoxRpfGZmZjw/tab1Cfp7kgXa6m4NVTuAHQCTk5M1NTU1hlKk/gaDAZ6fWsv63HVz09yUTPd+c9d+ADhuqN/jgBt7HEeS1EOfoP8McGa3fCbw6aH2V3Z33zwTuHVuikeSdN9b0tRNkl3AFHBMkgPAW4C3A59IsgX4HvCSrvslwPOBa4GfAa8acc2SpGVYUtBX1eZ72HTKAn0LeE2foiRJo+M3YyWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGLemZsQtJ8iTg40NNTwD+Hng48JfAD7v2N1fVJSuuUJLUy4qDvqquATYBJFkH/AC4GHgV8J6qetdIKpQk9TKqqZtTgOuq6rsjGk+SNCIrvqKf5wxg19D6a5O8EtgNvKGqfjJ/hyRbga0AExMTDAaDEZUijdbMzIznp9a0VFW/AZIHAjcCJ1bVTUkmgB8BBbwNOLaqzr63MSYnJ2v37t296pDGZTAYMDU1tdplSHeTZE9VTS7WbxRTN6cBX62qmwCq6qaqOlxVvwQ+CDxjBMeQJK3QKIJ+M0PTNkmOHdr2YmDfCI4hSVqhXnP0SR4C/Anw6qHmdyTZxOzUzQ3ztkmS7mO9gr6qfgY8al7bK3pVJEkaKb8ZK0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWu1zNjAZLcANwOHAYOVdVkkkcCHwfWM/uA8JdW1U/6HkuStHyjuqI/uao2VdVkt34ucEVVnQBc0a1LklbBuKZuTgcu7JYvBF40puNIkhYxiqAv4LIke5Js7domquogQPf+mBEcR5K0Ar3n6IFnVdWNSR4DXJ7kW0vZqfujsBVgYmKCwWAwglKk0ZuZmfH81JrWO+ir6sbu/eYkFwPPAG5KcmxVHUxyLHDzAvvtAHYATE5O1tTUVN9SpLEYDAZ4fmot6zV1k+ShSY6eWwaeB+wDPgOc2XU7E/h0n+NIklau7xX9BHBxkrmxPlpVlyb5CvCJJFuA7wEv6XkcSdIK9Qr6qroeOGmB9h8Dp/QZW5I0Gn4zVpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrcioM+yXFJppPsT3J1ktd37W9N8oMke7vX80dXriRpufo8HPwQ8Iaq+mqSo4E9SS7vtr2nqt7VvzxJUl8rDvqqOggc7JZvT7IfeOyoCpMkjcZI5uiTrAeeCvxX1/TaJFclOT/JI0ZxDEnSyvSZugEgycOAi4C/qqrbknwAeBtQ3fu7gbMX2G8rsBVgYmKCwWDQtxRpLGZmZjw/taalqla+c/IA4LPA56vqnxfYvh74bFVtvLdxJicna/fu3SuuQxqnwWDA1NTUapch3U2SPVU1uVi/PnfdBNgJ7B8O+STHDnV7MbBvpceQJPXXZ+rmWcArgG8k2du1vRnYnGQTs1M3NwCv7lWhJKmXPnfd/AeQBTZdsvJyJEmj5jdjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMaNLeiTnJrkmiTXJjl3XMeRJN27sQR9knXA+4DTgCcDm5M8eRzHksZl165dbNy4kVNOOYWNGzeya9eu1S5JWpGjxjTuM4Brq+p6gCQfA04Hvjmm40kjtWvXLrZt28bOnTs5fPgw69atY8uWLQBs3rx5lauTlmdcUzePBb4/tH6ga5PWhO3bt7Nz505OPvlkjjrqKE4++WR27tzJ9u3bV7s0adnGdUWfBdrqVzokW4GtABMTEwwGgzGVIi3f/v37OXz4MIPBgJmZGQaDAYcPH2b//v2eq1pzxhX0B4DjhtYfB9w43KGqdgA7ACYnJ2tqampMpUjLt2HDBtatW8fU1BSDwYCpqSmmp6fZsGEDnqtaa8Y1dfMV4IQkxyd5IHAG8JkxHUsauW3btrFlyxamp6c5dOgQ09PTbNmyhW3btq12adKyjeWKvqoOJXkt8HlgHXB+VV09jmNJ4zD3gevrXvc69u/fz4YNG9i+fbsfxGpNSlUt3mvMJicna/fu3atdhrSguakb6UiTZE9VTS7Wz2/GSlLjDHpJapxBL0mNM+glqXEGvSQ17oi46ybJD4HvrnYd0j04BvjRahchLeC3qurRi3U6IoJeOpIl2b2UW9ikI5VTN5LUOINekhpn0EuL27HaBUh9OEcvSY3zil6SGmfQa01Ksj7JvmX0PyfJKxfpc1aS997DtjfPW39wki90z0desSQPTHJlknE9G0Iy6HX/UFXnVdWHewzx5nnrZwOfrKrDPcakqn4OXAG8rM840r0x6LWWrUvywSRXJ7msu8r+7SSXJtmT5ItJfgcgyVuTvLFbfnqSq5J8Kck75/2fwW92+387yTu6/m8HHpxkb5KPdP1eDnx6bqckf5PkG0m+3vUnySDJe7or9v3dcT/Zjf1PQ8f8VDeeNBYGvdayE4D3VdWJwE+Bv2D2DpnXVdXTgDcC719gvw8B51TVHwDzr8g3MXt1/RTgZUmOq6pzgTuralNVvbx7atoTquoGgCSnAS8Cfr+qTgLeMTTez6vqOcB5zP5heA2wETgryaO6PvuAp/f5FyHdG+cFtZZ9p6r2dst7gPXAHwL/mvz/8+kfNLxDkocDR1fVf3ZNHwVeONTliqq6tev7TeC3gO/PO+4xzP5hmfNc4ENV9TOAqrplaNvcIzS/AVxdVQe7sa9n9rnKP66qw0l+nuToqrp9qf/w0lIZ9FrL7hpaPgxMAD+tqk33sk/uZdtCYy7038idwK/PG/Oe7lOeG++X88b+5byxHwT87yK1SSvi1I1achvwnSQvAcisk4Y7VNVPgNuTPLNrOmOJY/8iyQOGxliXZC7sLwPOTvKQ7riPXE7R3RTOD6vqF8vZT1oqg16teTmwJcnXgauB0xfoswXYkeRLzF6N37qEcXcAVw19GHsZ8EcAVXUps1M0u5PsZfazgeU4GbhkmftIS+Y3Y3W/k+RhVTXTLZ8LHFtVr1/mGE8F/rqqXjGCej4JvKmqruk7lrQQ5+h1f/SCJG9i9vz/LnDWcgeoqq8lmU6yrs+99N0dPJ8y5DVOXtFLUuOco5ekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN+z8LtqavbifzqwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df.boxplot(column='height(cm)')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"I have drawn the boxplot of the height(cm) column in the above cell. It confirms that there is a value of 0.0 in the \n",
"height(cm) column. It is not possible as height cannot be 0. So, we need to take care of that."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADrpJREFUeJzt3X+MZWddx/H3x10ELA2ULIzbH7BESpyySMWxwbDoTBbLL00hSNIlgQqjCwmtBiW6OBpqcEyjkib+arIwtTWGIeWPQoWKlDpDWSnCVktZOiAbKLC0aUFI08Fau8vXP+ZUZqfT+XXn7sz0eb+Sydz73HPPeXYzec/Jc8+9k6pCkvT492MbPQFJ0qlh8CWpEQZfkhph8CWpEQZfkhph8CWpEQZfkhph8CWpEQZfkhqxfaMnMN+OHTtq165dGz0NaVE/+MEPOO200zZ6GtKj3Hbbbd+tqmcst92mCv6uXbs4fPjwRk9DWtT09DTDw8MbPQ3pUZJ8YyXbuaQjSY0w+JLUCIMvSY0w+JLUCIMvSY0w+NIyJicn2b17N3v37mX37t1MTk5u9JSkNdlUl2VKm83k5CRjY2NMTExw4sQJtm3bxujoKAD79u3b4NlJq+MZvrSE8fFxJiYmGBkZYfv27YyMjDAxMcH4+PhGT01aNYMvLWFmZoY9e/acNLZnzx5mZmY2aEbS2hl8aQmDg4McOnTopLFDhw4xODi4QTOS1s7gS0sYGxtjdHSUqakpjh8/ztTUFKOjo4yNjW301KRV80VbaQmPvDB72WWXMTMzw+DgIOPj475gqy0pVbXRc/h/Q0ND5YenabPyw9O0WSW5raqGltvOJR1JasSKg5/k6iT3JTkyb+zyJN9Ocnv39ap5j70rydEkX0ny8vWeuCRpdVZzhn8N8IpFxq+sqvO7rxsBkpwHXAw8v3vO3ybZ1utkJUlrt+LgV9UtwPdWuPlFwAer6qGq+jpwFLhgDfOTJK2T9bhK59IkbwIOA79bVd8HzgI+O2+bY93YoyTZD+wHGBgYYHp6eh2mJK2/2dlZfz61pfUa/KuA9wDVfX8v8BYgi2y76OVAVXUQOAhzV+l4FYQ2K6/S0VbX01U6VXVvVZ2oqh8C7+NHyzbHgHPmbXo2cHcvx5Ik9aan4CfZOe/ua4FHruC5Abg4yROTPAc4F/hcL8eSJPVmxUs6SSaBYWBHkmPAu4HhJOczt1xzF/BWgKr6UpLrgDuB48Dbq+rE+k5dkrQaKw5+VS32XvKJJbYfB/wMWUnaJHynrSQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1YsXBT3J1kvuSHJk39vQkNyX5avf9jG48Sf4yydEkdyR5UT8mL0laudWc4V8DvGLB2AHg5qo6F7i5uw/wSuDc7ms/cFVv05Qk9WrFwa+qW4DvLRi+CLi2u30t8Jp5439fcz4LPC3Jzl4nK0lau17X8Aeq6h6A7vszu/GzgG/N2+5YNyZJ2iDb+7TfLDJWi26Y7Gdu2YeBgQGmp6f7NCWpN7Ozs/58akvrNfj3JtlZVfd0Szb3dePHgHPmbXc2cPdiO6iqg8BBgKGhoRoeHu5xSlJ/TE9P48+ntrJel3RuAC7pbl8CfGTe+Ju6q3VeDNz/yNKPJGljrPgMP8kkMAzsSHIMeDdwBXBdklHgm8Dru81vBF4FHAX+G3jzOs5ZkrQGKw5+Ve17jIf2LrJtAW9f66QkSevPd9pKUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1wuBLUiMMviQ1ol8fniZtaslin++3/ubegyhtDp7hq0lVteqvZ//+R1f9HGkzMfiS1AiDL0mNMPiS1AiDL0mNMPiS1Agvy9SW98I//gT3P/jwKTnWrgMf6+v+n/rkJ/CFd1/Y12OoXQZfW979Dz7MXVe8uu/HORV/07bfv1DUNpd0JKkRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGuHHI2vLO33wAC+49sCpOdi1/d396YMA/f+oZ7XJ4GvLe2DmCj8PX1oBl3QkqRGe4etx4ZSdGX+8/3/iUOoXg68tby3LOUn6MJNHq6pTchxpJdZlSSfJXUm+mOT2JIe7sacnuSnJV7vvZ6zHsaT1UFWr/pqamlr1c6TNZD3X8Eeq6vyqGuruHwBurqpzgZu7+5KkDdLPF20v4kcXsV0LvKaPx5IkLWO9gl/AJ5LclmR/NzZQVfcAdN+fuU7HkiStwXq9aPuSqro7yTOBm5J8eaVP7H5B7AcYGBhgenp6naYkra/Z2Vl/PrWlrUvwq+ru7vt9Sa4HLgDuTbKzqu5JshO47zGeexA4CDA0NFT9fmOLtFan4o1XUj/1vKST5LQkpz9yG7gQOALcAFzSbXYJ8JFejyVJWrv1OMMfAK7vrmveDnygqj6e5PPAdUlGgW8Cr1+HY0mS1qjn4FfV14AXLjL+X8DeXvcvSVoffpaOJDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSIwy+JDXC4EtSI/oe/CSvSPKVJEeTHOj38SRJi+tr8JNsA/4GeCVwHrAvyXn9PKYkaXH9PsO/ADhaVV+rqv8FPghc1OdjSpIW0e/gnwV8a979Y92YJOkU297n/WeRsTppg2Q/sB9gYGCA6enpPk9JWpvZ2Vl/PrWl9Tv4x4Bz5t0/G7h7/gZVdRA4CDA0NFTDw8N9npK0NtPT0/jzqa2s30s6nwfOTfKcJD8OXAzc0OdjSpIW0dcz/Ko6nuRS4J+BbcDVVfWlfh5TkrS4fi/pUFU3Ajf2+ziSpKX5TltJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RG9BT8JJcn+XaS27uvV8177F1Jjib5SpKX9z5VSVIvtq/DPq6sqr+YP5DkPOBi4PnAmcAnkzyvqk6sw/EkSWvQryWdi4APVtVDVfV14ChwQZ+OJUlagfUI/qVJ7khydZIzurGzgG/N2+ZYNyZJ2iDLLukk+STwk4s8NAZcBbwHqO77e4G3AFlk+3qM/e8H9gMMDAwwPT29knlLp9zs7Kw/n9rSlg1+Vb1sJTtK8j7go93dY8A58x4+G7j7MfZ/EDgIMDQ0VMPDwys5nHTKTU9P48+ntrJer9LZOe/ua4Ej3e0bgIuTPDHJc4Bzgc/1cixJUm96vUrnz5Kcz9xyzV3AWwGq6ktJrgPuBI4Db/cKHUnaWD0Fv6reuMRj48B4L/uXJK0f32krSY0w+NIyJicn2b17N3v37mX37t1MTk5u9JSkNVmPd9pKj1uTk5OMjY0xMTHBiRMn2LZtG6OjowDs27dvg2cnrY5n+NISxsfHmZiYYGRkhO3btzMyMsLExATj4748pa3H4EtLmJmZYc+ePSeN7dmzh5mZmQ2akbR2Bl9awuDgIIcOHTpp7NChQwwODm7QjKS1M/jSEsbGxhgdHWVqaorjx48zNTXF6OgoY2NjGz01adV80VZawiMvzF522WXMzMwwODjI+Pi4L9hqS0rVop9ptiGGhobq8OHDGz0NaVF+lo42qyS3VdXQctu5pCNJjTD4ktQIgy9JjTD4ktQIgy9JjdhUV+kk+Q7wjY2eh/QYdgDf3ehJSIt4dlU9Y7mNNlXwpc0syeGVXPombVYu6UhSIwy+JDXC4Esrd3CjJyD1wjV8SWqEZ/iS1AiDryYleX+S85bZ5pokv7bI+K4kb1gw9rNJ3t/dvjzJO1cxlxckuWal20trZfDVpKr6jaq6c41P3wW8YcHYHwB/tca5fBE4O8mz1jgfaUUMvra0JL+X5Le621cm+Zfu9t4k/5DkwiS3Jvn3JB9K8pTu8ekkQ93t0ST/2Y29L8lfzzvELyb5TJKvzTvbvwJ4aZLbk7wjyenAz1TVFxaZ328m+ackT07y80nu6Obz50mOzNv0H4GL1/9/SPoRg6+t7hbgpd3tIeApSZ4A7AG+CPwh8LKqehFwGPid+U9OcibwR8CLgV8GfnrB/nd2+/oV5kIPcAD4dFWdX1VXdsc9suB5JLkU+FXgNVX1IPB3wNuq6heAEws2Pzzv3yH1hcHXVncb8HPdWfZDwK3MBfilwIPAecC/JrkduAR49oLnXwB8qqq+V1UPAx9a8PiHq+qH3fLPwGPMYSfwnQVjbwReCbyuqh5K8jTg9Kr6TPf4BxZsfx9w5vL/XGnt/BOH2tKq6uEkdwFvBj4D3AGMAD8FfB24qaqW+nuEWeYQD61g2weBJy0YOwKcD5zdzWO54zyp24/UN57h6/HgFuCd3fdPA28Dbgc+C7wkyXMBkvxEkucteO7ngF9KckaS7cDrVnC8B4DT592fAZ67YJv/AN4K3JDkzKr6PvBAkhd3jy9cr38eiywLSevJ4Ovx4NPMLavcWlX3Av/D3Br7d4BfByaT3MHcL4CT1uir6tvAnwL/BnwSuBO4f5nj3QEcT/KFJO+oqi8DT+2Wlebv+xBzv4g+lmQHMAocTHIrc2f8848zAnxs1f9yaRV8p62al+QpVTXbneFfD1xdVdevch/vAB6oqvcvd5zu9gFgZ1X9dpInAp8C9lTV8bX/S6SleYYvweXdi7pHmFtv//Aa9nEVJ6/3L+bV3aWcR5h7UflPuvFnAQeMvfrNM3xJaoRn+JLUCIMvSY0w+JLUCIMvSY0w+JLUCIMvSY34P4zFA/GN3r0aAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df.boxplot(column='weight(kg)')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"I have drawn the boxplot of the weight(kg) column in the above cell. The above boxplot confirms our findings that there is a negative value of -60 and a very high value of 160 in the weight(kg) column. \n",
"\n",
"Both are invalid values. Weight cannot be negative and also weight cannot be so high. So, we need to take care of these invalid values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Scatter plot\n",
"\n",
"\n",
"Scatter plot help us to explore relationship between two numeric variables. It help us to identify potentially bad data.\n",
"\n",
"We can draw a scatter plot of height(cm) and weight(kg) column as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGHFJREFUeJzt3X2QXXWd5/H3B0IwQR5EWkXANDr4iE9M4+Bj+QAirorrrjNk3RXHnmLdcVxnLGfEcWucrbJ2fVh1tWbUihNQtzSKLiJV4wjIqsyuojTKQzAiGQgaidLq2LomSxLy3T/uaby0p7tvHu5Dp9+vqq577u+ec+43pzv3c8/vnPM7qSokSZrrkGEXIEkaTQaEJKmVASFJamVASJJaGRCSpFYGhCSplQEhSWplQEiSWhkQkqRWK4ZdwP447rjjanx8fNhlSNKScv311/+0qsYWm29JB8T4+DhTU1PDLkOSlpQkd/Yyn11MkqRWBoQkqZUBIUlqZUBIkloZEJKkVn0LiCQXJbk7ycY57W9IcmuSW5K8u6v9rUk2N6+d3a+6JKlfpqe3c91125ie3j7sUg6Ifp7m+jHgb4BPzDYkeR5wLvCkqronyUOa9scD5wFPAB4OfDnJo6vq3j7WJ0kHzIYNm5icvIKVKw9h5849rF9/NmvXPm7YZe2Xvu1BVNU1wM/nNP8H4J1VdU8zz91N+7nAp6vqnqq6A9gMPK1ftUnSgTQ9vZ3JySvYsWM3MzM72bFjN5OTVyz5PYlBH4N4NPDsJN9M8rUkpzftJwA/7Jpva9P2W5JckGQqydT09HSfy5WkxW3ZMsPKlff/OD3ssEPYsmVmSBUdGIMOiBXAg4AzgD8HLkkSIC3zVtsKqmpdVU1U1cTY2KJXiktS342PH83OnXvu17Zr1x7Gx48eUkUHxqADYitwaXV8C9gDHNe0n9Q134nAXQOuTZL2ydjYatavP5tVq1Zw1FErWbVqBevXn83Y2Ophl7ZfBj0W02XA84GvJnk0sBL4KXA58Kkk76NzkPoU4FsDrk2S9tnatY/jzDPXsGXLDOPjRy/5cIA+BkSSDcBzgeOSbAXeDlwEXNSc+roTOL+qCrglySXAd4HdwOs9g0nSUjM2tvqgCIZZ6Xw+L00TExPlaK6StHeSXF9VE4vN55XUkqRWBoQkqZUBIUlqZUBIkloZEJKkVgaEJKmVASFJamVASJJaGRCSpFYGhCSplQEhSWplQEiSWhkQkqRWBoQkqZUBIUlqZUBIkloZEJKkVgaEJKmVASFJamVASJJaGRCSpFYGhCSplQEhSWplQEiSWhkQkqRWBoQkqZUBIUlqZUBIklr1LSCSXJTk7iQbW157c5JKclzzPEk+mGRzkpuSnNavuiRJvennHsTHgBfNbUxyEnAW8IOu5nOAU5qfC4AP97EuSVIP+hYQVXUN8POWl94P/AVQXW3nAp+ojmuBY5Ic36/aJEmLG+gxiCQvA35UVTfOeekE4Iddz7c2bZKkIVkxqDdKshp4G/DCtpdb2qqljSQX0OmG4hGPeMQBq0+SdH+D3IN4FHAycGOSLcCJwLeTPIzOHsNJXfOeCNzVtpKqWldVE1U1MTY21ueSJWn5GlhAVNXNVfWQqhqvqnE6oXBaVf0YuBx4dXM20xnATFVtG1RtkqTf1s/TXDcA3wAek2RrkskFZv8icDuwGfgo8Mf9qkuS1Ju+HYOoqrWLvD7eNV3A6/tViyRp73kltSSplQEhSWplQEiSWhkQkqRWBoQkqZUBIUlqZUBIkloZEJKkVgaEJKmVASFJamVASJJaGRCSpFYGhCSplQEhSWplQEiSWhkQkqRWBoQkqZUBIUlqZUBIkloZEJKkVgaEJKmVASFJamVASJJaGRCSpFYGhCSplQEhSWplQEiSWhkQkqRWBoQkqVXfAiLJRUnuTrKxq+09Sb6X5KYkn09yTNdrb02yOcmtSc7uV12SpN70cw/iY8CL5rRdBZxaVU8Cvg+8FSDJ44HzgCc0y3woyaF9rE2StIi+BURVXQP8fE7blVW1u3l6LXBiM30u8Omquqeq7gA2A0/rV22SpMWtWGyGJCfS+Xb/bODhwA5gI/D3wD9U1Z59fO/XAp9ppk+gExiztjZtkqQhWXAPIsnFwEXATuBdwFrgj4Ev0+kK+t9JnrO3b5rkbcBu4JOzTS2z1TzLXpBkKsnU9PT03r61JKlHi+1BvLeqNra0bwQuTbISeMTevGGS84GXAC+oqtkQ2Aqc1DXbicBdbctX1TpgHcDExERriEiS9t+CexDzhEP36zuranOvb5bkRcBbgJdV1fauly4HzktyeJKTgVOAb/W6XknSgbfoMQiAJDfz210+M8AU8I6q+lnLMhuA5wLHJdkKvJ3OWUuHA1clAbi2ql5XVbckuQT4Lp2up9dX1b379k+SJB0I+U0vzwIzJe8G7gU+1TSd1zz+EnhWVb20P+UtbGJioqampobx1pK0ZCW5vqomFpuvpz0I4JlV9cyu5zcn+T9V9cwk/3bfSpQkjbJer4N4YJLfm32S5GnAA5unu9sXkSQtZb3uQfwRcFGS2VD4FfBHSY4A/mtfKpMkDVWvAXFTVT0xydF0jlv8IsmxVfVr4JI+1idJGpJeu5guTbKiqmaacHgYnXGVJEkHqV4D4jLgc0kOTTIOXEkz0J4k6eDUUxdTVX20uWr6MmAc+PdV9fV+FiZJGq4FAyLJm7qf0hkO4wbgjCRnVNX7+lmcJGl4FtuDOHLO88/P0y5JOsgsGBBV9Z8HVYgkabQsNtz3uiSnzvPaEUlem+RV/SlNkjRMi3UxfQj4qyRPpDPE9zTwADqjrR5F514Rn5x/cUnSUrVYF9MNwO83V1BPAMfTuaPcpqq6dQD1SZKGpNcrqSer6gPdDUneOLdNknTw6PVCufNb2l5zAOuQJI2Yxa6DWAv8G+DkJJd3vXQk8Fs3CZIkHTwW62L6OrANOA54b1f7r4Cb+lWUJGn4FjtIfSdwJ/D0wZQjSRoVPR2DSPKKJLclmUnyyyS/SvLLfhcnSRqeXs9iejfw0qra1M9iJEmjo9ezmH5iOEjS8rLYWUyvaCanknyGznDf98y+XlWX9rE2SdIQLdbF9NKu6e3AC7ueF2BASNJBarGzmP5wUIVIkkZLTwepk3ywpXkGmKqqLxzYkiRJo6DXg9QPAJ4C3Nb8PAk4FphM8t/7VJskaYh6Pc31d4DnV9VugCQfBq4EzgJu7lNtkqQh6nUP4gTgiK7nRwAPr6p76TqrSZJ08NibC+VuSPJVIMBzgP+S5Ajgy32qTZI0RD3tQVTVeuAZdK6DuAx4VlX9XVX9uqr+vG2ZJBcluTvJxq62Y5Nc1QzbcVWSBzXtSfLBJJuT3JTktP3/p0mS9sdi96R+bPN4Gp27yf0Q+AHwsB4+xD8GvGhO24XA1VV1CnB18xzgHDq3MT0FuAD4cO//BElSPyzWxfQmOh/Y7215rYDnz7dgVV2TZHxO87nAc5vpjwNfBd7StH+iqgq4NskxSY6vqm2L1CdJ6pPFLpS7oHl83gF6v4fOfuhX1bYkD2naT6CzdzJra9NmQEjSkPQ63PfqJP8pybrm+SlJXnIA60hLW81TywVJppJMTU9PH8ASJEndej3N9WJgJ50D1dD5hv+OfXi/nyQ5HqB5vLtrfSd1zXcicFfbCqpqXVVNVNXE2NjYPpQgSepFrwHxqKp6N7ALoKp20P6tfzGXA+c30+cDX+hqf3VzNtMZwIzHHyRpuHq9DmJnklU03T5JHsUiF8gl2UDngPRxSbYCbwfeCVySZJLO2VCvbGb/IvBiYDOdUWMdJFCShqzXgHg78CXgpCSfBJ4JvGahBapq7TwvvaBl3gJe32MtkqQB6DUgXg38PfA54HbgjVX1075VJUkaul4D4mLgWXQG53sknWE3rqmqD/StMknSUPUUEFX1v5J8DTgdeB7wOuAJgAEhSQepXm8YdDWdEVy/AfwjcHpV3b3wUpKkpazX01xvonMdxKl0bhZ0anNWkyTpINVrF9OfASR5IJ1TUC8GHgYc3r/SJEnD1GsX058AzwZ+F7gTuIhOV5Mk6SDV61lMq4D3AdfP3nZUknRw67WL6T39LkSSNFp6PUgtSVpmDAhJUisDQpLUyoCQJLUyICRJrQwISVIrA0KS1MqAkCS1MiAkSa0MCElSKwNCkpaY6entXHfdNqant/f1fQwISVpCNmzYxJo16zjrrM+yZs06NmzY1Lf3MiAkaYmYnt7O5OQV7Nixm5mZnezYsZvJySv6tidhQEjSErFlywwrV97/Y/uwww5hy5aZvryfASFJS8T4+NHs3Lnnfm27du1hfPzovryfASFJS8TY2GrWrz+bVatWcNRRK1m1agXr15/N2Njqvrxfr3eUkySNgLVrH8eZZ65hy5YZxseP7ls4gAEhSUvO2NjqvgbDLLuYJEmtDAhJUquhBESSP0tyS5KNSTYkeUCSk5N8M8ltST6TZOUwapMkdQw8IJKcAPxHYKKqTgUOBc4D3gW8v6pOAf4ZmBx0bZKk3xhWF9MKYFWSFcBqYBvwfOBzzesfB14+pNokSQwhIKrqR8B/A35AJxhmgOuBX1TV7ma2rcAJg65NkvQbw+hiehBwLnAy8HDgCOCclllrnuUvSDKVZGp6erp/hUrSMjeMLqYzgTuqarqqdgGXAs8Ajmm6nABOBO5qW7iq1lXVRFVNjI2NDaZiSVqGhhEQPwDOSLI6SYAXAN8FvgL862ae84EvDKE2SVJjGMcgvknnYPS3gZubGtYBbwHelGQz8GBg/aBrkyT9xlCG2qiqtwNvn9N8O/C0IZQjSWrhldSSNEIGdTvRXhgQkjQiBnk70V4YEJI0AgZ9O9FeGBCSNAIGfTvRXhgQkjQCBn070V4YEJI0AgZ9O9FeeEc5SRoRg7ydaC8MCEkaIYO6nWgv7GKSJLUyICRJrQwISVIrA0KS1MqAkCS1MiAkSa0MCElSKwNCktTKgJAktTIgJEmtDAhJUisDQpLUyoCQJLUyICRJrQwISVIrA0KS1MqAkCS1MiAkSa0MCElSKwNCktTKgJCkBUxPb+e667YxPb192KUM3FACIskxST6X5HtJNiV5epJjk1yV5Lbm8UHDqE2SZm3YsIk1a9Zx1lmfZc2adWzYsGnYJQ3UsPYgPgB8qaoeCzwZ2ARcCFxdVacAVzfPJWkopqe3Mzl5BTt27GZmZic7duxmcvKKZbUnMfCASHIU8BxgPUBV7ayqXwDnAh9vZvs48PJB1yZJs7ZsmWHlyvt/RB522CFs2TIzpIoGbxh7EI8EpoGLk3wnyd8lOQJ4aFVtA2geHzKE2iQJgPHxo9m5c8/92nbt2sP4+NE9LX8wHLsYRkCsAE4DPlxVTwV+zV50JyW5IMlUkqnp6el+1ShpmRsbW8369WezatUKjjpqJatWrWD9+rMZG1u96LIHy7GLVNVg3zB5GHBtVY03z59NJyB+B3huVW1Lcjzw1ap6zELrmpiYqKmpqX6XLGkZm57ezpYtM4yPH91TOExPb2fNmnXs2LH7vrZVq1Zw550X9LT8ICS5vqomFptv4HsQVfVj4IdJZj/8XwB8F7gcOL9pOx/4wqBrk6S5xsZWc/rpx/f84X4wHbtYMaT3fQPwySQrgduBP6QTVpckmQR+ALxySLVJ0m/pdU9if49djJKhBERV3QC07d68YNC1SNJiNmzYxOTkFaxceQg7d+5h/fqzWbv2ca3zzh67mJy8gsMOO4Rdu/b0fOxi1Az8GMSBtK/HIPa2T1HS8rWvxxRG+XOm12MQw+piGpq9+SYgSbPHFHbs+E3b7DGFhT74x8ZWj1ww7K1lNRaTV0ZK2lsH0zGFvbWsAuJgOrtA0mDsz/UQS92y6mJazt8EJO27tWsfx5lnrhnZYwr9sqz2IJbzNwFJ+2dvr4c4GCyrPQhYvt8EJGlvLbuAgIPj7AJJ6rdl1cUkSeqdASFJamVASJJaGRCSpFYGhCSp1ZIerC/JNHDnPi5+HPDTA1hOvy2lepdSrbC06rXW/llK9e5vrWuqamyxmZZ0QOyPJFO9jGY4KpZSvUupVlha9Vpr/yylegdVq11MkqRWBoQkqdVyDoh1wy5gLy2lepdSrbC06rXW/llK9Q6k1mV7DEKStLDlvAchSVrAsgyIJC9KcmuSzUkuHHY93ZKclOQrSTYluSXJG5v2v07yoyQ3ND8vHnats5JsSXJzU9dU03ZskquS3NY8PmgE6nxM1/a7Ickvk/zpKG3bJBcluTvJxq621m2Zjg82f8c3JTltBGp9T5LvNfV8PskxTft4kh1d2/gjI1DrvL/3JG9ttuutSc4eZK0L1PuZrlq3JLmhae/ftq2qZfUDHAr8E/BIYCVwI/D4YdfVVd/xwGnN9JHA94HHA38NvHnY9c1T8xbguDlt7wYubKYvBN417Dpb/g5+DKwZpW0LPAc4Ddi42LYEXgz8AxDgDOCbI1DrC4EVzfS7umod755vRLZr6++9+f92I3A4cHLzeXHosOud8/p7gb/q97ZdjnsQTwM2V9XtVbUT+DRw7pBruk9VbauqbzfTvwI2AScMt6p9ci7w8Wb648DLh1hLmxcA/1RV+3qhZV9U1TXAz+c0z7ctzwU+UR3XAsckOX4wlbbXWlVXVtXu5um1wImDqmch82zX+ZwLfLqq7qmqO4DNdD43BmahepME+H1gQ7/rWI4BcQLww67nWxnRD+Ak48BTgW82TX/S7LpfNApdNl0KuDLJ9UkuaNoeWlXboBN6wEOGVl2787j/f7BR3bYw/7Yc9b/l19LZw5l1cpLvJPlakmcPq6g52n7vo75dnw38pKpu62rry7ZdjgGRlraRO5UryQOB/wn8aVX9Evgw8CjgKcA2OruYo+KZVXUacA7w+iTPGXZBC0myEngZ8NmmaZS37UJG9m85yduA3cAnm6ZtwCOq6qnAm4BPJTlqWPU15vu9j+x2bazl/l9u+rZtl2NAbAVO6np+InDXkGppleQwOuHwyaq6FKCqflJV91bVHuCjDHiXdyFVdVfzeDfweTq1/WS2u6N5vHt4Ff6Wc4BvV9VPYLS3bWO+bTmSf8tJzgdeAryqmk7yprvmZ8309XT69R89vCoX/L2P5HYFSLICeAXwmdm2fm7b5RgQ1wGnJDm5+SZ5HnD5kGu6T9O/uB7YVFXv62rv7lv+l8DGucsOQ5Ijkhw5O03nIOVGOtv0/Ga284EvDKfCVvf7Bjaq27bLfNvycuDVzdlMZwAzs11Rw5LkRcBbgJdV1fau9rEkhzbTjwROAW4fTpX31TTf7/1y4Lwkhyc5mU6t3xp0ffM4E/heVW2dbejrth3kkflR+aFz9sf36STt24Zdz5zankVnd/Ym4Ibm58XA/wBubtovB44fdq1NvY+kc8bHjcAts9sTeDBwNXBb83jssGtt6loN/Aw4uqttZLYtneDaBuyi8012cr5tSacr5G+bv+ObgYkRqHUznf772b/djzTz/qvm7+NG4NvAS0eg1nl/78Dbmu16K3DOKPwdNO0fA143Z96+bVuvpJYktVqOXUySpB4YEJKkVgaEJKmVASFJamVASJJaGRBaVpqRL3u+ziHJ65K8epF5XpPkb+Z57S/nPF/VDIdwaK81zLPelUmuaS6ckvrCgJAWUFUfqapP7Mcq/nLO89cCl1bVvfuxTqoz0OTVwB/sz3qkhRgQWo4OTfLRdO63cWXzrf5RSb7UDDj4j0keC/fdM+DNzfTpzcBu32jue9C9J/LwZvnbkry7mf+dwKpmjP7ZMYleRddV5Un+Ip17adzYzE+SryZ5f7OHsKl530ubdb+j6z0va9Yn9YUBoeXoFOBvq+oJwC/oXIm6DnhDVf0u8GbgQy3LXUznKtanA3P3AJ5C59v8E4E/SHJSVV0I7Kiqp1TVq5qhXR5ZVVsAkpxDZ+ju36uqJ9O578OsnVX1HOAjdALl9cCpwGuSPLiZZyNw+v5sCGkh9l9qObqjqm5opq+nc8OVZwCf7QyFBXRuFnOfdO6MdmRVfb1p+hSdAelmXV1VM82836VzI6LuIaMBjqMTSLPOBC6uZsyiquoe/392fLCbgVuqGWMpye10BpL7WVXdm2RnkiOrc+8Q6YAyILQc3dM1fS/wUOAXVfWUBZZpGwJ6oXW2/d/aATxgzjrnG+tmdn175qx7z5x1Hw78v0Vqk/aJXUwS/BK4I8kr4b57PT+5e4aq+mfgV82oqdAZBbgXu5rh22fXcWiS2ZC4EnhtktXN+x67N0U3XU3TVbVrb5aTemVASB2vAiaTzI5K23Yb2klgXZJv0Pn2P9PDetcBN3UdpL6Szoi9VNWX6HQlTaVzA/o372XNzwO+uJfLSD1zNFepR0keWFX/t5m+kM7w0G/cy3U8FXhTVf27A1DPpcBbq+rW/V2X1MZjEFLv/kWSt9L5f3Mn8Jq9XUFVfSfJV5Icuj/XQjRnRF1mOKif3IOQJLXyGIQkqZUBIUlqZUBIkloZEJKkVgaEJKmVASFJavX/ATtRAr/O9YazAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df.plot(kind='scatter',x='height(cm)', y='weight(kg)', c='DarkBlue')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Interpretation**\n",
"\n",
"The above scatter plot does not depict anything due to lack of data values. We need more data to identify relationship or pattern between data values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Findings of EDA and Visual EDA\n",
"\n",
"\n",
"We can summarize the findings of EDA and visual EDA as follows:-\n",
"\n",
"\n",
"\n",
"1.\tThe dataset has 10 rows and 10 columns.\n",
"\n",
"\n",
"\n",
"2.\tThe age and sex columns are combined together with an underscore. There should be two separate columns of age and sex.\n",
"\n",
"\n",
"\n",
"3.\tAll the invalid values (the values coded as \"xx\") and missing values in height (cm), weight (kg), spend_A, spend_B and spend_C columns are coded as \"NaN\". The use of the keyword errors='coerce' in pd.to_numeric() method enable us to convert all the invalid values into NaN.\n",
"\n",
"\n",
"\n",
"4.\tThe data types of columns height (cm), weight(kg) and spend_C columns are converted into float64.\n",
"\n",
"\n",
"\n",
"5.\tIn the height (cm) column, there is a value of 0.0. It is not possible as height cannot be 0. So, we need to resolve it.\n",
"\n",
"\n",
"\n",
"6.\tIn the weight (kg) column, there is a negative value of -60 and a very high value of 160. Both are invalid values. Hence, we need to solve the issue.\n",
"\n",
"\n",
"\n",
"7.\tThe three columns spend_A, spend_B and spend_C denote spending at three supermarkets A,B and C. These columns must contain positive real numbers. The missing values in these columns denote nothing spend in that market. We need to handle these missing values properly.\n",
"\n",
"\n",
"\n",
"8.\tIn the spend_B column, there is a negative value -100. The amount spent cannot be negative. So, we need to take care of that.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Split ‘age_sex’ column \n",
"\n",
"\n",
"We should split the 'age_sex' column into two separate columns. \n",
"\n",
"We can do this using the **df.str.split()** function as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"df[['age','sex']] = df.age_sex.str.split(\"_\", expand = True)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age_sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" <th>sex</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37_M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>37</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38_M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>38</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35_M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>35</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40_M</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>40</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39_M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>39</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age_sex section height(cm) weight(kg) age spend_A \\\n",
"0 Rahul Pandey 37_M B 175.4 85.6 37 1500.0 \n",
"1 Rakesh Kumar 38_M B NaN NaN 38 NaN \n",
"2 Nitesh Mondol 35_M B 165.1 NaN 35 500.0 \n",
"3 Arvind Dwivedi 40_M A 0.0 55.6 40 300.0 \n",
"4 Nikhil Shikharwar 39_M B 160.8 160.0 39 200.0 \n",
"\n",
" spend_B spend_C sex \n",
"0 200.0 300.0 M \n",
"1 1000.0 NaN M \n",
"2 -100.0 200.0 M \n",
"3 300.0 NaN M \n",
"4 NaN 300.0 M "
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that now we have two separate columns for age and sex.\n",
"\n",
"Now, there is no need for the age_sex column. So, we should drop that column.\n",
"\n",
"We can drop 'age_sex' column using the **df.drop()** method as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"df.drop(['age_sex'], axis=1, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>age</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" <th>sex</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>37</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>38</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>35</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>40</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>39</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>-60.0</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>35</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>34</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>36</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>38</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" <td>M</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname section height(cm) weight(kg) age spend_A \\\n",
"0 Rahul Pandey B 175.4 85.6 37 1500.0 \n",
"1 Rakesh Kumar B NaN NaN 38 NaN \n",
"2 Nitesh Mondol B 165.1 NaN 35 500.0 \n",
"3 Arvind Dwivedi A 0.0 55.6 40 300.0 \n",
"4 Nikhil Shikharwar B 160.8 160.0 39 200.0 \n",
"5 Ashutosh Merothiya A NaN -60.0 36 100.0 \n",
"6 Shubhro Das C 170.3 75.5 35 NaN \n",
"7 Suvendu Das C 155.1 56.8 34 1000.0 \n",
"8 Swadesh Kumar C 154.2 60.6 36 100.0 \n",
"9 Arun Vishwakarma A 162.1 58.7 38 600.0 \n",
"\n",
" spend_B spend_C sex \n",
"0 200.0 300.0 M \n",
"1 1000.0 NaN M \n",
"2 -100.0 200.0 M \n",
"3 300.0 NaN M \n",
"4 NaN 300.0 M \n",
"5 400.0 NaN M \n",
"6 500.0 500.0 M \n",
"7 100.0 NaN M \n",
"8 200.0 300.0 M \n",
"9 200.0 200.0 M "
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the 'age_sex' column has been dropped from the dataframe.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Reorder the column labels\n",
"\n",
"Before that, we should reorder the columns for more pleasing visual appearance.\n",
"\n",
"We can do it as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"df = df[['fname','lname','age','sex','section','height(cm)','weight(kg)','spend_A','spend_B','spend_C']]"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>-60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.4 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B NaN NaN NaN \n",
"2 Nitesh Mondol 35 M B 165.1 NaN 500.0 \n",
"3 Arvind Dwivedi 40 M A 0.0 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.8 160.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A NaN -60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.3 75.5 NaN \n",
"7 Suvendu Das 34 M C 155.1 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.2 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.1 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 -100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 \n",
"5 400.0 NaN \n",
"6 500.0 500.0 \n",
"7 100.0 NaN \n",
"8 200.0 300.0 \n",
"9 200.0 200.0 "
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we should move further to deal with missing and negative numerical values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Dealing with negative numerical values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have seen that, in the weight(kg) column, there is a negative value of -60. It is invalid value because weight cannot be negative. There is a high probability that weight is 60 kg and it is mistyped as -60. So, I will replace the negative value \n",
"of -60 with positive value of 60.\n",
"\n",
"We can do it as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"# Suppress settingwithcopywarning\n",
"\n",
"pd.set_option('mode.chained_assignment', None)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"df['weight(kg)'].replace(-60, 60, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>-100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.4 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B NaN NaN NaN \n",
"2 Nitesh Mondol 35 M B 165.1 NaN 500.0 \n",
"3 Arvind Dwivedi 40 M A 0.0 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.8 160.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A NaN 60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.3 75.5 NaN \n",
"7 Suvendu Das 34 M C 155.1 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.2 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.1 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 -100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 \n",
"5 400.0 NaN \n",
"6 500.0 500.0 \n",
"7 100.0 NaN \n",
"8 200.0 300.0 \n",
"9 200.0 200.0 "
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the negative value of -60 is replaced with positive value of 60."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, in the spend_B column, there is a negative value -100. The amount spent cannot be negative. So, we need to replace this negative value of -100 with positive value of 100.\n",
"\n",
"We can do it as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"df['spend_B'].replace(-100,100, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.4</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.1</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>0.0</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.8</td>\n",
" <td>160.0</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.3</td>\n",
" <td>75.5</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.1</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.2</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.1</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.4 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B NaN NaN NaN \n",
"2 Nitesh Mondol 35 M B 165.1 NaN 500.0 \n",
"3 Arvind Dwivedi 40 M A 0.0 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.8 160.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A NaN 60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.3 75.5 NaN \n",
"7 Suvendu Das 34 M C 155.1 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.2 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.1 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 \n",
"5 400.0 NaN \n",
"6 500.0 500.0 \n",
"7 100.0 NaN \n",
"8 200.0 300.0 \n",
"9 200.0 200.0 "
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, we can see that the negative value of -100.0 in spend_B column is replaced with positive value of 100.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 13. Dealing with outliers\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the height(cm) column, there is a value of 0.0. It is not possible as height cannot be 0. So, we need to resolve it.\n",
"\n",
"I will replace the 0.0 value with the mean of the height(cm) column. It can be done as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"mean = df['height(cm)'].mean()"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"df['height(cm)'].replace(0.0, mean, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800</td>\n",
" <td>160.0</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300</td>\n",
" <td>75.5</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.400 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B NaN NaN NaN \n",
"2 Nitesh Mondol 35 M B 165.100 NaN 500.0 \n",
"3 Arvind Dwivedi 40 M A 142.875 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.800 160.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A NaN 60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.300 75.5 NaN \n",
"7 Suvendu Das 34 M C 155.100 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.200 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.100 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 \n",
"5 400.0 NaN \n",
"6 500.0 500.0 \n",
"7 100.0 NaN \n",
"8 200.0 300.0 \n",
"9 200.0 200.0 "
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the data value of 0.0 in height(cm) column is replaced by a proper height value."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the weight(kg) column, there is a very high absurd value of 160. It is not possible to have so much weight. Hence, it is invalid value. There is a high chance that the weight is 60 kg and it is mistakenly typed as 160 kg. So, I will replace the 160 \n",
"data value with 60.\n",
"\n",
"It can be done as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"df['weight(kg)'].replace(160.0, 60.0, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1000.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800</td>\n",
" <td>60.0</td>\n",
" <td>200.0</td>\n",
" <td>NaN</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>NaN</td>\n",
" <td>60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300</td>\n",
" <td>75.5</td>\n",
" <td>NaN</td>\n",
" <td>500.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.0</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.0</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.400 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B NaN NaN NaN \n",
"2 Nitesh Mondol 35 M B 165.100 NaN 500.0 \n",
"3 Arvind Dwivedi 40 M A 142.875 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.800 60.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A NaN 60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.300 75.5 NaN \n",
"7 Suvendu Das 34 M C 155.100 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.200 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.100 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.0 300.0 \n",
"1 1000.0 NaN \n",
"2 100.0 200.0 \n",
"3 300.0 NaN \n",
"4 NaN 300.0 \n",
"5 400.0 NaN \n",
"6 500.0 500.0 \n",
"7 100.0 NaN \n",
"8 200.0 300.0 \n",
"9 200.0 200.0 "
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the data value of 160.0 in the weight(kg) column is replaced by 60.0 and thus I handle the outliers. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 14.\tDealing with missing numerical values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following commands help us to deal with missing numerical values.\n",
"\n",
"\n",
"`df.isnull()`\n",
"\n",
"The above command checks whether each cell in a dataframe contains missing values or not. If the cell contains missing value, it returns True otherwise it returns False. \n",
"\n",
"\n",
"`df.isnull.sum()`\n",
"\n",
"The above command returns the total number of missing values in each column in the dataset.\n",
"\n",
"\n",
"\n",
"**isna()** and **notna()** functions to detect ‘NA’ values\n",
"\n",
"\n",
"Pandas provides isna() and notna() functions to detect ‘NA’ values. These are also methods on Series and DataFrame objects.\n",
"\n",
"\n",
"Examples of isna() and notna() commands\n",
"\n",
"\n",
"We can detect ‘NA’ values in the dataframe with the following command:-\n",
"\n",
"\n",
"`df.isna()`\n",
"\n",
"\n",
"\n",
"We can detect ‘NA’ values in a particular column in the dataframe\n",
"\n",
"\n",
"`pd.isna(df[‘col_name’])`\n",
"\n",
"\n",
"`df[‘col_name’].notna()`\n",
"\n",
"\n",
"So, we can check the number of missing values in each column in the dataset as follows:-\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"fname 0\n",
"lname 0\n",
"age 0\n",
"sex 0\n",
"section 0\n",
"height(cm) 2\n",
"weight(kg) 2\n",
"spend_A 2\n",
"spend_B 1\n",
"spend_C 4\n",
"dtype: int64"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are lots of missing values in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fill missing values with a test statistic\n",
"\n",
"\n",
"In this method, we fill the missing values with a test statistic like mean, median or mode of the particular feature the missing value belongs to. One can also specify a forward-fill or back-fill to propagate the next values backward or previous value forward.\n",
"\n",
"\n",
"\n",
"### Filling missing values with a test statistic like mean\n",
"\n",
"\n",
"`mean = df['col_name'].mean()`\n",
"\n",
"\n",
"`df['col_name'].fillna(value = median, inplace = True )`\n",
"\n",
"\n",
"\n",
"### We can also use replace() in place of fillna()\n",
"\n",
"\n",
"`df[‘col_name’].replace(to_replace = NaN, value = median, inplace = True)`\n",
"\n",
"\n",
"If we choose this method, then we should compute the median value on the training set and use it to fill the missing values in the training set. Then we should save the median value that we have computed. Later, we will replace missing values in the test set with the median value to evaluate the system.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can calculate the missing variables and use it to fill the missing values as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"mean_height = df['height(cm)'].mean()\n",
"\n",
"df['height(cm)'].fillna(mean_height, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"mean_weight = df['weight(kg)'].mean()\n",
"\n",
"df['weight(kg)'].fillna(mean_weight, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"mean_spend_A = df['spend_A'].mean()\n",
"\n",
"df['spend_A'].fillna(mean_spend_A, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"mean_spend_B = df['spend_B'].mean()\n",
"\n",
"df['spend_B'].fillna(mean_spend_B, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"mean_spend_C = df['spend_C'].mean()\n",
"\n",
"df['spend_C'].fillna(mean_spend_C, inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have calculated the mean variables and use them to fill the missing values."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"fname 0\n",
"lname 0\n",
"age 0\n",
"sex 0\n",
"section 0\n",
"height(cm) 0\n",
"weight(kg) 0\n",
"spend_A 0\n",
"spend_B 0\n",
"spend_C 0\n",
"dtype: int64"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Again check for missing values\n",
"\n",
"df.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are no missing values in the dataframe. \n",
"\n",
"We can confirm this with **ASSERT** statement as follwos:-"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 15. Check with ASSERT statement"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we can check for missing values programmatically. If we drop or fill missing values, we expect no missing values. \n",
"We can write an assert statement to verify this. So, we can use an assert statement to programmatically check that no \n",
"missing or unexpected ‘0’ value is present. This gives confidence that our code is running properly.\n",
"\n",
"\n",
"Assert statement will return nothing if the value being tested is true and will throw an AssertionError if the value is false.\n",
"\n",
"Asserts\n",
"\n",
"\n",
"•\tassert 1 == 1 (return Nothing if the value is True)\n",
"\n",
"\n",
"•\tassert 1 == 2 (return AssertionError if the value is False)\n",
"\n",
"\n",
"We can check with assert statement as follows:-\n"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"#assert that there are no missing values in the dataframe\n",
"\n",
"assert pd.notnull(df).all().all()\n"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"#assert all values are greater than 0\n",
"\n",
"assert (df >=0).all().all()\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The assert statement returns nothing. So, we can conclude that there are no missing values in the dataset and all the values \n",
"are greater than zero."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can confirm this by looking at the dataframe."
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>spend_A</th>\n",
" <th>spend_B</th>\n",
" <th>spend_C</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400000</td>\n",
" <td>85.6</td>\n",
" <td>1500.0</td>\n",
" <td>200.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.734375</td>\n",
" <td>64.1</td>\n",
" <td>537.5</td>\n",
" <td>1000.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100000</td>\n",
" <td>64.1</td>\n",
" <td>500.0</td>\n",
" <td>100.000000</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875000</td>\n",
" <td>55.6</td>\n",
" <td>300.0</td>\n",
" <td>300.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800000</td>\n",
" <td>60.0</td>\n",
" <td>200.0</td>\n",
" <td>333.333333</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>160.734375</td>\n",
" <td>60.0</td>\n",
" <td>100.0</td>\n",
" <td>400.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300000</td>\n",
" <td>75.5</td>\n",
" <td>537.5</td>\n",
" <td>500.000000</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100000</td>\n",
" <td>56.8</td>\n",
" <td>1000.0</td>\n",
" <td>100.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200000</td>\n",
" <td>60.6</td>\n",
" <td>100.0</td>\n",
" <td>200.000000</td>\n",
" <td>300.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100000</td>\n",
" <td>58.7</td>\n",
" <td>600.0</td>\n",
" <td>200.000000</td>\n",
" <td>200.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) spend_A \\\n",
"0 Rahul Pandey 37 M B 175.400000 85.6 1500.0 \n",
"1 Rakesh Kumar 38 M B 160.734375 64.1 537.5 \n",
"2 Nitesh Mondol 35 M B 165.100000 64.1 500.0 \n",
"3 Arvind Dwivedi 40 M A 142.875000 55.6 300.0 \n",
"4 Nikhil Shikharwar 39 M B 160.800000 60.0 200.0 \n",
"5 Ashutosh Merothiya 36 M A 160.734375 60.0 100.0 \n",
"6 Shubhro Das 35 M C 170.300000 75.5 537.5 \n",
"7 Suvendu Das 34 M C 155.100000 56.8 1000.0 \n",
"8 Swadesh Kumar 36 M C 154.200000 60.6 100.0 \n",
"9 Arun Vishwakarma 38 M A 162.100000 58.7 600.0 \n",
"\n",
" spend_B spend_C \n",
"0 200.000000 300.0 \n",
"1 1000.000000 300.0 \n",
"2 100.000000 200.0 \n",
"3 300.000000 300.0 \n",
"4 333.333333 300.0 \n",
"5 400.000000 300.0 \n",
"6 500.000000 500.0 \n",
"7 100.000000 300.0 \n",
"8 200.000000 300.0 \n",
"9 200.000000 200.0 "
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are no missing or negative values in the dataframe. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 16. Reshaping the data into tidy data format"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When we take a closer look at the dataframe, we can see that our dataframe is not in the tidy data format.\n",
"\n",
"The columns spend_A, spend_B and spend_C contain values of amount spent rather than variables. We should reorganize our \n",
"dataframe into tidy data format.\n",
"\n",
"We can convert it into the tidy data format using the **pd.melt()** function as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>fname</th>\n",
" <th>lname</th>\n",
" <th>age</th>\n",
" <th>sex</th>\n",
" <th>section</th>\n",
" <th>height(cm)</th>\n",
" <th>weight(kg)</th>\n",
" <th>expenditure</th>\n",
" <th>amount</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400000</td>\n",
" <td>85.6</td>\n",
" <td>spend_A</td>\n",
" <td>1500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.734375</td>\n",
" <td>64.1</td>\n",
" <td>spend_A</td>\n",
" <td>537.500000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100000</td>\n",
" <td>64.1</td>\n",
" <td>spend_A</td>\n",
" <td>500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875000</td>\n",
" <td>55.6</td>\n",
" <td>spend_A</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800000</td>\n",
" <td>60.0</td>\n",
" <td>spend_A</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>160.734375</td>\n",
" <td>60.0</td>\n",
" <td>spend_A</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300000</td>\n",
" <td>75.5</td>\n",
" <td>spend_A</td>\n",
" <td>537.500000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100000</td>\n",
" <td>56.8</td>\n",
" <td>spend_A</td>\n",
" <td>1000.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200000</td>\n",
" <td>60.6</td>\n",
" <td>spend_A</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100000</td>\n",
" <td>58.7</td>\n",
" <td>spend_A</td>\n",
" <td>600.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400000</td>\n",
" <td>85.6</td>\n",
" <td>spend_B</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.734375</td>\n",
" <td>64.1</td>\n",
" <td>spend_B</td>\n",
" <td>1000.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100000</td>\n",
" <td>64.1</td>\n",
" <td>spend_B</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875000</td>\n",
" <td>55.6</td>\n",
" <td>spend_B</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800000</td>\n",
" <td>60.0</td>\n",
" <td>spend_B</td>\n",
" <td>333.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>160.734375</td>\n",
" <td>60.0</td>\n",
" <td>spend_B</td>\n",
" <td>400.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300000</td>\n",
" <td>75.5</td>\n",
" <td>spend_B</td>\n",
" <td>500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100000</td>\n",
" <td>56.8</td>\n",
" <td>spend_B</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200000</td>\n",
" <td>60.6</td>\n",
" <td>spend_B</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100000</td>\n",
" <td>58.7</td>\n",
" <td>spend_B</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Rahul</td>\n",
" <td>Pandey</td>\n",
" <td>37</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>175.400000</td>\n",
" <td>85.6</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Rakesh</td>\n",
" <td>Kumar</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.734375</td>\n",
" <td>64.1</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>Nitesh</td>\n",
" <td>Mondol</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>165.100000</td>\n",
" <td>64.1</td>\n",
" <td>spend_C</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Arvind</td>\n",
" <td>Dwivedi</td>\n",
" <td>40</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>142.875000</td>\n",
" <td>55.6</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>Nikhil</td>\n",
" <td>Shikharwar</td>\n",
" <td>39</td>\n",
" <td>M</td>\n",
" <td>B</td>\n",
" <td>160.800000</td>\n",
" <td>60.0</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>Ashutosh</td>\n",
" <td>Merothiya</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>160.734375</td>\n",
" <td>60.0</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>Shubhro</td>\n",
" <td>Das</td>\n",
" <td>35</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>170.300000</td>\n",
" <td>75.5</td>\n",
" <td>spend_C</td>\n",
" <td>500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>Suvendu</td>\n",
" <td>Das</td>\n",
" <td>34</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>155.100000</td>\n",
" <td>56.8</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>Swadesh</td>\n",
" <td>Kumar</td>\n",
" <td>36</td>\n",
" <td>M</td>\n",
" <td>C</td>\n",
" <td>154.200000</td>\n",
" <td>60.6</td>\n",
" <td>spend_C</td>\n",
" <td>300.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>Arun</td>\n",
" <td>Vishwakarma</td>\n",
" <td>38</td>\n",
" <td>M</td>\n",
" <td>A</td>\n",
" <td>162.100000</td>\n",
" <td>58.7</td>\n",
" <td>spend_C</td>\n",
" <td>200.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" fname lname age sex section height(cm) weight(kg) expenditure \\\n",
"0 Rahul Pandey 37 M B 175.400000 85.6 spend_A \n",
"1 Rakesh Kumar 38 M B 160.734375 64.1 spend_A \n",
"2 Nitesh Mondol 35 M B 165.100000 64.1 spend_A \n",
"3 Arvind Dwivedi 40 M A 142.875000 55.6 spend_A \n",
"4 Nikhil Shikharwar 39 M B 160.800000 60.0 spend_A \n",
"5 Ashutosh Merothiya 36 M A 160.734375 60.0 spend_A \n",
"6 Shubhro Das 35 M C 170.300000 75.5 spend_A \n",
"7 Suvendu Das 34 M C 155.100000 56.8 spend_A \n",
"8 Swadesh Kumar 36 M C 154.200000 60.6 spend_A \n",
"9 Arun Vishwakarma 38 M A 162.100000 58.7 spend_A \n",
"10 Rahul Pandey 37 M B 175.400000 85.6 spend_B \n",
"11 Rakesh Kumar 38 M B 160.734375 64.1 spend_B \n",
"12 Nitesh Mondol 35 M B 165.100000 64.1 spend_B \n",
"13 Arvind Dwivedi 40 M A 142.875000 55.6 spend_B \n",
"14 Nikhil Shikharwar 39 M B 160.800000 60.0 spend_B \n",
"15 Ashutosh Merothiya 36 M A 160.734375 60.0 spend_B \n",
"16 Shubhro Das 35 M C 170.300000 75.5 spend_B \n",
"17 Suvendu Das 34 M C 155.100000 56.8 spend_B \n",
"18 Swadesh Kumar 36 M C 154.200000 60.6 spend_B \n",
"19 Arun Vishwakarma 38 M A 162.100000 58.7 spend_B \n",
"20 Rahul Pandey 37 M B 175.400000 85.6 spend_C \n",
"21 Rakesh Kumar 38 M B 160.734375 64.1 spend_C \n",
"22 Nitesh Mondol 35 M B 165.100000 64.1 spend_C \n",
"23 Arvind Dwivedi 40 M A 142.875000 55.6 spend_C \n",
"24 Nikhil Shikharwar 39 M B 160.800000 60.0 spend_C \n",
"25 Ashutosh Merothiya 36 M A 160.734375 60.0 spend_C \n",
"26 Shubhro Das 35 M C 170.300000 75.5 spend_C \n",
"27 Suvendu Das 34 M C 155.100000 56.8 spend_C \n",
"28 Swadesh Kumar 36 M C 154.200000 60.6 spend_C \n",
"29 Arun Vishwakarma 38 M A 162.100000 58.7 spend_C \n",
"\n",
" amount \n",
"0 1500.000000 \n",
"1 537.500000 \n",
"2 500.000000 \n",
"3 300.000000 \n",
"4 200.000000 \n",
"5 100.000000 \n",
"6 537.500000 \n",
"7 1000.000000 \n",
"8 100.000000 \n",
"9 600.000000 \n",
"10 200.000000 \n",
"11 1000.000000 \n",
"12 100.000000 \n",
"13 300.000000 \n",
"14 333.333333 \n",
"15 400.000000 \n",
"16 500.000000 \n",
"17 100.000000 \n",
"18 200.000000 \n",
"19 200.000000 \n",
"20 300.000000 \n",
"21 300.000000 \n",
"22 200.000000 \n",
"23 300.000000 \n",
"24 300.000000 \n",
"25 300.000000 \n",
"26 500.000000 \n",
"27 300.000000 \n",
"28 300.000000 \n",
"29 200.000000 "
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.melt(frame=df, id_vars=['fname','lname','age','sex','section','height(cm)','weight(kg)'],\n",
" value_vars=['spend_A','spend_B','spend_C'], var_name='expenditure', value_name='amount')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, when we look at our data, we can see that there are no missing or negative values. There are no outliers in our data. Lastly, our data is in the tidy data format."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This concludes the data cleaning process for friends dataset."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment