Skip to content

Instantly share code, notes, and snippets.

@a-y-khan

a-y-khan/pandas_1_0_highlights.ipynb

Created February 27, 2020 04:12
Show Gist options
  • Save a-y-khan/c7c5239142d6532ef5d665f48fb908f7 to your computer and use it in GitHub Desktop.
Save a-y-khan/c7c5239142d6532ef5d665f48fb908f7 to your computer and use it in GitHub Desktop.
Download ZIP
Pyladies lightning talk: interesting new features in Pandas 1.0
Display the source blob
Display the rendered blob
Raw
pandas_1_0_highlights.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pandas 1.0 Updates\n",
"\n",
"Release notes: https://pandas.pydata.org/docs/whatsnew/v1.0.0.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python 2 support dropped\n",
"\n",
"* Python 3.6.1 minimum supported version\n",
"\n",
"## Notable deprecations\n",
"\n",
"Deprection notes: https://pandas.pydata.org/docs/whatsnew/v1.0.0.html#deprecations\n",
"\n",
"* Pandas.datetime removed, use Python datetime instead\n",
"* Sparse classes SparseSeries, SparseDataFrame removed\n",
" * Sparse data support moved to Series, DataFrame"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataFrame.to_markdown()\n",
"\n",
"* Render DataFrame as Markdown table string.\n",
"* Requires Python **tabulate** package."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>show_id</th>\n",
" <th>type</th>\n",
" <th>title</th>\n",
" <th>director</th>\n",
" <th>cast</th>\n",
" <th>country</th>\n",
" <th>date_added</th>\n",
" <th>release_year</th>\n",
" <th>rating</th>\n",
" <th>duration</th>\n",
" <th>listed_in</th>\n",
" <th>description</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>81145628</td>\n",
" <td>Movie</td>\n",
" <td>Norm of the North: King Sized Adventure</td>\n",
" <td>Richard Finn, Tim Maltby</td>\n",
" <td>Alan Marriott, Andrew Toth, Brian Dobson, Cole...</td>\n",
" <td>United States, India, South Korea, China</td>\n",
" <td>September 9, 2019</td>\n",
" <td>2019</td>\n",
" <td>TV-PG</td>\n",
" <td>90 min</td>\n",
" <td>Children &amp; Family Movies, Comedies</td>\n",
" <td>Before planning an awesome wedding for his gra...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>80117401</td>\n",
" <td>Movie</td>\n",
" <td>Jandino: Whatever it Takes</td>\n",
" <td>NaN</td>\n",
" <td>Jandino Asporaat</td>\n",
" <td>United Kingdom</td>\n",
" <td>September 9, 2016</td>\n",
" <td>2016</td>\n",
" <td>TV-MA</td>\n",
" <td>94 min</td>\n",
" <td>Stand-Up Comedy</td>\n",
" <td>Jandino Asporaat riffs on the challenges of ra...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>70234439</td>\n",
" <td>TV Show</td>\n",
" <td>Transformers Prime</td>\n",
" <td>NaN</td>\n",
" <td>Peter Cullen, Sumalee Montano, Frank Welker, J...</td>\n",
" <td>United States</td>\n",
" <td>September 8, 2018</td>\n",
" <td>2013</td>\n",
" <td>TV-Y7-FV</td>\n",
" <td>1 Season</td>\n",
" <td>Kids' TV</td>\n",
" <td>With the help of three human allies, the Autob...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" show_id type title \\\n",
"0 81145628 Movie Norm of the North: King Sized Adventure \n",
"1 80117401 Movie Jandino: Whatever it Takes \n",
"2 70234439 TV Show Transformers Prime \n",
"\n",
" director \\\n",
"0 Richard Finn, Tim Maltby \n",
"1 NaN \n",
"2 NaN \n",
"\n",
" cast \\\n",
"0 Alan Marriott, Andrew Toth, Brian Dobson, Cole... \n",
"1 Jandino Asporaat \n",
"2 Peter Cullen, Sumalee Montano, Frank Welker, J... \n",
"\n",
" country date_added release_year \\\n",
"0 United States, India, South Korea, China September 9, 2019 2019 \n",
"1 United Kingdom September 9, 2016 2016 \n",
"2 United States September 8, 2018 2013 \n",
"\n",
" rating duration listed_in \\\n",
"0 TV-PG 90 min Children & Family Movies, Comedies \n",
"1 TV-MA 94 min Stand-Up Comedy \n",
"2 TV-Y7-FV 1 Season Kids' TV \n",
"\n",
" description \n",
"0 Before planning an awesome wedding for his gra... \n",
"1 Jandino Asporaat riffs on the challenges of ra... \n",
"2 With the help of three human allies, the Autob... "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"\n",
"netflix_titles_df = pd.read_csv('netflix_titles.csv')\n",
"display(netflix_titles_df.head(3))"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"| | show_id | type | title | director | cast | country | date_added | release_year | rating | duration | listed_in | description |\n",
"|---:|----------:|:--------|:----------------------------------------|:-------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------|:------------------|---------------:|:---------|:-----------|:-----------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| 0 | 81145628 | Movie | Norm of the North: King Sized Adventure | Richard Finn, Tim Maltby | Alan Marriott, Andrew Toth, Brian Dobson, Cole Howard, Jennifer Cameron, Jonathan Holmes, Lee Tockar, Lisa Durupt, Maya Kay, Michael Dobson | United States, India, South Korea, China | September 9, 2019 | 2019 | TV-PG | 90 min | Children & Family Movies, Comedies | Before planning an awesome wedding for his grandfather, a polar bear king must take back a stolen artifact from an evil archaeologist first. |\n",
"| 1 | 80117401 | Movie | Jandino: Whatever it Takes | nan | Jandino Asporaat | United Kingdom | September 9, 2016 | 2016 | TV-MA | 94 min | Stand-Up Comedy | Jandino Asporaat riffs on the challenges of raising kids and serenades the audience with a rousing rendition of \"Sex on Fire\" in his comedy show. |\n",
"| 2 | 70234439 | TV Show | Transformers Prime | nan | Peter Cullen, Sumalee Montano, Frank Welker, Jeffrey Combs, Kevin Michael Richardson, Tania Gunadi, Josh Keaton, Steve Blum, Andy Pessoa, Ernie Hudson, Daran Norris, Will Friedle | United States | September 8, 2018 | 2013 | TV-Y7-FV | 1 Season | Kids' TV | With the help of three human allies, the Autobots once again protect Earth from the onslaught of the Decepticons and their leader, Megatron. |\n"
]
}
],
"source": [
"markdown_table = netflix_titles_df.head(3).to_markdown()\n",
"print(markdown_table)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Render Markdown table in code cell!\n"
]
},
{
"data": {
"text/markdown": [
"| | show_id | type | title | director | cast | country | date_added | release_year | rating | duration | listed_in | description |\n",
"|---:|----------:|:--------|:----------------------------------------|:-------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------|:------------------|---------------:|:---------|:-----------|:-----------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| 0 | 81145628 | Movie | Norm of the North: King Sized Adventure | Richard Finn, Tim Maltby | Alan Marriott, Andrew Toth, Brian Dobson, Cole Howard, Jennifer Cameron, Jonathan Holmes, Lee Tockar, Lisa Durupt, Maya Kay, Michael Dobson | United States, India, South Korea, China | September 9, 2019 | 2019 | TV-PG | 90 min | Children & Family Movies, Comedies | Before planning an awesome wedding for his grandfather, a polar bear king must take back a stolen artifact from an evil archaeologist first. |\n",
"| 1 | 80117401 | Movie | Jandino: Whatever it Takes | nan | Jandino Asporaat | United Kingdom | September 9, 2016 | 2016 | TV-MA | 94 min | Stand-Up Comedy | Jandino Asporaat riffs on the challenges of raising kids and serenades the audience with a rousing rendition of \"Sex on Fire\" in his comedy show. |\n",
"| 2 | 70234439 | TV Show | Transformers Prime | nan | Peter Cullen, Sumalee Montano, Frank Welker, Jeffrey Combs, Kevin Michael Richardson, Tania Gunadi, Josh Keaton, Steve Blum, Andy Pessoa, Ernie Hudson, Daran Norris, Will Friedle | United States | September 8, 2018 | 2013 | TV-Y7-FV | 1 Season | Kids' TV | With the help of three human allies, the Autobots once again protect Earth from the onslaught of the Decepticons and their leader, Megatron. |"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.display import Markdown as md\n",
"\n",
"print(\"Render Markdown table in code cell!\")\n",
"\n",
"md(markdown_table)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataFrame.to_json() indent\n",
"\n",
"* Pretty print JSON output"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"show_id\":{\n",
" \"0\":81145628,\n",
" \"1\":80117401,\n",
" \"2\":70234439\n",
" },\n",
" \"type\":{\n",
" \"0\":\"Movie\",\n",
" \"1\":\"Movie\",\n",
" \"2\":\"TV Show\"\n",
" },\n",
" \"title\":{\n",
" \"0\":\"Norm of the North: King Sized Adventure\",\n",
" \"1\":\"Jandino: Whatever it Takes\",\n",
" \"2\":\"Transformers Prime\"\n",
" },\n",
" \"director\":{\n",
" \"0\":\"Richard Finn, Tim Maltby\",\n",
" \"1\":null,\n",
" \"2\":null\n",
" },\n",
" \"cast\":{\n",
" \"0\":\"Alan Marriott, Andrew Toth, Brian Dobson, Cole Howard, Jennifer Cameron, Jonathan Holmes, Lee Tockar, Lisa Durupt, Maya Kay, Michael Dobson\",\n",
" \"1\":\"Jandino Asporaat\",\n",
" \"2\":\"Peter Cullen, Sumalee Montano, Frank Welker, Jeffrey Combs, Kevin Michael Richardson, Tania Gunadi, Josh Keaton, Steve Blum, Andy Pessoa, Ernie Hudson, Daran Norris, Will Friedle\"\n",
" },\n",
" \"country\":{\n",
" \"0\":\"United States, India, South Korea, China\",\n",
" \"1\":\"United Kingdom\",\n",
" \"2\":\"United States\"\n",
" },\n",
" \"date_added\":{\n",
" \"0\":\"September 9, 2019\",\n",
" \"1\":\"September 9, 2016\",\n",
" \"2\":\"September 8, 2018\"\n",
" },\n",
" \"release_year\":{\n",
" \"0\":2019,\n",
" \"1\":2016,\n",
" \"2\":2013\n",
" },\n",
" \"rating\":{\n",
" \"0\":\"TV-PG\",\n",
" \"1\":\"TV-MA\",\n",
" \"2\":\"TV-Y7-FV\"\n",
" },\n",
" \"duration\":{\n",
" \"0\":\"90 min\",\n",
" \"1\":\"94 min\",\n",
" \"2\":\"1 Season\"\n",
" },\n",
" \"listed_in\":{\n",
" \"0\":\"Children & Family Movies, Comedies\",\n",
" \"1\":\"Stand-Up Comedy\",\n",
" \"2\":\"Kids' TV\"\n",
" },\n",
" \"description\":{\n",
" \"0\":\"Before planning an awesome wedding for his grandfather, a polar bear king must take back a stolen artifact from an evil archaeologist first.\",\n",
" \"1\":\"Jandino Asporaat riffs on the challenges of raising kids and serenades the audience with a rousing rendition of \\\"Sex on Fire\\\" in his comedy show.\",\n",
" \"2\":\"With the help of three human allies, the Autobots once again protect Earth from the onslaught of the Decepticons and their leader, Megatron.\"\n",
" }\n",
"}\n"
]
}
],
"source": [
"print(netflix_titles_df.head(3).to_json(indent=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Better DataFrame.info output\n",
"\n",
"* Pandas 0.25, for example:\n",
"\n",
"![](pandas_0_25_3_df_info.png)\n",
"\n",
"### DataFrame.info improvements\n",
"\n",
"* Column summary line numbers\n",
"* Clearer headers"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 6234 entries, 0 to 6233\n",
"Data columns (total 12 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 show_id 6234 non-null int64 \n",
" 1 type 6234 non-null object\n",
" 2 title 6234 non-null object\n",
" 3 director 4265 non-null object\n",
" 4 cast 5664 non-null object\n",
" 5 country 5758 non-null object\n",
" 6 date_added 6223 non-null object\n",
" 7 release_year 6234 non-null int64 \n",
" 8 rating 6224 non-null object\n",
" 9 duration 6234 non-null object\n",
" 10 listed_in 6234 non-null object\n",
" 11 description 6234 non-null object\n",
"dtypes: int64(2), object(10)\n",
"memory usage: 584.6+ KB\n"
]
},
{
"data": {
"text/plain": [
"None"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(netflix_titles_df.info())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Add boolean column for testing"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([True, False, nan], dtype=object)"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 6234 entries, 0 to 6233\n",
"Data columns (total 13 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 show_id 6234 non-null int64 \n",
" 1 type 6234 non-null object\n",
" 2 title 6234 non-null object\n",
" 3 director 4265 non-null object\n",
" 4 cast 5664 non-null object\n",
" 5 country 5758 non-null object\n",
" 6 date_added 6223 non-null object\n",
" 7 release_year 6234 non-null int64 \n",
" 8 rating 6224 non-null object\n",
" 9 duration 6234 non-null object\n",
" 10 listed_in 6234 non-null object\n",
" 11 description 6234 non-null object\n",
" 12 tv_rating 6224 non-null object\n",
"dtypes: int64(2), object(11)\n",
"memory usage: 633.3+ KB\n"
]
},
{
"data": {
"text/plain": [
"None"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"netflix_titles_df['tv_rating'] = netflix_titles_df['rating'].str.upper().str.contains('TV')\n",
"display(netflix_titles_df['tv_rating'].unique())\n",
"display(netflix_titles_df.info())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Updated datatypes\n",
"\n",
"### String datatype\n",
"\n",
"\n",
"* ***Experimental!!!***\n",
"* Convert objects using DataFrame.convert_dtypes(), Series.convert_dtypes()\n",
" * converts **string**, **integer**, **boolean** types by default"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 6234 entries, 0 to 6233\n",
"Data columns (total 13 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 show_id 6234 non-null Int64 \n",
" 1 type 6234 non-null string \n",
" 2 title 6234 non-null string \n",
" 3 director 4265 non-null string \n",
" 4 cast 5664 non-null string \n",
" 5 country 5758 non-null string \n",
" 6 date_added 6223 non-null string \n",
" 7 release_year 6234 non-null Int64 \n",
" 8 rating 6224 non-null string \n",
" 9 duration 6234 non-null string \n",
" 10 listed_in 6234 non-null string \n",
" 11 description 6234 non-null string \n",
" 12 tv_rating 6224 non-null boolean\n",
"dtypes: Int64(2), boolean(1), string(10)\n",
"memory usage: 608.9 KB\n"
]
},
{
"data": {
"text/plain": [
"None"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"netflix_titles_converted_df = netflix_titles_df.convert_dtypes()\n",
"display(netflix_titles_converted_df.info())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### pd.NA scalar\n",
"\n",
"* ***Experimental!!!***\n",
"* Consistent missing value indicator\n",
" * Instead of `np.nan`, `None`, `pd.NaT` (datetime)\n",
"* Supported by nullable integer types\n",
" * `Int64` vs numpy `int64`, `Int32`, etc.\n",
"* Supported by **boolean** types"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Select all rows with missing value (old)\n",
"\n",
"* `isnull` is alias of `isna`"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>show_id</th>\n",
" <th>type</th>\n",
" <th>title</th>\n",
" <th>director</th>\n",
" <th>cast</th>\n",
" <th>country</th>\n",
" <th>date_added</th>\n",
" <th>release_year</th>\n",
" <th>rating</th>\n",
" <th>duration</th>\n",
" <th>listed_in</th>\n",
" <th>description</th>\n",
" <th>tv_rating</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>180</th>\n",
" <td>70259208</td>\n",
" <td>TV Show</td>\n",
" <td>Transformers: Rescue Bots</td>\n",
" <td>NaN</td>\n",
" <td>Lacey Chabert, Elan Garfias, Maurice LaMarche,...</td>\n",
" <td>Canada, United States</td>\n",
" <td>September 19, 2018</td>\n",
" <td>2016</td>\n",
" <td>TV-Y7</td>\n",
" <td>1 Season</td>\n",
" <td>Kids' TV</td>\n",
" <td>Follow the adventure of four young transformer...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>223</th>\n",
" <td>81176196</td>\n",
" <td>TV Show</td>\n",
" <td>The Rap Game</td>\n",
" <td>NaN</td>\n",
" <td>Jermaine Dupri</td>\n",
" <td>United States</td>\n",
" <td>September 15, 2019</td>\n",
" <td>2016</td>\n",
" <td>TV-14</td>\n",
" <td>1 Season</td>\n",
" <td>Reality TV</td>\n",
" <td>Young lyricists between the ages of 12 and 16 ...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>446</th>\n",
" <td>80198511</td>\n",
" <td>TV Show</td>\n",
" <td>Dancing Queen</td>\n",
" <td>NaN</td>\n",
" <td>Justin \"Alyssa Edwards\" Johnson</td>\n",
" <td>NaN</td>\n",
" <td>October 5, 2018</td>\n",
" <td>2018</td>\n",
" <td>TV-14</td>\n",
" <td>1 Season</td>\n",
" <td>Reality TV</td>\n",
" <td>Snatching trophies. Getting gorgeous. Turning ...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3276</th>\n",
" <td>80189619</td>\n",
" <td>TV Show</td>\n",
" <td>A Taiwanese Tale of Two Cities</td>\n",
" <td>NaN</td>\n",
" <td>Tammy Chen, James Wen, Peggy Tseng, Denny Huang</td>\n",
" <td>NaN</td>\n",
" <td>January 27, 2019</td>\n",
" <td>2018</td>\n",
" <td>TV-MA</td>\n",
" <td>1 Season</td>\n",
" <td>International TV Shows, TV Dramas</td>\n",
" <td>A Taipei doctor and a San Francisco engineer s...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3694</th>\n",
" <td>80140364</td>\n",
" <td>TV Show</td>\n",
" <td>A Good Wife</td>\n",
" <td>NaN</td>\n",
" <td>Tien Hsin, Christopher Lee, Darren Chiu, Shara...</td>\n",
" <td>Taiwan</td>\n",
" <td>January 1, 2017</td>\n",
" <td>2013</td>\n",
" <td>TV-14</td>\n",
" <td>1 Season</td>\n",
" <td>International TV Shows, Romantic TV Shows, TV ...</td>\n",
" <td>As her seemingly idyllic life begins to crumbl...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4337</th>\n",
" <td>80225085</td>\n",
" <td>TV Show</td>\n",
" <td>Fast &amp; Furious Spy Racers</td>\n",
" <td>NaN</td>\n",
" <td>Tyler Posey, Charlet Chung, Jorge Diaz, Camill...</td>\n",
" <td>United States</td>\n",
" <td>December 26, 2019</td>\n",
" <td>2019</td>\n",
" <td>TV-Y7-FV</td>\n",
" <td>1 Season</td>\n",
" <td>Kids' TV</td>\n",
" <td>A government agency recruits teen driver Tony ...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4362</th>\n",
" <td>80145086</td>\n",
" <td>Movie</td>\n",
" <td>When Hari Got Married</td>\n",
" <td>Ritu Sarin, Tenzing Sonam</td>\n",
" <td>NaN</td>\n",
" <td>United Kingdom, India, United States</td>\n",
" <td>December 25, 2016</td>\n",
" <td>2013</td>\n",
" <td>NR</td>\n",
" <td>75 min</td>\n",
" <td>Documentaries, International Movies</td>\n",
" <td>A small-town taxi driver is arranged to be mar...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4529</th>\n",
" <td>80214013</td>\n",
" <td>TV Show</td>\n",
" <td>Black</td>\n",
" <td>NaN</td>\n",
" <td>Seung-heon Song, A-ra Go, EL, Dong-jun Kim, Wo...</td>\n",
" <td>South Korea</td>\n",
" <td>December 15, 2017</td>\n",
" <td>2017</td>\n",
" <td>TV-MA</td>\n",
" <td>1 Season</td>\n",
" <td>Crime TV Shows, International TV Shows, Korean...</td>\n",
" <td>A Grim Reaper, a detective and a woman who for...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5244</th>\n",
" <td>81072883</td>\n",
" <td>TV Show</td>\n",
" <td>The Outsiders</td>\n",
" <td>NaN</td>\n",
" <td>Dylan Kuo, Ady An, Lan Cheng-Lung, Michael Cha...</td>\n",
" <td>Taiwan</td>\n",
" <td>April 30, 2019</td>\n",
" <td>2004</td>\n",
" <td>TV-MA</td>\n",
" <td>1 Season</td>\n",
" <td>Crime TV Shows, International TV Shows, Romant...</td>\n",
" <td>A trio of delinquent brothers fends off neighb...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5522</th>\n",
" <td>80196139</td>\n",
" <td>Movie</td>\n",
" <td>Fishpeople</td>\n",
" <td>Keith Malloy</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>April 1, 2018</td>\n",
" <td>2017</td>\n",
" <td>TV-14</td>\n",
" <td>49 min</td>\n",
" <td>Documentaries, Sports Movies</td>\n",
" <td>In this exploration of the life-changing power...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" show_id type title \\\n",
"180 70259208 TV Show Transformers: Rescue Bots \n",
"223 81176196 TV Show The Rap Game \n",
"446 80198511 TV Show Dancing Queen \n",
"3276 80189619 TV Show A Taiwanese Tale of Two Cities \n",
"3694 80140364 TV Show A Good Wife \n",
"4337 80225085 TV Show Fast & Furious Spy Racers \n",
"4362 80145086 Movie When Hari Got Married \n",
"4529 80214013 TV Show Black \n",
"5244 81072883 TV Show The Outsiders \n",
"5522 80196139 Movie Fishpeople \n",
"\n",
" director \\\n",
"180 NaN \n",
"223 NaN \n",
"446 NaN \n",
"3276 NaN \n",
"3694 NaN \n",
"4337 NaN \n",
"4362 Ritu Sarin, Tenzing Sonam \n",
"4529 NaN \n",
"5244 NaN \n",
"5522 Keith Malloy \n",
"\n",
" cast \\\n",
"180 Lacey Chabert, Elan Garfias, Maurice LaMarche,... \n",
"223 Jermaine Dupri \n",
"446 Justin \"Alyssa Edwards\" Johnson \n",
"3276 Tammy Chen, James Wen, Peggy Tseng, Denny Huang \n",
"3694 Tien Hsin, Christopher Lee, Darren Chiu, Shara... \n",
"4337 Tyler Posey, Charlet Chung, Jorge Diaz, Camill... \n",
"4362 NaN \n",
"4529 Seung-heon Song, A-ra Go, EL, Dong-jun Kim, Wo... \n",
"5244 Dylan Kuo, Ady An, Lan Cheng-Lung, Michael Cha... \n",
"5522 NaN \n",
"\n",
" country date_added release_year \\\n",
"180 Canada, United States September 19, 2018 2016 \n",
"223 United States September 15, 2019 2016 \n",
"446 NaN October 5, 2018 2018 \n",
"3276 NaN January 27, 2019 2018 \n",
"3694 Taiwan January 1, 2017 2013 \n",
"4337 United States December 26, 2019 2019 \n",
"4362 United Kingdom, India, United States December 25, 2016 2013 \n",
"4529 South Korea December 15, 2017 2017 \n",
"5244 Taiwan April 30, 2019 2004 \n",
"5522 NaN April 1, 2018 2017 \n",
"\n",
" rating duration listed_in \\\n",
"180 TV-Y7 1 Season Kids' TV \n",
"223 TV-14 1 Season Reality TV \n",
"446 TV-14 1 Season Reality TV \n",
"3276 TV-MA 1 Season International TV Shows, TV Dramas \n",
"3694 TV-14 1 Season International TV Shows, Romantic TV Shows, TV ... \n",
"4337 TV-Y7-FV 1 Season Kids' TV \n",
"4362 NR 75 min Documentaries, International Movies \n",
"4529 TV-MA 1 Season Crime TV Shows, International TV Shows, Korean... \n",
"5244 TV-MA 1 Season Crime TV Shows, International TV Shows, Romant... \n",
"5522 TV-14 49 min Documentaries, Sports Movies \n",
"\n",
" description tv_rating \n",
"180 Follow the adventure of four young transformer... True \n",
"223 Young lyricists between the ages of 12 and 16 ... True \n",
"446 Snatching trophies. Getting gorgeous. Turning ... True \n",
"3276 A Taipei doctor and a San Francisco engineer s... True \n",
"3694 As her seemingly idyllic life begins to crumbl... True \n",
"4337 A government agency recruits teen driver Tony ... True \n",
"4362 A small-town taxi driver is arranged to be mar... False \n",
"4529 A Grim Reaper, a detective and a woman who for... True \n",
"5244 A trio of delinquent brothers fends off neighb... True \n",
"5522 In this exploration of the life-changing power... True "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"netflix_titles_df[pd.isna(netflix_titles_df).any(axis=1)].sample(10).sort_index()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Select all rows with missing value (new)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>show_id</th>\n",
" <th>type</th>\n",
" <th>title</th>\n",
" <th>director</th>\n",
" <th>cast</th>\n",
" <th>country</th>\n",
" <th>date_added</th>\n",
" <th>release_year</th>\n",
" <th>rating</th>\n",
" <th>duration</th>\n",
" <th>listed_in</th>\n",
" <th>description</th>\n",
" <th>tv_rating</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>811</th>\n",
" <td>81036448</td>\n",
" <td>TV Show</td>\n",
" <td>Miniforce X</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>South Korea</td>\n",
" <td>October 1, 2019</td>\n",
" <td>2018</td>\n",
" <td>TV-Y7-FV</td>\n",
" <td>1 Season</td>\n",
" <td>Kids' TV, Korean TV Shows</td>\n",
" <td>Zenos wants to use the Earth’s negative energy...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1204</th>\n",
" <td>81215856</td>\n",
" <td>Movie</td>\n",
" <td>100 Things to do Before High School</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Isabela Moner, Jaheem Toombs, Owen Joyner, Jac...</td>\n",
" <td>United States</td>\n",
" <td>November 2, 2019</td>\n",
" <td>2014</td>\n",
" <td>TV-G</td>\n",
" <td>44 min</td>\n",
" <td>Movies</td>\n",
" <td>Led by seventh-grader C.J., three students who...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2155</th>\n",
" <td>80135296</td>\n",
" <td>Movie</td>\n",
" <td>Naledi: A Baby Elephant's Tale</td>\n",
" <td>Ben Bowie, Geoff Luck</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>United States, Botswana</td>\n",
" <td>March 17, 2017</td>\n",
" <td>2016</td>\n",
" <td>TV-PG</td>\n",
" <td>91 min</td>\n",
" <td>Documentaries</td>\n",
" <td>Scientists and animal keepers fight to save an...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2494</th>\n",
" <td>80244783</td>\n",
" <td>TV Show</td>\n",
" <td>Malibu Rescue: The Series</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Ricardo Hurtado, Breanna Yde, Jackie R. Jacobs...</td>\n",
" <td>United States</td>\n",
" <td>June 3, 2019</td>\n",
" <td>2019</td>\n",
" <td>TV-G</td>\n",
" <td>1 Season</td>\n",
" <td>Kids' TV, TV Action &amp; Adventure, TV Comedies</td>\n",
" <td>On the heels of Junior Rescue training, Team F...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2794</th>\n",
" <td>80102266</td>\n",
" <td>TV Show</td>\n",
" <td>Dharmakshetra</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Kashmira Irani, Chandan Anand, Dinesh Mehta, A...</td>\n",
" <td>India</td>\n",
" <td>July 5, 2018</td>\n",
" <td>2014</td>\n",
" <td>TV-14</td>\n",
" <td>1 Season</td>\n",
" <td>International TV Shows, TV Dramas, TV Sci-Fi &amp;...</td>\n",
" <td>After the ancient Great War, the god Chitragup...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3204</th>\n",
" <td>80128657</td>\n",
" <td>Movie</td>\n",
" <td>Growing Up Coy</td>\n",
" <td>Eric Juhola</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>United States</td>\n",
" <td>January 6, 2017</td>\n",
" <td>2016</td>\n",
" <td>TV-PG</td>\n",
" <td>82 min</td>\n",
" <td>Documentaries, LGBTQ Movies</td>\n",
" <td>Filmmakers follow a Colorado family's highly p...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4036</th>\n",
" <td>81063049</td>\n",
" <td>TV Show</td>\n",
" <td>The Bomb Squad</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Christopher Eccleston</td>\n",
" <td>United Kingdom</td>\n",
" <td>February 1, 2019</td>\n",
" <td>2011</td>\n",
" <td>TV-MA</td>\n",
" <td>1 Season</td>\n",
" <td>British TV Shows, Docuseries</td>\n",
" <td>This documentary follows British bomb disposal...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5671</th>\n",
" <td>80018294</td>\n",
" <td>TV Show</td>\n",
" <td>Marvel's Daredevil</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Charlie Cox, Deborah Ann Woll, Elden Henson, R...</td>\n",
" <td>United States</td>\n",
" <td>October 19, 2018</td>\n",
" <td>2018</td>\n",
" <td>TV-MA</td>\n",
" <td>3 Seasons</td>\n",
" <td>Crime TV Shows, TV Action &amp; Adventure</td>\n",
" <td>Blinded as a young boy, Matt Murdock fights in...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6074</th>\n",
" <td>80092654</td>\n",
" <td>TV Show</td>\n",
" <td>Occupied</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Henrik Mestad, Ane Dahl Torp, Ingeborga Dapkun...</td>\n",
" <td>Norway, Sweden</td>\n",
" <td>December 31, 2019</td>\n",
" <td>2017</td>\n",
" <td>TV-MA</td>\n",
" <td>2 Seasons</td>\n",
" <td>International TV Shows, TV Dramas, TV Thrillers</td>\n",
" <td>In the near future, Russia initiates a \"silk g...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6193</th>\n",
" <td>80049277</td>\n",
" <td>TV Show</td>\n",
" <td>Ash vs. Evil Dead</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>Bruce Campbell, Lucy Lawless, Ray Santiago, Da...</td>\n",
" <td>United States</td>\n",
" <td>April 29, 2019</td>\n",
" <td>2018</td>\n",
" <td>TV-MA</td>\n",
" <td>3 Seasons</td>\n",
" <td>TV Action &amp; Adventure, TV Comedies, TV Horror</td>\n",
" <td>Thirty years later, Ash Williams – demon hunte...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" show_id type title \\\n",
"811 81036448 TV Show Miniforce X \n",
"1204 81215856 Movie 100 Things to do Before High School \n",
"2155 80135296 Movie Naledi: A Baby Elephant's Tale \n",
"2494 80244783 TV Show Malibu Rescue: The Series \n",
"2794 80102266 TV Show Dharmakshetra \n",
"3204 80128657 Movie Growing Up Coy \n",
"4036 81063049 TV Show The Bomb Squad \n",
"5671 80018294 TV Show Marvel's Daredevil \n",
"6074 80092654 TV Show Occupied \n",
"6193 80049277 TV Show Ash vs. Evil Dead \n",
"\n",
" director \\\n",
"811 <NA> \n",
"1204 <NA> \n",
"2155 Ben Bowie, Geoff Luck \n",
"2494 <NA> \n",
"2794 <NA> \n",
"3204 Eric Juhola \n",
"4036 <NA> \n",
"5671 <NA> \n",
"6074 <NA> \n",
"6193 <NA> \n",
"\n",
" cast \\\n",
"811 <NA> \n",
"1204 Isabela Moner, Jaheem Toombs, Owen Joyner, Jac... \n",
"2155 <NA> \n",
"2494 Ricardo Hurtado, Breanna Yde, Jackie R. Jacobs... \n",
"2794 Kashmira Irani, Chandan Anand, Dinesh Mehta, A... \n",
"3204 <NA> \n",
"4036 Christopher Eccleston \n",
"5671 Charlie Cox, Deborah Ann Woll, Elden Henson, R... \n",
"6074 Henrik Mestad, Ane Dahl Torp, Ingeborga Dapkun... \n",
"6193 Bruce Campbell, Lucy Lawless, Ray Santiago, Da... \n",
"\n",
" country date_added release_year rating \\\n",
"811 South Korea October 1, 2019 2018 TV-Y7-FV \n",
"1204 United States November 2, 2019 2014 TV-G \n",
"2155 United States, Botswana March 17, 2017 2016 TV-PG \n",
"2494 United States June 3, 2019 2019 TV-G \n",
"2794 India July 5, 2018 2014 TV-14 \n",
"3204 United States January 6, 2017 2016 TV-PG \n",
"4036 United Kingdom February 1, 2019 2011 TV-MA \n",
"5671 United States October 19, 2018 2018 TV-MA \n",
"6074 Norway, Sweden December 31, 2019 2017 TV-MA \n",
"6193 United States April 29, 2019 2018 TV-MA \n",
"\n",
" duration listed_in \\\n",
"811 1 Season Kids' TV, Korean TV Shows \n",
"1204 44 min Movies \n",
"2155 91 min Documentaries \n",
"2494 1 Season Kids' TV, TV Action & Adventure, TV Comedies \n",
"2794 1 Season International TV Shows, TV Dramas, TV Sci-Fi &... \n",
"3204 82 min Documentaries, LGBTQ Movies \n",
"4036 1 Season British TV Shows, Docuseries \n",
"5671 3 Seasons Crime TV Shows, TV Action & Adventure \n",
"6074 2 Seasons International TV Shows, TV Dramas, TV Thrillers \n",
"6193 3 Seasons TV Action & Adventure, TV Comedies, TV Horror \n",
"\n",
" description tv_rating \n",
"811 Zenos wants to use the Earth’s negative energy... True \n",
"1204 Led by seventh-grader C.J., three students who... True \n",
"2155 Scientists and animal keepers fight to save an... True \n",
"2494 On the heels of Junior Rescue training, Team F... True \n",
"2794 After the ancient Great War, the god Chitragup... True \n",
"3204 Filmmakers follow a Colorado family's highly p... True \n",
"4036 This documentary follows British bomb disposal... True \n",
"5671 Blinded as a young boy, Matt Murdock fights in... True \n",
"6074 In the near future, Russia initiates a \"silk g... True \n",
"6193 Thirty years later, Ash Williams – demon hunte... True "
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"netflix_titles_converted_df[pd.isna(netflix_titles_converted_df).any(axis=1)].sample(10).sort_index()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Missing values in boolean Series"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"object\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>show_id</th>\n",
" <th>type</th>\n",
" <th>title</th>\n",
" <th>director</th>\n",
" <th>cast</th>\n",
" <th>country</th>\n",
" <th>date_added</th>\n",
" <th>release_year</th>\n",
" <th>rating</th>\n",
" <th>duration</th>\n",
" <th>listed_in</th>\n",
" <th>description</th>\n",
" <th>tv_rating</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2411</th>\n",
" <td>80144119</td>\n",
" <td>Movie</td>\n",
" <td>My Honor Was Loyalty</td>\n",
" <td>Alessandro Pepe</td>\n",
" <td>Leone Frisa, Paolo Vaccarino, Francesco Miglio...</td>\n",
" <td>Italy</td>\n",
" <td>March 1, 2017</td>\n",
" <td>2015</td>\n",
" <td>NaN</td>\n",
" <td>115 min</td>\n",
" <td>Dramas</td>\n",
" <td>Amid the chaos and horror of World War II, a c...</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6231</th>\n",
" <td>80116008</td>\n",
" <td>Movie</td>\n",
" <td>Little Baby Bum: Nursery Rhyme Friends</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2016</td>\n",
" <td>NaN</td>\n",
" <td>60 min</td>\n",
" <td>Movies</td>\n",
" <td>Nursery rhymes and original music for children...</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" show_id type title \\\n",
"2411 80144119 Movie My Honor Was Loyalty \n",
"6231 80116008 Movie Little Baby Bum: Nursery Rhyme Friends \n",
"\n",
" director cast \\\n",
"2411 Alessandro Pepe Leone Frisa, Paolo Vaccarino, Francesco Miglio... \n",
"6231 NaN NaN \n",
"\n",
" country date_added release_year rating duration listed_in \\\n",
"2411 Italy March 1, 2017 2015 NaN 115 min Dramas \n",
"6231 NaN NaN 2016 NaN 60 min Movies \n",
"\n",
" description tv_rating \n",
"2411 Amid the chaos and horror of World War II, a c... NaN \n",
"6231 Nursery rhymes and original music for children... NaN "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(netflix_titles_df['tv_rating'].dtype)\n",
"netflix_titles_df[netflix_titles_df['tv_rating'].isna()].sample(2).sort_index()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"boolean\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>show_id</th>\n",
" <th>type</th>\n",
" <th>title</th>\n",
" <th>director</th>\n",
" <th>cast</th>\n",
" <th>country</th>\n",
" <th>date_added</th>\n",
" <th>release_year</th>\n",
" <th>rating</th>\n",
" <th>duration</th>\n",
" <th>listed_in</th>\n",
" <th>description</th>\n",
" <th>tv_rating</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>211</th>\n",
" <td>70129452</td>\n",
" <td>Movie</td>\n",
" <td>Louis C.K.: Hilarious</td>\n",
" <td>Louis C.K.</td>\n",
" <td>Louis C.K.</td>\n",
" <td>United States</td>\n",
" <td>September 16, 2016</td>\n",
" <td>2010</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>84 min</td>\n",
" <td>Movies</td>\n",
" <td>Emmy-winning comedy writer Louis C.K. brings h...</td>\n",
" <td>&lt;NA&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6231</th>\n",
" <td>80116008</td>\n",
" <td>Movie</td>\n",
" <td>Little Baby Bum: Nursery Rhyme Friends</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>2016</td>\n",
" <td>&lt;NA&gt;</td>\n",
" <td>60 min</td>\n",
" <td>Movies</td>\n",
" <td>Nursery rhymes and original music for children...</td>\n",
" <td>&lt;NA&gt;</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" show_id type title director \\\n",
"211 70129452 Movie Louis C.K.: Hilarious Louis C.K. \n",
"6231 80116008 Movie Little Baby Bum: Nursery Rhyme Friends <NA> \n",
"\n",
" cast country date_added release_year rating \\\n",
"211 Louis C.K. United States September 16, 2016 2010 <NA> \n",
"6231 <NA> <NA> <NA> 2016 <NA> \n",
"\n",
" duration listed_in description \\\n",
"211 84 min Movies Emmy-winning comedy writer Louis C.K. brings h... \n",
"6231 60 min Movies Nursery rhymes and original music for children... \n",
"\n",
" tv_rating \n",
"211 <NA> \n",
"6231 <NA> "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(netflix_titles_converted_df['tv_rating'].dtype)\n",
"netflix_titles_converted_df[netflix_titles_converted_df['tv_rating'].isna()].sample(2).sort_index()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Count missing values"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"show_id 0\n",
"type 0\n",
"title 0\n",
"director 1969\n",
"cast 570\n",
"country 476\n",
"date_added 11\n",
"release_year 0\n",
"rating 10\n",
"duration 0\n",
"listed_in 0\n",
"description 0\n",
"tv_rating 10\n",
"dtype: int64"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"netflix_titles_converted_df.isna().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Integer missing value types\n",
"\n",
"* `np.nan` not compatible with integer type\n",
"* Pandas integer types use `pd.NA`"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cannot convert float NaN to integer\n"
]
},
{
"data": {
"text/plain": [
"0 1.0\n",
"1 2.0\n",
"2 NaN\n",
"dtype: float64"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"try:\n",
" test_series = pd.Series([1, 2, np.nan], dtype='int64')\n",
" display(test_series)\n",
"except ValueError as exc:\n",
" print(exc)\n",
"\n",
"test_series = pd.Series([1, 2, np.nan])\n",
"display(test_series)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 <NA>\n",
"dtype: Int64"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 <NA>\n",
"dtype: Int64"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"test_series = pd.Series([1, 2, np.nan], dtype='Int64')\n",
"display(test_series)\n",
"\n",
"test_series = pd.Series([1, 2, pd.NA], dtype='Int64')\n",
"display(test_series)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 <NA>\n",
"dtype: object"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 <NA>\n",
"dtype: Int64"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"test_series = pd.Series([1, 2, pd.NA])\n",
"display(test_series)\n",
"\n",
"test_series = pd.Series([1, 2, np.nan]).convert_dtypes()\n",
"display(test_series)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment