Skip to content

Instantly share code, notes, and snippets.

@aculich

aculich/remove-empty-columns.csv

Last active July 6, 2026 05:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save aculich/fb2769414850d20911eb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save aculich/fb2769414850d20911eb to your computer and use it in GitHub Desktop.
Download ZIP
remove-empty-columns
Raw
remove-empty-columns.csv
foo bar baz
a 1
b 2
c
4
e 5
Display the source blob
Display the rendered blob
Raw
remove-empty-columns.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To drop all empty columns (but still keeping the headers) using the Python [Pandas library](http://pandas.pydata.org/) we can use the following 4-line script to read in the csv file, drop the columns where **all** the elements are missing, and save the data to a new csv file."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from pandas.io.parsers import read_csv\n",
"data = read_csv('remove-empty-columns.csv')\n",
"filtered_data = data.dropna(axis='columns', how='all')\n",
"filtered_data.to_csv('empty-columns-removed.csv', index=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As shown below, the sample data included in the csv file has 3 columns which contain missing values.\n",
"\n",
"The second column, labeled **bar**, is completely empty except the header; columns like this should be dropped. The other columns contain data, but should not be dropped even though they contain some missing values."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>foo</th>\n",
" <th>bar</th>\n",
" <th>baz</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>a</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>b</td>\n",
" <td>NaN</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>c</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>e</td>\n",
" <td>NaN</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" foo bar baz\n",
"0 a NaN 1.0\n",
"1 b NaN 2.0\n",
"2 c NaN NaN\n",
"3 NaN NaN 4.0\n",
"4 e NaN 5.0"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the [pandas.DataFrame.dropna()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html) function with the **columns** axis we can drop any column where **all** the entries are **NaN** (missing values)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>foo</th>\n",
" <th>baz</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>a</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>b</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>c</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>e</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" foo baz\n",
"0 a 1.0\n",
"1 b 2.0\n",
"2 c NaN\n",
"3 NaN 4.0\n",
"4 e 5.0"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"filtered_data = data.dropna(axis='columns', how='all')\n",
"filtered_data"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Raw
remove-empty-columns.py
#!/usr/bin/env python
# coding: utf-8
# To drop all empty columns (but still keeping the headers) using the Python [Pandas library](http://pandas.pydata.org/) we can use the following 4-line script to read in the csv file, drop the columns where **all** the elements are missing, and save the data to a new csv file.
# In[1]:
from pandas.io.parsers import read_csv
data = read_csv('remove-empty-columns.csv')
filtered_data = data.dropna(axis='columns', how='all')
filtered_data.to_csv('empty-columns-removed.csv', index=False)
# As shown below, the sample data included in the csv file has 3 columns which contain missing values.
#
# The second column, labeled **bar**, is completely empty except the header; columns like this should be dropped. The other columns contain data, but should not be dropped even though they contain some missing values.
# In[2]:
data
# Using the [pandas.DataFrame.dropna()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html) function with the **columns** axis we can drop any column where **all** the entries are **NaN** (missing values).
# In[3]:
filtered_data = data.dropna(axis='columns', how='all')
filtered_data
@EricPeter

EricPeter commented Jun 18, 2019

Copy link
Copy Markdown

It works thanx

@ganeshan

ganeshan commented Jan 23, 2020

Copy link
Copy Markdown

The statement
filtered_data.to_csv('empty-columns-removed.csv')
will write first column with index values and the column header will be empty. To prevent writing index column, use this code instead.
filtered_data.to_csv('empty-columns-removed.csv', index=False)
Thanks very much for the script.

@JayProXter

JayProXter commented Jan 7, 2021

Copy link
Copy Markdown

doesn't work for me ;(

@zhixe

zhixe commented Mar 18, 2021

Copy link
Copy Markdown

doesn't work for me :(

@aculich

aculich commented Mar 18, 2021

Copy link
Copy Markdown
Author

@zhixe @JayProXter i've updated the script based on the suggestion from @ganeshan

Does it work for you now? If not, can you let me know what errors or unexpected behaviors you are seeing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment