File utilities examples

useful_file_utils.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples of useful file utilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The os library for listing files and making directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say I have a directory on my computer filled with *.ab1* files that I need to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "abi_dir = '/Users/triznam/Downloads/rev'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then use os.listdir to get a list of all of the files in that directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in os.listdir(abi_dir):\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice Macs includes this weird \".DS_Store\" file in every folder, so you should build in some sort of filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in os.listdir(abi_dir):\n",
    "    if file.endswith('.ab1'):\n",
    "        print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, say for example, you want to move all of these files to another directory. First, let's make a new directory with *os.makedirs*. This command is similar to `mkdir -p` in Unix, where is automatically creates parent directories if needed, so I used this instead of *os.mkdir*.\n",
    "\n",
    "I also include a check to see if the directory already exists, since it will error out if it has already been made. I honestly didn't realize this until I got through writing this up, and then tried Restart and Run All on the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "good_abis_path = '/Users/triznam/Documents/ab1_files/good_ones/'\n",
    "if not os.path.exists(good_abis_path):\n",
    "    os.makedirs(good_abis_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copying files with shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some reason, the command to copy files is in a different library than os 🤷‍♂️, so we'll have to import the *shutil* library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run through the same for-loop as before to copy each file to the new directory. The *shutil.copyfile* command requires the full path, so we'll also need to use *os.path.join* to combine the directory with the file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current filepath: /Users/triznam/Downloads/rev/647767_HCO2198_E01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647767_HCO2198_E01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647766_HCO2198_C01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647766_HCO2198_C01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647768_HCO2198_F01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647768_HCO2198_F01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647769_HCO2198_G01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647769_HCO2198_G01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647764_HCO2198_A01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647764_HCO2198_A01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647762_HCO2198_D01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647762_HCO2198_D01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647765_HCO2198_B01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647765_HCO2198_B01.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647763_HCO2198_B07.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647763_HCO2198_B07.ab1\n",
      "--------\n",
      "Current filepath: /Users/triznam/Downloads/rev/647770_HCO2198_H01.ab1\n",
      "New filepath: /Users/triznam/Documents/ab1_files/good_ones/647770_HCO2198_H01.ab1\n",
      "--------\n"
     ]
    }
   ],
   "source": [
    "for file in os.listdir(abi_dir):\n",
    "    if file.endswith('.ab1'):\n",
    "        current_file_path = os.path.join(abi_dir, file)\n",
    "        print('Current filepath: {}'.format(current_file_path))\n",
    "        new_file_path = os.path.join(good_abis_path, file)\n",
    "        print('New filepath: {}'.format(new_file_path))\n",
    "        shutil.copyfile(current_file_path, new_file_path)\n",
    "        print('-'*8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a `ls` command to manually check that the files are now in the new directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "647762_HCO2198_D01.ab1 647765_HCO2198_B01.ab1 647768_HCO2198_F01.ab1\r\n",
      "647763_HCO2198_B07.ab1 647766_HCO2198_C01.ab1 647769_HCO2198_G01.ab1\r\n",
      "647764_HCO2198_A01.ab1 647767_HCO2198_E01.ab1 647770_HCO2198_H01.ab1\r\n"
     ]
    }
   ],
   "source": [
    "!ls ~/Documents/ab1_files/good_ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zipping a directory with shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a *zipfile* library in the Python standard library, but I just learned while Googling that there's an easier version right in shutil which is more straightforward, so let's use that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say I wanted to zip up the \"good\" ab1 files and put it on the Desktop, I would use the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/triznam/Desktop/good_ones.zip'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shutil.make_archive('/Users/triznam/Desktop/good_ones', #The filepath of the zipfile (minus .zip)\n",
    "                    'zip', #What we want to zip it with -- also can use \n",
    "                    good_abis_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}