Skip to content

Instantly share code, notes, and snippets.

@pmarreck
Last active December 5, 2023 02:00
Show Gist options
  • Save pmarreck/91124e761e45d6860834eb046d6b782d to your computer and use it in GitHub Desktop.
Save pmarreck/91124e761e45d6860834eb046d6b782d to your computer and use it in GitHub Desktop.
Split a text file into a directory of files each of which contains 1 line from the original file
#!/usr/bin/env bash
# Ensure a file name is provided
if [ -z "$1" ]
then
echo "No file name provided. Usage: ./split_file.bash <filename>"
exit 1
fi
# Extract the directory and base name from the file
dir_name=$(dirname "$1")
base_name=$(basename "$1")
file_name="${base_name%.*}"
# Create directory to store the split files
mkdir -p "$dir_name/$file_name"
# Read the file line by line
while IFS= read -r line
do
# Hash the line with sha256 and use it as the file name
file_hash=$(echo -n "$line" | cksum | awk '{print $1}')
# Write the line to the new file
echo "$line" > "$dir_name/$file_name/$file_hash.txt"
done < "$1"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment