Created
November 24, 2023 16:18
-
-
Save do-me/28cb01fbb34dd371471ed67787eb0a11 to your computer and use it in GitHub Desktop.
Pandas function to generate unique hash for each row
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import hashlib | |
| # Function to generate hash for each row | |
| def generate_hash(row): | |
| hash_object = hashlib.sha256() | |
| # Convert each element in the row to a string and update the hash object | |
| for value in row.values: | |
| hash_object.update(str(value).encode('utf-8')) | |
| # Return the hexadecimal representation of the hash | |
| return hash_object.hexdigest()[:30] | |
| # Apply the function to each row and create a new column 'hash' | |
| df['hash'] = df.apply(generate_hash, axis=1) | |
| df | |
| # Useful: check duplicate rows | |
| df[df.duplicated(keep="first")] | |
| # Keep only first duplicate | |
| df.drop_duplicates("OI",keep="first").reset_index(drop=True) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment