JohnArchieMckown/nodups.awk

Last active August 29, 2015 14:21

Star (0) You must be signed in to star a gist
Fork (1) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/JohnArchieMckown/d4e051e3f48d60186370.js"></script>
Save JohnArchieMckown/d4e051e3f48d60186370 to your computer and use it in GitHub Desktop.

Download ZIP

AWK code to remove duplicates lines. This does not require sorting then input, but may use a lot of memory. It simply uses an "associative array" in which the index is the data line. From an tweet by Tim Chase (@Gumnos)

Raw

	#!/bin/awk
	!a[$0]++

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment