Skip to content

Instantly share code, notes, and snippets.

@jeffreybaird
Created October 9, 2012 21:58
Show Gist options
  • Select an option

  • Save jeffreybaird/3861705 to your computer and use it in GitHub Desktop.

Select an option

Save jeffreybaird/3861705 to your computer and use it in GitHub Desktop.
require 'csv'
def remove_dupes
uniq_csv = []
CSV.foreach("input.csv","r+") {|csv| uniq_csv << csv }
uniq_csv.uniq!.compact.compact
end
CSV.open("ouput.csv","w+") do |csv|
remove_dupes.each { |entry| csv << entry }
end
@gstark
Copy link
Copy Markdown

gstark commented Oct 10, 2012

If the source CSV isn't too large, could you have just done:

def remote_dupes
   CSV.read("input.csv").uniq
end

(I don't know your data so the #compact and perhaps the double #compact is required)

Also, there is no reason to use the self modifying version "uniq!" where I think you more mean the returning version of "uniq"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment