How to remove duplicate entries from a file without sorting

 

Usually whenever we want to remove duplicate entries or lines from a file, we need to sort the entries and then eliminate the duplicates using "uniq" command.

But if we want to remove the duplicates and preserve the entries in the same order or sequence, here is the way:

 Sample file:

$ cat sample.txt AAAA FFFF BBBB BBBB CCCC AAAA FFFF DDDD

Using the "uniq" command without sorting the file will not remove all duplicates.

$ uniq sample.txt AAAA FFFF BBBB CCCC AAAA FFFF DDDD

The "sort" command has an option (-u) to sort and uniq, but we will lose the original sequence of the entries

$ sort -u sample.txt AAAA BBBB CCCC DDDD FFFF

Using sort and then uniq commands will remove the duplicates, but sequence?? (Same as above)

$ sort sample.txt | uniq AAAA BBBB CCCC DDDD FFFF

Finally, here is the solution using AWK:

$ awk ' !x[$0]++' sample.txt AAAA FFFF BBBB CCCC DDDD