Lately while using Go I had an interesting situation, I had a Slice which contained duplicate integer values and I needed to find a way to get rid of the duplicates.

Having worked with other languages I found that the solution could in some cases, be a one liner. This is the case for C#, where one can leverage Linq.

Using Linq one can simply tell the compiler what you want. Of course Linq hides the implementation, but that after all is the underlying philosophy of Linq. Tell the compiler ‘what’ you want and not ‘how’ you want it, let the compiler take care of the ‘how’ for you.

C#

Here is how you would remove duplicate values from an List in C# using Linq.

using System;
using System.Collections.Generic;
using System.Linq;

var duplicates = new List<int>() { 113, 95, 113};
var unique = duplicates.Distinct().ToList();

Console.WriteLine(String.Join(", ", unique));

Nodejs

We could also take a look at Javascript which does not have a built in way of removing duplicates, but because it’s Javascript there are so many useful utility libraries out there which one can use. For example if we look at lodash you can use the uniq function to achieve the same goal.

const _ = require("lodash")

_.uniq([113, 95, 113]);

Go

Unfortunately there is no such thing as Linq for Go, and I haven’t found any utility libraries for Go which can do the same as the examples above. This means that you will have to roll your own implementation when it comes to Go. The simplest way I know is the following.

package main

import (
"fmt"
)

func distinct(duplicates []int) []int {
    keys := make(map[int]bool)
    list := []int{}
    for _, entry := range duplicates {
        if _, value := keys[entry]; !value {
            keys[entry] = true
            list = append(list, entry)
        }
    }
    return list
}

func main() {
    duplicates := []int{113, 95, 113}
    unique := distinct(duplicates)
    fmt.Println(unique)
}

Of course it’s not a cheap way of doing it, the extra map could be avoided with more complicated solutions but then you might end up with difficult to read code, and code which others are not able to maintain. That is why I like simple solutions.

Finally one drawback of this solution is that you will need a different implementation for each type of Slice you plan to work with, be it int or string or any other type, which leads to a lot of code duplication, which we all know is bad.