function getEachForSource(source) {
    if (source.length < 40) {
      return UniqueArrayWrapper.prototype.eachNoCache;
    } else if (source.length < 100) {
      return UniqueArrayWrapper.prototype.eachArrayCache;
    } else {
      return UniqueArrayWrapper.prototype.eachSetCache;
    }
  }
My latest findings here...

So I hadn't really given the set-based approach enough credit. The main issue
was that my Set implementation was totally not optimized at all. After pretty
heavily optimizing it (just take a look; it's a monstrosity now!), it now
becomes the fastest option for much smaller values of N.