<p>The obvious problem is that if you use a cryptographic pseudorandom number generator (PRNG), the correctness of your algorithm is conditional on a complexity conjecture. However, usually this can be avoided, because the full strength of cryptographic pseudorandmness is usually a huge overkill for streaming. If your streaming algorithm uses a small amount of space, then what you need from your PRNG is that its output cannot be distinguished from a stream of random bits by a small space algorithm. Such a PRNG was <a href="http://www.math.ucsd.edu/~sbuss/CourseWeb/Math268_2013W/Nisan_PRG.pdf" rel="nofollow">constructed by Nisan</a>, and the proof of pseudorandomness is <em>unconditional</em>. </p>

<p>A seminal paper in streaming algorithms that used Nisan's generator is <a href="http://www.cs.dartmouth.edu/~ac/Teach/CS49-Fall11/Papers/indyk-stable.pdf" rel="nofollow">Indyk's work on sketches</a> of $L_1$. Since the sketch uses Cauchy random variables for which the mean and the variance do not exist, the usual moment calculations do not work, so it is hard to imagine how to use $k$-wise independence. That's why the heavy hammer of Nisan's generator comes handy.</p>

<p>A problem with these heavy hammers is that they come at a cost in space. The seed needs to be stored in memory, and for Nisan's generator the seed length is roughly $S\log R$ where $R$ is the number of random bits and $S$ is the space complexity of the algorithm. So you end up multiplying the space complexity by roughly the logarithm of the stream length $n$. If you can use $k$-wise independence you usually <em>add</em> a term like $O(k\log n)$ to the space complexity. </p>

CC BY-SA 3.0
2014-12-03T22:14:20.573Z
Thanks!  However, I *really* don't care about avoiding complexity conjectures.  Truly, I don't.  Therefore, Nisan's PRNG is solving a problem I don't have.  I'd prefer to take the practitioner's perspective, not a complexity theorist's: assume we are willing to accept strong assumptions (e.g., SHA3 can be modeled as a random oracle).  Then what performance is achievable with sketches?  Is it better than with Nisan's generator?  e.g., crypto hashes won't incur any cost in space, so won't they be preferable to using Nisan's PRNG or to $k$-wise independent hashes, from a practical perspective?
D.W.
CC BY-SA 3.0
2014-12-03T23:00:29.630Z
Just to make sure, SHA3 being like a random oracle is afaik not known even if you assume standard complexity conjectures. But ok, let's say you have a random oracle. Then algorithms based on stable distributions (sketches for hamming norm, $L_1$, $L_2$) and Flajolet-Martin random counting become somewhat simpler and easier to analyze. For example $L_2$ is just JL-lemma. I think for all these problems there are optimal or very close to optimal space algorithms that do not assume magic boxes, but they can be more complicated.
Sasho Nikolov
CC BY-SA 3.0
2014-12-03T23:32:11.840Z
Sasho, yes, that's exactly what I had in mind.  So does anything change?  Does it change the decision of which sketch to use?  Does it make a significant difference to the space needed or running time?  Sounds like you're saying it doesn't change the best achievable performance, it just means you can achieve that best achievable performance using simpler constructions -- is that right?  If so, cool, that's exactly the sort of thing I was asking about!  (Yeah, I know random oracle assumptions don't follow from any complexity conjecture.)  Thank you!
D.W.
CC BY-SA 3.0
2014-12-03T19:57:03.073Z
2014-12-03T19:57:03.073Z
Sasho Nikolov
<p>Another reason for not using cryptographic algorithms in practice is speed. In the streaming setting, we typically do not want to spend too long processing each item in the stream. Computing k cryptographic hash functions will be much more expensive than computing k fast non-cryptographic hash functions, e.g. MurmurHash.</p>

<p>In practice, I think most people use Kirsch and Mitzenmacher's result (<a href="http://www.eecs.harvard.edu/~michaelm/postscripts/rsa2008.pdf" rel="nofollow">http://www.eecs.harvard.edu/~michaelm/postscripts/rsa2008.pdf</a>) to reduce the number of hashes, but I would imagine that computing two SHA3 hashes would still take significantly longer than computing two Murmur hashes.</p>

CC BY-SA 3.0
2014-12-04T11:15:58.183Z
2014-12-04T11:15:58.183Z
bandini
<p>I've been reading about sketches for processing streaming data (the CountMin sketch, the Count sketch, the tug-of-war sketch, FM sketches, etc.).  They use hash functions that are required to be 2-independent or 4-independent or $k$-independent for some small $k$.</p>

<p>However from cryptography we have fast hash functions that are very good candidates to essentially perfect: $k$-independent for every value of $k$ you could ever wish for, and so on.  We can essentially treat them as uniformly chosen from the set of all functions: as far as we know, no polynomial-time computation can tell that this is not the case.  I suspect many practitioners might be happy to use these cryptographic hash functions, if it led to improved performance or better parameters for the sketches.</p>

<p>Does it change anything about the design and analysis of sketches, if we assume we have perfect hash functions like this?  Can we get asymptotically better parameters if we assumed our hash function was perfect?  Or, can we get simpler sketch constructions?</p>

<p>(For instance, I've noticed that the analysis of existing sketches is often somewhat conservative: e.g., it uses the Markov bound or Chebyshev bound.  If we had a perfect hash function we could use a Chernoff-style bound.  Are there any sketches where this makes a significant difference, or is it just a small constant factor in the bound?)</p>

<ds.algorithms><hash-function><online-algorithms><streaming-algorithms>
Sketches, using ideal hash functions
D.W.
# Sketches, using ideal hash functions

(Asked by: username_0 on 2014-12-03)

I've been reading about sketches for processing streaming data (the CountMin sketch, the Count sketch, the tug-of-war sketch, FM sketches, etc.). They use hash functions that are required to be 2-independent or 4-independent or $k$-independent for some small $k$.

However from cryptography we have fast hash functions that are very good candidates to essentially perfect: $k$-independent for every value of $k$ you could ever wish for, and so on. We can essentially treat them as uniformly chosen from the set of all functions: as far as we know, no polynomial-time computation can tell that this is not the case. I suspect many practitioners might be happy to use these cryptographic hash functions, if it led to improved performance or better parameters for the sketches.

Does it change anything about the design and analysis of sketches, if we assume we have perfect hash functions like this? Can we get asymptotically better parameters if we assumed our hash function was perfect? Or, can we get simpler sketch constructions?

(For instance, I've noticed that the analysis of existing sketches is often somewhat conservative: e.g., it uses the Markov bound or Chebyshev bound. If we had a perfect hash function we could use a Chernoff-style bound. Are there any sketches where this makes a significant difference, or is it just a small constant factor in the bound?)

**Tags:** ds.algorithms, hash-function, online-algorithms, streaming-algorithms

---

## 2 answers

### Answer by username_1 on 2014-12-03. Score: 1

The obvious problem is that if you use a cryptographic pseudorandom number generator (PRNG), the correctness of your algorithm is conditional on a complexity conjecture. However, usually this can be avoided, because the full strength of cryptographic pseudorandmness is usually a huge overkill for streaming. If your streaming algorithm uses a small amount of space, then what you need from your PRNG is that its output cannot be distinguished from a stream of random bits by a small space algorithm. Such a PRNG was [constructed by Nisan](http://www.math.ucsd.edu/~sbuss/CourseWeb/Math268_2013W/Nisan_PRG.pdf), and the proof of pseudorandomness is *unconditional*.

A seminal paper in streaming algorithms that used Nisan's generator is [Indyk's work on sketches](http://www.cs.dartmouth.edu/~ac/Teach/CS49-Fall11/Papers/indyk-stable.pdf) of $L\_1$. Since the sketch uses Cauchy random variables for which the mean and the variance do not exist, the usual moment calculations do not work, so it is hard to imagine how to use $k$-wise independence. That's why the heavy hammer of Nisan's generator comes handy.

A problem with these heavy hammers is that they come at a cost in space. The seed needs to be stored in memory, and for Nisan's generator the seed length is roughly $S\log R$ where $R$ is the number of random bits and $S$ is the space complexity of the algorithm. So you end up multiplying the space complexity by roughly the logarithm of the stream length $n$. If you can use $k$-wise independence you usually *add* a term like $O(k\log n)$ to the space complexity.

> **Comments:**
> **username_0**: Thanks!  However, I *really* don't care about avoiding complexity conjectures.  Truly, I don't.  Therefore, Nisan's PRNG is solving a problem I don't have.  I'd prefer to take the practitioner's perspective, not a complexity theorist's: assume we are willing to accept strong assumptions (e.g., SHA3 can be modeled as a random oracle).  Then what performance is achievable with sketches?  Is it better than with Nisan's generator?  e.g., crypto hashes won't incur any cost in space, so won't they be preferable to using Nisan's PRNG or to $k$-wise independent hashes, from a practical perspective?  
> **username_1**: Just to make sure, SHA3 being like a random oracle is afaik not known even if you assume standard complexity conjectures. But ok, let's say you have a random oracle. Then algorithms based on stable distributions (sketches for hamming norm, $L_1$, $L_2$) and Flajolet-Martin random counting become somewhat simpler and easier to analyze. For example $L_2$ is just JL-lemma. I think for all these problems there are optimal or very close to optimal space algorithms that do not assume magic boxes, but they can be more complicated.  
> **username_0**: Sasho, yes, that's exactly what I had in mind.  So does anything change?  Does it change the decision of which sketch to use?  Does it make a significant difference to the space needed or running time?  Sounds like you're saying it doesn't change the best achievable performance, it just means you can achieve that best achievable performance using simpler constructions -- is that right?  If so, cool, that's exactly the sort of thing I was asking about!  (Yeah, I know random oracle assumptions don't follow from any complexity conjecture.)  Thank you!  

---

### Answer by username_2 on 2014-12-04. Score: 1

Another reason for not using cryptographic algorithms in practice is speed. In the streaming setting, we typically do not want to spend too long processing each item in the stream. Computing k cryptographic hash functions will be much more expensive than computing k fast non-cryptographic hash functions, e.g. MurmurHash.

In practice, I think most people use Kirsch and Mitzenmacher's result (<http://www.eecs.harvard.edu/~michaelm/postscripts/rsa2008.pdf>) to reduce the number of hashes, but I would imagine that computing two SHA3 hashes would still take significantly longer than computing two Murmur hashes.


---

