Jitter is how much variation there is in your latency. Basically, it represents how consistent your delay is. For example, if your average ping is 50ms, but half of the packets made the trip in 30ms and half in 70ms, you would have an average jitter of 20ms.
Jitter can uniquely negatively affect your connection because it can cause packets to arrive out of order. Sometimes this may result in things actually being handled backwards, or it may mean that the first, slower packet must be waited on, causing many packets to experience the delay and filling up the server's buffers. If the buffers get full or the wait gets too long, the abnormally slow packet may be dropped.
This most affects VoIP. While consistent latency may be handled okay and only cause some delay in people's speech being received, if jitter exceeds the buffer, it will cause parts of one's sentences to be dropped or scrambled.