Code:  static int smacker_decode_tree(BitstreamContext *bc, HuffContext *hc,
                                uint32_t prefix, int length)
 {
     if (!bitstream_read_bit(bc)) { // Leaf
         if(hc->current >= 256){
             av_log(NULL, AV_LOG_ERROR, "Tree size exceeded!\n");
            return AVERROR_INVALIDDATA;
        }
        if(length){
            hc->bits[hc->current] = prefix;
            hc->lengths[hc->current] = length;
        } else {
            hc->bits[hc->current] = 0;
            hc->lengths[hc->current] = 0;
        }
        hc->values[hc->current] = bitstream_read(bc, 8);
        hc->current++;
        if(hc->maxlength < length)
            hc->maxlength = length;
        return 0;
    } else { //Node
        int r;
        length++;
        r = smacker_decode_tree(bc, hc, prefix, length);
        if(r)
            return r;
        return smacker_decode_tree(bc, hc, prefix | (1 << (length - 1)), length);
    }
}

Vulnerability Type: DoS Overflow 
CWE ID: CWE-119
Summary: In Libav through 11.11 and 12.x through 12.1, the smacker_decode_tree function in libavcodec/smacker.c does not properly restrict tree recursion, which allows remote attackers to cause a denial of service (bitstream.c:build_table() out-of-bounds read and application crash) via a crafted Smacker stream.
Commit Message: smacker: add sanity check for length in smacker_decode_tree()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Bug-Id: 1098
Cc: libav-stable@libav.org
Signed-off-by: Sean McGovern <gseanmcg@gmail.com>