variable_cache.py compatibility for v4.57.2 / python3.12
#12
by
NePe
- opened
No description provided.
Can't find a better way to fix the cache.
Usage:
past_key_values = model.getVariableCache(batch_size=1, max_cache_len=4096)
model.generate(... ,past_key_values=past_key_values)