Does it possible to create a version without MTP layer to save some VRAM

by adonishong - opened Aug 2

Aug 2

Appreciate for your work, does it possible to create a version without MTP layer to save some VRAM as described in title?

Aug 13

I think vLLM breaks with quantized MTP layer currently so it would break compatibility?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment