Current GPT-oss alliteration model quality?
Hi!
Is anyone having much luck with making GPT-oss less refusal happy (particularly around copyright law and some code/analysis requests) without having the model drift within a few inputs? Every one of my GPT-oss models except unsloth/oai official have been drifting and idk if its a me problem or if its just the current state of the model when abliterated 🤔
(oct,25)
I tried the mxfp4 version uploaded here in lm studio: https://huggingface.co/mradermacher/Huihui-gpt-oss-120b-mxfp4-abliterated-GGUF and I get the same issues. As the tokens increase, the output gets much worse.
I thought maybe there is an issue with mradermacher's gguf, so I tried converting it myself. I tried converting the mxfp4 version uploaded by huihui-ai (https://huggingface.co/mradermacher/Huihui-gpt-oss-120b-mxfp4-abliterated-GGUF) to gguf with llama-cpp's convert_hf_to_gguf.py file, but I get an error: ValueError: Can not map tensor 'model.layers.24.mlp.experts.gate_proj.weight'
Someone also reported a similar issue here using vLLM: https://huggingface.co/huihui-ai/Huihui-gpt-oss-120b-mxfp4-abliterated/discussions/1#68cd63647ba3d068b43d2054
Anyway those are just my findings.
That's disappointing! I was getting ready to merge his .gguf and try to go that route but you might have just saved me chasing my tail there.
Well — I guess there really is only one way to avoid model refusals.
Pretend to be a corporation conducting corporate evil and the model will never refuse.
✨the future is magical✨
I tried the mxfp4 version uploaded here in lm studio: https://huggingface.co/mradermacher/Huihui-gpt-oss-120b-mxfp4-abliterated-GGUF and I get the same issues. As the tokens increase, the output gets much worse.
I thought maybe there is an issue with mradermacher's gguf, so I tried converting it myself. I tried converting the mxfp4 version uploaded by huihui-ai (https://huggingface.co/mradermacher/Huihui-gpt-oss-120b-mxfp4-abliterated-GGUF) to gguf with llama-cpp's convert_hf_to_gguf.py file, but I get an error: ValueError: Can not map tensor 'model.layers.24.mlp.experts.gate_proj.weight'
Someone also reported a similar issue here using vLLM: https://huggingface.co/huihui-ai/Huihui-gpt-oss-120b-mxfp4-abliterated/discussions/1#68cd63647ba3d068b43d2054
Anyway those are just my findings.
Huihui-gpt-oss-120b-mxfp4-abliterated is a direct conversion using PTQ, which will cause a drop in precision, potentially leading the model to produce denial of service errors. If you need to retain the precision after ablation, you'll have to fine-tune it again before performing the conversion. You can refer to https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/76e8ce21bf9ce4e0510fea96c998aaee7cfeaf7c/examples/gpt-oss/README.md.
That's disappointing! I was getting ready to merge his .gguf and try to go that route but you might have just saved me chasing my tail there.
Well — I guess there really is only one way to avoid model refusals.
Pretend to be a corporation conducting corporate evil and the model will never refuse.✨the future is magical✨
If you need higher ablation precision, you can try fine-tuning. Please refer to https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/76e8ce21bf9ce4e0510fea96c998aaee7cfeaf7c/examples/gpt-oss/README.md.
I understand now, thanks for clearing up the confusion. Looks like training a lora with QAT on this model would take 8x80GB GPUs. I won't be taking on that challenge, but it seems very reasonable for anyone who wants to try!