Kwaipilot

/

KAT-Dev

like 176

Follow

Kwaipilot 329

This seem to be a problem of quantization from mradermacher, the DevQuasar version performs better then GLM4.5-Air and Qwen3-coder30B

Thanks for the detailed feedback! We’ve open-sourced an FP8 version that avoids the Q8 quantization issues. Please give it a try and let us know how it works for you.
https://huggingface.co/Kwaipilot/KAT-Dev-FP8

Cant test .

• The RTX 3090 (Ampere architecture) does not support FP8 (8-bit floating point) math.
• AMD Radeon Instinct MI50 INT8: Not supported in hardware on MI50. Vega 20 does not have native INT8 matrix acceleration
• AMD Ryzen AI Max+ 395 processor does not support FP8 (8-bit floating point) operations. Its AI acceleration is primarily focused on INT8 precision, facilitated by the integrated XDNA 2 Neural Processing Unit (NPU).

So a quantized version Q8, or a INT8 for RTX 3090 is best option so most users can use this. AMD Ryzen AI Max+ 395 supports INT8 too.

Can you release KAT-Dev-72B-Exp INT8 too or quantized Q8?

Appreciate your suggestion — the Q8 quantized build of KAT-Dev-72B-Exp is scheduled to be ready this week.

Appreciate your suggestion — the Q8 quantized build of KAT-Dev-72B-Exp is scheduled to be ready this week.

So the issue is only with Q8 quants, not lower like Q6 ?

We plan to roll out the update later this month.

Can you please roll out IQ4 quant too? For those having low VRAM.
🤗🤗🤗🤗🤗🤗🤗🤗🤗
Like here: https://huggingface.co/bartowski/Kwaipilot_KAT-Dev-GGUF
4-bit

IQ4_XS
17.7 GB

Q4_K_S
18.8 GB

IQ4_NL
18.7 GB

Q4_0
18.7 GB

Q4_1
20.6 GB

Q4_K_M
19.8 GB

Thanks for your attention and suggestion! We’ll support these quantization formats in the future. 😊

To the authors make sure you use this script by Mungert, a big impact for Q8 model. Model is smarter !
https://github.com/Mungert69/GGUFModelBuilder/blob/main/model-converter/tensor_list_builder.py

@blankreg
I tested only Q8 as I need for coding, lower quant is not worth the time and power used. In my case 5x 3090

@Reverger
For some reason only Mungert version of Q8 is working way better with Roocode, provides better code output (less failed tool calls or none at all, better code that works better) compared to Bartowski or others.
https://huggingface.co/Mungert/KAT-Dev-GGUF

@akierum

For some reason only Mungert version of Q8 is working way better with Roocode, provides better code output (less failed tool calls or none at all, better code that works better) compared to Bartowski or others.

Very helpful notice, thank you!

I readressed it to Unsloth here:
https://huggingface.co/spaces/unsloth/README/discussions/15#68ee23eba7599344193e2caa

Somebody still needs to dig into tool calling problems with roocode.

My lastest template fix (or rather KAT-DEV fix itself) still gives these errors:

{% macro render_extra_keys(json_dict, handled_keys) %}
{%- if json_dict is mapping %}
{%- for json_key in json_dict if json_key not in handled_keys %}
{%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
{%- else %}
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{% endmacro %}

{%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}

{%- if not tools is defined %}
{%- set tools = [] %}
{%- endif %}

{%- if system_message is defined %}
{{- "system\n" + system_message + "" }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- "system\nYou are a helpful AI assistant that can interact with a computer to solve tasks." }}
{%- endif %}
{%- endif %}

{%- if tools is iterable and tools | length > 0 %}
{{- "\n\n# Tools\n\nYou have access to the following functions:\n\ntools>" }}
{%- for tool in tools %}
{%- if tool.function is defined %}
{%- set tool = tool.function %}
{%- endif %}
{{- "\n\n" ~ tool.name ~ "" }}
{%- if tool.description is defined %}
{{- '\n' ~ (tool.description | trim) ~ '' }}
{%- endif %}
{{- '\n' }}
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{{- '\n' }}
{{- '\n' ~ param_name ~ '' }}
{%- if param_fields.type is defined %}
{{- '\n' ~ (param_fields.type | string) ~ '' }}
{%- endif %}
{%- if param_fields.description is defined %}
{{- '\n' ~ (param_fields.description | trim) ~ '' }}
{%- endif %}
{%- set handled_keys = ['name', 'type', 'description'] %}
{{- render_extra_keys(param_fields, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{%- endif %}
{% set handled_keys = ['type', 'properties'] %}
{{- render_extra_keys(tool.parameters, handled_keys) }}
{{- '\n' }}
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
{{- render_extra_keys(tool, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{{- "\n" }}
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...> block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your content, do not tell the function calls\n' }}
{%- endif %}

{%- if system_message is defined %}
{{- '\n' }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- '\n' }}
{%- endif %}
{%- endif %}

{%- for message in loop_messages %}
{%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
{{- '' + message.role }}
{%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
{{- '\n' + message.content | trim + '\n' }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n\n<function=' + tool_call.name + '>\n' }}
{%- if tool_call.arguments is defined %}
{%- for args_name, args_value in tool_call.arguments|items %}
{{- '<parameter=' + args_name + '>\n' }}
{%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
{{- args_value }}
{{- '\n\n' }}
{%- endfor %}
{%- endif %}
{{- '\n' }}
{%- endfor %}
{{- '\n' }}
{%- elif message.role == "user" %}
{{- 'user\n' + message.content + '\n' }}
{%- elif message.role == "system" %}
{{- 'system\n' + message.content + '\n' }}
{%- elif message.role == "assistant" %}
{{- 'assistant\n' + message.content + '\n' }}
{%- elif message.role == "tool" %}
{%- set tool_content = message.content %}
{%- if tool_content.startswith('[tool_call] ') %}
{%- set tool_content = tool_content[10:] | trim }
{%- elif tool_content.startswith('[tool_name] ') %}
{%- set tool_content = tool_content[11:] | trim }
{%- endif %}
{{- 'tool>\n' + tool_content + '\n\n' }}
{%- else %}
{{- '' + message.role + '\n' + message.content + '\n' }}
{%- endif %}
{%- endfor %}

{%- if add_generation_prompt %}
{{- 'assistant\n' }}
{%- endif %}

when debug mode listing for file that does not exist:

Error
Roo tried to use search_files without value for required parameter 'path'. Retrying...

code mode

Roo said
Now I'll create the manifest.json file compatible with Chrome v109. Chrome v109 requires manifest version 3, which has several important differences from version 2:

Checkpoint

Error
Roo tried to use write_to_file without value for required parameter 'path'. Retrying...

roo code mode tries to read folder as file:

Roo said
Now I need to update the background.js file. In Chrome v109 with Manifest V3, the background script runs as a service worker, which has some differences from the previous event pages. The current code should work mostly as-is, but I need to make sure it follows service worker best practices:

Roo wants to read this file "project folder"

Error
Roo tried to use read_file without value for required parameter 'args (containing valid file paths)'. Retrying...

Error
Roo tried to use apply_diff without value for required parameter 'path'. Retrying...

Error
Roo tried to use apply_diff without value for required parameter 'path'. Retrying...