Upload folder using huggingface_hub

deploy_guidance.md

@@ -0,0 +1,106 @@
+# Kimi-K2-Thinking Deployment Guide
+
+> [!Note]
+> This guide only provides some examples of deployment commands for Kimi-K2-Thinking, which may not be the optimal configuration. Since inference engines are still being updated frequenty,  please continue to follow the guidance from their homepage if you want to achieve better inference performance.
+
+> kimi_k2 reasoning parser and other related features have been merged into vLLM/sglang and will be available in the next release. For now, please use the nightly build Docker image.
+
+
+## vLLM Deployment
+
+The smallest deployment unit for Kimi-K2-Thinking INT4 weights with 256k seqlen on mainstream H200 platform is a cluster with 8 GPUs with Tensor Parallel (TP).  
+Running parameters for this environment are provided below. For other parallelism strategies, please refer to updates of official documents. 
+
+
+### Tensor Parallelism
+
+Here is a sample launch command with TP=8:
+
+``` bash
+
+vllm serve $MODEL_PATH \
+  --served-model-name kimi-k2-thinking \
+  --trust-remote-code \
+  --tensor-parallel-size 8 \
+  --enable-auto-tool-choice \
+  --max-num-batched-tokens 32768 \
+  --tool-call-parser kimi_k2 \
+  --reasoning-parser kimi_k2
+```
+
+**Key parameter notes:**
+- `--enable-auto-tool-choice`: Required when enabling tool usage.
+- `--tool-call-parser kimi_k2`: Required when enabling tool usage.
+- `--reasoning-parser kimi_k2`: Required for correctly processing reasoning content. 
+- `--max-num-batched-tokens 32768`: Using chunk prefill to reduce peak memory usage.
+
+
+## SGLang Deployment
+
+Similarly, here are the examples using TP in SGLang for Deployment. 
+
+### Tensor Parallelism
+
+Here is the simple example code to run TP8 on H200 in a sigle node:
+
+``` bash
+python -m sglang.launch_server --model-path $MODEL_PATH --tp 8 --trust-remote-code  --tool-call-parser kimi_k2 --reasoning_parser kimi_k2
+```
+
+**Key parameter notes:**
+- `--tool-call-parser kimi_k2`: Required when enabling tool usage.
+- `--reasoning_parser kimi_k2`: Required for correctly processing reasoning content. 
+
+
+## KTransformers Deployment
+
+### KTransformers+SGLang Inference Deployment
+
+Launch with KTransformers + SGLang for CPU+GPU heterogeneous inference:
+
+``` bash
+python -m sglang.launch_server \
+  --model path/to/Kimi-K2-Thinking/ \
+  --kt-amx-weight-path path/to/Kimi-K2-Instruct-CPU-weight/ \
+  --kt-cpuinfer 56 \
+  --kt-threadpool-count 2 \
+  --kt-num-gpu-experts 200 \
+  --kt-amx-method AMXINT4 \
+  --trust-remote-code \
+  --mem-fraction-static 0.98 \
+  --chunked-prefill-size 4096 \
+  --max-running-requests 37 \
+  --max-total-tokens 37000 \
+  --enable-mixed-chunk \
+  --tensor-parallel-size 8 \
+  --enable-p2p-check \
+  --disable-shared-experts-fusion
+```
+
+Achieves 577.74 tokens/s Prefill and 45.91 tokens/s Decode (37-way concurrency) on 8× NVIDIA L20 + 2× Intel 6454S.  
+
+More details: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2-Thinking.md
+
+
+### KTransformers+LLaMA-Factory Fine-tuning Deployment
+
+You can use below command to run LoRA SFT with KT+llamafactory. 
+
+``` bash 
+# For LoRA SFT
+USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml
+# For Chat with model after LoRA SFT
+llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml
+# For API with model after LoRA SFT
+llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml
+```
+
+This achieves end-to-end LoRA SFT Throughput: 46.55 token/s on 2× NVIDIA 4090 + Intel 8488C with 1.97T RAM and 200G swap memory.  
+
+More details refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/SFT_Installation_Guide_KimiK2.md.
+
+## Others
+
+Kimi-K2-Thinking reuses the `DeepSeekV3CausalLM` architecture and convert it's weight into proper shape to save redevelopment effort. To let inference engines distinguish it from DeepSeek-V3 and apply the best optimizations, we set `"model_type": "kimi_k2"` in `config.json`.
+
+If you are using a framework that is not on the recommended list, you can still run the model by manually changing `model_type` to "deepseek_v3" in `config.json` as a temporary workaround. You may need to manually parse tool calls in case no tool call parser is available in your framework.

tool_call_guidance.md

@@ -0,0 +1,258 @@
+## Tool Calling
+To enable the tool calling feature, you may need to set certain tool calling parser options when starting the service. See [deploy_guidance](./deploy_guidance.md) for details.
+In Kimi-K2, a tool calling process includes:
+- Passing function descriptions to Kimi-K2
+- Kimi-K2 decides to make a function call and returns the necessary information for the function call to the user
+- The user performs the function call, collects the call results, and passes the function call results to Kimi-K2
+- Kimi-K2 continues to generate content based on the function call results until the model believes it has obtained sufficient information to respond to the user
+
+### Preparing Tools
+Suppose we have a function `get_weather` that can query the weather conditions in real-time. 
+This function accepts a city name as a parameter and returns the weather conditions. We need to prepare a structured description for it so that Kimi-K2 can understand its functionality.
+
+```python
+def get_weather(city):
+    return {"weather": "Sunny"}
+
+# Collect the tool descriptions in tools
+tools = [{
+    "type": "function",
+    "function": {        
+        "name": "get_weather", 
+        "description": "Get weather information. Call this tool when the user needs to get weather information", 
+         "parameters": {
+              "type": "object",
+              "required": ["city"], 
+              "properties": { 
+                  "city": { 
+                      "type": "string", 
+                      "description": "City name", 
+                }
+            }
+        }
+    }
+}]
+
+# Tool name->object mapping for easy calling later
+tool_map = {
+    "get_weather": get_weather
+}
+```
+### Chat with tools
+We use `openai.OpenAI` to send messages to Kimi-K2 along with tool descriptions. Kimi-K2 will autonomously decide whether to use and how to use the provided tools. 
+If Kimi-K2 believes a tool call is needed, it will return a result with `finish_reason='tool_calls'`. At this point, the returned result includes the tool call information. 
+After calling tools with the provided information, we then need to append the tool call results to the chat history and continue calling Kimi-K2. 
+Kimi-K2 may need to call tools multiple times until the model believes the current results can answer the user's question. We should check `finish_reason` until it is not `tool_calls`.
+
+The results obtained by the user after calling the tools should be added to `messages` with `role='tool'`.
+
+```python
+import json
+from openai import OpenAI
+model_name='moonshotai/Kimi-K2-Thinking'
+client = OpenAI(base_url=endpoint, 
+                        api_key='xxx')
+
+messages = [
+{"role": "user", "content": "What's the weather like in Beijing today? Let's check using the tool."}
+]
+finish_reason = None
+while finish_reason is None or finish_reason == "tool_calls":
+    completion = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        temperature=0.3,
+        tools=tools, 
+        tool_choice="auto",
+    )
+    choice = completion.choices[0]
+    finish_reason = choice.finish_reason
+    # Note: The finish_reason when tool calls end may vary across different engines, so this condition check needs to be adjusted accordingly
+    if finish_reason == "tool_calls": 
+        messages.append(choice.message)
+        for tool_call in choice.message.tool_calls: 
+            tool_call_name = tool_call.function.name
+            tool_call_arguments = json.loads(tool_call.function.arguments) 
+            tool_function = tool_map[tool_call_name] 
+            tool_result = tool_function(tool_call_arguments)
+            print("tool_result", tool_result)
+
+            messages.append({
+                "role": "tool",
+                "tool_call_id": tool_call.id,
+                "name": tool_call_name,
+                "content": json.dumps(tool_result), 
+            })
+print('-' * 100)
+print(choice.message.content)
+```
+### Tool Calling in Streaming Mode
+Tool calling can also be used in streaming mode. In this case, we need to collect the tool call information returned in the stream until we have a complete tool call. Please refer to the code below:
+
+```python
+messages = [
+    {"role": "user", "content": "What's the weather like in Beijing today? Let's check using the tool."}
+]
+finish_reason = None
+msg = ''
+while finish_reason is None or finish_reason == "tool_calls":
+    completion = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        temperature=0.3,
+        tools=tools,
+        tool_choice="auto",
+        stream=True 
+    )
+    tool_calls = []
+    for chunk in completion:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            msg += delta.content
+        if delta.tool_calls:
+            for tool_call_chunk in delta.tool_calls:
+                if tool_call_chunk.index is not None:
+                    # Extend the tool_calls list
+                    while len(tool_calls) <= tool_call_chunk.index:
+                        tool_calls.append({
+                            "id": "",
+                            "type": "function",
+                            "function": {
+                                "name": "",
+                                "arguments": ""
+                            }
+                        })
+
+                    tc = tool_calls[tool_call_chunk.index]
+
+                    if tool_call_chunk.id:
+                        tc["id"] += tool_call_chunk.id
+                    if tool_call_chunk.function.name:
+                        tc["function"]["name"] += tool_call_chunk.function.name
+                    if tool_call_chunk.function.arguments:
+                        tc["function"]["arguments"] += tool_call_chunk.function.arguments
+
+        finish_reason = chunk.choices[0].finish_reason
+    # Note: The finish_reason when tool calls end may vary across different engines, so this condition check needs to be adjusted accordingly
+    if finish_reason == "tool_calls":
+        for tool_call in tool_calls:
+            tool_call_name = tool_call['function']['name']
+            tool_call_arguments = json.loads(tool_call['function']['arguments'])
+            tool_function = tool_map[tool_call_name] 
+            tool_result = tool_function(tool_call_arguments)
+            messages.append({
+                "role": "tool",
+                "tool_call_id": tool_call['id'],
+                "name": tool_call_name,
+                "content": json.dumps(tool_result),
+            })
+        # The text generated by the tool call is not the final version, reset msg
+        msg = ''
+
+    print(msg)
+```
+### Manually Parsing Tool Calls
+The tool call requests generated by Kimi-K2 can also be parsed manually, which is especially useful when the service you are using does not provide a tool-call parser. 
+The tool call requests generated by Kimi-K2 are wrapped by `<|tool_calls_section_begin|>` and `<|tool_calls_section_end|>`, 
+with each tool call wrapped by `<|tool_call_begin|>` and `<|tool_call_end|>`. The tool ID and arguments are separated by `<|tool_call_argument_begin|>`. 
+The format of the tool ID is `functions.{func_name}:{idx}`, from which we can parse the function name.
+
+Based on the above rules, we can directly post request to the completions interface and manually parse tool calls.
+
+```python
+import requests
+from transformers import AutoTokenizer
+messages = [
+    {"role": "user", "content": "What's the weather like in Beijing today? Let's check using the tool."}
+]
+msg = ''
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+while True:
+    text = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        tools=tools,
+        add_generation_prompt=True,
+    )
+    payload = {
+        "model": model_name,
+        "prompt": text,
+        "max_tokens": 512
+    }
+    response = requests.post(
+        f"{endpoint}/completions",
+        headers={"Content-Type": "application/json"},
+        json=payload,
+        stream=False,
+    )
+    raw_out = response.json()
+
+    raw_output = raw_out["choices"][0]["text"]
+    tool_calls = extract_tool_call_info(raw_output)
+    if len(tool_calls) == 0:
+        # No tool calls
+        msg = raw_output
+        break
+    else:
+        for tool_call in tool_calls:
+            tool_call_name = tool_call['function']['name']
+            tool_call_arguments = json.loads(tool_call['function']['arguments'])
+            tool_function = tool_map[tool_call_name]
+            tool_result = tool_function(tool_call_arguments)
+
+            messages.append({
+                "role": "tool",
+                "tool_call_id": tool_call['id'],
+                "name": tool_call_name,
+                "content": json.dumps(tool_result), 
+            })
+print('-' * 100)          
+print(msg)
+```
+Here, `extract_tool_call_info` parses the model output and returns the model call information. A simple implementation would be:
+```python
+def extract_tool_call_info(tool_call_rsp: str):
+    if '<|tool_calls_section_begin|>' not in tool_call_rsp:
+        # No tool calls
+        return []
+    import re
+    pattern = r"<\|tool_calls_section_begin\|>(.*?)<\|tool_calls_section_end\|>"
+    
+    tool_calls_sections = re.findall(pattern, tool_call_rsp, re.DOTALL)
+    
+    # Extract multiple tool calls
+    func_call_pattern = r"<\|tool_call_begin\|>\s*(?P<tool_call_id>[\w\.]+:\d+)\s*<\|tool_call_argument_begin\|>\s*(?P<function_arguments>.*?)\s*<\|tool_call_end\|>"
+    tool_calls = []
+    for match in re.findall(func_call_pattern, tool_calls_sections[0], re.DOTALL):
+        function_id, function_args = match
+        # function_id: functions.get_weather:0
+        function_name = function_id.split('.')[1].split(':')[0]
+        tool_calls.append(
+            {
+                "id": function_id,
+                "type": "function",
+                "function": {
+                    "name": function_name,
+                    "arguments": function_args
+                }
+            }
+        )  
+    return tool_calls
+```
+
+## FAQ
+
+#### Q1: I received special tokens like '<|tool_call_begin|>' in the 'content' field instead of a normal tool_call.
+
+This indicates a tool-call crash, which most often occurs in multi-turn tool-calling scenarios due to incorrect tool-call ID. K2 expects the ID to follow the format `functions.func_name:idx`, where `functions` is a fixed string; `func_name` is the actual function name, like `get_weather`, and `idx` is a global counter that starts at 0 and increments with each function invocation.
+Please check all tool-call IDs in the message list.
+
+
+#### Q2: My tool-call ID is incorrect—how can I fix it?
+
+First, make sure your code and chat template are up to date with the latest version from the Hugging Face repo.
+If you're using vLLM or SGLang and they are generating random tool-call IDs, upgrade them to the latest release. For other frameworks, you must either parse the tool-call ID from the model output and set it correctly in the server-side response, or rewrite every tool-call ID according to the rules above on the client side before sending the messages to Kimi K2.
+
+#### Q3: My tool call id is correct, but I still get crashed in multiturn tool call.
+
+Please describe your situation in the [discussion](https://huggingface.co/moonshotai/Kimi-K2-Thinking/discussions)