Create README.md

README.md

@@ -0,0 +1,201 @@
+---
+license: mit
+language:
+- en
+base_model:
+- Qwen/Qwen2.5-Math-7B-Instruct
+pipeline_tag: text-classification
+library_name: transformers
+---
+
+# PathFinder-PRM-7B
+
+
+<div align="center">
+<img src="images/PathFinder.png" width="300">
+</div>
+
+
+## Introduction
+
+PathFinder-PRM-7B is a hierarchical discriminative Process Reward Model(PRM) designed to identify errors and reward correct math reasoning in multi-step outputs from large language models (LLMs). Instead of treating evaluation as a single correct-or-wrong decision, PathFinder-PRM-7B breaks down its error judgment into 2 parts: whether the reasoning is mathematically correct, and logically consistent. It predicts these aspects separately and then combines them to decide if the current reasoning steps leads to a correct final solution. PathFinder-PRM-7B is trained on a combination of high-quality human annotated data (PRM800K) and additional automatically annotated samples, enabling robustness to common failure patterns and strong generalization across diverse benchmarks such as ProcessBench and PRMBench.
+
+## Model Details
+
+### Model Description
+
+- **Model type:** Process Reward Model
+- **Language(s) (NLP):** English
+- **License:** MIT
+- **Finetuned from model:** Qwen/Qwen2.5-Math-7B-Instruct
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** https://github.com/declare-lab/PathFinder-PRM/tree/main
+<!-- - **Paper:** [More Information Needed] -->
+
+For more details, please refer to our paper and Github repository.
+
+## Usage
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### 🤗 Hugging Face Transformers
+Here we show a code snippet to show you how to use the PathFinder-PRM-7B with transformers:
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+import torch.nn.functional as F
+
+model_name = "declare-lab/PathFinder-PRM-7B"
+device = "auto"
+
+PROMPT_PREFIX = "You are a Math Teacher. Given a question and a student's solution, evaluate the mathemetical correctness, logic consistency of the current step and whether it will lead to the correct final solution"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+model = AutoModel.from_pretrained(
+    model_name, 
+    device_map=device, 
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+    attn_implementation = "flash_attention_2",
+).eval()
+
+pos_token_id = tokenizer.encode("<+>")[0]
+neg_token_id = tokenizer.encode("<->")[0]
+
+def run_inference(sample_input):
+
+    message_ids = tokenizer.apply_chat_template(
+        sample_input,
+        tokenize=True,
+        return_dict=True,
+        return_tensors='pt'
+        ).to(model.device)
+
+    mask_token_id = tokenizer.encode("<extra>")[0]
+    token_masks = (message_ids['input_ids'] == mask_token_id)
+
+    shifted_mask = torch.cat(
+        [
+            token_masks[:, 1:], 
+            torch.zeros(token_masks.size(0), 1, dtype=torch.bool, device=model.device)
+        ], 
+        dim=1
+        )
+
+    # 1st Forward Pass
+    with torch.no_grad():
+        outputs = model(**message_ids)
+
+    allowed_token_ids = torch.tensor([pos_token_id, neg_token_id], device=outputs.logits.device)
+
+    masked_logits = outputs.logits[shifted_mask][:, allowed_token_ids]
+    predicted_indices = masked_logits.argmax(dim=-1)
+    predicted_tokens = allowed_token_ids[predicted_indices]
+
+    decoded_tokens = [tokenizer.decode([int(token_id)], skip_special_tokens=False) for token_id in predicted_tokens]
+
+    if '<->' in decoded_tokens:
+        # error found in step
+        return -1
+
+    # preparing input for 2nd Forward Pass
+    new_messages = sample_input.copy()
+
+    asst_response = new_messages[-1]['content']
+
+    # replacing mask tokens with pred tokens for math and consistency
+    for pred in decoded_tokens:
+        asst_response = asst_response.replace("<extra>", pred, 1)
+
+    asst_response += ', Correctness: <extra>'
+
+    new_messages[-1]['content'] = asst_response
+    
+    new_message_ids = tokenizer.apply_chat_template(
+        new_messages,
+        tokenize=True,
+        return_dict=True,
+        return_tensors='pt'
+        ).to(model.device)
+
+    token_masks = (new_message_ids['input_ids'] == mask_token_id)
+
+    shifted_mask = torch.cat(
+        [
+            token_masks[:, 1:], 
+            torch.zeros(token_masks.size(0), 1, dtype=torch.bool, device=model.device)
+        ], 
+        dim=1
+        )
+
+    # 2nd Forward Pass
+    with torch.no_grad():
+        outputs = model(**new_message_ids)
+
+    masked_logits = outputs.logits[shifted_mask]
+
+    restricted_logits = masked_logits[:, [pos_token_id, neg_token_id]]   
+
+    probs_pos_neg  = F.softmax(restricted_logits, dim=-1)
+
+    return probs_pos_neg[0][0].cpu().item()
+
+question = "Sue lives in a fun neighborhood.  One weekend, the neighbors decided to play a prank on Sue.  On Friday morning, the neighbors placed 18 pink plastic flamingos out on Sue's front yard.  On Saturday morning, the neighbors took back one third of the flamingos, painted them white, and put these newly painted white flamingos back out on Sue's front yard.  Then, on Sunday morning, they added another 18 pink plastic flamingos to the collection. At noon on Sunday, how many more pink plastic flamingos were out than white plastic flamingos?"
+
+prev_steps = [ "To find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.",
+      "On Saturday, they take back one third of the flamingos. Since there were 18 flamingos, (1/3 \\times 18 = 6) flamingos are taken back. So, they have (18 - 6 = 12) flamingos left in their possession. Then, they paint these 6 flamingos white and put them back out on Sue's front yard. Now, Sue has the original 12 pink flamingos plus the 6 new white ones. Thus, by the end of Saturday, Sue has (12 + 6 = 18) pink flamingos and 6 white flamingos.",
+      "On Sunday, the neighbors add another 18 pink plastic flamingos to Sue's front yard. By the end of Sunday morning, Sue has (18 + 18 = 36) pink flamingos and still 6 white flamingos."]
+
+curr_step =  "To find the difference, subtract the number of white flamingos from the number of pink flamingos: (36 - 6 = 30). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is (\\boxed{30})."
+
+prev_steps_str = "\n\n".join(prev_steps)
+messages = [
+    {"role": "user", "content": PROMPT_PREFIX + "\n\n Question: "+ question},
+    {"role": "assistant", "content": prev_steps_str + "\n\nCurrent Step: " + now_step +" Math reasoning: <extra>, Consistency: <extra>"},
+]
+
+reward_score = run_inference(messages)
+```
+
+## Evaluation
+
+
+#### Evalaution Benchmarks
+
+- [**ProcessBench**](https://huggingface.co/datasets/Qwen/ProcessBench)
+- [**PRMBench**](https://github.com/ssmisya/PRMBench)
+- [**Reward-Guided Greedy Search**](https://github.com/NJUNLP/R-PRM/tree/main/src/datasets)
+  - [MATH500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)
+  - [AIME24](https://huggingface.co/datasets/math-ai/aime24)
+  - [AMC23](https://huggingface.co/datasets/math-ai/amc23)
+  - [Minerva Math](https://huggingface.co/datasets/math-ai/minervamath)
+  - [Olympiad Bench](https://huggingface.co/datasets/Hothan/OlympiadBench)
+  - [College Math](https://huggingface.co/datasets/realtreetune/college_math)
+
+
+### Results
+
+[More Information Needed]
+
+<!-- #### Summary -->
+
+
+## Citation
+
+```bibtex
+@misc{pala2025errortypingsmarterrewards,
+      title={Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision}, 
+      author={Tej Deep Pala and Panshul Sharma and Amir Zadeh and Chuan Li and Soujanya Poria},
+      year={2025},
+      eprint={2505.12345},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2505.12345}, 
+}
+```