Improve model card for MovieCORE dataset

README.md

@@ -1,22 +1,155 @@
 ---
-license: mit
+base_model:
+- lmsys/vicuna-7b-v1.1
 datasets:
 - MovieCORE/MovieCORE
 - Enxin/MovieChat-1K-test
-base_model:
-- lmsys/vicuna-7b-v1.1
+license: mit
+pipeline_tag: video-text-to-text
 ---
 
-# HERMES: temporal-coHERent long-forM understanding with Episodes and Semantics
+<div align="center">
+  <img src="https://github.com/joslefaure/MovieCORE/raw/main/assets/moviecore_icon.png" alt="MovieCORE Icon" width="150"/>
+  
+  # MovieCORE: COgnitive REasoning in Movies
+  
+  **A Video Question Answering Dataset for Probing Deeper Cognitive Understanding of Movie Content**
+  
+  [![arXiv](https://img.shields.io/badge/arXiv-2508.19026-b31b1b.svg)](https://arxiv.org/abs/2508.19026)
+  [![Hugging Face Paper](https://img.shields.io/badge/%F0%9F%A4%97%20Paper-HuggingFace-blue)](https://huggingface.co/papers/2508.19026)
+  [![Hugging Face Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-HuggingFace-yellow.svg)](https://huggingface.co/datasets/MovieCORE/MovieCORE)
+  [![GitHub Code](https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&)](https://github.com/joslefaure/moviecore)
+  [![Project Page](https://img.shields.io/badge/Project%20Page-Website-green.svg)](https://joslefaure.github.io/assets/html/moviecore.html)
+  [![License](https://img.shields.io/badge/License-MIT-green.svg)](https://github.com/joslefaure/MovieCORE/blob/main/LICENSE)
+  
+  ![MovieCore Dataset Teaser](https://github.com/joslefaure/MovieCORE/raw/main/assets/poster_teaser.png)
+</div>
+
+## 📖 Overview
+
+MovieCORE is a comprehensive video question answering (VQA) dataset specifically designed to evaluate and probe deeper cognitive understanding of movie content. Unlike traditional VQA datasets that focus on surface-level visual understanding, MovieCORE challenges models to demonstrate sophisticated reasoning about narrative structures, character development, thematic elements, and complex temporal relationships within cinematic content.
+
+## 🗂️ Data Preparation
+
+The MovieCORE dataset builds upon video content from MovieChat. To get started:
+
+### Video Data
+Download the video files from MovieChat's HuggingFace repositories:
+- **Training Data**: [MovieChat-1K Train](https://huggingface.co/datasets/Enxin/MovieChat-1K_train)
+- **Test Data**: [MovieChat-1K Test](https://huggingface.co/datasets/Enxin/MovieChat-1K-test)
+
+### Annotations
+Access our annotations on HuggingFace:
+- **MovieCORE Annotations**: [🤗 HuggingFace Dataset](https://huggingface.co/datasets/MovieCORE/MovieCORE/tree/main)
+
+Extract and organize the data according to your model's requirements, then use our annotations for evaluation.
+
+## 🚀 Quick Start
+
+### Installation
+```bash
+git clone https://github.com/joslefaure/MovieCORE.git
+cd MovieCORE
+```
+
+## 🎯 Baselines
+- We have provided the script to run [HERMES](https://github.com/joslefaure/HERMES) (ICCV'25) on MovieCORE. Please check out the linked project.
+
+## 📊 Evaluation Dimensions
 
-[Project Page](https://joslefaure.github.io/assets/html/hermes.html) |
-[Model Paper](https://huggingface.co/papers/2408.17443) |
-[Github](https://github.com/joslefaure/HERMES)
+MovieCORE employs a comprehensive multi-dimensional evaluation framework to assess model performance across different aspects of cognitive understanding:
 
-To use these models for inference, please look at our inference code and guidelines in this [Github Repo](https://github.com/joslefaure/HERMES)
+| Dimension | Description |
+|-----------|-------------|
+| **🎯 Accuracy** | Measures semantic similarity between predicted and ground truth answers |
+| **📋 Comprehensiveness** | Assesses coverage of all key aspects mentioned in the ground truth |
+| **🧠 Depth** | Evaluates level of reasoning and insight demonstrated in predictions |
+| **🔍 Evidence** | Checks quality and relevance of supporting evidence provided |
+| **🔗 Coherence** | Measures logical flow, organization, and clarity of responses |
 
+Each dimension provides unique insights into different cognitive capabilities required for deep video understanding.
 
-# References
+## 💻 Usage
 
-- [MovieCORE](https://huggingface.co/papers/2508.19026)
+### Evaluation Script
+
+Evaluate your model's performance on MovieCORE using our evaluation script:
+
+```bash
+export OPENAI_API_KEY='your_openai_api_key'
+python evaluate_moviecore.py --pred_path path/to/your/predictions.json
+```
+
+### 📝 Input Format
+
+Your predictions should follow this JSON structure:
+
+```json
+{
+    "video_1.mp4": [
+        {
+            "question": "How does the video depict the unique adaptations of the species in the Sahara Desert, and what roles do these species play in their ecosystem?",
+            "answer": "The ground truth answer.",
+            "pred": "Your model's prediction.",
+            "classification": "the question classification"
+        },
+        {
+            "question": "The second question for video 1?",
+            "answer": "The ground truth answer.",
+            "pred": "Your model's prediction.",
+            "classification": "the question classification"
+        }
+    ],
+    "video_2.mp4": [
+        {
+            "question": "The only question for video 2",
+            "answer": "The ground truth answer.",
+            "pred": "Your model's prediction.",
+            "classification": "the question classification"
+        }
+    ]
+}
+```
+
+### 📈 Output
+
+The evaluation script provides:
+- Overall scores across all dimensions
+- Classification-specific performance metrics
+- Detailed breakdowns for comprehensive analysis
+
+## 📚 Citation
+
+If you use MovieCORE in your research, please cite our paper:
+
+```bibtex
+@misc{faure2025moviecorecognitivereasoningmovies,
+      title={MovieCORE: COgnitive REasoning in Movies}, 
+      author={Gueter Josmy Faure and Min-Hung Chen and Jia-Fong Yeh and Ying Cheng and Hung-Ting Su and Yung-Hao Tang and Shang-Hong Lai and Winston H. Hsu},
+      year={2025},
+      eprint={2508.19026},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2508.19026}, 
+}
+```
+
+## 🤝 Contributing
+
+We welcome contributions to MovieCORE! Please feel free to:
+- Report issues or bugs
+- Suggest improvements or new features
+- Submit baseline implementations
+- Provide feedback on the evaluation framework
+
+## 📄 License
+
+This dataset is provided under the MIT License. See [LICENSE](https://github.com/joslefaure/MovieCORE/blob/main/LICENSE) for more details.
+
+---
 
+<div align="center">
+  <p>🎬 <strong>Advancing Video Understanding Through Cognitive Evaluation</strong> 🎬</p>
+  
+  **[\ud83d\udcd6 Paper](https://arxiv.org/abs/2508.19026v1) | [\ud83e\udd17 Dataset](https://huggingface.co/datasets/MovieCORE/MovieCORE) | [\ud83d\udcbb Code](https://github.com/joslefaure/moviecore)**
+</div>