Spaces:

Jiaqi-hkust
/

hawk

Running on Zero

App Files Files Community

hawk / README.md

Jiaqi-hkust

Initial commit on new branch

ef0f225 9 months ago

preview code

raw

history blame

7.26 kB

Hawk: Learning to Understand Open-World Video Anomalies

This is the official repository for Hawk.

Jiaqi Tang^, Hao Lu^, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang,
Bin Guo, Jiangbo Lu, Qifeng Chen and Ying-Cong Chen*

^: Equal contribution. *: Corresponding Author.

🔍 Motivation - Have eyes like a Hawk!

🚩 Current VAD systems are often limited by their superficial semantic understanding of scenes and minimal user interaction.
🚩 Additionally, the prevalent data scarcity in existing datasets restricts their applicability in open-world scenarios.

📢 Updates

✅ Feb 24, 2025 - We release the training and demo code of Hawk.
✅ Feb 24, 2025 - We release the dataset (video + annotation) of Hawk. Check this Huggingface link for DOWNLOAD.
✅ Step 26, 2024 - Hawk is accepted by NeurIPS 2024.
✅ June 29, 2024 - We release the dataset (annotation) of Hawk. Check this Google Cloud link for DOWNLOAD.

▶️ Getting Started

🪒 Installation

Create environment by following steps:

apt install ffmpeg
conda env create -f environment.yml
conda activate hawk

🏰 Pretrained and Fine-tuned Model

The following checkpoints are utilized to run Hawk：

Checkpoint Link Note

Video-LLaMA-2-7B-Finetuned link Used as initial weights for training.

Hawk_Pretrained link Pretrained on the WebViD

Hawk_Finetuned link Fine-tuned on Hawk dataset
If you want to use the pretrained model, please use the Hawk_Pretrained checkpoint.
If you wish to leverage the model for our anomaly understanding, please opt for the Hawk_Finetuned checkpoint.

Checkpoint	Link	Note
Video-LLaMA-2-7B-Finetuned	link	Used as initial weights for training.
Hawk_Pretrained	link	Pretrained on the WebViD
Hawk_Finetuned	link	Fine-tuned on Hawk dataset

⏳ Domo

The configuration files for demo.

Replace the following part as your own path:

  # Use LLaMA-2-chat as base modal

  # Some ckpts could be download from Video_LLaMA-2-7B-Finetuned
  # https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned
  llama_model: ".../Video-LLaMA-2-7B-Finetuned/llama-2-7b-chat-hf"

  # Hawk Weight (Pretrained or Finetuned)
  ckpt: '.../checkpoint.pth'

Then, run the script:

python app.py \
    --cfg-path configs/eval_configs/eval.yaml \
    --model_type llama_v2 \
    --gpu-id 0

🖥️ Training

💾 Dataset Preparation

For your convenience, we now provide the video and annotations for the Hawk dataset. You can download them using the Hugglingface: DOWNLOAD.
Traditional Data Acquisition Method:
- DOWNLOAD all video datasets for their original dources.
1. CUHK_Avenue
2. DoTA
3. Ped1
4. Ped2
5. ShanghaiTech
6. UBNormal
7. UCF_Crime
Google Drive Link to DOWNLOAD our annotations.
Data Structure: each forder contains one annotation file (e.g. CUHK Avenue, DoTA, etc.). The All_Mix directory contains all of datasets in training and testing.

The dataset is organized as follows:

(Hawk_data)

Annotation
├── All_Mix
│   ├── all_videos_all.json
│   ├── all_videos_test.json
│   └── all_videos_train.json
│    
├── CUHK_Avenue
│   └── Avenue.json
├── DoTA
│   └── DoTA.json
├── Ped1
│   ├── ...
├── ...
└── UCF_Crime
│   └── ...
│    
Videos
├── CUHK_Avenue
│   └── Avenue.json
├── DoTA
│   └── DoTA.json
├── Ped1
│   ├── ...
├── ...
│    
readme

Note：the data path should be redefined.

🔨 Configuration

The configuration files for training including two stages.

Replace the following part as your own path:

llama_model: ".../Video-LLaMA-2-7B-Finetuned/llama-2-7b-chat-hf"

# The ckpt of vision branch after stage1 pretrained, (only for stage 2)
ckpt: ".../checkpoint.pth"

🖥️ To Train

Then, run the script:

# for pretraining
NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port='10000' train.py --cfg-path  ./configs/train_configs/stage1_pretrain.yaml

# for fine-tuning
NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port='12001' train.py --cfg-path  ./configs/train_configs/stage2_finetune.yaml

Resource Usage: Training (stage 1 and stage 2): 4 * RTX A6000 48G

🌐 Citations

The following is a BibTeX reference:

@inproceedings{atang2024hawk,
  title = {Hawk: Learning to Understand Open-World Video Anomalies},
  author = {Tang, Jiaqi and Lu, Hao and Wu, Ruizheng and Xu, Xiaogang and Ma, Ke and Fang, Cheng and Guo, Bin and Lu, Jiangbo and Chen, Qifeng and Chen, Ying-Cong},
  year = {2024},
  booktitle = {Neural Information Processing Systems (NeurIPS)}
}

📧 Connecting with Us?

If you have any questions, please feel free to send email to [email protected].

📜 Acknowledgment

This work is supported by the National Natural Science Foundation of China (No. 62206068) and the Natural Science Foundation of Zhejiang Province, China under No. LD24F020002.

Also, this project is inspired by Video-LLaMA.