Spaces:

Jiaqi-hkust
/

hawk

Running on Zero

File size: 7,258 Bytes

ef0f225

<div align="center">

# Hawk: Learning to Understand Open-World Video Anomalies

<div align="center">

### This is the official repository for [Hawk](https://arxiv.org/pdf/2405.16886).

[Jiaqi Tang^](https://jqt.me/), [Hao Lu^](https://scholar.google.com/citations?user=OOagpAcAAAAJ&hl=en), [Ruizheng Wu](https://scholar.google.com/citations?user=OOagpAcAAAAJ&hl=en), [Xiaogang Xu](https://xuxiaogang.com/), [Ke Ma](https://scholar.google.com.hk/citations?user=yXGNGS8AAAAJ&hl=en), [Cheng Fang](), 
\
[Bin Guo](http://www.guob.org/), [Jiangbo Lu](https://sites.google.com/site/jiangbolu), [Qifeng Chen](https://cqf.io/) and [Ying-Cong Chen*](https://www.yingcong.me/)

^: Equal contribution.
*: Corresponding Author.

[![made-for-VSCode](https://img.shields.io/badge/Made%20for-VSCode-1f425f.svg)](https://code.visualstudio.com/) [![Visits Badge](https://badges.strrl.dev/visits/jqtangust/hawk)](https://badges.strrl.dev)



<img src="figs/icon.png" alt="Have eyes like a HAWK!" width="80">
</div>
</div>

## 🔍 **Motivation** - Have eyes like a Hawk!
- 🚩 Current VAD systems are often limited by their superficial semantic understanding of scenes and minimal user interaction. 
- 🚩 Additionally, the prevalent data scarcity in existing datasets restricts their applicability in open-world scenarios.

  <div align="center">
    <img src="figs/motivation1.png" alt="Hawk">
  </div>


## 📢 **Updates**

- ✅ Feb 24, 2025 - We release the **training and demo code** of **Hawk**.
- ✅ Feb 24, 2025 - We release the **dataset (video + annotation)** of **Hawk**. Check this Huggingface link for [DOWNLOAD](https://huggingface.co/datasets/Jiaqi-hkust/hawk).
- ✅ Step 26, 2024 - **Hawk** is accepted by NeurIPS 2024.
- ✅ June 29, 2024 - We release the **dataset (annotation)** of Hawk. Check this Google Cloud link for [DOWNLOAD](https://drive.google.com/file/d/1WCnizldWZvtS4Yg5SX7ay5C3kUQfz-Eg/view?usp=sharing).


## ▶️ **Getting Started**

### 🪒 *Installation*
- Create environment by following steps:
  ```
  apt install ffmpeg
  conda env create -f environment.yml
  conda activate hawk
  ```

### 🏰 *Pretrained and Fine-tuned Model*


- The following checkpoints are utilized to run Hawk：

  | Checkpoint       | Link | Note |
  |:------------------|-------------|-------------|
  | Video-LLaMA-2-7B-Finetuned | [link](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned/tree/main) | Used as initial weights for training.|
  | **Hawk_Pretrained** | [link](https://huggingface.co/Jiaqi-hkust/hawk) | Pretrained on the [WebViD](https://github.com/m-bain/webvid)|
  | **Hawk_Finetuned** | [link](https://huggingface.co/Jiaqi-hkust/hawk) | Fine-tuned on [Hawk dataset](https://huggingface.co/datasets/Jiaqi-hkust/hawk)|

- If you want to use the pretrained model, please use the **Hawk_Pretrained** checkpoint. 
- If you wish to leverage the model for our anomaly understanding, please opt for the **Hawk_Finetuned** checkpoint.


## ⏳ **Domo**

- The configuration files for [`demo`](/configs/eval_configs/eval.yaml).

- Replace the following part as your own path:
  ```
    # Use LLaMA-2-chat as base modal

    # Some ckpts could be download from Video_LLaMA-2-7B-Finetuned
    # https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned
    llama_model: ".../Video-LLaMA-2-7B-Finetuned/llama-2-7b-chat-hf"

    # Hawk Weight (Pretrained or Finetuned)
    ckpt: '.../checkpoint.pth' 
  ```

- Then, run the script:
  ```
  python app.py \
      --cfg-path configs/eval_configs/eval.yaml \
      --model_type llama_v2 \
      --gpu-id 0
  ```

- GUI
  <div align="center">
    <img src="figs/demo.png" alt="Hawk">
  </div>

## 🖥️ **Training**

### 💾 *Dataset Preparation*

-  **For your convenience, we now provide the video and annotations for the Hawk dataset. You can download them using the Hugglingface: [DOWNLOAD](https://huggingface.co/datasets/Jiaqi-hkust/hawk).**

- Traditional Data Acquisition Method:

  - DOWNLOAD all video datasets for their original dources.
  1. [CUHK_Avenue](https://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html)
  2. [DoTA](https://github.com/MoonBlvd/Detection-of-Traffic-Anomaly)
  3. [Ped1](http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm)
  4. [Ped2](http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm)
  5. [ShanghaiTech](https://svip-lab.github.io/dataset/campus_dataset.html)
  6. [UBNormal](https://github.com/lilygeorgescu/UBnormal/)
  7. [UCF_Crime](https://www.crcv.ucf.edu/projects/real-world/)

- Google Drive Link to [DOWNLOAD](https://drive.google.com/file/d/1WCnizldWZvtS4Yg5SX7ay5C3kUQfz-Eg/view?usp=sharing) our annotations.

- Data Structure: each forder contains one annotation file (e.g. CUHK Avenue, DoTA, etc.). The `All_Mix` directory contains all of datasets in training and testing.

- The dataset is organized as follows:

    ```
    (Hawk_data)

    Annotation
    ├── All_Mix
    │   ├── all_videos_all.json
    │   ├── all_videos_test.json
    │   └── all_videos_train.json
    │    
    ├── CUHK_Avenue
    │   └── Avenue.json
    ├── DoTA
    │   └── DoTA.json
    ├── Ped1
    │   ├── ...
    ├── ...
    └── UCF_Crime
    │   └── ...
    │    
    Videos
    ├── CUHK_Avenue
    │   └── Avenue.json
    ├── DoTA
    │   └── DoTA.json
    ├── Ped1
    │   ├── ...
    ├── ...
    │    
    readme

    ```
    Note：the data path should be redefined.


### 🔨 *Configuration*

- The configuration files for [`training`](/configs/train_configs) including two stages.

- Replace the following part as your own path:

  ```
  llama_model: ".../Video-LLaMA-2-7B-Finetuned/llama-2-7b-chat-hf"

  # The ckpt of vision branch after stage1 pretrained, (only for stage 2)
  ckpt: ".../checkpoint.pth" 
  ```

### 🖥️ *To Train*

- Then, run the script:
  ```
  # for pretraining
  NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port='10000' train.py --cfg-path  ./configs/train_configs/stage1_pretrain.yaml

  # for fine-tuning
  NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port='12001' train.py --cfg-path  ./configs/train_configs/stage2_finetune.yaml
  ```

  *Resource Usage: Training (stage 1 and stage 2): 4 * RTX A6000 48G*

## 🌐 **Citations**

**The following is a BibTeX reference:**

``` latex
@inproceedings{atang2024hawk,
  title = {Hawk: Learning to Understand Open-World Video Anomalies},
  author = {Tang, Jiaqi and Lu, Hao and Wu, Ruizheng and Xu, Xiaogang and Ma, Ke and Fang, Cheng and Guo, Bin and Lu, Jiangbo and Chen, Qifeng and Chen, Ying-Cong},
  year = {2024},
  booktitle = {Neural Information Processing Systems (NeurIPS)}
}
```

## 📧 **Connecting with Us?**

If you have any questions, please feel free to send email to `[email protected]`.


## 📜 **Acknowledgment**
This work is supported by the National Natural Science Foundation of China (No. 62206068) and the Natural Science Foundation of Zhejiang Province, China under No. LD24F020002.

Also, this project is inspired by [Video-LLaMA](https://github.com/DAMO-NLP-SG/Video-LLaMA).