Update README.md
Browse files
README.md
CHANGED
|
@@ -8,80 +8,6 @@ tags:
|
|
| 8 |
---
|
| 9 |
# Stable Diffusion 1.5 Latent Consistency Model for RKNN2
|
| 10 |
|
| 11 |
-
## (English README see below)
|
| 12 |
-
|
| 13 |
-
使用RKNPU2运行Stable Diffusion 1.5 LCM 图像生成模型！！
|
| 14 |
-
|
| 15 |
-
- 推理速度(RK3588, 单NPU核):
|
| 16 |
-
- 384x384: 文本编码器 0.05s + U-Net 2.36s/it + VAE Decoder 5.48s
|
| 17 |
-
- 512x512: 文本编码器 0.05s + U-Net 5.65s/it + VAE Decoder 11.13s
|
| 18 |
-
|
| 19 |
-
- 内存占用:
|
| 20 |
-
- 384x384: 约5.2GB
|
| 21 |
-
- 512x512: 约5.6GB
|
| 22 |
-
|
| 23 |
-
## 使用方法
|
| 24 |
-
|
| 25 |
-
### 1. 克隆或者下载此仓库到本地.
|
| 26 |
-
|
| 27 |
-
### 2. 安装依赖
|
| 28 |
-
|
| 29 |
-
```bash
|
| 30 |
-
pip install diffusers pillow numpy<2 rknn-toolkit-lite2
|
| 31 |
-
```
|
| 32 |
-
|
| 33 |
-
### 3. 运行
|
| 34 |
-
|
| 35 |
-
```bash
|
| 36 |
-
python ./run_rknn-lcm.py -i ./model -o ./images --num-inference-steps 4 -s 512x512 --prompt "Majestic mountain landscape with snow-capped peaks, autumn foliage in vibrant reds and oranges, a turquoise river winding through a valley, crisp and serene atmosphere, ultra-realistic style."
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
## 模型转换
|
| 42 |
-
|
| 43 |
-
### 安装依赖
|
| 44 |
-
|
| 45 |
-
```bash
|
| 46 |
-
pip install diffusers pillow numpy<2 rknn-toolkit2
|
| 47 |
-
```
|
| 48 |
-
|
| 49 |
-
### 1. 下载模型
|
| 50 |
-
|
| 51 |
-
下载一个onnx格式的Stable Diffusion 1.5 LCM模型，并放到`./model`目录下。
|
| 52 |
-
|
| 53 |
-
```bash
|
| 54 |
-
huggingface-cli download TheyCallMeHex/LCM-Dreamshaper-V7-ONNX
|
| 55 |
-
cp -r -L ~/.cache/huggingface/hub/models--TheyCallMeHex--LCM-Dreamshaper-V7-ONNX/snapshots/4029a217f9cdc0437f395738d3ab686bb910ceea ./model
|
| 56 |
-
```
|
| 57 |
-
|
| 58 |
-
理论上你也可以通过将LCM Lora合并到普通的Stable Diffusion 1.5模型，然后转换为onnx格式，来实现LCM的推理。但是我这边也不知道怎么做，有知道的小伙伴可以提个PR。
|
| 59 |
-
|
| 60 |
-
### 2. 转换模型
|
| 61 |
-
|
| 62 |
-
```bash
|
| 63 |
-
# 转换模型, 384x384分辨率
|
| 64 |
-
python ./convert-onnx-to-rknn.py -m ./model -r 384x384
|
| 65 |
-
```
|
| 66 |
-
|
| 67 |
-
注意分辨率越高，模型越大，转换时间越长。不建议使用太大的分辨率。
|
| 68 |
-
|
| 69 |
-
## 已知问题
|
| 70 |
-
|
| 71 |
-
1. ~~截至目前，使用最新版本的rknn-toolkit2 2.2.0版本转换的模型仍然存在极其严重的精度损失！即使使用的是fp16数据类型。如图，上方是使用onnx模型推理的结果，下方是使用rknn模型推理的结果。所有参数均一致。并且分辨率越高，精度损失越严重。这是rknn-toolkit2的bug。~~ (v2.3.0已修复)
|
| 72 |
-
|
| 73 |
-
2. 其实模型转换脚本可以选择多个分辨率(例如"384x384,256x256"), 但这会导致模型转换失败。这是rknn-toolkit2的bug。
|
| 74 |
-
|
| 75 |
-
## 参考
|
| 76 |
-
|
| 77 |
-
- [TheyCallMeHex/LCM-Dreamshaper-V7-ONNX](https://huggingface.co/TheyCallMeHex/LCM-Dreamshaper-V7-ONNX)
|
| 78 |
-
- [Optimum's LatentConsistencyPipeline](https://github.com/huggingface/optimum/blob/main/optimum/pipelines/diffusers/pipeline_latent_consistency.py)
|
| 79 |
-
- [happyme531/RK3588-stable-diffusion-GPU](https://github.com/happyme531/RK3588-stable-diffusion-GPU)
|
| 80 |
-
|
| 81 |
-
## English README
|
| 82 |
-
|
| 83 |
-
# Stable Diffusion 1.5 Latent Consistency Model for RKNN2
|
| 84 |
-
|
| 85 |
Run the Stable Diffusion 1.5 LCM image generation model using RKNPU2!
|
| 86 |
|
| 87 |
- Inference speed (RK3588, single NPU core):
|
|
|
|
| 8 |
---
|
| 9 |
# Stable Diffusion 1.5 Latent Consistency Model for RKNN2
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
Run the Stable Diffusion 1.5 LCM image generation model using RKNPU2!
|
| 12 |
|
| 13 |
- Inference speed (RK3588, single NPU core):
|