File size: 775 Bytes
94a426d |
1 2 3 4 5 6 7 8 9 10 11 12 |
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| seed_oss 36B Q8_0 | 35.78 GiB | 36.15 B | CUDA | 35 | pp8 | 17.77 ± 0.91 |
| seed_oss 36B Q8_0 | 35.78 GiB | 36.15 B | CUDA | 35 | tg128 | 2.84 ± 0.02 |
build: 92bb442ad (7040)
|