>>109429637
Faster now
| Cache depth | Mine | llama.cpp ROCm | llama.cpp Vulkan |
|---:|---:|---:|---:|
| 8 | 286.34 tok/s | 187.74 tok/s | 74.87 tok/s |
| 128 | 284.49 tok/s | 185.32 tok/s | 74.93 tok/s |
| 2,048 | 255.65 tok/s | 147.81 tok/s | 28.54 tok/s |
| 8,192 | 198.75 tok/s | 89.23 tok/s | 67.49 tok/s |
| 32,512 | 106.36 tok/s | 30.45 tok/s | 51.19 tok/s |
For reference again:
>windows, rocm backend, 9070xt, qwen3-0.6b in bf16
>>109429648
Thanks anon