RTX A4000 results:
- llama.cpp (fresh pull and compile)
- INFO [ print_timings] generation eval time = 20826.11 ms / 523 runs ( 39.82 ms per token, 25.11 tokens per second) | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 t_token_generation=20826.106 n_decoded=523 t_token=39.82047036328872 n_tokens_second=25.112711901111037
INFO [ print_timings] total time = 20948.52 ms | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 t_prompt_processing=122.417 t_token_generation=20826.106 t_total=20948.523
INFO [ update_slots] slot released | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 n_ctx=16640 n_past=871 n_system_tokens=0 n_cache_tokens=871 truncated=false
32K tokens OOMed, 16K was fine
INFO [ print_timings] generation eval time = 20826.11 ms / 523 runs ( 39.82 ms per token, 25.11 tokens per second) | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 t_token_generation=20826.106 n_decoded=523 t_token=39.82047036328872 n_tokens_second=25.112711901111037
INFO [ print_timings] total time = 20948.52 ms | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 t_prompt_processing=122.417 t_token_generation=20826.106 t_total=20948.523
INFO [ update_slots] slot released | tid="140598100832256" timestamp=1723757763 id_slot=0 id_task=102 n_ctx=16640 n_past=871 n_system_tokens=0 n_cache_tokens=871 truncated=false
It ripped through writing a short python program. Very good! Card has a max TDP of 140W. I like this card. It's the perfect "just a little more" card without having a too shitty memory bus or core count, or being too old for flash attention.