>>101953380
>It doesn't affect the "older" quants you made like Q8_0?
It doesn't. I could redo them and probably make them marginally better with the new logic (keeping small tensors in FP32 as per the lcpp defaults) but I don't want to push another shitty sidegrade like the FP32 compute idea on everyone lol.
>Will it change the speed of the legacy quants?
Hopefully for the better, I switches from array indices to torch.split. I wanna look into actual cuda/c++ kernels though, not sure how ass shipping those would be.
>That's why I'm hoping for a Q8_K desu, that shit will be virtually fp16
Bad news on that, llama-quantize doesn't accept that as a valid output format, not even for LLMs. May need to wait for future support on their end.
./llama-quantize /mnt/neke/models/mistral-large-2-f16.bin q8_K
main: missing ftype