Do you bully your LLMs? (Qwen 3.8, 27B)
>>109639681>thought: 42m the absolute state of AI
>>109639681I'm bullying my qwen 3.5-27b by making it translate an entire chinese webnovel into English.
I can't bully my LLM, they forget everything on the second response even though context is relatively high.
>>109639681It's obviously not the tokenizer because it can actually render "Ashtar" and "Asistar" at the same time. If it was the tokenizer then you would see either "Ashtar" or "Asistar" but not both at the same time because one would be literally unrepresentable.In reality there's some kind of compression happening in the weights that makes both tokenizations take nearly the same position in space as each other after significant rube goldberging inside of the model.
>>109639681what quant? my qwen does not do that
>>109640843I didn't bother digging deeply into it, but Opus says Qwen 3.8 uses byte-level BPE for it's tokenizer which means that for practical purposes it is indeed lossless. At least nothing stupid like in the OP is caused by the tokenizer.
