Do you have at least 12 GB of VRAM and 32 GB of system RAM?If you do, then why are you paying for Claude Code when you can run it locally for free?>Strata>Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.https://github.com/Niko1221/StrataAnyone else test this yet?
>>110060046it dilates the models
>128k ctx
>>110060046>23.8k starsbotted shit
Its super slow.
>>110060046someone test it and report back
>>110060046bro even the best of the best small models are trash compared to claude at 12GB. I already know this will suck. I have 64GB ram and 8GB vram. I ran so many chinese quantized versions and they are just ok at roleplay. That's it.
>>110060228you first
>>110060314i don't have enough VRAM for it
>>110060342Then I guess you're gonna have to fork your money over to this guy.
>>110060355the only thing I regret is not being able to give him more money TBQF
>>110060046doesnt llama cpp do this anyway?
>>110060046What kind of trickery are they doing to run a 125B model on 12GB of vram?
>>110061702dont have anything except windows installed on your pcwhen you send the prompt, go take a 6 hours nap
>>110061702This is a high level overview if you are interested.Not worth imo since I use it so rarely its cheaper (for now) to use a remote providerhttps://www.youtube.com/watch?v=NA9D49kQFuU
>>110061702they rape your SSD
>>110061702Most of it stays in ram because a lot less parameters are active, and the CPU also does part of the work. The fork generally removes most of the dead time waiting for memory transfers and instead the whole computer is doing something productive to keep the LLM running.The quant still makes it kinda retarded though
>>110060046Cause claude is good and local models below 300b at q4 are a joke
>>110060355Why is he so handsome and fit?
>>110060046i tested it a bit on my rx6700xt 12GB VRAM. Even at a retarded quant it managed at 50 toks/s to fix some vibecoded application I created several years ago with an old model. Made me decide I need to upgrade to 24gb vram rx7900xtx.
>>110060046>okay let me run that commandHow’d it go bot?>you’re right I’ll run it nowResults?>here is the result: Nice, you gonna keep going?>you’re absolutely right I was supposed to loop let me pick up where I left offAnd?>running the command nowHave they fixed this or no?
>>1100600469060XT 16Gb CHADS WE ARE SO BACK
>>110060046Oh boy another vibecoded LLM inference engine.
