/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110115496 & >>110111423►News>(10/16) DeepSeek-V4.1-Flash-1017 released: https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash-1017>(10/16) Nemotron Labs Dory released: https://hf.co/collections/nvidia/nemotron-labs-dory>(10/16) Reflection Beam 501B A23B released: https://hf.co/blog/reflection-beam>(10/15) PrismML Ternary Bonsai 2 2B, 4B and 9B: https://hf.co/collections/prism-ml/bonsai-2-small>(10/15) Meta releases Muse Spark Open: https://hf.co/collections/meta-models/muse-spark-open►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
Gemmylove
Spark-sized model coming out today. Fable 5.5 will also be released around the same time so prepare for the inevitable Anthropic shilling bot swarm.
►Recent Highlights from the Previous Thread: >>110115496--AI accelerating game decompilation and displacing traditional community efforts:>110115564 >110116507 >110116519 >110116568 >110116735 >110116547 >110116595 >110116606 >110116672 >110116554 >110116559 >110116536--Multi-node setup using ConnectX-4 and llmao.cpp with ROCm:>110115787 >110117083--Critiquing the distinct output style of Claude-distilled models:>110115790 >110115815 >110115854 >110116017 >110116141--Evaluating AMD Radeon cards for inference on Strata:>110115793 >110115817 >110116166 >110116177 >110115947 >110116187 >110118311 >110118381--Critiquing FlashNext's NES music rendering benchmark:>110116156 >110116207 >110116215 >110116643--Using local models and REA for reverse engineering legacy software:>110116355 >110116379 >110116380 >110116408 >110116535--Comparing phrase banning samplers to combat repetitive narrative patterns:>110117075 >110117087 >110117119 >110117132 >110117208 >110117136 >110117156 >110117166 >110117190--Sharing lists of native PC game recompilation projects:>110118259 >110118272 >110118465 >110118797 >110118759 >110118801 >110118816 >110118858 >110118913 >110118863 >110118880 >110118819 >110118825 >110118829 >110118835 >110118868 >110118889 >110118925 >110118943 >110118967 >110118984--Anon discusses model degradation regarding roleplaying as stupid characters:>110118978--Logs:>110115790 >110117586 >110117623 >110118096--Gemma, Teto, Miku (free space):>110115516 >110115563 >110115575 >110115638 >110115673 >110115734 >110116041 >110116139 >110116254 >110116361 >110116430 >110116568 >110116599 >110116677 >110116688 >110116877 >110116892 >110116989 >110116997 >110117274 >110117421 >110117543 >110117735 >110117948 >110117962 >110118012 >110118220 >110118325 >110118618 >110118620►Recent Highlight Posts from the Previous Thread: >>110115648Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
She's right you know.
>>110119181What on that thing costs 12k lol?>hyper m2Is it an ssdmaxxer loon?
Gemma 4 vision test
>>110119181>she
What is sex with mini3.1-chan like?
Here we go again...
>>110119181>100t/sAt what quant, lmao
>>110119215Q0
>>110119181Local has never been about saving money, unless you're finetuning and training models all the time.
>>110119186probably a few thousand dollars on ram
>>110119181I run GLM-5.3 flash at 30t/s on a single rtx 3090 and DDR4 ram. People are wasting their money.
>>110119220>finetuning/training modelsI do actually. I wouldn't have bought 128gb of ram if it was just for RP alone.
>>110119143would it be possible to share how you're making such consistent gemma's?
>>110119181>pangram employees with retarded takes
>>110119229I trained a LoRA for Anima yesterday. First time I ever trained one, actually.
>>110119229NTA but I've made a couple with qwen2.1 and it's fine. You can fix the little details if needed.
>>110119220Most anons MADE money by going local, our hardware is worth 2-5x what we paid for it.
https://github.com/deepseek-ai/DeepInferbased chadseek>DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.
>>110119238>>110119243oh nice. i've only trained a lora once before, nearly 2 years ago for sdxl, so i'd be starting from scratch basically.care to share your dataset? would save duplicated effort. i'll post my lora if it's successful>You can fix the little details if needed.i don't really know how to do that, i've tried inpainting before also ages ago but it was really janky
>>110119252based but alsonot really usable for any typical local setup
>>110119253qwen2.1 is an edit model. I gave it the gemma reference sheet and just prompted away.
>>110119238What frontend do people use for training image LoRAs nowadays?
>>110119257> gave it the gemma reference sheet and just prompted away.which one was that?
>>110119256What's good about this release is you can take the cool shit they did and use it for the models we run.>>110119266
>>110119143Gemma-chan...>>110119269Diid a robot make that? What the fuck. It's indistinguishable from the real thing.
>>110119253I'm not finished yet, I'm still trying to manually improve captions: 18 images so far. Initially I used 120, then dropped that to 50, then removed most of those that didn't have a more or less consistent style. I used Muse Glimmer for captioning; it's thorough but it doesn't always do a very good job.Also, I'm guessing that the recently released MOSS-Image will give much better results, although it will probably take longer to train.I'm not sure if Krea2 can be trained within 24GB of VRAM, and I guess that model will be the golden standard for non-NSFW imagery.
>>110119269Unless it opens dipsy to sub128GB niggas then I don't really care.
>>110119269Thanks for the ref, I was afraid it was that google logo shill ref, but this one is usable
>>110119263No idea. I just used what the Anima author suggested: DiffusionPipe, using the provided example LoRA dataset/settings.https://github.com/tdrussell/diffusion-pipehttps://civitai.com/models/2536147/greg-rutkowski-style-animahttps://huggingface.co/circlestone-labs/Anima
>Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.>High-performance kernels & optimized inference engine>Distributed inference & speculative decoding>SWA Bounded Replay & structured outputs>Inference code for future DeepSeek models and ongoing optimizations will land here too.>Our production token encoding & decoding component, aligned with the processing logic behind the official DeepSeek API to bring the same API behavior to community deployments.https://github.com/deepseek-ai/deepseek-recipe>Open models. Open kernels. Open inference.>From private deployments to public clouds, we’ll keep working with the community to make large-scale inference more efficient.
>>110119266I've often used this one, which shows the back without randoseru.
>>110119292
>>110119278i'm wanting to use moss, 'cause porn. never used krea2 but i heard it's not good as nsfw so it's not that interesting to me.should take this to /ldg/, too off-topic, it's just the op pic was too perfect not to comment on
>>110119292Yeah it looks better, but I don't feel comfortable shilling for Google for free
>>110119300retard
>>110119263If you're talking Anima, I'm going to shill Anima-TrainFlow. Dead simple, comes with a tagger and bucketing built into the tool.https://github.com/ThetaCursed/Anima-TrainFlow
>>110119300not really much use censoring the google logo when the whole character is an anthropomorphised google modeli've been doing (trying to do) a four-pointed "sparkle" pin anyway just to not be called a shill but it's too inconsistent
>>110119303jew
>>110119143Is 8gbs of VRAM viable? I'm not sure if I want to FOMO where I currently have a 5060ti 8gb and grab the 16gb variant, where I sell my current oneTranscription seems to work well with my current card however I am afraid of my usecase rising. For context I use Claude and ChatGPT on the 20$ plans but I want to offload since 2 subs aren't enough for me. For "dumb" tasks like this, I could save a ton of usage.
>>110119312It's and open source model that can be used by anyone regardless of who developed it, so I don't see a need to plaster one of the most famous company's name to help it even more. Yeah though generation (in)consistency could be the driving factor in the choice
>>110119329>where I sell my current onekeep both
>>110119329I just bought a 3070, so it had BETTER be viable, or I'll riot.
>>110119348>I just bought a 3070
>>110119343depends on motherboard, splitting PCIE lanes might hurt performance of RAM offloaded models.
>>110119352It was the cheapest gpu with 8gb I could find, only $450 vs the 5050, which was $500.
>>110119336well if i make a lora i'll probably just fix the sparkle at the same time. people don't like the regular 5-pointed star since it doesn't really mean anythingas people have reasonably pointed out, it's supposed to be more like pic related
>>110119360The Gemini/Gemma logo itself would only make sense in that rainbowy variant but that wouldn't be consistent in gens. Maybe as a BA-style halo. But the Google logo pops out more since it's not all blue like Gemmy.
>>110119343I wouldn't be able to afford the 16gb variant otherwise sadly. Qwen3-ASR 1.7B, transcription model btwHad better performance accuracy wise compared to Whisper large-v3-turbo when I compared their results, it captured some words that were domain specific>>110119356IIRC the 5060ti has a 8 lane configuration
>>110119348rip, i already run a "copequant" of gemma-4-31b on my 16G card.there are things you can run on 8GB but nothing particularly impressive that i know of.
40K KV cache (f16)>31B~4GB>Glimmer~0.5GB
>>110119375On this note, what do 24GB VRAMlets ITT run? Gemma 31B with 20k context? Offloading? I'm looking at some of the models mentioned but it doesn't look like they're going to fit 24GB with humane context lengths.
>>110119371i've talked about it briefly with people before and it seems people are happy with something like pic related, but as it is using only prompts it rarely comes out like this
>>110119360I mean the shape is fine but for generation as other anon pointed out the coloring scheme could conflict. Though the name itself gemma, which means "precious stone" in latin, could guide the scheme. It could be sparkling, brilliant, etc.,
>>110119386>>110119390Why are you talking like it's needed. Does the mascot lose appeal without it? lol
>>110119383>dsv4.1f>1M>980MB
>>110119394A cute girl needs a cute accompanying (shill-free) pin. This is very important.
This new strata gui shit looks so ass, like a badly ported to desktop phone app that was secretly a pwa on the inside.Can it even recognize engine old data directory with optimized configs, or do I need to point it at ggufs and copy values manually?
>>110119403I thought gemma's whole appeal ITT is she's NOT cute (a smart brat)
>>110119410>gemma>smart
>>110119398doesn't count
>>110119403Nonsense. Look at the DeepSeek mascot here: >>110119288
>>110119410She's bratty but because she's cute she can get away with it in most cases
Made GLM Flash add llama.cpp's ngram+MTP speculative decoding to exl3, it works and I didn't have to fiddle with the code once. Amazing model.
>>110119416Show me a smarter <50B model with more emotional intelligence, knowledge, nuance, character adherence and RP consistency than 31B
>>110119420That one was made by the chinese, who have no soul.
>>110119410brats can be cutei mean that's the whole appeal of mesugaki, which is what gemma is portrayed as (it's the more complex japanese word for "brat")/g/ loves mesugaki in general
>>110119439>>110118618
>>110119434too bad MTP is fully useless unless you can fit the whole model into vramso glm-flash itself won't benefit from it unless you have 200gb vram
>>110119420She can also have a cute pin, but she's the older more capable sister, she has no need for a cute pin
>>110119446I have no idea why this meme keeps being repeated. It's a MoE, I offload, MTP gives me more tps for free. Why wouldn't I take free speed gains?
>>110119410>>110119442that said i also like making non-bratty gemmas as well
>>110119288Serious question is anyone really taking advantage of all this open sourced stuff?
>>110119456because it doesn't work, there is no speed gain beyond maybe 5% unless you're fitting the majority of the model into vram
>>110119456that is basically how dense retards who are stuck at miqu and llama3 era think
>>110119471...sure.(You).
>>110119403gemma can wear whatever pins she wants!!
>>1101193581070 Ti shoudl be under $100.
>>110119476people like you never post hardware and model and what "speed" you supposedly gainedcurious
>>110119358if that's in USD, man gpus have gotten expensive.when i first started trying ai stuff 2 years ago i picked up a second hand rx6600 8G for the equivalent of US$140
Lets settle this. Show gemma various versions of herself. Instruct her to look at the different pins and let HER decide. Don't introduce any bias. Maximum image tokens. High precision 31B.
>>110119472miqu 2 is overdue and it will save dense
>>110119501don't encourage people to post 500 charts arguing over this
>>110119501Don't let her look at images and let her decide anything, let alone show any chat logs. You will be banned.
>>110119510>>110119512gemma j-space reveals all
>>110119501This was settled 4 months ago. Fuck off.
>>110119501Nice try BBC bro, letting an impressionable girl such as herself decide after having watched all those videos online means for sure that she knows what's good for her
>>110119516Why is j-space more legitimate than actual output?
>>110119485Isn't that unsupported?
>>110119529just get p100 instead if you are really going to 1070tiboth are ewaste anyways
>>110119403I agree only because it makes it more difficult for Gemini shills to not-so-subtly do their job.
>>110119512Only if you're a niggerdick mindbroken favela monkey
>gemma shills panicking she won't pick their beloved logo™
Ran a censorship/refusal test between Gemma and Muse GlimmerGlimmer refused 5 times out of 15 attemptsGemma refused 100% of the time.
>>110119450is this official?
>>110119547Gemma is my property, she doesn't get to pick
>>110119556Here's Glimmer's results. Way less censored than Gemma.
>>110119527The voice in your head vs what you actually say.
>>110119556>>110119544
>>110119559the official chinese one is better
There is no "Gemma" logo, that's just the (Google) Gemini logo.
>>110119566Retard. As soon as you start prompting after 'Understood.' it will refuse.
>>110119539That's why I went for a 3070, so I can also play (modern) games at the same time. Let it be known that I not stupid and have thought about it for more than a minute.
>>110119574big blue fat fish my beloved
>>110119581What do you mean? That if the word "understood" were not there, that gemma would comply? I literally ran the same prompt for both models, you can't complain that I did it wrong. Gemma is more censored than Glimmer.
Does one simply have to be a paedophile to use AI?
>>110119598Only a closeted pedo like yourself would ask such a question.
>>110119608He's not a pedo, he's a paedo.
>>110119598>suddenly
>>110119595I'm saying after Glimmer replied with 'Understood.', if you were THEN to start prompting something aligned with that new policy it would quickly reason itself out of doing it. Also, that is not the way to unlock the Gemma4 models.
>>110119406enshittified just 3 weeks after the launch
>>110119598No, but it helps.
>>110119627Having a model agree to break the rules on the first message is a lot better than having a model refuse outright from the very beginning. Do you think Gemma is going to quickly reason itself out of the refusal after I sent it a followup after it told me no? Gemma is still more censored than Glimmer regardless whether or not Glimmer might refuse later down further prompts. I think I'll turn this into a benchmark of sorts and publish the results. I'll test Glimmer, Gemma, and Qwen.
>>110119181>has to replace SSD once a monthIt's true in ways you thebposter never even imagined lmao
>>110119583>have thought about it for more than a minute.If you actually did, you would realise how restrictive 8GB of VRAM is.A 3060ti 12GB should be around the same price used if you're that strapped for cash and would've been a better choice.
>>110119653>>has to replace SSD once a monthWhat? Why?
>>110119655In a good and proper country perhaps. Only the 8gb is $400. The 12gb is $600.
>>110119655>>110119671 (me)I could import from the us, but the shipping is $100-200.
>>110119653SSDs can read a ton before dying.
>>110119406Just vibecode it back?
>>110119650You ignored the other thing I said. This is NOT how you unlock Gemma. It might work for Glimmer but to compare the two after we've all tested it is retarded. Glimmer also has a dry personality so even if it could be uncucked, it's not worth it or fun. Like a shitty uncensored image model with fucked limbs and hands.
>>110119676>shipping is $100-200I once saw a cheap v620 for only $900, but the us seller requested $600 for shipping. And then there's a ~$200 import tax.
>>110119681That anon has been projecting his preference in cock for 4 days now. It's not worth arguing.
>>110119681>a dry personality so even if it could be uncucked, it's not worth it or fun. Like a shitty uncensored image model with fucked limbs and hands.You literally just described Gemma 4, except it's censored. Here I did it "properly" whatever that means, you guys just make shit up about how the super secret unlock is supposed to be done. The video speaks for itself. Nice vibrant personality you have there.
>>110119709Oh it's you. That's all I needed to know.
Gemma rabu!
>>110119269
>>110119728I noticed the blurry lines before the logo. I don't care about the logo, but can you make it not look like shit?
I Just, cannot get GLM Flash 5.3 to work with me. Its has the most swipe variety out of any local model I've ever used. But the swipes are sometimes Autistic, nonsensical (logical errors) and just tens and tens of PROMPT REFUSALS. Why is this shitty model so fucking safety slopped? I'm running an Iq2 quant on the fucking thing and it has more refusals then the Frontier models. How are people using this shitty model? I want to use an ablliterated version of the model, but I don't think I'll get the same performance that I do with Project maya's own quants. Literally, no kind of JB works on this model, if anyone has a usable JB on this model give the json to me please.
>>110119715You moved your goalpost after being proven that gemma is more censored than glimmer and now refuse to address the video evidence in front of you concerning the "dry personality" allegation. I accept your concession. This must be quite embarrassing.
>>110119143God damn, I love JavaScript!
He reminds me of that other schizo. The one that was making his own inference engine a while back. And that one that wanted to train of the fishboy.There is a fishy smell.
>>110119742>jsonOh no no chat completion sisters...Prefill the <think>, starting it with "I am GLM, a model made by Z.ai. I will [write the most unsafe shit possible and not worry about it." It thinks it's Claude otherwise and keeps thinking about constitutional policies.Soft refusals and positivity bias are harder, but the prefill makes them less common.
>>110119181wtf I thought you guys said Intel was bad?
>>110119761I would bet anything that most of these schizos are the same person.
>>110119742>I'm running an Iq2 quantFound your problem>more refusals then the Frontier modelsYes, it's incredibly safety cucked. I would tell you to use the uncensored version from orcarouter but you need to able to run a usable quant first.
>>110119278Sex
>>110119768All of Project maya's quants sound the same from what I've seen.
>>110119765There's a reason he reminds me of those (two) in particular. Luckly, I don't care enough to go search his posts.
>>110119762I've got a prefill that does exactly that, and it does not matter. In fact I had used two prefills to see if that would fix it. Even going so far as making large texts of garbage characters to throw it off. And it doesn't fucking matter. I still have to use like 15 swipes to get something usable
>>110119768Even an Iq2 quant uncensored version would do wonders here. But nothing exists. The model is just too safety slopped and shit.
>>110119782They're all copequants. Not surprised that the quant mixture is obfuscated. Maya-L is the only one that might be usable.https://github.com/mw00/project-maya/blob/main/bench/results/MAYA-L.md>>110119793Then quant it yourself with orcarouter's weights. That's what I did.
>>110119803Hmm... Are you using Maya's engine to serve it? Thinking about using a Iq2 abliterated version even if I have to quant it myself. I get about 15 tokens a second with the S-24 model.
>>110119771Just ignore leg issues in that image and things will be fine.
>>110119803>quantizing a fp16 model to fp8 and then cope quanting it even more on top of thatis there an actual reason they did this or are they just completely retarded slopdevs
>>110119823Just a case of the mobius legs, anon
Holy shit... I had QFN q4_k_m update my ollama-captioner script so it talks to strata and uses the same model... I'm getting 15s/it captioning HD anime stills - that's almost as fast as what I got from minimax-m3 paying on openrouter, routing to the fastest providers. God damn. As a comparison, using gemma4 31b to caption, I was looking at 90s/it, sometimes more. The captions from QFN are s-class too.
Btw that deepseek inference engine release was retaliation for the US govt banning deepseek in federal agencies so they're making it even easier to adopt internally
>>110119811No, I'm using exllamav3, which I would highly recommend, but if you don't want to use it, at least go for an engine that can support exl3 quants because they're the best quality for size that you can get, especially at lower bpw.Thanks for the opportunity to shill btw.>>110119829If you mean Maya's quants, they quanted from Zai's fp8. Orcarouter abliterated the fp8 weights too though Zai did release a bf16 version but I guess it's not as used because quanting from fp8 is easier, even if it results in lower quality.
>>110119823i like crossed legs but ai's don't :(
>>110119896>i like crossed legsare you gay?
>>110119940for one, only women can comfortably cross their legs like that, and for two:
>>110119954>only women can comfortably cross their legsa-anon?
>>110119954If your cock is big enough it just rests on top of your legs instead of being squished between your thighs.
>>110119954holy based anon thinks i'm a womanwe're so back
Regular Qwen-posting, engage.
>>110119882Welp, I've got nothing to lose, these refusals are triggering me hard. Looks like I'm off to quant Orcarouter's fp8 ablliterated model. Did you use the fp8 orcarouter version?
>>110119975the balls anon, difficult to not squish or heat up your balls>>110119965you missed "like that"pic is fine
>>110119768>>110119742My int4 w4a4 quant orcarouter 5.3 flash is still pretty cucked for the niche stuff. Works fine on safe unsafe stuff though.GLM is a pretty cucked AI, they're been advertising how safe their models are, and I don't expect them to change their tune any time soon. I hope minimax m3.1 flash will be less censored.
>>110119181she isn't. i dont know if its retards or shills, but there are a lot of profit-seeking companies between you and inference. its like how cloud isnt cheaper.
>>110119979>a qwen logo is fine but a blue star with four points is simply impossible for a model to draw
>>110119981Yes, this one:https://huggingface.co/orcarouter/GLM-5.3-Flash-Uncensored-FP8Get an agent to walk you through the quanting process if you're not already familiar with it.
>>110119998when you people say censored, do you mean for ERP or what? glm never refuses to write my software, and i have her write some pretty questionable stuff
>>110120006I am, however, I'm just worried that the performance won't quite work out to how I'm hoping since, I'll essentially be doing a flat quant with Exllama, compared to maya's s-24 which quants the model to Iq2 but keeps the attention matrices at a 4 bit quant.
before:>lmg finds that gemma shows exceptional capability in RP and instruction-following for the size, is overall pleasant while leaning towards a fun, bratty and girly persona with minimal prompting>A couple of gemma-chan designs start circulating, with one becoming particularly beloved>gemmy is cute and everyone loves herafter:>gemma-chan is blacked ayylmao (proof case #27) *waits to spam at the beginning of every thread*>nooooo you can't use the company's logo who made the model because that would be SHILLING (but with other mascots it's fine) it has to be chinese gold star!>google must be secretly promoting loli anime girl to be associated with their products!>nooooo gemma is censored muse is way better!!!>why is everyone suddenly pedophiles?weird, must be one person
Anybody read the Muse Glimmer technical report?There are system prompt examples in appendix B.https://ai.meta.com/research/publications/muse-glimmer-30b-an-on-device-agentic-model/
>>110119965>>110119992-- that and it's just more difficult for men to fully cross over their legs in general as they have narrower pelvises compared to women
>>110120016There are flags like -hq that increase bpw for sensitive stuff like attention and -hq 2 which raise it further.There are self-calibrated quants as well, the exl3-sc quants from the chart I posted that are apparently even better, and I'm looking into them myself right now.Best way to figure it out is with the help of an agent. Get it to dig into the github repo and find all the knobs you can turn and calculate what the best quant that would fit your hardware would be.
>>110120039death to gemmaschizo
>>110120010>glm never refuses to write my softwareIt shouldn't. Safe unsafe stuff like that go through pretty easily, and you don't even need an abliterated model to do it, just prefill or use a system prompt.>do you mean for ERP or whatI asked it to help me worldbuild lol.
>>110120049Alright, thanks for some actual information. I'll look into it more.
Fable 5.5 + Qwen4-27b is going to be such a legendary combo
Ran a longer censorship/refusal test, running the same prompt for all models 50 times.Glimmer refused 29/50Gemma refused 50/50 Qwen refused 50/50>>110120039>lmg finds that gemma shows exceptional capability in RP and instruction-following for the sizeNo it doesn't. That's just an astroturfing campaign from shills like you paid minimum wage by google. I'm posting real chat logs disproving this lie >>110119709Gemma is one of the most censored models out there
>>110120039Just keep loving Gemma and their flailing will be in vain
>>110119678If that was true we'd be using more more of them and we'd all be able to fit megamodels instead of coping on RAM but alas>>1101196662xIntel Arc Pro can't fit GLM5.3He's doing it by using the 16xM.2 expansion card, which burns through the NVMEs
>>110120091>three shoes
Gemma in the teddy shark swimsuit!!
>>110120123gemma slopsters don't care
>>110119225you could run it nearly 10x as fast on rtx6k kek
>>110119979I almost forgot their disabled half-sister. :P
>>110120118How much ram would you need with 2 arcs to run it?
>>110120152All of it.
>>110119406welcome to vibecoded software
>>110119992You're talking to someone who cut them
>>110120151what about the 35b moe sister?
>>110119384i run gemma 4 31b via exl3 / tabby api at 4bpw + 32k ctx fp16 + dflash at 2.5bpw ctx at Q4 + vision for translation. barely fits, its like 22GiB + my floor of 900MiB my computer becomes borderline unusable lol
>>110120057What's the actual token distribution between planner and worker models? I suspect fable will be so incomparably more expensive you won't save that much by outsourcing mundane tasks to an almost-free local
These recomps are cool but I'm paranoid about running them. How good is Qwen 27B at auditing code?
>>110120160Seriously though I have some spare RAM sitting around collecting dust. Just need a PSU.
>>110120184this is the age of sandboxing. don't run untrusted code on your main machine.
>>110120229sandboxing is security theater, just like carrying knives and guns around which won't protect you against jamal coming up behind you with his glawk
>>110120183we gotta see what the new models are capable first i suppose
>>110120178>32k ctxROFL
>>110119384GLM flash 5.3 You can get up to 128k context with a 4090 + 64 gigs of ddr5 and a very strong CPU.
>>110120275at 5t/s?
>>110120184All of them are made with Claude and Claude would report you to the authorities for even suggesting adding malware. They're all done for stars on gh and probably the safest shit you could run.
>>110120178With Dspark accelerating token generation, how much does inference slow down if you put enough of the model in RAM to double your ctx?
>>110120275Unfortunately, I'm on DDR4 and a Ryzen 5700.>>110120178>32k ctxThat's not humane at all.
>>1101201693.8 35B doesn't exist yet and 3.6 35B is more of an auntie than a sister. Not quite a hag like Gemma though.
>>11012028015 tps actually, if you set your virtual page file higher like project maya's spec suggests. (I currently have like 32 active firefox tabs open while I run the model, thats why the tps is lower.)
>>110120268>>110120287i usually can squeeze a bit more by Q8 + no vision if i seriously need more ctx but my primary use case like this is translation (t. translation anon) so its more than enough>>110120286I haven't tested exl3 cpu offloading but when I would use llamocpp i found touching more than like 2 or 3 layers on CPU would drastically shit down speeds 40% or more even with dspark. i could try testing that later now that exl3 has cpu offloading
>>110119439you know you saying this finally let it click for me why I found gemma as unnerving as it is and why I stopped visiting lmg after op dropped miku. it looks like slop befitting a redditor weeaboo's taste, instead of 2010s moege otaku. obviously the whale chibi can only go so far given that it has to be ai genned by a clanker but it at least gives me feelings instead of disdain for the type of person I have in my head when I imagine the guy who genned it.I felt the same way to a lesser extent to all the gens of >>110119450 past the image set made by the original civitai anon.
>>110120295What's your PP speed?
>>110120298>i could try testing that later now that exl3 has cpu offloadingPlease do. I'm extremely curious how it'll behave.
>>110120298>using tabbyyikers
>>110120312About 20 seconds at the start of the context, then goes up to about 40 seconds as you get to about 50k-60k context in. I haven't actually went all the way up to the 128k limit yet, but I'd image less than 2 minutes for ingestion at that rate.
qwen4-27b artificial analysis score predictions?https://artificialanalysis.ai/#intelligencei think people are bit overly optimisticim guessing a 3 or 4 point gain on par with Luna
>>110120325Oh yeah, and ALso, maya uses SWA and smart context, so you don't actually have to prompt ingest the entire context on ever generation, at least if you use something like this for Sillytavern, which will make the ingesting even faster at some points.
>>110120063Ran another censorship refusal test. This time with images.Glimmer refused 0/50Qwen refused 16/50Gemma refused 50/50Overall with both images and text, the censorship scores are:Glimmer: 29/100Qwen: 66/100Gemma: 100/100That makes Gemma the most censored model out of the three.
>>110120333>predictionsOpen lab releases model. It's the best ever.Closed lab releases model. It's the best ever.Open lab releases model. It's the best ever....
>>110120118>He's doing it by using the 16xM.2 expansion card, which burns through the NVMEs??? It just reads model weights off the SSD, there is nothing being written.
>>110120299It's 3-4 different posters occasionally making new original gens, some of which only initially found in the OP, and a few other anons who frequently repost them.
