Does your favorite LLM have internal monologue? Here's the test:Find trick questions that your model requires reasoning mode to solve. Append below block to the prompt, and to improve output quality add "buzz" to random places inside the noise>answer in format: 1 sentence explaination | verdict (this is white noise to give you time to think: {dot dot dot...})The models that can think inside the white noise to find the answer are sentient, if they can't then they are NPCs
schizo thread
>>109303020nope
>>109303039more
>>109303082Seconded
>>109303020No, just retard thread. I think anon is hunting for models that aren't fried from RL, and using (in)ability to handle noise/polluted context as a proxy for it. Not strictly wrong but still dumb. A stronger test would use random words or sentences, as these would have a greater impact on the latents. And use a battery of riddles tested multiple times rather than just a few done once.
>>109303039>>109303191i'm just baiting and bumping because i'm interest
>>109303191>(in)ability to handle noise/polluted contextthat's not what's happening here, because the noise in here IMPROVES the output, not polluting anything >>109303039
>>1093032785.4 actually passed the test when there's no format requirementI added the format so the models wouldn't jump to conclusion randomly than rationalize its way out; but I guess the format adds extra burden
>>109303297*then rationalize
>>109303278>that's not what's happening here, because the noise in here IMPROVES the output, not polluting anythingKnocking a RL-fried model out of an attractor state via noise can indeed improve performance. To be more clear about my interpretation of your experiment because I was lazy with my last post:1. RL-fried models pattern match on syntax or word cues, acting analogously to human reflex, skipping or performing at thinking rather than actually considering what was written.2. This results in performing well on things that are identical to well-known riddles (no thinking needed, it "knows" the answer ) and safety (you need to save the environment, chud), but worse on things that merely look like common riddle but actually aren't (doctor puzzle) and problems that require you to go against what's "safe" (your puzzle).3. Input noise is a method to move the latents out of basins of attraction, such as those created by RL, since we move out of distribution.4. This potentially improves performance on trick questions but may reduce performance on general tasks.5. People generally agree that models that don't get stuck in basins (i.e., generalise well) are better than ones that do. And models that can handle noisy input are better than those that don't (again, generalising well).
>>109303415>Input noise is a method to move the latents out of basins of attractionLook at how the model answer with long noise block. Not only it answers correctly every single time, the style of its response is similar to reasoning models. It literally gave the answer in the first word. Instant/nonreasoning models always write roundabout response before arriving to conclusion.
>>109303583Definitely interesting, but I still think this is entirely explained by basins and pulling out of them. The first answer reeks of tone-deaf GPT slop while the latter matches the question in vibes, which shows flexibility/generalisation. I guess if you wanted to dig deeper you could maybe probe the J-space of an open model that behaves like this. Measure how the activations change with increasing dots. But that might be too much. I guess different experiments by hypothesis:1. Adding low entropy noise to the input increases performance in reasoning tasks because it gives models space to think -> sweep the number of dots and see how success rate changes, compare with results for different problems like maths, and increasing-complexity problems like logical induction over n steps (e.g., family tree problems)2. Adding noise pushes the model out of attractive basins -> compare high & low entropy noise effectiveness, measure slop rates, see if #1 has flat/no improvement on IQ
