Gpt 5.6 better than Mythos 5 that's really good
>>109142465who cares about benchod marks?
>>109142465Bloody benchold bastard beach
>>109142485What is your argument against benchmarks?
>>109142465It's unironically over for software. And all of that happened just in the past 7 months. It's fucking depressing.
>>109142535That they cannot actually measure rationality and intellect, which is where the real value of AI lies. We as a class of entities do not even have an agreeably objective system of determining such ourselves. All you're doing with benchmarks is verifying whether they can meet arbitrarily set and predefined goals, which is a measure of automation and baseline competency; but baseline is bare minimum and concerns known quantities, which doesn't necessarily track how something deals with unknowns needing to be discovered.Benchmarks are misleading and to the largest degrees irrelevant.
>>109142535these are language models. what is the benchmark for language? there is none. the only thing benchmarks measure is performance of a prompt engineered for a specific task, which is an extremely narrow description of the potential applications of a language model. fixating on benchmarks is how you get shit like VW benchmaxxing their cars to pass emission tests, and that's a domain where the solution space is rather restrained. for natural language the solution space is so vast that the only thing benchmarks do is become a perverse incentive for labs to focus on it in order to maximize it not for the utility of the bench but literally just better marketing when their model does better versus the next competitor. there's still innumerable other ways you could state another problem that the model probably sucks at because it hasn't been maximized for it. so it becomes whack-a-mole, the model became good at the bench which incidentally made it better for some tasks, but in real world use turns out it became an annoying redditor sycophant that's irritating to work with.
