Do you think we could all pull our computing resources to run Kimi K 3.1 in a distributed way?
>>110290462I remember seeing someone post a thing that already does this, like p2p LLM hosting
>>110290524There used to be one back in the Llama 2 days but it died
>>110290462the speed of it would be extremely painful
>>110290553for you
>>110290462>pullDid you mean ‘pool’, Pradesh?
>>110292088yeah
>>110290553not really I think, k3.1 works at 7168 hidden size, with speculative decoding you could get an average of 2 tokens per forward pass, 14kb in fp8 which with an open connection so some small overhead at 10mb/s could move at 50 times per second. Most of the time would be spent computing on the GPU even assuming each node only holds 1 layer.For a moe like this, most likely each node would get a few experts. So each forward pass just waits for one hidden state and outputs a token.
>>110294573The problem would be network latency. An average of say 100ms per participant with each participant holding a layer would add up quick (about 10 seconds per token).
>>110290462Found this study:https://neurips.cc/virtual/2023/poster/71336>abstractLarge language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals — a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents. https://github.com/bigscience-workshop/petals
>>110290462Yes absolutely>>110290553Not if you also distributed the training, theoretically it would get faster and faster
>>110290462https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.mdllama.cpp has a experimental feature that does this, apparently
>>110290462https://en.wikipedia.org/wiki/Speed_of_light>one foot 1.0nsno
>>110295850fuck this gay earth I'll go back to ai porn thanks to thisnow i just need to learn how to do the same but with image/video/audio
>>110295850awesome
Unity engine has a lot of stuff already in place for this, it could even be done anonymously.
>>110296095it's like bitcoin but with AI
>>110296110i believe bitcoin was an op to get hardware moving in the direction AI needed>new tech stack requires massive computation of a specific type>will require research $$$ to unlock>introduce new pre-technology that will pay for it via money laundering feesor, "dirty money" needed a way to enter the legitimate market. It was a standing reserve of cash. Crypto lets them wash the cash. But, the mechanism requires certain technical hardware.So, to create an ecosystem of hardware of the type ai needed, they designed crypto to also need it, but made crypto extremely profitable (from a money laundering perspective).its hard to explain, ive never tried to explain my idea before. but essentially crypto was an op to unlock ai hardware and have Putin pay for it all. pic is unrelated OC
>>110295850>>110295978https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/rpc.mdsame thing but for images
>>110296188I love the demiurge now. Nigga's smart.
