Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D
Really depends on the details of your goals, but why not just VMs?
The "problem" with Pi-like devices is that they're usually not very "normal". The process of provisioning is different, IIRC they only "recently" supported booting off something other than the SD Card, and in the case of the Zeros, you'll either be using Wifi or an external USB Ethernet dongle (over USB OTG no less). Sometimes they need specially compiled version of linux, so you're stuck far from mainline (this was a big component of the RPis success) This may be distracting from your goals of learning about clustering.
I suspect the $10 Pi Zero is about as cheap as you'll get though, depending on your personal costs of case + ethernet dongle + USB power supply, etc.
Instead of having to use lots of dongles and usb ethernet, I just wired them all up using brass rods, a small 5V power supply in the base, and boot them over WiFi (just the kernel and wifi config on the sd card).
Raspberry pi's are cheap, easily available, and there is an absolutely massive trove of information about them on the internet. And the scale means that the linux implementation is very stable and "just works" to a degree that is extremely hard for other SBCs to match.
Sure, VMs are the logical choice, but not everything has to be logical. Real hardware does feel more real :-)
You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.
Wanted to reply to you directly also to increase the chance you see this because I think I had exactly the same intrusive thought as you, and actually built such a cluster recently. Would love to hear what you think: https://x.com/andreer/status/2007509694374691230/photo/1
I think that's amazing, surprisingly physically nice looking, and now you've gone and reminded me that Plan 9 is an option, which is kinda another tangent I didn't need:), but IIRC Plan 9 is really low resource so it might be a good fit (aside it lending itself to distributed computing). Have you written up the build anywhere?
No, not beyond what I already put in the twitter thread. I wanted to wait until I had some cool distributed software running on it too, but then I ran into trouble with the plan 9 wifi drivers for the rpi being unstable and so I'm still working on fixing that. It does serve as a great test bed for that purpose too, as with 8 nodes I can get much more reliable data about how often the driver gets stuck
I feel you, but something like an R730 or 7810 with a pair of E5-2690 v4 and 128GB RAM can be had for under $400. Not the most power efficient, but you'd have to run it quite a while to make up the difference in energy cost. Plus there's way less work in getting it all set up.
I have a bunch of old Rpi 3B's in a 19" 1RU rack space i use for toy clustering. PoE splitters from a PoE switch to power them which also provides the network uplink. All very neat and self contained and when it comes times to upgrade/replace if they all fail (i have a couple of spares) I can just pull out the 1RU for disposal.
Intel NUC's are probably much better value for money these days but the 3B's were pretty cheap at the time I bought them.
I work in e-waste recycling, and specialize in micro desktops. The most common model I have, Optiplex Micro 3050 (7th gen i5, 8gb RAM, 128gb SSD), goes for about $60 each.
