I highly discourage anyone to use Ubuntu Core as part of their solution. It is a secure environment, yes, but it is a nightmare to configure (snaps) and for everything, expect for the most basic stuff, you will need what is called a "brand store", which costs about $20k/year, regardless if you have 1 or 100 devices.
(We could not even make one snap talk to another snap, checking another snap status etc).
We had commercial discussions w/ Canonical and we basically moved away.
I like Ubuntu, and looked into using Ubuntu Core for an IoT architecture a few years ago - but the "brand store", which IIRC is mandatory for private deployments, are complete madness.
Plus, while the concept behind Snaps is interesting, in practice they seem to be nearly universally reviled.
> Plus, while the concept behind Snaps is interesting, in practice they seem to be nearly universally reviled.
It might be worth mentioning that a more open, community driven and decentralized option offering mostly the same concepts exists in the way of Flatpak and Flathub.
And that Linux Mint, a very good derivation of Ubuntu LTS, made the choice to ditch Snap and integrate Flatpak instead in their version of the Software Store.
it's reviled because nobody wants to learn yet another deployment tool. apt has been reliable enough for installing packages and good enough UX that people can't be bothered
Was going to say the same thing. IMO the snap concept would make it worth learning something new for some use cases - but forced automatic snap updates, high memory usage, long startup times and permission issues are not selling it. Moreover, Canonical is well aware of how snap is perceived, and has done diddly squat to fix it.
You can’t really compare the two wrt IoT. For Debian you need an additional OTA update mechanism. Luckily rauc is in the Debian repos but you still need to setup everything yourself: A/B partitioning etc.
I'm surprised they ended up going with a non-immutable distro. They say their Debian-derived system is as robust as Ubuntu Core, but seemingly immediately contradict this by saying they're using apt to manage updates.
apt updates are not transactional, so abort the update mid way and you have a partially applied update. Hopefully the packager was careful enough to ensure your system still boots and reapplying the update cleans everything up.
We had the some problem evaluating it for our startup. We were planning on using it to deploy ROS in the field and leverage the benefits. I was quoted $20k/year to get started and I think that also included 10 or 20 devices in the field. Then additional costs for that.
There was so many hacks to get it working with ROS since we rely on hardware for robotics to communicate with sensors, actuators, and network. Basically broke the security model to get our application working. We didn't go with it and went Debian as well :D
Been a huge fanboy since Warthog, but as of 20.04 I've sought refuge elsewhere. Turns out, Ubuntu isn't nearly as user friendly as it was back then, in fact, it's one of the least user friendly distros these days! Everybody progressed and you deserve t check it out yourself!
Not the one you're replying to, but started my Linux experience with Ubuntu 6.10, was using it all the way up to ~17, and started using Arch Linux after that since during my time of using Ubuntu, I learnt enough Linux that I could setup my own system (and of course, with the help of the awesome Arch Wiki).
Worth noting that Arch has had something resembling the OpenBSD installer for some time. Trying it out doesn't require reading 20 wiki articles as it did back in my day™. You answer a few questions, wait a few minutes, and it's done.
Wow, that's great, haven't seen that before! Granted, it was some time ago I did a setup from scratch.
Seems that installer is like 90% on the way to be useful as a general installer. At a glance, seems to missing things like networking setup (as most people use WiFi these days, it seems), but at least it takes care of most things you need for a install.
If you want to set it up and forget about it, just use any RHEL clone (AlmaLinux is by far the fastest with updates: it took them like a week to ship RHEL 9 after the official release). You set up the system, install & enable `dnf-automatic`, and forget that it exists for the next 10 years.
If AlmaLinux (or any other clone) dies for some reason, you can always move to another clone without re-installation (using their tool `elevate`).
Buying a Brand Store is also the only official way to have full control of snap updates. Someone at Canonical must have been taking notes from Microsoft -- take away control from users and sell it back in an "enterprise edition".
No issue with Canonical making some money from their OSS efforts, but as-is the pricing seems crazy. At the least, I wish they'd do something like make brand stores free for less than 100 devices. That would also serve to hook people in, then they pay as they grow.
Ideally though, Canonical would let anyone run their own "brand store" on their own hardware.
I completely sprinting away with anything canonical. SUSE/openSUSE/Micro RHEL/Fedora/CoreOS nothing else...no more canonical, too many disappointments and trickery into pay massive at the end of the day.
That is really ridiculous. I'm sure some companies out there will pay it without blinking, but how are the little people supposed to evaluate things, heck a former boss at a Fortune 500 I know for a fact will say no thank you, and move on. I would just use DEB packages and setup a package repository myself, what does snap bring that regular debian packaging does not?
> what does snap bring that regular debian packaging does not
Snap is like using a python virtualenv, while apt is like using global pip. We all know which one of these usually results in a broken installation.
Apt debs more often than not rely on some specific version of a third party package which becomes a headache when there's another package that needs a different version of it. Snap in theory solves that, making the deps local to the package you're installing.
If I had a dollar for every time I had apt bullshit me with "requires package, but it will not be installed" or something of the sort I'd probably have something over $20 which isn't that much but still infuriatingly high.
As an alternative just package everything up in an AppImage and use a AppRun shim script.. Then just deploy the AppImage to whatever. The LXD Snap bundles quite a few things, there's no reason you couldn't do the same with AppImage.
>If I had a dollar for every time I had apt bullshit me with "requires package, but it will not be installed" or something of the sort I'd probably have something over $20 which isn't that much but still infuriatingly high.
Completely unrelated but the exactness of this turn of phrase here delighted me so much and without your permission I’d like to steal it for my own use in my daily conversations (without credit).
It's an adaptation of Doofenshmirtz's line from Phineas and Ferb:
> "Wow! If I had a nickel for every time I was doomed by a puppet, I'd have two nickels. Which isn't a lot, but it's weird that it happened twice. Right?"
My favorite distro in this regard was openSUSE, Debian and Ubuntu are #2's for me. I just can't install it anymore, it doesn't support the right drivers before the install process finishes, which causes issues. I'm not a fan of having to do anything to trick a GUI based installer to work.
Honestly wondering if a System76 box or laptop would play nicely with openSUSE.
We are talking about IoT devices where (hopefully) the company knows exactly what is installed and what not, so I don't think it is a big issue. There may still be conflicts to solve, but you can manage them when you prepare the deb package, not when you install it
If you've only got one device then it probably doesn't make sense. But if you have a number of devices, you probably have a business and need to make sure that you can reliably and securely update them remotely. How many developer/ops hours do you think that would take? 20k doesn't buy you much in dev hours these days.
Do you still have to have a canonical sso account just to login to your own devices?
EDIT: The answer appears to still be yes. I don't know who canonical is making this for, but no company is going to want to deploy IoT devices that hand over the root of trust for authentication to a third party. And I have to believe that many individuals/hobbyists that would otherwise be excited to use this will choose another distro for the same reason.
Canonical seems to be out of touch with what people actually want, and is happy to build things it likes that will ultimately be irrelevant.
For individuals and hobbyists Ubuntu Server would likely make more sense than Ubuntu Core. I used to recommend it for use on headless Raspberry Pi's over Raspberry Pi OS. Use of cloud-init and netplan made it much easier for deployment of headless RPi's.
Unfortunately for 22.04 they made some changes on the RPi which severely impacted my deployments causing me to move away from Ubuntu on the RPi and to no longer recommend it to others.
What were the changes in 22.04 on the Pi server images that severely impacted you?
(disclosure: I'm probably the one that made those changes, so apologies for that, but I'd like to know what they were in case there's anything I can do about them)
Maybe severely was the wrong word to use. I use RPi's in my home lab so any impact no matter how significant is limited in real world impact. In this case it was limited to loss of access to several services in my home lab for a few hours due to a complete loss of network connectivity to the server and a few hours of my time troubleshooting the issue.
The major issue was caused by moving some of the kernel modules to linux-modules-extra package. It caused an issue when I upgraded because the 8021q module needed for vlan support was moved to this package. I lost network connectivity to the first Pi I upgraded due to this. The Pi in question had an ip address assigned to the interface and to vlans configured for the interface. When networkd could not create the vlans on the interface it failed without assigning the ip address to the interface itself. I filed a ticket regarding this.
I consider the ability to use vlans to be a fairly basic expectation of a server os especially when it previously worked out of the box. I would also think vlan support to be an expected feature for the IoT gateway role pushed as appropriate for Ubuntu Core. Luckily I was able to connect to the serial console, troubleshoot and eventually correct the issue. When the module which supports a commonly used part of the ethernet family of protocols is moved to an optional package, I would expect testing of the impact to connectivity to existing deployments to take place. This led me to lose some faith in the testing done of Ubuntu on the RPi and the stability of the platform going forward. Switching platforms was easy to do in my case and so I did.
Hopefully the recommendations I made in my ticket of either moving the module back to linux-modules or adding an explicit warning to the release notes happens. The current warning regarding some modules having been moved to linux-modules-extra, without any detailing of them including the functionality they tie back to, is completely inadequate.
Fair comment; and you're correct that (to the best of my knowledge) we don't specifically test VLANs on the pi platform at the moment (the majority of the automated testing performed directly on pis is pi-specific hardware compatibility: boot, wifi, bluetooth, etc. etc.)
Something vaguely similar came up back in hirsute with the veth module (preventing containers from configuring ethernet): another thing that wasn't pi specific. That got resolved by moving veth back (which will almost certainly be the resolution in this case). I've found what I suspect is the ticket you mentioned (LP: #1973485) and though it looks like the kernel team have picked it up, it's not resolved yet. I'll see what I can do to get it some more priority (at the very least it should be dealt with prior to the point release).
I've added VLANs to the release notes in the meantime, however, I can't add the full list of functionality because the list is frankly enormous (well over 1000 modules).
They have no clue what the market wants. Nobody is going to shell out $15k-20k to build a solution.
We lost about 4 months betting on it and eventually moved away. We are glad we did.
In our case we were building a solution from scratch. It is very hard to justify upfront costs like that, involving open-source and early in the project.
IoT solution costs should move up according to scale. That's why AWS, Azure, and others have grown their IoT offerings. Nobody starts paying $20k upfront to use AWS IoT.
I read that post and as an outsider and one that knows very little about ubuntu core, I seem to understand you need a brand store to access direct hardware functionality, is that so? so you need to pay to be able to use GPIOs, I2C and other hardware peripherals from your snaps? on an IoT OS?!
I would also chip in along the same lines as the other comments here and strongly advise against using Ubuntu Core in any application. It's a tempting idea when you first look at it, but in practice it is seriously difficult to build an application with, comes with a lot of constraints, and requires you forking over repeated large sums of money to Canonical with no way out.
We had a much better experience with https://www.balena.io – quite a different service, but achieves a sweet-spot combination of having an open core with good management tooling that I'm more than happy to pay for.
I've never used anything in the product space, but I've read through some of the docs because IoT deployment interests me in the context of most network devices having terrible deployment and update strategies.
My favorite sales pitch was always Mender (https://mender.io) because it can be as simple as .img files and slice based booting. Unfortunately they changed their pricing a couple years ago and went from a hobbyist friendly pricing structure where you could scale down to $120 / year to charging for scale and features where the minimum cost to use delta updates is $3k per year. If you want something like mutual TLS authentication it's "call us" pricing.
When I looked at Balena I thought the whole architecture looked like a mess of complexity and containers when a simple .img would suffice for most use cases (IMO). It's also too expensive to get into without having a well understood target market for whatever IoT product you're planning to build. IE: No hobbyists or enthusiasts that just want to learn without having something sellable planned out.
IMHO all of the IoT deployment solutions charging tons of money for features and scale are a good example of where platforms like Cloudflare could reshape the industry. I think by going "all in" on Cloudflare Pages/Functions/Workers, etc. someone could build a "Cloudflare Native" platform that can afford to dominate the unserved (low-end) portion of those markets, but with the ability to scale up to accommodate the large deployments these companies are trying to attract as customers.
I would kill for firewalls and switches that use a slice based, watchdog monitored deployment system like Mender has, but with pricing that's closer to the reality of the small business world.
There is also balenaHub, which hosts many community projects and introduces the concept of "open fleets", bring your device and join the fleet, instead of managing it yourself https://hub.balena.io/fleets
I still don't think the pricing makes sense for anything that doesn't have a solid plan and a sales team. If you graph the cost per device / month it'll be free for 10 users, then a huge jump when you need a paid plan, then an increasing slope until you hit the peak at $14.39 per device / month on the Production plan, then a decreasing slope trending towards $1.00 per device / month (for essentials).
In something like the digital signage market, where IoT device management makes sense, I can buy hosted solutions that include the signage SaaS for around $10 per device / month. On Balena Pilot I'd be paying $6.50 per device / month (initially) and competing against large providers who have costs closer to $1.
The pricing page looks even worse than that at a glance. The thing I see immediately is $1439 / month for 100 devices. The plan is named Production and it's the only one where I can calculate the per device cost without thinking (divide by 100), so it gives me the expectation of $15 per device / month until I read the fine print and pull out a calculator to figure out an actual cost average based on projections.
My gut reaction after spending 5 minutes on the pricing page is that I would need to get into the range of 500 devices before I could be cost competitive unless I'm in a specialized market where I can justify charging a really large monthly fee per device.
A hobbyist trying to deploy a few dozen devices is probably going to have a tough time doing cost recovery in the beginning. If someone is trying to manage a couple dozen devices as a pure cost (just simplified maintenance), I think it would take a lot to justify moving away from doing things manually. I deal with a lot of small business network devices and I value a firmware update on a network device as 3 minutes of labor.
I also have a huge dislike for tiered features and functionality, but that's endemic to the tech industry in general and I can't imagine any established companies being able to move away from it. The reason I dislike it is that anything I would build would be on the scale of a weekend side project and I feel like I have to adjust my strategy based on the scale of my project. For example, I might start with manual device management to keep costs low initially and then move through the different tiers of something like Balena as I grow.
Compared to something like the old Mender pricing where IIRC it was a linear cost per device with all features included and a minimum cost of $10 per month, the current pricing, scaling, and tiering strategies are awful for small developers. I know anything I build as a side project has a limited chance of being successful. I don't want to pre-plan a cost minimizing scaling strategy, but I don't want to YOLO it and hope for the best if if I end up with something successful.
What I'd want out of a platform like Balena is a setup where I can just build the best solution on day one and throw money at it to scale if it becomes necessary. However, the reality is I'll probably end up stuck in those low scale scenarios I described above, so, IMHO, that pricing is built for you to make money off of me failing, not for mutual benefit.
To be fair, I assume Mender couldn't sustain their old pricing, but I think platforms like Cloudflare could make that kind of pricing more tenable in the future. At least I hope so.
Annoyingly, along with this, the Ubuntu "Minimal" network installer is dead (no 22.04 release) — presumably to "encourage" people to move over to this.
At this point all my from-scratch installs are usually Debian-based where they used to be Ubuntu-based.
Canonical seems to completely have lost touch with what the community wants, and with the non-free installer, I find Debian working equally well OOB as Ubuntu has done in the past.
The only machines I still keep on Ubuntu are the ones already running it where migrating them will be too much hassle.
For my latest laptop, out of curiosity, I also tried Arch. It seems to work fairly well, but I haven’t had enough experience with it yet to “dare” put it on servers.
I'm using Manjaro on my laptop and it's been smooth so far, aside from an initial wifi chipset hiccup. I'm using it also on a laptop-as-a-router. Disabled all the graphical stuff. It's been very good, thus far and only 213 MB of RAM being used after disabling some (but certainly not all) unnecessary services. I do with they had a minimal install for x86_64 like they do for ARM.
Ubuntu Core seems to be Snap As A Service. Snap is mostly open source (everything except the actual store backend because "it would be complicated" to do that).
If you fork it and remove the connection to Ubuntu's cloud ecosystem, you'd probably be end with a barebones Ubuntu Server install. When all you need is Ubuntu Server with Livepatch, you're probably better off using that.
Look for balenaOS (http://balena.io) if you want an alternative IoT solution. We looked into ubuntu core, which seems like an overkill in complexity. With balena you can get started immetiately with all the tools you already know, like docker etc and have a solution that works...
While balena is great for getting up and running quick, you'll quickly run into issues when more specific application requirements arise, or you need to tighten down security.
It's fantastic for small-scale IoT deployments, but you very much need to play nice in their ecosystem, and there's many features missing for deployment and version management still, and balenaOS and the balena supervisor can often introduce breaking quirks, and managing networking is a nightmare. We're very happy with how it allowed us to get off the ground and running quickly, but are moving to a more "traditional" build using yocto on its own moving forwards.
Another solution would be Apertis (www.apertis.org). Unlike Ubuntu Core, it uses standard technologies like OSTree, Flatpak etc to provide similar features.
It is a Debian derivative distribution with a stabilised subset of packages and a recent LTS kernel. Target images use non-GPL-3 software, which some industries require. It also comes with infrastructure for building images, atomic updates and so on.
There's a lot of concept documents on the website going into details on those.
When using OSTree, you can build images using regular packages installed with apt, and then deploy them atomically while allowing to rollback the update if necessary. Alternatively you can choose to use Flatpak to deploy applications.
I really never understood this thing about Ubuntu as a minimalist OS for servers and embedded. Almost no matter what, it's going to be more bloated and complicated than just about any other flavor of Linux you can think of. Sure, it works, and there's the factor of familiarity, but just... why? Make Ubuntu the best desktop or mobile OS it can be. Without the original selling point that it fixed problems downstream from Debian and made install easier, what exactly is the point? Ubuntu does some things well, but they've never been good at keeping anything minimal.
Almost more interesting than this announcement by itself is that Canonical is now providing a PREEMPT_RT patched kernel for Ubuntu in general. That's super cool.
What IoT devices are running Ubuntu, and how is it optimized? That it has snap containers? I’m guessing by IoT here they don’t mean light switches running on an embedded processor, but industrial equipment?
It does contain raspberry pi's but the list isn't all that great.
I had trouble finding 32 bit ARM precompiled binary distributions from ubuntu (that is not a rpi), and had to compile my own kernel + inline it with debian to get 32 bit arm support.
Yes, you can actually. I haven't done this myself, but there are defconfigs for mmu less devices (stm32f4 series for instance). You need some more RAM though, but most evaluation kits don't provide this. Have a look here: https://elinux.org/STM32
I'm planning to give this a spin on an EVK that I got from work if I find the time.
Linux? yes. Popular distro? not really. Lack of MMU means that all the processes (and shared objects) use the same memory space, so unless you have some way of randomizing where things go (like SELinux), two processes will step on each other's data (MPU will not trigger segfault, as both are allowed to r/w same addr - IIUC). But even when using randomizing, you're playing russian rulette.
drivers, as you said, for filesystems for instance,
syscalls available that we know and love :),
somewhat easier deployment (elf instead of raw bin),
other abstractions that linux provides,
multiprocessing (if done carefully),
real multithreading,
