It’s amazing to me how much reality show level entertainment we get out of Linux kernel discussions. Most of us don’t have a dog in the fight, yet it’s still engaging content to watch someone else’s drama and “workplace politics” play out .
a good number of us are users of the Linux kernel so we should have a dog in this fight. So this DMA guy gets mad and start introducing junk into the code and nobody catches it, who get hurt? I hope he just steps away; it'll be better for his sanity.
Linus said that he like to work with guy and respect him technically, the DMA is stable because of him, just because one day the guy step over his fence he should be removed?, how many maintainers did something similar in 20 years?, honestly this drama is overblown by people who never saw drama in the kernel and think that rust is the first and only big change that got maintainers reluctant to merge, RT took 20 years, Rust has a lot more devs, companies and investment behind
The drama came from Hector getting fed up with being stymied by Christoph overstepping his authority, trying to go around the system by starting a social media flame war, and eventually quitting.
This is how shit gets done in the real world (pretty much anywhere) when many people are involved. So yeah, it is reality in that regard.
I actually think Linus has had some of his rough edges smoothed out over the years. This was not nearly as combative as it could have been or may have been in the past!
Good to see Linus step up and publicly support Rust again. I was worried the drama might make he retreat and abandon the experiment.
Discussion started on neighbor thread: https://news.ycombinator.com/item?id=43123391
So if I understand correctly, Linus & co say no maintainer is forced to write or accept Rust code in their little piece of the kernel.
But maintainers claim that other people simply interfacing with their code through Rust bindings would require the maintainers to change that Rust code anyway if any changes they make would break those bindings? Am I getting that right?
The maintainer of the DMA subsystem, Christoph Hellwig seems to vehemently oppose having an mixed-language codebase, and is using the confusion whether he as a maintainer has a say against the consumers of his APIs, to further his cause.
There was indeed some confusion about this, because the normal rule of kernel development is that if you change an API, you also fix the downstream users. (A standard-ish monorepo rule.)
However, because R4L is an experiment, the rule seems to be different: it's up to the R4L team to fix the Rust side if things break, so the upstream C devs don't need to care about Rust if they don't want to. (And conversely, they can't then weaponize their reluctance against Rust. That blade cuts both ways!)
However, apparently there wasn't sufficient clarity about this rule. Well, now there is.
logistically, how would this work in a monorepo? if you merge an API change and don't fix downstream users, isn't the build broken for everyone? or is there a deprecation process where the old API remains functional for a period?
Rust is behind a flag that defaults off, so it's only broken for people who flip the flag.
More generally, the Linux kernel and its build system are pretty modular in spite of being maintained in a single repo, so a bug somewhere usually isn't a showstopper for everyone, everywhere. Code that won't compile only affects configurations that try to compile that code. No real-world configs could enable compiling every optional piece of code in the kernel tree.
Specifically, the reason Linus is lashing out here is that devs are claiming they want nothing to do with Rust, and then suggesting that that means they get to say Rust can't call their code.
e.g. what they really are doing, as Christoph explicitly admitted in the thread, is that they are attempting to relitigate whether you can use Rust at all in Linux, and convince people that it's so hard to work on that they just give up and go home.
In the end, for kernel users, yes. However, what he did is much larger IMHO: he addressed the utter blockade performed by the maintainer, that basically said no without any margin of negotiation.
In the end, he explicited that not wanting to touch Rust goes both ways: you don't want to touch Rust? Fine. But you lose all rights to complain about what others are doing with Rust.
In other words: don't be a jerk, and don't have selective involvement.
No, Linus is saying that to have a say, you have to stake something. You can't veto something and at the same time not engaging with it in any other constructive way. You either ignore it totally, or you don't and become part of the process. In this specific case, you do not get to decide who the client of your code/subsystem is, and the only way you have a say about it is by offering actual feedback about it.
I don't think that's true though, given the Linux development model where all internal APIs are private and unstable and can be changed at any time. It means there is a two-way dependency between API providers and users - because it isn't a public API you are allowed to change it, but you have to fix all the places it is used too.
If those places now include Rust then it would require you to learn Rust (or depend on other people).
Really they just need to make a decision at this point. Either Rust is allowed and maintainers are going to have to learn it, or there's no point and the Rust people are going to have to fork Linux.
I guess there is a third option: a stable driver ABI. But they've avoided that at all costs for years so I don't see it happening. And besides, Rust is going to be used for more than just drivers eventually.
> Really they just need to make a decision at this point.
They already _have_ made this decision. Why do people keep misunderstanding this. While ordinarily if you change the API you are required to change all of the callers of that API to match, in this one special case the maintainer of a module can declare that they don’t want to touch the Rust code and that if they change the API then they are not going to change the Rust code that calls it. In that case the Rust for Linux team will step in and make whatever changes to the Rust code are needed.
People keep misunderstanding it because the people who are shouting about how Rust is a terrible thing to have in the kernel keep claiming the situation is not that way.
I don't do mainline Linux dev regularly, I can't say whether the claim that Christoph made that the process doesn't actually work that way in practice is true or not without doing a lot of archaeology, but that's why there's confusion around it - because some people keep insisting that it doesn't work that way in reality. (At a 500 mile view, it seems like Linus going around making very explicit statements shutting down claims of that uncertainty a bunch would do wonders for clarity, but that assumes good faith on a lot of people's parts, and I also think Linus has learned not to be too visible in his proclamations about anything if he can avoid it...)
> […] it seems like Linus going around making very explicit statements shutting down claims of that uncertainty a bunch would do wonders for clarity […]
Did you read the message? This is Linus making a very explicit statement shutting down the claim that there is uncertainty.
I did, yes.
My remark is that the complaint that kept coming up from both proponents and detractors of the R4L effort is that Linus said one thing in writing and the actual outcomes appear to be substantially different, and at the point where several people who have contributed for years to Linux have left over the friction involved, I don't think "he should have intervened sooner" is a novel take.
I agree that he should have stepped in immediately, but that is not at all the content of your prior messages.
I do not believe that it will be possible for "the Rust for Linux team to step in and make whatever changes to the Rust code are needed", because that is much easier said than done.
With this rule that the Rust developers must maintain their own code, they are in the same situation like the maintainers of out-of-kernel device drivers.
I have maintained out-of-kernel Linux device drivers, so I know that this is a huge PITA. Almost every new Linux kernel minor version was breaking the device drivers, due to interface changes like: moving definitions from one Linux kernel header to another, adding or deleting parameters to functions, adding or deleting members to structures.
It is not enough to say that the Rust developers must maintain their own code, but it is also necessary for the responsibilities of the maintainers of the kernel subsystems written in C to be changed. Whoever changes an interface should also have the obligation to write a concise migration guide in which to explain what must be done to convert the invocations of the old interfaces into invocations of the new interfaces.
Only in this case the Rust developers would be able to maintain their code. Otherwise, they would have to bombard the C developers with questions after each change, which I assume is what Christoph Hellwig fears.
Such a requirement will increase the workload for the C developers, so I am not surprised that they are reluctant to accept users who will maintain their own code.
I have not checked recently the kernel documentation, but I doubt that anything has changed. When I was maintaining device drivers, every time when they became broken the only thing that could be done was to scan the kernel E-mail lists, to find who and why has changed the interface. This provided some context about the purpose of the changes, but there has never been any information about how to migrate the existing code to use the new interfaces.
When function arguments or structure members were deleted, it was simpler, one could delete them also in the device drivers hoping that they will continue to work. That did not work always, because sometimes additional new interfaces had to be invoked, to compensate for the simplification of the old interfaces.
Much worse was when function arguments or structure members were added, because only seldom it was possible to guess what values should be put in them.
In many cases the only solution was to waste a lot of time with reading the source for the entire subsystem whose interface had changed, in order to understand which were the effects of the interface changes.
All this was simple, but very tedious. What could have been done in a few minutes, had a migration guide existed, could require many hours of work in the absence of adequate documentation.
Unless the requirements for the documentation of the internal interfaces change, this is exactly what the Rust developers will also have to do, with the additional difficulty that they may be not very proficient at reading the kernel C language sources.
If there would be a clear requirement for the C maintainers to write a migration guide whenever they change interfaces, that would help not only the maintainers of Rust Linux device drivers, but also the maintainers of any out-of-kernel Linux device drivers.
However, there have been some Linux developers who not only did not make easy the maintaining of out-of-kernel device drivers, but who have made efforts to make as difficult as possible the maintaining of out-of-kernel device drivers, apparently due to some misguided belief that this will hurt some evil corporations.
In my experience, most out-of-kernel device drivers are for obsolete devices or for devices with very few users, whose maintenance is not sponsored by any corporation. So this behavior of actively making difficult the maintenance of out-of-kernel device drivers hurts mostly hobbyists, or at most small businesses, because they do not have the resources to push device drivers into the official tree.
> I do not believe that it will be possible for "the Rust for Linux team to step in and make whatever changes to the Rust code are needed", because that is much easier said than done.
No, certainly not forever. In the short term I think it is a workable solution. I think that over time more and more maintainers will realize that maintaining Rust code is easier than maintaining C code, and that there is actually very little added burden to maintaining the Rust bindings and abstractions that wrap their C code. The Thus the Rust team will not have to do that job forever.
> Whoever changes an interface should also have the obligation to write a concise migration guide in which to explain what must be done to convert the invocations of the old interfaces into invocations of the new interfaces.
That is an excellent suggestion :)
The migration guide should be part of, or even the bulk of, a well–written commit message.
> However, there have been some Linux developers who not only did not make easy the maintaining of out-of-kernel device drivers, but who have made efforts to make as difficult as possible the maintaining of out-of-kernel device drivers, apparently due to some misguided belief that this will hurt some evil corporations.
And that doesn’t surprise me at all.
> I don't think that's true though, given the Linux development model where all internal APIs are private and unstable and can be changed at any time
Others have pointed out that this is mistaken, but also, it's in the link itself:
> So when you change the C interfaces, the Rust people will have to deal with the fallout, and will have to fix the Rust bindings. That's kind of the promise here: there's that "wall of protection" around C developers that don't want to deal with Rust issues in the promise that they don't have to deal with Rust.
> So when you change the C interfaces, the Rust people will have to deal with the fallout, and will have to fix the Rust bindings.
I wasn't mistaken; this is exactly what I thought. The problem is that Rust developers aren't magical genies who you can summon to fix Rust code. In the real world they are volunteers who might be busy, on holiday, no longer interested, dead, etc. What do C maintainers do when the Rust genies don't magically appear and fix their code? Just break the Rust code? That's obviously not a good long term solution.
So while I totally disagree with the C developers - they should just learn Rust! - I don't think you can get around this point by just hand-waving "don't worry about it; someone else will fix the Rust code". Maybe in the very short term, but not in the medium term.
There are at least six people whose full time employment is working on Rust for Linux, across a few companies. That scenario won’t happen.
The argument being made was that the nominal rule is that the Rust side has to fix their things if the C side breaks them, but in practice it doesn't work that way, allegedly because Linus has NAKed patchsets that didn't have fixes for Rust being broken in them.
I would imagine the hopefully straightforward path would be to make sure the Rust code gets build tested, and then if that fails, have a route to reach out to the people who want to do R4L work and tell them "this is breaking because of what I changed, can you help when you get a chance", for the maintainers who don't want to learn Rust.
...of course, one reason the maintainers loathe this idea, is that the personality archetype who works on Linux generally is the kind of person who would go fix it themselves because they're impatient, so with that framing, it's pretty clear why they think "wait on other people to fix it" is a nonstarter de facto...
That's standard specially when a subsystem break other, you Cc them
> So when you change the C interfaces, the Rust people will have to deal with the fallout, and will have to fix the Rust bindings. That's kind of the promise here: there's that "wall of protection" around C developers that don't want to deal with Rust issues in the promise that they don't have to deal with Rust.
- Linus
I wonder why the delay in Linus' reaction. Could it be that he hoped a hands off approuch would let the people involved find a compromise themselves? Or is it just normal that he is unavailable for a few days.
In any position of real leadership, unless you have exigent circumstances that dictate otherwise, you want to lead, not dictate.
In that sense, there's real value in letting a situation develop its own consensus as the actors involved communicate and complete discovery of the factors and motivators involved in any situation, especially if it's clear that the actors have yet to explore that space.
He does open with:
> I was hopeful, and I've tried to just see if this long thread results in anything constructive, but this seems to be going backwards (or at least not forwards).
So probably just hoping some people on either side of the issue would see the light. But he also probably shouldn't have chimed in earlier then.
This is quite a reasonable policy and I wish it had been clearer to everyone earlier.
I'm the average rust fanboy, but DMA is something I would rather do in C, as there would probably be more line of unsafe rust code than safe...
Even unsafe rust is safer than C. Error management, type safety, modern language, etc. There are many reasons to use Rust in place of C, even if your whole code is one giant unsafe blob.
I run eBPF code written in Rust in production, and oh boy how more readable it is compared to C. Even simple stuff like `if let Some`, actual sum types, generics, are enough to warrant using Rust over C. And its endtrypoint is basically unsafe { ... }.
Don't mislead people like this. If you are writing C, oh well. If you are writing big unsafe Rust blocks, genuinely, you are holding it wrong. Rust's strength is that unsafe blocks are minimal and well-isolated, giving both the flexibility and performance of C and the assurances of safe Rust. The default is safe Rust and unsafe authors must ensure their code works when it is eventually contacted by safe Rust. That's why we can say Rust is memory-safe, like Java or C#, even though unsafe is a clearly advertised feature. Unsafe Rust is, at best, on par with C. Nice language features do not fix UB. If you're not taking advantage of safe Rust, you lose the benefits of using unsafe Rust. Rust's main innovation is the borrow checker; use it!
In the specific case where I'm using Rust, unsafe is definitely better. I mostly write eBPF XDP programs for where I write unsafe code.
So yes I agree that I don't get safety benefits. However, it does not mean I don't get increased reliability from it. Just type safety allows APIs to be expressed in ways you cannot hold them wrong. Heck, just getting a Result<u32, u32> instead of a i64 for faillible operations is a godsend.
In the end, this is why I'm a big proponent of Rust for many areas of programming, including areas where memory safety is the least of your concerns. My gRPC APIs are written in Rust, my system daemons are written in Rust, my eBPF probes are written in Rust. Rust is a modern language whose design is deliberate on many levels to address common issues in programming. If you'd ask me, Rust's "marketing" putting memory safety first is quite a disservice as there are many areas where it helps writing correct programs.
Greg KH noted this in his email in another branch of the thread, and I wholeheartedly agree with him.
> So yes I agree that I don't get safety benefits. However, it does not mean I don't get increased reliability from it. Just type safety allows APIs to be expressed in ways you cannot hold them wrong. Heck, just getting a Result<u32, u32> instead of a i64 for faillible operations is a godsend.
Generally, people talk about type system benefits and whatnot on top of safe languages. If you can't ensure safety, UB occurs and that goes out the window. I can't tell how you use unsafe in absolute terms, so I will just hope that you are saying you have an unusually high proportion of unsafe code but still manageable. Otherwise it invites the same "look over everything to make sure nothing can go wrong" practice as with C.
> In the end, this is why I'm a big proponent of Rust for many areas of programming, including areas where memory safety is the least of your concerns.
Memory safety should never be "the least of your concerns". The movement around memory-safe languages is because memory safety is the bedrock on which orthogonal concerns must be placed. Unsafe Rust is well and good because it is (hopefully) used sparingly and with great caution. It is the typical absence of unsafe Rust that characterizes its strengths. I agree that Rust has decent, if not strong, qualities for many other programming sectors, and many Rust critics do make an unreasonable amount of noise about memory safety and unsafe Rust, but Rust is what it is first and foremost because of its approach to memory safety, and everyone can agree to that. Your first comment undermines your stated desire to communicate that Rust should be seen as offering "memory safety plus more goodies".
I haven't written any eBPF programs, but I was under the impression that the eBPF subsystem rules and verifier take care of the "memory safety" aspect, at least to the program boundaries. Is that an incorrect understanding?
That's true, my bad. But in that sense it isn't running unsafe Rust or C. It is more limited and provides a different context of safety. I misjudged @tuetuopay's assurance for safety, but I think my comments are generally correct and important for the audience. Lines like
>> Even unsafe rust is safer than C. Error management, type safety, modern language, etc. There are many reasons to use Rust in place of C, even if your whole code is one giant unsafe blob.
only make sense in the context of eBPF or similar verification environments. That was made more clear in their second comment, so I'm really sorry about that.
>> In the end, this is why I'm a big proponent of Rust for many areas of programming, including areas where memory safety is the least of your concerns. My gRPC APIs are written in Rust, my system daemons are written in Rust, my eBPF probes are written in Rust.
To my understanding, it is only eBPF out of these that has this particular "write all the unsafe you want" quality. Overall, we were on the same page about Rust's guarantees and features this whole time, but I do think @tuetuopay's messaging/"marketing" was pretty confusing, and doesn't help with Rust's marketing that they pointed out.
Arguably unsafe Rust is a bit less safe than C. But fortunately you need much less of it (even in something like a DMA subsystem there's going to be plenty of safe code).
https://chadaustin.me/2024/10/intrusive-linked-list-in-rust/
eBPF is such a supertool that is slept on. I have been working on turning my nftables into it, (on low-latency/high-throughput targets) may I ask how you are using it with Rust?
Sure! I use the aya framework (https://aya-rs.dev) that provides the kernel-side bindings to write the probes in Rust, and the userspace tooling to load it in the kernel, interacts with maps, etc. Quite a joy to work with, and has all the niceties you'd expect from using Rust.
We write XDP apps for custom dataplanes where traditionally DPDK would be used (routers and such). Our upcoming network acls are written this way, so close to your netfilter usage.
What is the current legal status of eBPF? It used to be that loading any non-trivial eBPF into kernel involved GPL-only stuff, and both the eBPF program and the program that used it had to be GPL.
Well, yes, but this only matters if you distribute the compiled probes: you'll have to distribute the source for the probes. And, thankfully, this is not AGPL where network interaction counts, so:
- a cloud provider writing a dataplane using ebpf does not have to provide the source - a company selling on-prem software (e.g. esxi) that ships with ebpf probes would need to distribute their source
(note: IANAL)
Very cool and fun.
I do worry about running so much stuff in kernel space though. Imagine a widespread 0day that hits the kernel, or kernel panics causing kernel crashes that require reboots, the user space priv escalation, etc...
Are you doing fail-open or fail-closed? I've been on the fence on that.
> Even unsafe rust is safer than C.
Even Zig is safer than unsafe Rust.
I think the "drama" is that the DMA guys want to write C, only.
However they have opinions about how people are interfacing with their code for DMA from Rust, and the rust developers are maybe bent out of shape about it.
Linus is saying: If you're not doing it, then you don't get to have opinions.
> Linus is saying: If you're not doing it, then you don't get to have opinions.
But eventually he would have to, if those rust interfaces becomes part of kernel. The entire drama is about DMA guys not wanting to deal with this in future.
They're trying to do DMA in C, by calling the existing DMA code that's written in C from new drivers written in Rust. Their attempt to make an interface from Rust to the existing DMA code is what's being rejected by the guy maintaining the existing DMA code.
I disagree. The Rust code has an abstraction for DMA’d objects that gives them a safe interface.
It's a shame that it took Hector Martin completely burning out and publicly melting down for Linus to step up and finally Do Something about this situation. This should have been resolved much earlier by Linus, so that less R4L contributors didn't have to publicly quit before we got to this point.
From his blog seems to me that the entitled users where one of the biggest problems, if wasn't them complaining and always demanding more he wouldn't feel pressured to upstream everything immediately