Linus Torvalds & Dirk Hohndel: Linux, Rust and Maintainers

[0:00] Now I'm pleased to introduce our next speaker for conversation. Please welcome--please welcome to the stage, Dirk Hohndel, the head of the Open Source Program Office at Verizon and the creator of Linux and Git, Linus Torvalds. Please welcome. Bye.

[0:31] Thank you. Good morning. It's so funny to see all these cell phones pointed at us. I want to take out my cell phone and take a picture of all of you, but I left my phone over there. I'm Dirk Hohndel, I run open source at Verizon, and to quote Jim Zemlin, this is that other guy. Yes. Yeah, I'm the other guy, and I've done this before, and we've done this. Dirk said this is apparently the 23rd time we've done it.

[1:04] And the reason we do it this way is I do not enjoy giving public talks. I'm sure a lot of you understand that it's not the most fun thing in the world, so we've come up with this format where Dirk prepares. questions that I don't know beforehand, and we have more of a discussion in front of you, and that makes me much comfortable and hopefully it works for you people too. Yeah, and usually it works well. And occasionally he throws me curveballs that make me go, what do I do now? So let's see which one this one is. So I'm thrilled to be back in Japan. It's been a few years.

[1:43] sadly. We did our traditional sushi OD last night and it's It's always good to be here. Thank you so much for for having us. The timing changed. It used to be that the Japan conferences were like in June. And so being here pre-Christmases are actually kind of fun. So I'm-- quite happy. - And we've been lucky with the weather too. - Yes, it's been gorgeous. So Linus, you released Linux 6.7 RC4 a couple of days ago.

[2:16] And one of the fun things that came out in this is that we are going to have a Christmas New Year's Eve kernel. Well done. Good timing. Well, I mean, we have releases every... two and a half months, I think 10 weeks is basically what I try to aim for. So this happens every time that Christmas comes around, and we have The merge window, which is my busy time, is right around Christmas, which really destroys Christmas for me. Or we have the quiet period.

[2:54] when we're just trying to find and fix the last few bugs. And this year, happily, Christmas for me is the quiet period when the last few weeks of the release are going on and we're just waiting to make sure that we have nothing big going on that is a show stopper. And that works really well for me, but it means that All the... some maintainers and developers who are now preparing for the next version, 6.8, They will -- they will be in a slight panic just because they know that after Christmas my merge window opens, and that's when they are supposed to have all their stuff ready for the next version.

[3:39] So we'll probably delay that by a week or two just because We do that every year to make the timing work out better because nobody wants to work over Christmas. Yeah, but so with 6.7, what's exciting? Anything that sticks out to you. Um... We've actually had for the last, almost 20--no, not 20 years but 15 plus years, we've had a very regular release schedule. I'm happy to say that 6.7 is-- turning out to be one of the boring releases. Boring is good. - Boring is good. - You don't want to have too much excitement, because excitement usually means big bugs.

[4:23] 6.7 is-- - Bugs with a G, not with a CK. - Yes. Yeah. Thank you. Horrible. But 6.7 is actually slightly unusual. I think it may be the biggest release we've ever had when you just count number of commits. And, uh, The reason for that is actually we have a new file system that has been in the works for like a decade.

[4:56] There's a lot of new commits coming in for that. But it's still interesting to see that Not only are we keeping steady, we are still actually increasing the development speed. and the Kernel sizes tend to be going up, which Which is a good sign, but it obviously also means that there's a lot more work going on and a lot more work for maintainers to deal with. Yeah, besides Linus not liking my joke, I want to apologize to the simultaneous translators, because I have no idea how you would translate this into Japanese.

[5:36] So one of the things we've talked a lot about over the last few kernel releases is the introduction of Rust into the kernel And I think it has been-- relatively steady and quiet. What's your perception? - Well, the rest still is at the point where we don't, we have, The initial infrastructure got merged last year. It's been growing, but we don't have any part of the kernel that really depends on Rust yet. To me, Rust was One of those things that A, made technical sense, but to me personally even more important was that We need to not stagnate as as a kernel and as developers.

[6:22] And so I am always-- excited by trying something new and not getting too comfortable doing the same thing. I mean, I've been working on the kernel now for 32 years. Yeah, 32 years. And that's a long time to work on one single thing, but it's still interesting because it's not the same single thing. I mean, Linux 32 years ago was very different from what Linux is today, obviously. I actually-- often look for things where where we can do new things and we can do things differently, because it's so easy to get stuck in a rut and say, "This is working just fine," right?

[7:06] and rust. has not really shown itself as the next great big thing, but I think during next year, we'll actually be starting to integrate and some even major subsystems that are starting to actively use it. So it's one of those things. It's going to take years before it's a big part of the kernel, It's certainly shaping up to be one of those. - So you're writing Rust code yourself? You're reviewing Rust code? - Oh no. I have been reading Rust code a bit just so that I can make some kind of judgment calls on when something is too horrendous to be included in the kernel. But I have to admit, no.

[7:54] I mean, the colonel We rely on literally thousands of people. Every single release we have, a thousand people involved, and they're not the same thousand people. Quite often we have people, in fact, for the longest time, we've had the statistics be roughly that every release... about half the people involved Send just one patch. And a lot of them never show up again. They may have something small they wanted to fix that they cared about, and they were not really colonel people. They found it for some other reason.

[8:29] and they sent their small, patch to the colonel. And they were never interested in doing anything more. But then-- The other half. keeps coming back. And when it comes to rust, I'm not going to be the one who manages the rust cover because that's not my expertise, as is true of so many other parts of the kernel. I'm honestly I'm less of a programmer these days than I am, I call myself a technical lead because I'm not a manager.

[9:03] I don't manage people, I manage code. So I call myself a technical lead. I'm not, my day-to-day work is not programming, it is merging other people's codes and Rust will be one of those states. Yeah, so Jim in his keynote pointed out that open source developers sometimes are opinionated. sometimes they express their opinions. - I'm sorry? - You have had in the past a reputation for being expressive and not lately of course, but one of the things that I've been wondering about, so Who is currently on your naughty list? Who are the companies that might... Oh, I'm not going there.

[9:46] I went there... Can we get a sign of hands? Should he go there? No, no, no. It's one of those things you... give some company the finger 20 years ago. And that picture keeps coming up every day. So I learned my lesson. I think it's still the number one search result when you look for images of you. I think that's still The first kid, yes. But I mean, to be honest, I mean, I joke, but at the same time, the whole commercial environment has gone so much better over the last 20 years. We used to be in the situation that it was, hard finding documentation for hardware.

[10:27] And some companies would be actively hostile. And that's just not true anymore. I mean, now We have companies that write drivers for their own hardware, and sometimes it's so hard to find documentation because writing documentation is hard. And a lot of companies don't necessarily want to document some of the quirks or bugs in their hardware, but we're in such a good position when it comes to hardware support.

[10:57] compared to where we used to be. So you talked about the changing environment. The Kernel Summit was just a few weeks ago. And one of the topics that has been very common at the Kernel Summit for many years is this overall question of maintainer fatigue and how draining and stressful this role is. what are we doing to improve the situation, which is not just for the individual maintainers but for the Linux ecosystem and for the larger open source ecosystem, I guess.

[11:35] So I mean, I seem-- say, I mean this has been an thing that has been coming up for the longest time. It turns out it's hard to find maintainers. It's much easier to find developers. We have a lot of developers. Every release is mentioned. But to be a maintainer, Um, Some people think that you have to be this kind of super developer that can do it all, but that's not actually true. But to be a maintainer, you have to have a certain amount of good taste to--you can judge other people's code.

[12:08] and Some of that-- maybe innate, but a lot of it is just it takes practice. to look at other people's code and easily be able to tell, is this a good approach or a bad approach? it's usually just a matter of having done it for many years. So to be a maintainer and be good at your job, you have to do it for a long time. And we do have a lot of great maintainers, but the other part of it is You have to be there all the time.

[12:42] uh or you have to find other maintainers that you can work with so that you schedule around your vacations and things like that. For me, being there all the time is not a problem because I love doing what I'm doing. I was on vacation. a couple of months ago and I had my laptop with me And if I didn't have... my laptop with me, I would have been so bored. Because it is what I do, and it is what I've been doing for a long time.

[13:17] But I realize that's not the life for everybody, especially when you have to put in years of your life into this. So we don't have a solution to this. we've been talking about different solutions and many of our maintainer groups have formed different solutions, and it turns out one of the issues is People are hard. Code is easy. To write code you have a right answer and you have a wrong answer. People relationships are hard and being able to work with other developers, work with other maintainers, and work, especially when you have maintainers that work on different things and they have different goals, and they want to push their area in one direction and another maintainer comes in from another area and wants to pull it in another direction.

[14:09] It can be very stressful and It's one of those things where A lot of people seem to think that open source is all about Programming? But... A lot of it is about communication too. And it doesn't need to be the same person. Quite often it's good to have, you have your programmer that does not have a lot of social skills. It happens, right? and then you may need to have a maintainer who's the one who translates, and I say translates not, I mean in Japan, We have the language issue too in many open source programs.

[14:50] project, but when I say translates, I don't necessarily mean language. I mean the context. the reason for the code. Um, And that's again one of those jobs of a maintainer to kind of take take the developer output and translate it into the project and again makes Makes four. a tough job and a certain kind of person. And we're lucky to... to have as many main tears as we do, If you want to be a maintainer, Trust me, there's room at the top. But this is actually where I wanted to go next. Because if you look at the top, if you look at the people at the sub-maintainer level.

[15:37] Most of them these days are either like Greg and have no hair. or they are, that's rough. Or they are like the two of us and they have It's not completely gray yet. Some people kid themselves that the hair isn't completely gray yet. Others have accepted that. But Kidding aside, the the leadership, what you call at the top of the maintainer tree, we are certainly seeing a significant aging there. Obviously as time goes by that has to happen.

[16:14] If I look five years into the future, and a lot of people will start hitting the 60s, and the first ones will approach the 70s. Where do you see this going? It's a--to some degree, it's a good problem to have. I mean, so the Kernel Summit was what-- a month ago, something like that? - Three weeks. - Three weeks ago. And it was actually the first year when I personally reacted to the fact that yes, a lot of us are going gray.

[16:44] Thank you. But at the same time, part of the reason really is The maintainer summit is we try to limit it to about 30 people or so. Of those 30 people, at least three of them had been around for more than 30 years. So they had been around since like, the first year of Linux existing, and the fact that they are still around and still active, and still end up coming to maintainer summits means that yes, they are older and grayer, but it also means that we actually have a community where people do stick around.

[17:20] But that's a double-edged sword. Absolutely. And it's... For example, one of the things I liked about the rust side of the kernel was that there was one maintainer who was clearly much younger than most of the maintainers, and that was the rust maintainer. And, and, uh, we can clearly see that certain areas in the kernel bring in more young people. We had a--in the maintainer--at the maintainer summit, we had this clear division between the file system people who were very careful and very staid and they cared deeply about their code being 100% correct because if you have a bug in a file system, the data on your disk may be gone.

[18:13] So these people take themselves very seriously and their code very seriously. And then you have the driver people who are A bit more, OK. Especially the GPU people seem to be like, anything goes. And you did notice that the-- On the driver's side, have a much easier time finding young people and that is traditionally how we've grown a lot of maintainers including I mean, Greg with no hair.

[18:46] I apologize, Greg. You can beat me up later. I mean, my forehead is getting larger every year, so what can I say? It's not far away from me. Let's switch gears completely and talk about something else. You cannot possibly give a presentation today as Jim proved to us. without talking about artificial intelligence and large language models. Um, I typically say artificial intelligence is autocorrect on steroids. because all a large language model does is it predicts what's the most likely next word that you're going to use and then it extrapolates from there. So, not really very intelligent.

[19:26] Obviously the impact that it has on our lives and on the reality we live in is significant. Do you think we will see LLM written code that is submitted to you as a pull request? I'm convinced it's going to happen, yes. I mean, and it may well be happening already, maybe on a smaller scale where people really use it more as a, as a help in writing code, but It's... it's clearly is something where Automation has always helped people write code. I mean, this is not anything new at Right.

[20:07] machine code anymore. We don't even write assembler. And now we're moving on from C to Rust. So I don't see this as something as revolutionary as all the news talk is every day about AI. It's not an area I obviously-- work with. I'm still very low level. I got into kernels because I love the low level hardware details and that's why I'm still there. But so you say you expect this can help people write code, this can help people get started.

[20:42] But then if we look at the previous conversation and the challenges around code reviews and maintaining, so do you think that large language models will get to the point that they can help us review code, that they can help maintain subsystems? I hope, I hope, because that's certainly one of the areas where which I see them really being able to shine, to find the obvious stupid bugs. Because, I mean, how many people here are actually programmers in this room? - A lot. - A fair number. A lot of the bugs Ivory right a lot of the bugs I see other people right? They're not like I subtle bugs. A lot of them are just the stupid bugs that you did not think about and you don't need to and his kind of higher intelligence to find them.

[21:33] But having tools that warn-- I mean, we have compilers that warn about the obvious, really obvious ones, but having LLMs that warn about slightly more subtle-- cases where where it may just say, This pattern does not look like the regular pattern. Are you sure this is what you mean? I and The answer may be, no, that was not at all what I meant. You found an obvious bug. Thank you very much. So I do think that LLMs are going to be a big -- you call them disparagingly -- Wait.

[22:10] autocorrects on steroids and I actually think that They're way more than that and how most people work is we all are. autocorrects on steroids to some degree. And I see that as a tool that can help us be better at what we do. But I've always been optimistic. The whole -- Hopeful. Hopeful was the word you had. Yes, yes. Helpful, hopeful and humble. Hopeful and humble, that's my middle name.

[22:41] But -- On the other hand, I mean, I have been so optimistic that 32 years ago, I was stupid enough to think that you can write a better kernel than anybody else. So you have to kind of be a bit-- bit too optimistic at times to make a difference. My approach to your lens really has to be in that, hey, This is wonderful. This is going to... I love seeing the optimism. I don't necessarily share it. Now a lot of people disagree with me.

[23:13] But one of the things that I worry about in all this is we see the hallucinations. We see... And that's a technical term for LLMs. They do hallucinate and they do make up stuff. And so the more they are being put into the position where they will automatically do things without an actual human being there to catch them. the more this becomes scary. not scary as in they will rule the world and not in the sci-fi sense, but in the so many bugs that will happen and that will affect our lives or our code.

[23:49] Well, I see the bugs that happen without them every day. So that may be why I'm not so worried. I think we're doing those just fine on our own. And I don't think I can end the AI topic on a better highlight. Let's go from there to my next. topic I wanted to talk about, which is data. Which is, of course, very related to AI, actually. 30 years ago when we started on open source, it felt like code was the thing that controlled the future and the ability to collaborate, to have this, as Jim described it, billions of dollars worth of code that we have created is the determining factor of the future. That was certainly true in the 1990s.

[24:38] If I look at our lives today, it seems it's far more data than controls our world. What the Googles and Amazons and Apples of the world know about us. So, do you get engaged in the discussions about open data? So-- repeating that revolution just in a different realm? - No, no, so one of the things I enjoy most about open source has always been that I can let go and not care about the things that are not my side.

[25:11] I mean, that's... That's been true very much inside the kernel too, where there are AREAS. where I concentrate. my personal effort much more than others. And I let other people who maintain that side worry about. like networking, things like drivers. And the same is very much true of All the LF projects. Uh... I may be one of the more high profile employees of the Linux Foundation.

[25:43] But I keep myself into the kernel. - Which is named the Linux Foundation. - Yes, it's a great name. but I I enjoy the fact that open source and not even open source, the notion of openness has gotten so much more widely accepted. And I enjoy it particularly because I remember what he was. 30 years ago when I had started this project and people would ask me, Why? Right? And people would say, But how do you make money?

[26:20] Right. And this used to be a question and it never comes up anymore. So the fact that we are even talking about open data And nobody even asks, "Why would you do that?" I think that's the real, deal here. Openness has, I think, to some degree become the standard within the industry. And people kind of take for granted that when you have to have big projects, whether they are programming or data, and you end up having them so big that you need to share between companies.

[27:02] The easiest way to share is to just make it open and use a license that allows for completely wide sharing, and I mean this is what one of the things LF does. So the fact that LF does open data is great, but that does not mean that it's a thing I do. But I think that if you reflect on Jim's keynote, A lot of what the Linux Foundation is focused on is that collaboration beyond the individual, beyond the company, to collaborate and do things as a society.

[27:37] trying to be too hyperbolic here. There is a very huge role in having that neutral place that people can come together and do things that-- - Well, I mean, this, that is literally why I'm working at Linux Foundation. because I refuse to ever work at a Linux company. because I did not want to be in the situation where one company or one commercial entity would be the special place. So I agree 100%. You need to have a neutral place. And-- That's why I gave my name to the links below.

[28:15] So open data. What do you think Beyond that will be-- the the direction this openness is taking. We went from open code to open data. Where are we going with this? Where is the--? This is the five year, 10 year thing. And I always, my answer is always the same thing. I'm a plodding engineer. I look at the step in front of me. I don't make grandiose announcements of, this is what the world will be 10 years from now. - I remember there's this email of this guy. He says, "Hey, I started this fun little project.

[28:54] "It's just a little hobby." - Well, I mean, that kind of proves how bad I am at predicting the-- Yes, this is why there are a thousand people in this room, because you're so bad at planning, I guess. Yes, now I actually think A lot of people sometimes think that having of long-term plan is great. And I don't think it works, it doesn't work in politics. The whole five-year planning thing did not work so well in that area and I don't think it really works in technology very much. You have to have kind of an idea of where you'd like to go.

[29:33] But if you don't look... Right in front of you, you will just... fumble around and you'll never actually get to that top of the mountain, right? You may have a great big journey in mind, but it's still one step at a time. And that is particularly true in technology where you can't just change the world in one big leap. You have to take all the small steps to to improve. You are technical. slide until you get-- Well, and since you like to think in relative near-term future, So I figure I'll give you a challenge for for the end of our conversation today. So when the Open Source Summit comes back to Japan next year, Where should it be? I'm suggesting Okinawa. How about you? Well, that would... I mean, I actually enjoy Tokyo. I'm here with my wife, so we've been...

[30:33] walking around and doing odd shopping malls, and have been having a good time here. But Okinawa sounds good to me too. And on that note, thank you, everyone. Good to see you.