Showing posts with label kernel. Show all posts
Showing posts with label kernel. Show all posts

26 January 2014

Interview: Linus Torvalds - "I don't read code any more"


(This was originally published in The H Open in November 2012.)

I was lucky enough to interview Linus quite early in the history of Linux – back in 1996, when he was still living in Helsinki (you can read the fruits of that meeting in this old Wired feature.) It was at an important moment for him, both personally – his first child was born at this time – and in terms of his career. He was about to join the chip design company Transmeta, a move that didn't really work out, but led to him relocating to America, where he remains today.

That makes his trips to Europe somewhat rare, and I took advantage of the fact that he was speaking at the recent LinuxCon Europe 2012 in Barcelona to interview him again, reviewing the key moments for the Linux kernel and its community since we last spoke.

Glyn Moody: Looking back over the last decade and half, what do you see as the key events in the development of the kernel?

Linus Torvalds: One big thing for me is all the scalability work that we did. We've gone from being OK on 2 or 4 CPUs to the point where basically you can throw 4000 [at it] – you won't scale perfectly, but most of the time it's not the kernel that's the bottleneck. If your workload is somewhat sane we actually scale really well. And that took a lot of effort.

SGI in particular worked a lot on scaling past a few hundred CPUs. Their initial patches could just not be merged. There was no way we could take the work they did and use it on a regular PC because they added all this infrastructure to work on thousands of CPUs. That was way too expensive to do when you had only a couple.

I was afraid for the longest time that we would have the high-performance kernel for the big machines, and the source code would be separate from the normal kernel. People worked a lot on just making sure that we had a clean code base where you can say at compile time that, hey, I want the kernel that works for 4000 CPUs, and it generates the code for that, and at the same time, if you say no, I want the kernel that works on 2 CPUs, the same source code compiles.

It was something that in retrospect is really important because it actually made the source code much better. All the effort that SGI and others spent on unifying the source code, actually a lot of it was clean-up – this doesn't work for a hundred CPUs, so we need to clean it up so that it works. And it actually made the kernel more maintainable. Now on the desktop 8 and 16 CPUs are almost common; it used to be that we had trouble scaling to an 8, now it's like child's play.

But there's been other things too. We spent years again at the other end, where the phone people were so power conscious that they had ugly hacks, especially on the ARM side, to try to save power. We spent years doing power management in general, doing the kind of same thing - instead of having these specialised power management hacks for ARM, and the few devices that cellphone people cared about, we tried to make it across the kernel. And that took like five years to get our power management working, because it's across the whole spectrum.

Quite often when you add one device, that doesn't impact any of the rest of the kernel, but power management was one of those things that impacts all the thousands of device drivers that we have. It impacts core functionality, like shutting down CPUs, it impacts schedulers, it impacts the VM, it impacts everything.

It not only affects everything, it has the potential to break everything which makes it very painful. We spent so much time just taking two steps forward, one step back because we made an improvement that was a clear improvement, but it broke machines. And so we had to take the one step back just to fix the machines that we broke.

Realistically, every single release, most of it is just driver work. Which is kind of boring in the sense there is nothing fundamentally interesting in a driver, it's just support for yet another chipset or something, and at the same time that's kind of the bread and butter of the kernel. More than half of the kernel is just drivers, and so all the big exciting smart things we do, in the end it pales when compared to all the work we just do to support new hardware.

Glyn Moody: What major architecture changes have there been to support new hardware?

Linus Torvalds: The USB stack has basically been re-written a couple of time just because some new use-case comes up and you realise that hey, the original USB stack just never took that into account, and it just doesn't work. So USB 3 needs new host controller support and it turns out it's different enough that you want to change the core stack so that it can work across different versions. And it's not just USB, it's PCI, and PCI becomes PCIe, and hotplug comes in.

That's another thing that's a huge difference between traditional Linux and traditional Unix. You have a [Unix] workstation and you boot it up, and it doesn't change afterwards - you don't add devices. Now people are taking adding a USB device for granted, but realistically that did not use to be the case. That whole being able to hotplug devices, we've had all these fundamental infrastructure changes that we've had to keep up with.

Glyn Moody: What about kernel community – how has that evolved?

Linus Torvalds: It used to be way flatter. I don't know when the change happened, but it used to be me and maybe 50 developers - it was not a deep hierarchy of people. These days, patches that reach me sometimes go through four levels of people. We do releases every three months; in every release we have like 1000 people involved. And 500 of the 1000 people basically send in a single line change for something really trivial – that's how some people work, and some of them never do anything else, and that's fine. But when you have a thousand people involved, especially when some of them are just these drive-by shooting people, you can't have me just taking patches from everybody individually. I wouldn't have time to interact with people.

Some people just specialise in drivers, they have other people who they know who specialise in that particular driver area, and they interact with the people who actually write the individual drivers or send patches. By the time I see the patch, it's gone through these layers, it's seldom four, but it's quite often two people in between.

Glyn Moody: So what impact does that have on your role?

Linus Torvalds: Well, the big thing is I don't read code any more. When a patch has already gone through two people, at that point, I can either look at the patch and say: no, all your work was wasted, and micromanage at that level – and quite frankly I don't want to do that, and I don't have the capacity to do that.

So most of the time, when it comes to the major subsystem maintainers, I trust them because I've been working with them for 5, 10, 15 years, so I don't even look at the code. They tell me these are the changes and they give me a very high-level overview. Depending on the person, it might be five lines of text saying this is roughly what has changed, and then they give me a diffstat, which just says 15 lines have changed in that file, and 25 lines have changed in that file and diffstat might be a few hundred lines because there's a few hundred files that have changed. But I don't even see the code itself, I just say: OK, the changes happen in these files, and by the way, I trust you to change those files, so that's fine. And then I just say: I'll take it.

Glyn Moody: So what's your role now?

Linus Torvalds: Largely I'm managing people. Not in the logistical sense – I obviously don't pay anybody, but I also don't have to worry about them having access to hardware and stuff like that. Largely what happens is I get involved when people start arguing and there's friction between people, or when bugs happen.

Bugs happen all the time, but quite often people don't know who to send the bug report to. So they will send the bug report to the Linux Kernel mailing list – nobody really is able to read it much. After people don't figure it out on the kernel mailing list, they often start bombarding me, saying: hey, this machine doesn't work for me any more. And since I didn't even read the code in the first place, but I know who is in charge, I end up being a connection point for bug reports and for the actual change requests. That's all I do, day in and day out, is I read email. And that's fine, I enjoy doing it, but it's very different from what I did.

Glyn Moody: So does that mean there might be scope for you to write another tool like Git, but for managing people, not code?

Linus Torvalds: I don't think we will. There might be some tooling, but realistically most of the things I do tend to be about human interaction. So we do have tools to figure out who's in charge. We do have tools to say: hey, we know the problem happens in this area of the code, so who touched that code last, and who's the maintainer of that subsystem, just because there are so many people involved that trying to keep track of them any other way than having some automation just doesn't work. But at the same time most of the work is interaction, and different people work in different ways, so having too much automation is actually painful for people.

We're doing really well. The kind of pain points we had ten years ago just don't exist any more. And that's largely because we used to be this flat hierarchy, and we just fixed our tools, we fixed our work flows. And it's not just me, it's across the whole kernel there's no single person who's in the way of any particular workflow.

I get a fair amount of email, but I don't even get overwhelmed by email. I love reading email on my cellphone when I travel, for example. Even during breaks, I'll read email on my cellphone because 90% of them I can just read for my information that I can archive. I don't need to do anything, I was cc'd because there was some issue going on, I need to be aware of it, but I don't need to do anything about that. So I can do 90% of my work while travelling, even without having a computer. In the evening, when I go back to the hotel room, I'll go through [the other 10%].

Glyn Moody: 16 years ago, you said you were mostly driven by what the outside world was asking for; given the huge interest in mobiles and tablets, what has been their impact on kernel development?

Linus Torvalds: In the tablet space, the biggest issue tends to be power management, largely because they're bigger than phones. They have bigger batteries, but on the other hand people expect them to have longer battery life and they also have bigger displays, which use more battery. So on the kernel side, a tablet from the hardware perspective and a usage perspective is largely the same thing as a phone, and that's something we know how to do, largely because of Android.

The user interface side of a tablet ends up being where the pain points have been – but that's far enough removed from the kernel. On a phone, the browser is not a full browser - they used to have the mobile browsers; on the tablets, people really expect to have a full browser – you have to be able to click that small link thing. So most of the tablet issues have been in the user space. We did have a lot of issues in the kernel over the phones, but tablets kind of we got for free.

Glyn Moody: What about cloud computing: what impact has that had on the kernel?

Linus Torvalds: The biggest impact has been that even on the server side, but especially when it comes to cloud computing, people have become much more aware [of power consumption.] It used to be that all the power work originally happened for embedded people and cellphones, and just in the last three-four years it's the server people have become very power aware. Because they have lots of them together; quite often they have high peak usage. If you look at someone like Amazon, their peak usage is orders of magnitude higher than their regular idle usage. For example, just the selling side of Amazon, late November, December, the one month before Christmas, they do as much business as they do the rest of the year. The point is they have to scale all their hardware infrastructure for the peak usage that most of the rest of the year they only use a tenth of that capacity. So being able to not use power all the time [is important] because it turns out electricity is a big cost of these big server providers.

Glyn Moody: Do Amazon people get involved directly with kernel work?

Linus Torvalds: Amazon is not the greatest example, Google is probably better because they actually have a lot of kernel engineers working for them. Most of the time the work gets done by Google themselves. I think Amazon has had a more standard components thing. Actually, they've changed the way they've built hardware - they now have their own hardware reference design. They used to buy hardware from HP and Dell, but it turns out that when you buy 10,000 machines at some point it's just easier to design the machines yourself, and to go directly to the original equipment manufacturers and say: I want this machine, like this. But they only started doing that fairly recently.

I don't know whether [Amazon] is behind the curve, or whether Google is just more technology oriented. Amazon has worked more on the user space, and they've used a fairly standard kernel. Google has worked more on the kernel side, they've done their own file systems. They used to do their own drivers for their hard discs because they had some special requirements.

Glyn Moody: How useful has Google's work on the kernel been for you?

Linus Torvalds: For a few years - this is five or ten years ago - Google used to be this black hole. They would hire kernel engineers and they would completely disappear from the face of the earth. They would work inside Google, and nobody would ever hear from them again, because they'd do this Google-specific stuff, and Google didn't really feed back much.

That has improved enormously, probably because Google stayed a long time on our previous 2.4 releases. They stayed on that for years, because they had done so many internal modifications for their specialised hardware for everything, that just upgrading their kernel was a big issue for them. And partly because of the whole Android project they actually wanted to be much more active upstream.

Now they're way more active, people don't disappear there any more. It turns out the kernel got better, to the point where a lot of their issues just became details instead of being huge gaping holes. They were like, OK, we can actually use the standard kernel and then we do these small tweaks on top instead of doing these big surgeries to just make it work on their infrastructure.

Glyn Moody: Finally, you say that you spend most of your time answering email: as someone who has always seemed a quintessential hacker, does that worry you?

Linus Torvalds: I wouldn't say that worries me. I end up not doing as much programming as sometimes I'd like. On the other hand, it's like some kinds of programming I don't want to do any more. When I was twenty I liked doing device drivers. If I never have to do a single device driver in my life again, I will be happy. Some kind of headaches I can do without.

I really enjoyed doing Git, it was so much fun. When I started the whole design, started doing programming in user space, which I had not done for 15 years, it was like, wow, this is so easy. I don't need to worry about all these things, I have infinite stack, malloc just works. But in the kernel space, you have to worry about locking, you have to worry about security, you have to worry about the hardware. Doing Git, that was such a relief. But it got boring.

The other project I still am involved in is the dive computer thing. We had a break in on the kernel.org site. It was really painful for the maintainers, and the FBI got involved just figuring out what the hell happened. For two months we had almost no kernel development – well, people were still doing kernel development, but the main site where everybody got together was down, and a lot of the core kernel developers spent a lot of time checking that nobody had actually broken into their machines. People got a bit paranoid.

So for a couple of months my main job, which was to integrate work from other people, basically went away, because our main integration site went away. And I did my divelog software, because I got bored, and that was fun. So I still do end up doing programming, but I always come back to the kernel in the end.

25 January 2010

Why There is no Kernel Hacker Sell-Out

As you may have noticed, posting to this blog was light last week, as in non-existent (OK, so you didn't notice.) This was because I was engaged in some serious geeking-out at the LCA2010 conference. One of the talks that I saw came from Jon Corbet, who gave a run-down on recent changes to the Linux kernel. A statistic that he mentioned along the way has garnered much comment: the fact that "75% of the code comes from people paid to do it.” In particular, some have leapt on this figure as proof that kernel coders have “sold out”, and that the famed altruistic impulse behind free software is dead. I think this is nonsense.

On Open Enterprise blog.

23 April 2009

Tropical Disease Initiative Releases Kernel

No, really:

There is an urgent need for identifying new targets for drug discovery. This urgency is even more relevant for infectious diseases affecting third-world countries, which have been historically neglected by the pharmaceutical industry. For example, only ~10% of the R&D resources have been spent on illnesses that represent the 90% of the total disease burden in the world (Munos 2006), which translates in that just ~1% of newly developed drugs are for tropical diseases (Maurer et al. 2004).

At the beginning of the 90s, an initial Linux kernel conceived and created by Linus Torvalds paved the way for a wealth of open and free software programs and operating systems. Here we introduce what we believe can be regarded as an initial kernel for drug discovery with the hope that it will sparkle new ways for developing drugs against organisms that cause tropical diseases. The TDI kernel (v1.0) includes 297 potential drug targets against the 10 selected genomes and is freely and publicly accessible in a World Wide Web server, which was developed with Web2.0 tools for easy dissemination of the deposited data.

It's amazing how metaphors can spread.

You can browser the kernel - *their* kernel - here.

10 October 2008

Linus and the Art of the Kernel Release

In case you missed it, something of truly global importance happened last week. No, not the collapse of capitalism as we know it, something much more profound: Linus started a blog. His first post suggested that it won't be of much interest to the enterprise open source world, since it's really a *personal* blog....

On Open Enterprise blog.

24 June 2008

NVidia's Unenviable Pariahdom

Failing to take the hint, NVidia decides to remain a pariah:


Nvidia reiterated that it won’t provide open source drivers for Linux because the company claims there is no need for it.

Nvidia provides binary Linux drivers and has open sourced some drivers such as the nv X driver and other utilities that work with the proprietary driver, including the installer, config and settings.The company is a leading provider of graphics cards and software for the desktop and embraces a cross platform strategy.

“NVIDIA supports Linux, as well as the Linux community and has long been praised for the quality of the NVIDIA Linux driver. NVIDIA’s fully featured Linux graphics driver is provided as binary-only because it contains intellectual property NVIDIA wishes to protect, both in hardware and in software,” the company said in a statement released today, in response to Linux kernel developers’ criticism of vendors that produce only closed source drivers.

02 April 2008

Linux: Too Much of a Good Thing?

The transformation of the Linux Foundation from a rather sleepy, peripheral player into one of the main voices for open source has been fascinating to watch. It's certainly welcome, too, because one of the problems of Linux in particular, and open source in general, is that the distributed production has tended to lead to dissipation in terms of getting the message across.

Now, in addition to a useful series of interviews with open source luminaires, the Linux Foundation is getting into surveys:

The Linux Foundation (LF), the nonprofit organization dedicated to accelerating the growth of Linux, today announced it is publishing a new report written by kernel developers Jonathan Corbet and Greg Kroah-Hartman, and LF Director of Marketing Amanda McPherson.

The report titled “Linux Kernel Development: How Fast is it Going, Who is doing it and Who is Sponsoring it?” is available today. The paper finds that over the last three years the number of developers contributing to the kernel has tripled and that there has been a significant increase in the number of companies supporting kernel development.

Even though Linux has achieved near-ubiquity as a technology platform powering Internet applications, corporate servers, embedded and mobile devices and desktops, mainstream users know very little about how Linux is actually developed. This community paper exposes those dynamics and describes a large and distributed developer and corporate community that supports the expansion and innovation of the Linux kernel. The Linux kernel has become a common resource developed on a massive scale by companies who are fierce competitors in other areas.


Among its findings:

o Every Linux kernel is being developed by nearly 1,000 developers working for more than 100 different corporations. This is the foundation for the largest distributed software development project in the world.

o Since 2005, the number of active kernel developers has tripled, reflecting the growing importance of Linux in the embedded systems, server, and desktop markets.

o Between 70 and 95 percent of those developers are being paid for their work, dispelling the “hobbyist” myth present from the start of open source development.

...

o More than 70 percent of total contributions to the kernel come from developers working at a range of companies including IBM, Intel, The Linux Foundation, MIPS Technology, MontaVista, Movial, NetApp, Novell and Red Hat. These companies, and many others, find that by improving the kernel they have a competitive edge in their markets.

But one result seems slightly worrying to me:

o An average of 3,621 lines of code are added to the kernel tree every day, and a new kernel is released approximately every 2.7 months.

o The kernel, since 2005, has been growing at a steady state of 10 percent per year.

Surely that means that Linux is steadily becoming more and more bloated? I've always been of the view that one of Linux's great virtues is leanness, especially compared to a Certain Other operating system. While change can be good, I don't think that more is necessarily is better when it comes to lines of code. Perhaps the Linux Foundation's next project could be to study how much of the kernel could be trimmed away to return it to its earlier, svelte self.

07 November 2007

Why Linux is Great, Part 9392

What makes the code in the kernel so great is not that it goes in perfect, it's that we whittle all code in the treee down over and over again until it reaches it's perfect form.

So says Dave Miller, and he should know.

25 January 2007

Has Dave Miller Gone Nuts?

Dave Miller is one of the top kernel hackers. I had the pleasure of interviewing him for Rebel Code all those years ago (eek: how time flies). But something seems to have happened to him on the way to Australia:

I live for the warm nuts

20 September 2006

Why Linus is the Boss, er, Captain

Because he talks the talk:

She's good to go, hoist anchor!

Here's some real booty for all you land-lubbers.

There's not too many changes, with t'bulk of the patch bein' defconfig updates, but the shortlog at the aft of this here email describes the details if you care, you scurvy dogs.

Header cleanups, various one-liners, and random other fixes.

Linus "but you can call me Cap'n"

(Via Tuxmachines.org and ZDNet Australia.)

13 December 2005

Is KDE Winning?

Wow. Linus "my name is Linus and I am your god" Torvalds has spoken - and the answer is KDE.

If you haven't been following the great war between GNOME and KDE - and frankly, if you haven't, you probably made a wise decision - over what the face of GNU/Linux will be on the desktop, this news may not mean much. After all, Linus is just one voice, right?

Well, no: Linus is *the* voice when it comes to the Linux kernel, and his views carry enormous weight within the free software world. For this reason, he has always assiduously avoided coming down on one side of the other regarding desktop interfaces - until now.

As well as this flat-out endorsement from the man himself, there are other straws in the wind that suggest that KDE may emerge as the preferred desktop environment. For example, the amazing Knoppix live CD system, which has spawned dozens of other live CD systems, has always opted for KDE. More recently, the increasingly popular Ubuntu GNU/Linux distribution, which uses GNOME, has appeared in an KDE incarnation known as Kubuntu, and received the blessing of He Who Is Known As The SABDFL.

The reason this all matters is that 2006 may well prove the year that GNU/Linux makes its big breakthrough on the desktop. Just like 2005 and 2004.