Interview: Linus's latest lieutenant
A Q&A with Marcelo Tosatti, maintainer of the 2.4 Linux kernelRobert McMillan
(bob@linux-mag.com)
Editor at large, Linux Magazine
March 2002
Where did he come from, and what are his plans for 2.4? Marcelo Tosatti opens up to Robert McMillan about the joy, the fear, the challenges, and the rewards of being the Linux kernel maintainer. He also talks about his favorite hack.
Stepping into Alan Cox's shoes is no mean feat, even for the most accomplished of hackers, but that is just what a little-known, 18-year-old Brazilian hacker named Marcelo Tosatti did this fall when Linus Torvalds handed over maintenance of the Linux 2.4 kernel tree to Tosatti instead of Cox. According to all parties involved, Cox wanted more time to pursue other projects, and after having maintained the 2.0 and 2.2 trees, it was time for someone else to do the job.
Since taking over the job, Tosatti has managed three kernel releases, starting with 2.4.16; has answered, according to one account, close to 700 press queries; and has given the tersest Slashdot interview in history (see Resources later in this interview).
developerWorks caught up with Tosatti to ask him about his first four months on the job.
developerWorks: Were you really contacted by 700 journalists when you were made 2.4 maintainer?
Tosatti: 700 was a big number. It was not exactly 700, but a lot of journalists were trying to contact me. It was more like 30. Now they've stopped. The big problem with journalists, at least in my case, was that they were non-technical people who kept asking me, "What do you do? Who are you?" Not technical journalists, but the mainstream journalists.
dW: What was the most annoying question you were asked?
Tosatti: How you started working in computers.
dW: Okay, I'll cross that off my list, then. But I read that you had been working at Conectiva for four years, and that you're only 18 now. How old were you when you started there?
Tosatti: I was 14.
dW: How did you get a job at Conectiva at 14?
Tosatti: I started working for an ISP at 13, as a system administrator. And there I was administering some Linux boxes, so I became interested in working for a company where I could learn more about Linux, where I could work as a researcher. Then I e-mailed Conectiva, asking if they were interested in someone who knew a bit about Linux and who could work with their distribution.
dW: So was that a part-time job?
Tosatti: It was part-time, because I was studying.
dW: How did you first get into Linux?
Tosatti: Really, it was because I could look at the source. That was the only reason I got interested. I first installed Linux in my box just to see how it worked, but I knew I could have more control over the system than I had with Windows. That was basically the thing that drove me to use Linux. I already had used FreeBSD.
dW: How does one get made the maintainer of the Linux kernel?
Tosatti: It started with a discussion between Alan [Cox] and Linus [Torvalds]. Alan was saying, "I'm not interested in maintaining 2.4, because I want to do other projects." And then Linus said, "OK, who would you suggest?" And then Alan suggested me. Linus said, "That's OK with me." And then Linus forwarded me their discussion and he said, "are you interested?" And I said, "sure."
dW: Why did Alan pick you?
Tosatti: I don't know. I guess it's because I can deal with people easily. I do not start ego fights. I set a target and I try to achieve it. You'll see people on the Linux Kernel List fighting for ego reasons, you know, "I'm better than you." I just don't do that. I guess that's the biggest reason. There are other reasons, which Alan has mentioned on his diary, I guess because I have good technical skills too.
dW: Who would you have picked?
Tosatti: Aah! What a question! Hmm. Andrew Morton, I guess. And there are some other people too: Jeff Garzik, I guess.
dW: So how did you feel about being chosen?
Tosatti: Happy. In the beginning I got a bit afraid, because of the interviews. I was not used to people asking about my life all the time. I had five or six interviews a day... it was everything I did for two weeks. And at that time, I got a bit frightened, you know. Now I've seen that it was just a temporary thing.
dW: Alan Cox has said that he was a bit annoyed by some of the attitudes people had toward your appointment. He said that people tried to push you around a little bit. What happened?
Tosatti: I didn't know about that. I haven't noticed anything. I'm not sure who he's talking about. I haven't noticed any kind of disagreement from the 2.4 contributors.
dW: What was the first release you did as the 2.4 maintainer? 2.4.16?
Tosatti: Yes. 2.4.16 had basically no press. 2.4.15 had a file system corruption problem, so someone fixed the problem and I released 2.4.16 to fix that.
dW: How many releases have you managed so far?
Tosatti: Two full releases. [Tosatti's third release, 2.4.18, came out after this interview was conducted -- ed.]
dW: So what do you think of the job?
Tosatti: It's hard. Very hard. I guess the hardest part for me is reading -- most bug reports you get say "You have a bug," and not much else. So I have to ask, "What kind of hardware do you have?" The way the bug reports are done is very much of a problem. That's normal, because people who are reporting the bugs are not usually developers. It would be good if people could report them like developers, but they are not, so we have to keep asking, "What configuration are you using," and things like that.
dW: So are you thinking of ways of making the job easier?
Tosatti: In this regard, no. I don't think there's any solution. People just have problems and they report what they have, and I don't think there is anything that can be done. It's just culture. We keep saying, "Hey, we need a real bug report; please give us the information."
dW: So are there things that you're thinking about that could make the whole process better for everyone? Would CVS [Concurrent Versions System] work, for example?
Tosatti: I use CVS internally to maintain my tree, but I do not think that exporting it would be a good idea. We have to centralize -- at least that's my opinion now. All stuff has to come through me so I can read it carefully, and that's a problem with CVS.
For 2.5 it would be good to have some version control mechanism, which people could use to maintain their separate parts, but that would not work with 2.4. I simply cannot change the way 2.4 is. There are just bug fixes now.
I would like to change the way people submit patches. There are some people who submit big patches and do not really explain them. I would like to have the explanation of the problem, a problem description in the first place, then an explanation of how they've fixed the problem, and more comments like, "There could be another fix," or "With my approach there is this or that tradeoff." Some people just send patches without a good enough explanation.
dW: Do you think you are more likely to accept those kind of patches than Linus or less likely to accept them? I think people are wondering about how your style of maintaining the kernel is different.
Tosatti: I think I am less likely to ignore any patch. I read and reply, saying, "Hey, please be more specific," or "explain more."
One difference is sometimes I annoy developers by saying, "Hey, I'm just waiting for a fix from you to release the kernel." I guess Linus and Alan didn't do that.
dW: So you pester people?
Tosatti: Sometimes I pester people. For example, right now I am pestering Alexander Viro to send me a patch to release 2.4.18.
I guess I ignore fewer people than Linus. Actually, I try to never ignore people.
dW: You work right next to Rik van Riel, right?
Tosatti: Yes.
dW: So would you have done things differently with the switch to Andrea Arcangeli's virtual memory [VM] system? [Linus Torvalds switched to a new virtual memory system with the 2.4.10 kernel release, dropping code written by van Riel -- ed.]
Tosatti: I wouldn't have switched it at that time. Something about Linus that is a problem is that a lot of things come from his mood. He'll just say, "Oh, this is good: Apply. This is good: Apply." He should not accept some of the patches he does. He accepts some patches too early, without thinking too much about them. He does not have the time to really care about some stuff. For example, with Andrea's VM, he just got one VM and then he switched it to another: Boom! He didn't think about all the issues involved with this. I wouldn't have done that the way he did.
dW: Did sitting next to Rik as that was happening give you some insight into the issues of maintaining a kernel? Was that educational for you?
Tosatti: Yes, I think so. I did see the war between him and Andrea. In this respect, a big part of maintenance is about dealing with people. And usually when you're seeing Rik and Andrea, something which involves Rik and Andrea, you are going to learn something about people, because they're usually fighting. And I guess technically, I didn't learn too much from that, but in terms of managing people, I did.
dW: Do you think that had anything to do with Alan's decision not to maintain 2.4?
Tosatti: No, I don't think so. He had been working on the kernel for a long time already. After some time, you just get bored. I think he wanted to work on other projects.
dW: How long do you think you'll last as 2.4 maintainer?
Tosatti: I don't know. I'm not bored now, but it's a stressful thing.
dW: When you look at the work being done on the next major release of the Linux kernel, are there any areas that look like they might be as major, buggy, and complicated as 2.4's virtual memory?
Tosatti: I think the VM will be the most problematic part, as it was in 2.4. I say this because it's very hard to make VM work right for different cases. For example, it's hard to make it work in a 32 GB box and in an embedded system at the same time.
dW: So you're unhappy with the state of Linux's VM right now?
Tosatti: No. We have some problems. Andrea has some fixes on his aa tree for these problems, but I don't want to just grab his patches and plug them into my tree. I want to understand what problems these patches are fixing, and try to fix them on my tree. I'm not saying that I want to redo the work, it's just that some things in his code are not really clean and easy to understand. But Andrew Morton will be doing that work on the next pre series. He will be getting Andrea's VM work and making it integrate with my tree.
dW: How long until all of this will be working to your satisfaction in the 2.4 kernel?
Tosatti: It should happen in the next two major releases. The thing is, I would like to fix them myself, but I do not have time.
dW: Is that a priority for you?
Tosatti: Somewhat. My priorities are fixing some corruption bugs -- we're not sure if they're a hardware problem or not. We're not sure where the thing is, or whether it's present on all kernels, but it's memory corruption, which is a big problem because memory corruption may cause data corruption and crashes.
dW: Is that an Intel problem?
Tosatti: Yes. Apart from stability, I don't have any big concerns. If other people have, they may become my concerns. Right now I just want to make the thing stable.
dW: How will you tell whether or not you're doing a good job?
Tosatti: When I stop receiving bug reports.
dW: Do you think that will happen?
Tosatti: Stability is the important thing for me. I work for a company that provides enterprise service [Conectiva], right? And for them, it's very important to have stability. The servers must be stable, and not only for the enterprise people, but for a lot of things, it's important that the machine stays up, no matter what happens. And I guess that when I stop receiving bug reports because of Linux crashing because of load or instability problems, then I'll be happy.
dW: Do you think that you'll be maintaining the next major kernel's stable release?
Tosatti: I'm not sure, but right now I'm willing to do that, when it happens, and if Linus wants me to.
dW: Why aren't you sure?
Tosatti: Because I'm not sure if Linus will decide on another person to do the work. But I'm willing.
dW: I read an interview with you in Slashdot...
Tosatti: Oh, that Slashdot interview! I didn't like it. I was doing a hundred interviews a day, so I was like, "Aagh, no more interviews!" and I answered their questions very fast, and people got angry because of that.
dW: Well, one of the questions that was brought up was the idea of documenting your work in a clearer way.
Tosatti: You mean the changelog?
dW: Yes, do you think that could be addressed in any way, so people can more easily understand what has been changed with each release?
Tosatti: Well, it's hard. How are you going to explain to a non-developer what a specific bug fix is fixing? That is not easy. Let's say it fixes an SMP race. What are you going to say to an end-user if he doesn't know what an SMP race is? How are you going to explain it to him? It's quite hard. I would like to be able to do that -- to have more time to make the changelogs more verbose, but unfortunately I don't.
dW: Would you accept a volunteer to help you with that?
Tosatti: If someone wanted to volunteer, it would be welcome.
dW: Have you had much interaction with the developers from commercial companies?
Tosatti: Yes, they are contributing a lot. There are some IBM people, for example.
dW: Did companies have specific things they wanted to understand about you, when you first started the job?
Tosatti: Not that I know of. There were people from companies like IBM and SGI who knew me before I was made maintainer, but they didn't know me deeply, I guess. But nobody came to ask me anything.
dW: What are the hacks that you've done that you are most proud of?
Tosatti: I don't know. I don't remember all of them. I've worked with high availability for quite some time, you know the Linux HA Project? I've done some work on Heartbeat and DRBD (it's a block device which you mount on another machine; it replicates between the local machine and the remote machine). I guess I'm most proud of my VM hacks. I really like to mess with the VM and performance.
dW: So did you work on Rik van Riel's VM?
Tosatti: Yes. But I also did some 2.2 performance stuff, and I did some generic filesystem stuff. What I'm most proud of are my filesystem and VM performance hacks.
dW: At times Alan Cox seemed to make political statements with the release of his kernel. Recently, he censored some details of security fixes in the changelog in what seemed to be a protest of the U.S. Digital Millennium Copyright Act [DMCA] of 1998. You've said that you really didn't see the need for politics in the work that you were doing. Why not?
Tosatti: Because, in my opinion, I should try to develop and maintain Linux. I shouldn't put my personal... I shouldn't mess with things. This is Linux development; politics is something separate. I think that if there's some political issue in Linux, like the one Alan raised, it shouldn't be raised that way.
dW: How should it be dealt with?
Tosatti: It depends on the case, but in some other way that is not dependent on the kernel development.
dW: What is the problem with making statements like this?
Tosatti: I don't see any problem. It's just that in my opinion nothing should make the development slower, because I think one of the most important things in Linux development is its speed. I should not make it slower due to my personal opinions.
dW: Okay, so it could theoretically slow down development if you, for example, can't read the changelog because you're in the United States?
Tosatti: Yeah.
dW: So, where do you sit on the political side of all of this? What issues are you particularly concerned with?
Tosatti: I really care about these things. I think the biggest issue right now is the DMCA. I do not have much time to follow all of the political things, but I'm definitely interested in issues that threaten developers and researchers.
Resources
About the author
Robert McMillan is a freelance journalist and editor at large at Linux Magazine.
You can contact Robert at bob@linux-mag.com.
Copyright 2008