Get Real!
I don’t often write about work, mainly because much of what I do is mundane. But I had cause to port Ingo Molnar’s Realtime CPU Limit patch to an old-ish version of Linux used in some embedded equipment. I found it really useful, and now I can’t imagine being without it.
First, some background. The kernel is the program that is in overall charge of what happens on a computer. It spends most of its time organising any other programs on the computer that do actual useful things. About 2003 version 2.4.20 of Linux (which is a free kernel) was growing in popularity for desktop users. In 2004 there was much work to improve the ‘real time’ abilities of Linux, making system designers able to allocate the resources on the system to best ensure that things are done on time. The perceived quality, freedom from license fees and ‘real time’ capability led to 2.4.20 with some ‘real time’ patches being used on many projects.
Academics might argue over the meaning of real time, but from a pragmatic point of view we can see that it’s a bit of an illusion. It allows you to say that some tasks are more important than others, but in the end you can still have more work than will fit into the time available – no amount of organisation will resolve that problem. And this is where the interesting stuff starts.
The part of the kernel most affected by the real time changes was the scheduler. That is what divides up the time that the computer spends on the available tasks. Before real time, there was just simple priorities for tasks and the scheduler would make sure every task got to run for some time, with the higher priority ones getting the most time. This meant that some routine tasks that users take for granted, such as playing sounds or moving the pointer in response to the mouse movements, could seem sluggish on a computer that was busy.
The real time changes introduced a new class of tasks, with priorities above all other tasks. The new scheduler would always put these real time tasks before the non real time tasks, taking it in turn to do the important things then, if there was time, doing some non real time things.
The problems we found were tricky to diagnose. In one case there could really be more ‘important’ work than could be done in the time, so the system would neglect the other tasks completely. The same problem could be triggered by faulty software – if two real time tasks decided to chat to each other endlessly then the system would do nothing else. In these situations the system would not be usable and it would be impossible to find out what was going on. It would only end when the watchdog timer, an automatic safety device, kicked in and reset the system.
I don’t know if this was seen by other people but once I realised what was happening, a search of the ‘net revealed that Ingo Molnar, who had done much of the original real time work, had a patch for the more recent 2.6 kernel which cured the problem. The patch has since been incorporated into the 2.6 kernel that we now use on desktop PCs.
The patch works by monitoring the percentage of time that the machine spends doing ‘real time’ work. If that exceeds a predetermined limit then the real time behaviour is disabled until the percentage is below the limit. This means that the system degrades more gracefully under heavy load. All I had to do was port the patch from the 2.6 kernel to the 2.4 one that we use. I added in some warning messages so we could tell when the machine was too busy to cope with its workload, and some information so we could know exactly which tasks were the busy ones.
I was very pleased with the result – a system that instead of hanging then resetting would tell developers that their software was running out of control, and still allow them to get in with tools to see what those runaway tasks were doing. If you are using 2.4 ‘real time’ linux without the patch, I strongly recommend it! (Search for RT_CPU_LIMIT or Ingo Molnar)