adventures in booting

David Zeuthen david at fubar.dk
Mon Nov 29 01:49:46 UTC 2004


Hey,

So, I've looked a bit more into the booting process and how to optimize
it. Mostly based on the discussion triggered by Owen's boot poster
challenge, here

 http://www.redhat.com/archives/fedora-devel-list/2004-November/msg00447.html

and also some experiments that I did - basically replacing rhgb with gdm
as described here 

 http://www.redhat.com/archives/fedora-desktop-list/2004-November/msg00066.html

What I've done is a bit crude - I've replaced init(1) with a shell
script based on /etc/rc.d/rc.sysinit and tried to optimize specifically
for my system: IBM Thinkpad T41 laptop with a Pentium M processor at
1600MHz along with 512MB of RAM.

The results are pretty good I think, here is the general time line made
with a wallclock

00: exit grub; start booting the kernel
04: kernel prints audit()
11: initrd is mounted; Red Hat nash visible
     mount / ro (normal initrd procedure)
13: start bootchart logging; start readahead of approx 193MB files
     sleep until readahead is complete
24: readahead done; now
     create /dev and modprobe (in background)
     mount / rw, enable swap
     start xfs
     startx as user davidz in background
     start messagebus
     start hald
     start acpid
     start NetworkManager
32: X claims the display
34: GNOME desktop banner
40: GNOME desktop is usable (Nautilus desktop, panel fully populated)

Here is a bootchart made with the bootchart software from Ziga Mahkovec:

 http://people.redhat.com/davidz/bootchart.png

You may notice that I also start firefox after login and it starts very
very fast - that's because readahead loads all files used by Firefox -
in earlier experiments I've also added files from OpenOffice.org to
readahead and that meant I could start up OpenOffice.org Writer in about
three seconds. More below.

I've made the following observations

1. The kernel patch, linux-2.6.3-printopen.patch, wasn't really working
   well for me - it reported far to few files - instead I added a
   printk() to fs/namei.c:link_path_walk() 
   (disclaimer: I don't know much about the kernel so there may be a
   better solution than this). 

2. The data captured from link_path_walk() was massaged into a list
   of unique files to preload and sorted on sectors.

3. While capturing the data link_path_walk() and before processing
   I went through all the menus in the GNOME desktop (to make sure
   their icon and desktop files would be added to the list) as well as
   loading Firefox. The list contains 5189 unique files - 231 of these
   from my home directory - 103 of these from gconf in my home
   directory and 302 from gconf in /etc. 2267 were .png files and
   814 of them were .desktop files. 488 files had ".so" in their name.
   There was a total of 193MB of files (which says something about
   the footprint of the GNOME desktop on Fedora :-/)

4. Doing the readahead really helped the time from startx till a 
   usable desktop - less than 10 seconds!

5. Doing readahead on the 5189 files took about 45 seconds on my
   system, mostly because the files were scattered around the disk.
   Since I had a spare partition 17GB partition, I did this:
    a. format spare partition as ext3
    b. copy all readahead files to spare partition (193MB)
    c. copy rest of files from main partition to spare partition
       (about 9GB)
   Now the readahead is down to 11 seconds which averages out to
   be 18MB/s. On the other hand, I can still see (using fileblock)
   that the files in the readahead is still scattered out and hdparm
   says I should be able to get 33.87 MB/sec with no seeks.

6. I made a hack to cache /dev (a dev.tar file) and the list of modules
   to load. This could be used in production if the kernel could give
   us basically a hash value for the kobject hierarchy representing
   the hardware (perhaps even a 'tree /sys |md5sum' would suffice).
   This shaved some seconds of as well.

7. A number of things was started in parallel - I found that doing
   readahead while running modprobe wasn't helping anything; in fact
   it contributed negatively to performance (a bit to my surprise, I
   guess because the kernel was busy).

8. readahead on the right files is A Good Thing(tm). Booting my system
   without readahead on the partition with the readahead files scattered
   took 58 seconds (compared to 39 with readahead on the optimized
   partition)

    http://people.redhat.com/davidz/bootchart-without-readahead-scattered.png

   and without readahead on on the optimized partition it took 43
   seconds

    http://people.redhat.com/davidz/bootchart-without-readahead-nonscattered.png

   again compared to 39 seconds. As an added bonus, the readahead
   makes sure that e.g Firefox loads fast; all .png and .desktop files
   are in place for when using the menus. As mentioned, one could put
   very big apps like e.g. OO.o in the readahead set.

So, I think these numbers are good and there's still some room for
improvement; e.g. it takes ten seconds from grub to when the initrd is
mounted - surely the kernel can boot faster? It's after all 25% of the
time spent from grub until I have usable desktop.

The bad thing is that this approach is highly specific to my system (and
thus why I'm not posting an RPM with it :-), however I think it clearly
shows where improvements should be made; here are some random thoughts

 a. We should keep track of files being loaded and maintain the
    readahead fileset as appropriate. printk() doesn't seem like the
    right solution; perhaps a system daemon using inotify or the
    kernel events layer is the road ahead? This would enable us to
    readahead the KDE stuff if the user is e.g. using KDE a lot.

 b. ext3 should support operations for moving blocks around; e.g.
    optimize around the readahead fileset - when idle the system should
    rearrange the files to facilitate faster booting

 c. the start_udev and kmodule process could be cached as I did above

 d. The whole init(1) procedure seems dated; perhaps something more
    modern built on top of D-BUS is the right choice - SystemServices
    by Seth Nickell comes to mind [1]. Ideally services to be started
    would have dependencies such as 1) don't start the gdm service
    before /usr/bin/gdm is available; 2) the SSH service would only
    be active when NetworkManager says there is a network connection;
    /usr from LABEL=/usr would only be mounted when there is a volume
    with that label and so forth. Also, such a system would of course
    have support for LSB init scripts.
    (This is probably a whole project on it's own so I'm omitting
    detailed thinking on it for now)

Thanks a lot to Ziga Mahkovec for the bootchart software - it's been
very useful.

Have fun,
David

[1] : http://www.osnews.com/story.php?news_id=4711
      http://www.gnome.org/~seth/blog/2003/Sep/27




More information about the fedora-devel-list mailing list