Boot speedup with readahead

[Seth Vidal]

On Wed, 2008-09-10 at 20:48 -0400, Michael Stone wrote:
> I'm happy to explain in more detail, though I'll do so in my own order.
> 
> First, some introduction: my software [1] consists of a family of python
> programs (called "compilations", since each is a primitive compiler), designed
> to be run in a mock chroot, which convert collections of packages and hacks
> into publishables (for example a list of installed packages, a XO JFFS2 disk
> image, and a tarball of the filesystem contents). 
> 
>    [1]: http://wiki.laptop.org/go/Puritan
> 
> Its primary goal is to ease the task of creating releaseable disk images for
> OLPC XOs in a reproducible and verifiable fashion by storing all effective
> inputs to the build process in git commits. Its secondary goal is to overcome
> some of the infelicities of pilgrim with respect to error detection, cleanup,
> and debuggability. It's third goal is to be useful to expert Python programmers
> who maintain Linux builds for fixed hardware regardless of their preferred
> distro and package-format. It therefore exists in contrast to general-purpose
> tools like debian-installer, anaconda, revisor, and livecd-tools, which are
> distro-specific build tools with an interest in shielding their users from the
> tool's implementation's underlying error-detection, analysis, and correction
> mechanisms and workflows, often by means of a domain-specific language.
> 
> Smart is convenient to these goals in that it has a thoroughly documented shell
> interface which permits the following useful operations:
> 
> Smart permits me to control my selection of package repositories package more
> succinctly and less distro-specifically than yum; e.g.
> 
>    smart channel -y --remove-all
>    smart channel -y --add bootstrap type='rpm-md' baseurl='http://dev.laptop.org/~mstone/puritan-repo/'
>    smart channel -y --add bootstrap-f9 type='rpm-md' baseurl='http://dev.laptop.org/~mstone/puritan-f9-repo/'
>    smart channel -y --enable bootstrap
>    smart channel -y --enable bootstrap-f9
>    smart update
>    smart install -y olpc-crcimg python-msutils
> 
> I was able to write these basic commands without reference to the smart source
> code using only its man page; could I have done the same with yum?
> 
>    (In release compilations, the rpm repos being sourced would be included as
>     git submodules of the compilation commit thereby achieving full history good
>     reproducibility.)
> 
> Next, after adding two new repos (for olpc-specific packages and for general
> F-9 packages), I run
> 
>    for pkg in firefox seamonkey mozplugger kdebase kernel; do
>      smart priority --set $pkg olpc-joyride 100
>    done
> 


I think you can do all of the above in the yum shell

yum shell
>repo enable foo
>repo disable bar
>config assumeyes 1
>update
>install pkg1 pkg2 pkg3
>run

> in order to bias several packages toward the OLPC repo. I could do something
> similar in yum with the '--exclude' option or by modifying the yum repo
> configuration, but it would be much more annoying either way in that I'd need
> more complicated code to pipe the results of my configuration phase into the
> installation phase.

You can set the repo cost, too.

> 
> Next, mirror configuration:
> 
>    smart mirror --add http://koji.fedoraproject.org/static-repos/dist-olpc3-build-current/i386/ http://koji.fedoraproject.org/packages/
> 
> This command essentially lets me perform url-rewriting as needed. This ability
> is not _necessary_ for my purpose, but it sure is convenient from time to time.



> >> priority system, 
> >available in a yum plugin - though questionably useful to begin with.
> 
> Yum plugins seem, at first blush and in general rather than in specific, highly
> inconvenient for my purpose because of issues like the:
> 
>   * lack of uniform quality control 
- So all features in smart have complete uniform quality control? I've
read smart's bug reports. I don't think all features are tested
identically.

>   * absence of centralized documentation 

>   * potential lack of unified error handling conventions,
>   * perceived difficulty of installation and potential for version mismatch
>     between plugin and host,
>   * lack of a API stability guarantees

What on earth? We've had a few bugs but the yum api and the plugin api
has been stable for quite a while, now.



> Second, please acknowledge that I am, for the time being dedicated first to the
> XO hardware and second to Fedora. This being what it is, and there lots of
> other people out there who would like to get their pet distribution running on
> the XO hardware, it's important to me to be able to collaborate with others
> using the same tools that I (hope to soon) use during the rest of my day. Isn't
> that going to benefit Fedora insofar as it helps make Fedora packages work
> better on all 400k XOs we've shipped it on and insofar as it stimulates more
> people to make useful contributions to upstream projects?
> 

> Error-checking. Pilgrim was notorious for producing broken builds because
> pilgrim failed or was unable to detect situations in which yum was unable to
> install all the requested packages. Perhaps this is a documentation problem --
> I note that the yum man page contains no information about what error codes yum
> will or can be made to return (if any?).
> 
> In particular, 
> 
>    yum -d 10 -e 10 install grhlkjoei
> 
> returns 0. This needlessly complicates my life: I _DO NOT_ want to have to
> parse the yum output in order to learn that a problem occurred.

Then don't use shell! Use python.



> Regarding the possibility of using the yum python modules directly: 
> 
>   * what documentation exists?
pydoc has a fair bit in docstrings for yum. We have lots of good example
code to work from, too. Both James and I have been posting common
examples to get people started writing their own utilities using the yum
modules.

>   * what guarantees of API stability exist?

3.2.X is guaranteed api stable. If we've broken compat it is not
intentional.

>   * what sample code exists and is it sane or contorted?

http://illiterat.livejournal.com/6254.html
http://fedoraproject.org/wiki/SethVidal/YumCodeSnippets

all of the yum-utils are pretty good code examples, too.


Finally, I'd like to mention that excluding the 'make it work with
random package formats' everything you've described above is wheel
reinvention. You've respun what we've already done and had available in
fedora, rhel, rhl for years. It sure feels like there's a lot of NIH at
work in OLPC development and that's not productive for anyone and
certainly not very collaborative.

-sv