extras-buildsys README,1.8,1.9
Daniel Williams (dcbw)
fedora-extras-commits at redhat.com
Fri Jul 15 01:23:18 UTC 2005
Author: dcbw
Update of /cvs/fedora/extras-buildsys
In directory cvs-int.fedora.redhat.com:/tmp/cvs-serv23927
Modified Files:
README
Log Message:
Fix up the readme
Index: README
===================================================================
RCS file: /cvs/fedora/extras-buildsys/README,v
retrieving revision 1.8
retrieving revision 1.9
diff -u -r1.8 -r1.9
--- README 10 Jul 2005 06:08:24 -0000 1.8
+++ README 15 Jul 2005 01:23:15 -0000 1.9
@@ -1,4 +1,4 @@
-Fedora Extras Build System
+fFedora Extras Build System
System Requirements:
- Python 2.3 or 2.4
@@ -94,42 +94,39 @@
Architectural Overview:
------------------------------------------
-The build system is composed of a single build server, and multiple build
-clients. Clients run an XMLRPC server to which the build-server delivers
-build jobs. The build server runs an XMLRPC server to allow submission of
-jobs, and to retrieve basic status information about both clients and the
-build system as a whole.
+The build system is composed of a single build server, and multiple builders.
+Builders run an XMLRPC server to which the build server delivers build jobs.
+The build server runs an XMLRPC server to allow submission of jobs, and to
+relay basic status information about both builders and the build system as a
+whole to users.
-The Build Client:
+The Builder:
-usage: build-client <address> <architectures>
-ie : build-client localhost sparc sparcv9 sparcv8
+usage: /usr/bin/plague-builder -c <config_file>
-Currently, build clients are limited to building one job at a time. This
-limitation may be removed in the future. They do not queue pending jobs,
-but will reject build requests when something is already building. The build
-server is expected to queue and manage jobs at this time, and serialize
-requests to build clients. This may not be the case in the future.
+Currently, builders are limited to building one job at a time, though there is
+no restriction on running multiple builders on a single machine. They do not
+queue pending jobs, but will reject build requests when something is already
+building. The build server is expected to queue and manage jobs at this time,
+and serialize requests to builders.
main()
- `- Creates: XMLRPCBuildClientServer
+ `- Creates: XMLRPCBuilderServer
`- Creates: i386Arch, x86_64Arch, PPCArch, etc (subclasses
- of BuildClientMach)
+ of BuilderMock)
-The client creates an XMLRPC server object (XMLRPCBuildClientServer), and
-then processes requests in an infinite loop. Every so often (currently
-5 seconds) the server allows each build job that is still in process to
-update its status and perform work. The XMLRPCBuildClientServer keeps a
-list of local build jobs, which are architecture specific, and forwards
-requests and commands for each job to that specific job, keyed off a
-unique id.
+The builder creates an XMLRPC server object (XMLRPCBuilderServer), and
+then processes requests in an infinite loop. The build server queries each
+client periodically for its status and the ID of the currently building job, if
+any.
-Each build job (BuildClientMach and its architecture-specific subclasses like
+
+Each build job (BuilderMock and its architecture-specific subclasses like
i386Arch) proceeds through a number of states. Build jobs are periodically
-given time to do work (BuildClientMach.process()) by their BuildClientInstance
-(from XMLRPCBuildClientServer._process()), which is in turn periodically given
+given time to do work (BuilderMock.process()) by the builder controller
+(XMLRPCBuilderServer._process()), which is in turn periodically given
time by the client's main loop. During their processing time, build jobs
check their state, see if any actions have completed, and advance to the next
state if needed. Communication with mock and retrieval of status from mock
@@ -137,8 +134,8 @@
from talking to the build server.
All communication with the build server is done through SSL to ensure the
-identity of each party. Both the XMLRPC server and the result file server are
-SSL-enabled, and require SSL certificates and keys to operate. See later
+identity of each party. Both the XMLRPC server and the builder's file server
+are SSL-enabled, and require SSL certificates and keys to operate. See later
section in this document on how to configure SSL certificates for your build
system.
@@ -146,81 +143,72 @@
The Build Server:
-usage: build-server
+usage: /usr/bin/plague-server
The build server runs two threads. The first, the XMLRPC server
(XMLRPCBuildMaster class), accepts requests to enqueue jobs for build and
stuffs them into an sqlite database which contains all job details. The second
thread, the Build Master (BuildMaster class), pulls 'waiting' jobs from the
database and builds them. A third top-level object that runs in the same
-thread as the Build Master is the BuildClientManager, which keeps track of
-build clients (ArchWelders) and their status.
+thread as the Build Master is the BuilderManager, which keeps track of
+builders and their status.
main()
- |- Creates: XMLRPCBuildMaster
- |- Creates: BuildClientManager
- |- `- Creates: BuildClient (one for each remote build client)
- |- `- Creates: BuildClientJob (one for each build job
+ |- Creates: AuthedXMLRPCServer (system users talk to this object)
+ |- Creates: BuilderManager
+ |- `- Creates: Builder (one for each remote build client)
+ |- `- Creates: ArchJob (one for each build job
|- on each arch)
`- Creates: BuildMaster
- `- Creates: BuildJob (one for each build job)
+ `- Creates: PackageJob (one for each build job)
-The BuildClientManager object serves as a central location for all tracking and
+The BuilderManager object serves as a central location for all tracking and
status information about each build job on each arch. It creates a
-BuildClient instance for each remote build client. The BuildClient instance
-keeps track of specific jobs building on all architectures on that remote
-build client. It also serves as the XMLRPC client of the remote build
-client, proxying status information from it.
-
-BuildJobs must request that the BuildClientManager create a new BuildClientJob
-for each build on each architecture the BuildJob needs. If there is an
-available build client (since build clients only build one job at a time
-across all arches they support), the BuildClientManager will pass the request
-to the arch-specific BuildClient instance, which creates the new arch-specific
-BuildClientJob, and pass it back through the BuildClientManager to the parent
-BuildJob. If there is no available build client for the request, the BuildJob
-must periodically re-issue the build request to the BuildClientManager.
-
-BuildClientManager has a periodic processing routine that is called from the
-BuildMaster thread. This processing routine calls the BuildClient.process()
-routine on each BuildClient instance, which in turn updates its view of the
-remote build client's status. Thus, the BuildClientManager, through each
-BuildClient instance, knows the status and currently building job on each
-remote build client.
+Builder instance for each remote builder. The Builder instance keeps track
+of specific jobs building on all architectures on that remote builder.
+
+PackageJobs must queue requests with the BuilderManager for the architecture
+specific build jobs they need. The BuilderManager will wait until a Builder
+object is available to build the job, create the new ArchJob on the builder,
+and notify the parent PackageJob of its new ArchJob.
+
+The BuilderManager has a periodic processing routine that is called from the
+BuildMaster thread. This processing routine calls the Builder.process()
+routine of each Builder instance, which in turn updates its view of the remote
+builder's status. Thus, the BuilderManager, through each Builder instance,
+knows the status and currently building job on each remote builder.
-BuildJobs track a single SRPM build through the entire build system. They are
+PackageJobs track a single SRPM build through the entire build system. They are
created from the BuildMaster thread whenever the BuildMaster finds a job entry
-in the sqlite database with the status of 'waiting'. BuildJobs proceed through
+in the sqlite database with the status of 'waiting'. PackageJobs proceed through
a number of states: "initialize", "checkout", "make_srpm", "prep", "building",
-"finished", "cleanup", "failed", and "needsign".
+"finished", "addtorepo", "failed", "killed", and "needsign".
Flow goes like this:
initialize => checkout
checkout => make_srpm
make_srpm => prep
-prep => building
+prep => (queue architecture-specific job requests with the BuilderManager)
+<wait for first ArchJob to appear>
building
- All build jobs finished or failed? => finished
- otherwise => building
-finished => cleanup
-cleanup
+finished
- failed jobs? => failed
- - otherwise => needsign
-
-The BuildJob updates its status when it is periodically told to do so by the
-BuildManager. At this point, it will advance to the next state, or spawn build
-jobs that have not yet started if build clients for those architectures are
-now available. It stays in the "building" state until all jobs are first
-spawned, and then either completed or failed.
-
-All communication with build clients is done through SSL to ensure the
-identity of each party. When the client requests the SRPM to build, SSL
-is used. When the build server retrieves logs and RPMs from the build client,
-SSL is also used. This ensures that build clients can be more or less trusted,
-or at least that some random build client is not serving you packages that
-might contaminate your repository. See later section in this document on how
-to configure SSL certificates for your build system.
+ - otherwise => addtorepo
+addtorepo
+ <wait for Repo object to add our packages to the repo>
+ => repodone
+repodone => needsign
+
+All communication with builders is done through SSL to ensure the identity of
+each party. When the builder requests the SRPM to build, SSL is used. When the
+build server retrieves logs and RPMs from the builder, SSL is also used. This
+ensures that builders can be more or less trusted, or at least that some random
+builder is not serving you packages that might contaminate your repository. See
+later section in this document on how to configure SSL certificates for your
+build system.
Configuring SSL for your Build System
More information about the fedora-extras-commits
mailing list