[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[fedora-java] aot-compile-rpm



Over the past couple of days I've been writing a replacement for
aot-compile and find-and-aot-compile.  They were both good when they
were first written but the demands of Eclipse and JOnAS have exposed
numerous shortcomings.

Attached is a copy of aot-compile-rpm which I'd like to commit into
java-1.4.2-gcj-compat if everyone's agreeable.  It's an order of
magnitude more complex than aot-compile and find-and-aot-compile, but
it's advantages over them are manifold:

 IT'S MUCH MORE USABLE
 =====================
  Nativifying an rpm using aot-compile-rpm is a matter of
  copy-and-paste:

    1. Remove "BuildArch: noarch"
    2. Add "BuildRequires: java-1.4.2-gcj-compat >= 1.4.2.0-Xjpp"
       and "Requires(post,postun)" on same.
    3. Add "aot-compile-rpm" to the very end of %install.
    4. Add "/usr/bin/rebuild-gcj-db %{_libdir}" to %post and %postun.
    5. Add "%attr(-,root,root) %{_libdir}/gcj/%{name}" to %files.

  With aot-compile or find-and-aot-compile step 4 was much more
  complex.  I was always uneasy about nativifying things I didn't
  myself maintain, postgresql-jdbc for example, because I was wary of
  dropping a bunch of fragile code on someone else.  No longer.

 IT FINDS JARS BY SIGNATURE RATHER THAN BY EXTENSION
 ===================================================
  find-and-aot-compile identifies jarfiles by their extension, ".jar",
  so it misses ".war", ".ear", ".rar", and anything else the Java
  world happens to invent.  aot-compile-rpm identifies jarfiles by
  opening them, so it catches them no matter what they're called.

  It's already found some unexpected stuff.  Tomcat, for example, has
  a couple of servlets that are disabled by default because they are
  in jarfiles called ".renametojar".  aot-compile-rpm finds and
  compiles these, so if the user renames them to enable the servlets
  then they'll be running BC-compiled code.

 IT IGNORES SUBSETTED JARFILES
 =============================
  Several packages contain jarfiles which are a subset of others.
  MX4J, for example, has the APIs in mx4j-jmx.jar, the implementation
  in mx4j-impl.jar, and both together in mx4j.jar.  aot-compile-rpm
  recognises that compiling mx4j.jar will get every class in the other
  two jars too, so it'll only compile mx4j.jar.  So, aot-compile-rpm
  compiles MX4J in half the time (and generates half the bytes) that
  find-and-aot-compile does.

 IT WORKS AROUND THE PPC GO2 LIMIT
 =================================
  PPC machines are limited on the size of jarfiles that can be
  compiled in one go, affecting Eclipse and JacORB, as described
  at https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=158308.
  aot-compile-rpm splits large jarfiles on ppc to avoid this.

One limitation of aot-compile and find-and-aot-compile that I have not
addressed (yet) is SMP support.  Andrew Overholt suggested generating
a Makefile and letting make handle the complex stuff, which I think is
an excellent idea.  Implementing something along those lines is
something I'd like to do, but I need to get ppc64 and s390*
bootstrapped first.

Cheers,
Gary
#!/usr/bin/env python

import os
import sys
import zipfile

PATHS = {"libdir": "/usr/lib/gcj",
         "gcj":    "/usr/bin/gcj",
         "dbtool": "/usr/bin/gcj-dbtool"}

GCJFLAGS = os.environ.get("RPM_OPT_FLAGS", "").split() + [
    "-fPIC", "-findirect-dispatch"]
LDFLAGS = ["-Wl,-Bsymbolic"]

class Error(Exception):
    pass

def aot_compile_rpm(basedir, libdir):
    """Search basedir for jarfiles, then generate solibs and class
    mappings for them all in libdir."""
    dstdir = os.path.join(basedir, libdir.strip(os.sep))
    if not os.path.isdir(dstdir):
        os.makedirs(dstdir)
    for jar in weed_jars(find_jars(basedir)):
        aot_compile_jar(jar, dstdir, libdir)

def find_jars(dir):
    """Return a list of every jarfile under a directory.  Goes on
    magic rather than file extension so we hit wars, ears, rars and
    anything else they cooked up lately."""
    def visit(jars, dir, items):
        for item in items:
            path = os.path.join(dir, item)
            if os.path.islink(path) or not os.path.isfile(path):
                continue
            # could use zipfile.is_zipfile() but this is quicker
            if open(path, "r").read(2) != "PK":
                continue
            zf = zipfile.ZipFile(path, "r")
            try:
                zf.getinfo("META-INF/MANIFEST.MF")
            except KeyError:
                continue
            assert not jars.has_key(item)
            jars[item] = zf
    jars = {}
    os.path.walk(dir, visit, jars)
    jars = [(jar.filename, jar) for jar in jars.values()]
    jars.sort()
    return [jar for path, jar in jars]

def weed_jars(jars):
    """Remove any jarfiles that are completely contained within
    another.  This is more common than you'd think, and we only
    need one nativified copy of each class after all."""
    while True:
        for jar1 in jars:
            for jar2 in jars:
                if jar1 is jar2:
                    continue
                for item2 in jar2.infolist():
                    try:
                        item1 = jar1.getinfo(item2.filename)
                    except KeyError:
                        break
                    if item1.CRC != item2.CRC:
                        break
                else:
                    warn("subsetted %s" % jar2.filename)
                    jars.remove(jar2)
                    break
            else:
                continue
            break
        else:
            break
        continue
    return jars

def aot_compile_jar(jar, dir, libdir):
    """Generate the shared library and class mapping for one jarfile.
    If the shared library already exists then it will not be
    overwritten.  This is to allow optimizer failures and the like to
    be worked around."""
    basename = os.path.basename(jar.filename)
    soname = os.path.join(dir, basename + ".so")
    dbname = os.path.join(dir, basename + ".db")
    if os.path.exists(soname):
        warn("not recreating %s" % soname)
    else:
        sources = split_jarfile(jar, dir)
        if sources == [jar.filename]:
            # compile and link
            system([PATHS["gcj"], "-shared"] +
                   GCJFLAGS + LDFLAGS +
                   [jar.filename, "-o", soname])
        else:
            # compile
            objects = []
            for source in sources:
                object = os.path.join(dir, os.path.basename(source) + ".o")
                system([PATHS["gcj"], "-c"] +
                       GCJFLAGS +
                       [source, "-o", object])
                objects.append(object)
            # link
            system([PATHS["gcj"], "-shared"] +
                   GCJFLAGS + LDFLAGS +
                   objects + ["-o", soname])
            # clean up
            for item in sources + objects:
                os.unlink(item)
    # dbtool
    dbname = os.path.join(dir, basename + ".db")
    system([PATHS["dbtool"], "-n", dbname, "64"])
    system([PATHS["dbtool"], "-f", dbname, jar.filename,
            os.path.join(libdir, basename + ".so")])

def split_jarfile(src, dir, split = 1500):
    """In order to avoid #158308 we must split large jarfiles on PPC."""
    if os.environ.get("RPM_ARCH") != "ppc":
        return [src.filename]
    items = src.infolist()
    if len([i for i in items if i.filename.endswith(".class")]) < split:
        return [src.filename]
    warn("splitting %s" % src.filename)
    jarfiles, dst = [], None
    for item in items:
        if (dst is None or item.filename.endswith(".class") and size >= split):
            if dst is not None:
                dst.close()
            path = os.path.join(dir, "%s.%d.jar" % (
                os.path.basename(src.filename), len(jarfiles) + 1))
            jarfiles.append(path)
            dst = zipfile.ZipFile(path, "w", zipfile.ZIP_STORED)
            size = 0
        dst.writestr(item, src.read(item.filename))
        size += 1
    dst.close()
    return jarfiles

def system(command):
    """Execute a command."""
    prefix = os.environ.get("PS4", "+ ")
    prefix = prefix[0] + prefix
    print >>sys.stderr, prefix + " ".join(command)

    status = os.spawnv(os.P_WAIT, command[0], command)
    if status > 0:
        raise Error, "%s exited with code %d" % status
    elif status < 0:
        raise Error, "%s killed by signal %d" % -status

def warn(msg):
    """Print a warning message."""
    print >>sys.stderr, "%s: warning: %s" % (
        os.path.basename(sys.argv[0]), msg)

if __name__ == "__main__":
    try:
        name = os.environ.get("RPM_PACKAGE_NAME")
        if name is None:
            raise Error, "this script is designed for use in rpm specfiles"
        arch = os.environ.get("RPM_ARCH")
        if arch == "noarch":
            raise Error, "cannot be used on noarch packages"
        buildroot = os.environ.get("RPM_BUILD_ROOT")
        if buildroot in (None, "/"):
            raise Error, "bad $RPM_BUILD_ROOT"
        aot_compile_rpm(buildroot, os.path.join(PATHS["libdir"], name))
    except Error, e:
        print >>sys.stderr, "%s: error: %s" % (
            os.path.basename(sys.argv[0]), e)

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]