Issue #9 July 2005

Red Hat Global File System now supported with Oracle RAC

by Matthew T. O'Keefe

Introduction

Oracle® 9i RAC is a scalable, highly-available, clustered database that runs efficiently on Red Hat® Enterprise Linux® and low-cost servers and storage. Oracle 9i RAC and Oracle 10g clusters allow groups of Red Hat Enterprise Linux servers using shared storage to work together to more quickly process transactions and complete queries, without using expensive UNIX®-based symmetric multiprocessors. It is possible to achieve very high scalability on industry-standard servers with Oracle clustering and Red Hat Enterprise Linux: in December 2003, Red Hat Enterprise Linux and Oracle RAC set the world record for the TPC-C benchmark, achieving over 1 million transactions per minute on a shared storage cluster of 16 industry servers. Red Hat GFS is currently the only third-party cluster file system validated by Oracle for use with Oracle 9i RAC clusters.

To achieve the best performance and scalability using Oracle 9i RAC, it's necessary to have the right software to manage the shared storage hardware environment required by Oracle 9i RAC. The storage management issues with Oracle 9i RAC include the following:

  1. A separate Oracle installation must be performed on each Oracle 9i RAC node. More than 100,000 files must be installed on each node (this includes the Oracle home and root directories). Home and root directories must be separately managed for each node since these directories cannot be shared.
  2. Unless Oracle 9i RAC is deployed on top of a file system, a minimum of nine raw devices (one for each table space) must be managed, plus additional raw devices for user space. Raw devices cannot be resized. Raw devices appear as "unused" space and can be inadvertently written over by other applications if proper care is not taken.

The raw Devices approach: Oracle 9i RAC configuration

Let's look further at some of the challenges in deploying Oracle clustering directly on storage devices without a cluster file system. A typical Oracle 9i RAC installation for eight nodes is shown in Figure 1, Oracle 9i RAC cluster using raw devices for storage.

Figure 1. Oracle 9i RAC cluster using raw devices for storage

Several tasks must be performed by the system administrator when installing Oracle 9i RAC directly on raw storage devices. These tasks include:

  1. Creating multiple Oracle instances
    An Oracle home directory for each node (and node-specific information for each node) must be created. An eight-node configuration will have:
    • Eight local disks, one for each node (or similarly, eight physical or virtual storage devices on a storage network)
    • Eight Oracle home directories
    • Node-specific information on each of the eight local disks—requiring each local disk to be managed and protected
  2. A raw storage device must be created and used for each table space
    This approach results in a minimum of 10 raw devices to manage, including devices for Oracle tables (system, undo, redo, temp, users, tools, index) and user space data. In Red Hat Enterprise Linux 3, it's important to remember that only 255 raw devices may be created per Red Hat Enterprise Linux server.

    Note:
    Other servers that may share the same storage device (such as over a SAN) cannot see the raw devices that belong to Oracle 9i RAC. Therefore, Oracle data can be inadvertently overwritten by other servers that share the storage device. In addition, raw devices cannot be backed up using standard, file-based backup approaches.
  3. Unnecessary downtime and administrative effort
    Once created, it's important to remember that raw devices cannot be dynamically expanded. Therefore, the table spaces mapped to these devices cannot be expanded dynamically while the Oracle cluster remains online. Resizing raw partitions is difficult, and a backup version of the table must be written back to the larger partition once it is created offline. In addition, updates to the Oracle home directories on each node are required whenever the Oracle program files are updated.

A Solution: Red Hat Enterprise Linux, Red Hat GFS, and Oracle 9i RAC

Instead of mapping Oracle tables, logs, and archives directly to raw devices, a better approach is to map these structures to files. Since Oracle 9i RAC uses a cluster, its necessary to have a file system that works with shared storage as well. Such a file system is known as a cluster file system (CFS). Red Hat GFS is a CFS that works very well with Oracle clusters. A typical Oracle 9i RAC solution with Red Hat GFS is shown in Figure 2. Oracle 9i RAC cluster using Red Hat GFS for storage.

Figure 2. Oracle 9i RAC cluster using Red Hat GFS for storage

Oracle 9i RAC is made to access table data, logs, and archives through GFS, which clusters multiple nodes so that all nodes can have equal access to the same files. GFS supports direct I/O access to disks for maximum performance. Configuration and manageability is significantly improved.

The benefits with using the Red Hat Enterprise Linux, GFS, and Oracle 9i RAC include the following:

  1. A single Oracle instance: Support for a shared Oracle home and centralized node-specific information makes Oracle 9i RAC much easier to install and manage. Over 100,000 Oracle files and executables are installed in a single Oracle home directory. The Oracle home directory is then shared by all Oracle 9i RAC nodes. Node-specific information accessed by Oracle 9i RAC can be centrally stored under Red Hat GFS—eliminating the need to separately manage this information for each node.
  2. High-performance (direct_io) datafile storage: Create as many data files as are required by the database.
  3. Downtime to reconfigure and reformat disks for Oracle 9i RAC is eliminated and data integrity is assured.

Summary

Using Red Hat GFS to manage the shared storage environment required by Oracle clustering greatly simplifies the task of the database administrator and system administrator to support Oracle 9i RAC clusters. Red Hat Enterprise Linux and GFS make administrators more productive because all administrative tasks can be accomplished on a single cluster node in a consistent way for the entire cluster. Oracle and Red Hat are working closely together to insure that Red Hat infrastructure software, including Red Hat Enterprise Linux and GFS, are the best solution for managing the shared storage environments required by Oracle clustering.

A guide to installing Red Hat GFS with Oracle clusters can be found on redhat.com.

For more information on Red Hat GFS, refer to the following articles:

About the author

From 1990 to May 2000, Matthew O'Keefe taught and performed research in storage systems and parallel simulation software as a professor of electrical and computer engineering at the University of Minnesota. He founded Sistina Software in May of 2000 to develop storage infrastructure software for Linux, including the Global File System (GFS) and the Linux Logical Volume Manager (LVM). Sistina was acquired by Red Hat in December 2003, where Matthew now directs storage software strategy.