At the University of Reading, the Department of Meteorology needed a highly reliable, available, and scalable storage file system to manage data for its scientific research projects in weather, climate, and earth observation. With Red Hat® Gluster Storage, the department now has an enterprise-grade product—backed by world-class service and support—that saves departmental IT staff valuable research time they used to spend on maintenance and administration tasks.
Reading, Berkshire, United Kingdom
- Existing University of Reading architecture
One thing that gave me confidence was seeing that other organisations, including commercial ones that could lose a lot of money if their storage management wasn’t handled well, were using Red Hat Gluster Storage.Dan Bretherton, high-performance computing manager, Department of Meteorology, University of Reading, United Kingdom
The Department of Meteorology at the University of Reading is home to 60 postgraduate students and a thriving community of more than 200 research scientists conducting valuable work in atmospheric, oceanic, and climate science. This research is essential to understanding climate change and its impact on the environment.
In many cases, these scientists rely on access to high-performance computing (HPC) systems to create complex simulation models. That helps them predict likely outcomes with more precision, such as how weather patterns will move over hours, days, weeks, months, and years into the future.
Scientists process high volume of data
For heavy-duty computing tasks, department researchers have access to supercomputer and HPC services provided by several UK research councils and the European Centre for Medium-Range Weather Forecasts. The scientists must process the data created by these external production runs. For this, they use a local HPC cluster owned and operated by the department and managed by its high-performance computing manager, Dan Bretherton.
Manager must ensure performance, security
"My priorities are to ensure not only that the Department can store hundreds of terabytes of research data efficiently and securely, but that good performance is maintained as the I/O load from our growing compute cluster increases," said Bretherton.
For several years, Bretherton had been using GlusterFS—an open source, scale-out file system for network-attached storage (NAS). That enabled him to organise and manage volumes of research data from different scientists and projects.
The growth of the department's HPC cluster, and the amount of time that Bretherton was devoting to managing GlusterFS, made this approach unsustainable.
The department needed a supported, scale-out NAS product backed by enterprise-grade service and support. After some exploration, Bretherton chose Red Hat® Gluster Storage, a supported version of GlusterFS, which Red Hat purchased in 2011.
Confidence from experience
"When I started talking to Red Hat, I got the assurances I needed that the challenges I'd experienced with managing scale-out NAS could be solved," said Bretherton. "One thing that gave me confidence was seeing that other organisations, including commercial ones that could lose a lot of money if their storage management wasn't handled well, were using Red Hat Gluster Storage. Plus, while I had some familiarity with GlusterFS and knew that would be a help, I also knew I'd get the extra help I needed from Red Hat to fine-tune the product for our academic, data-intensive HPC environment."
Bretherton reported that the implementation of Red Hat Gluster Storage in spring 2013 went smoothly. The product has run well ever since, and the addition of new users and new volumes of data has caused no significant problems.
Today, the Department of Meteorology at the University of Reading is using Red Hat Gluster Storage to manage around 200 terabytes (TB) of research data. With the addition of two large new servers, the cluster's capacity has recently risen to around 300TB.
Bretherton is gradually adding data from older, stand-alone servers within the department to the product; in total, the Department hosts well over 1 petabyte of data. "If Red Hat Gluster Storage continues to work well for us, then eventually all of our data scattered around servers large and small could end up under its management," he said.
Organization, availability improve
Red Hat Gluster Storage's global namespace capability gives Bretherton a simple and consistent way to organise data volumes by research group and project, and to allocate more storage capacity to volumes when required.
Another major benefit for the department is high availability. "If a server needs taking down for maintenance, that doesn't mean that work with a particular dataset needs to grind to a halt," said Bretherton. "The replication feature ensures that every file exists on two separate servers, so the absence of one server for a period of time will be entirely invisible to the users."
Saving time and money
For Bretherton, one of the attractions of the scale-out NAS approach is that he can future-proof the department's HPC cluster—it lets him retire older servers that are reaching end-of-life and swap in new ones as needed, without ever having to shut the whole cluster down.
At the same time, the useful life of servers is extended when they're used as part of the cluster. Plus, the utilisation rates associated with their storage capacity are higher, so the University of Reading can be sure it is getting the maximum return on its hardware investments.
Manager workload eases
Red Hat Gluster Storage has helped to free up a lot of Bretherton's time. "Before we implemented the product, I was spending up to 50% of my time managing and maintaining GlusterFS on the department's behalf," he added. "Now, it's more like 15% and that's more manageable alongside my other responsibilities."
Red Hat makes room to grow
As the Department of Meteorology grows, so will its use of Red Hat Gluster Storage. The University of Reading's Academic Investment Project has recently led to 28 new academic appointments in the area of climate and environmental sciences, and that means 15 new faculty members will be joining the Department of Meteorology.
"New staff mean new research groups, new data sets and new storage requirements," said Bretherton. "I'm confident now that our storage infrastructure is sufficiently stable and well organised to accommodate the new demands we will face."Learn More