Northeast Georgia Health System improves uptime for Epic EHR

Northeast Georgia Health System (NGHS) cares for more than one million patients, and to do that, clinicians rely on records being available in Epic – an electronic health records system for hospitals and large practices. While Epic is used across the world, it’s known for being complex and time-consuming to manage; any downtime significantly disrupts the patient experience. To speed up patching and maintenance of Epic and other systems, NGHS uses Red Hat Ansible Automation Platform. This has eliminated unplanned outages, strengthened security, and increased the stability of NGHS’s IT infrastructure.

Benefits: 

• Protected patient records from unplanned outages with zero downtime
• Saved more than 295 FTE hours for the IT team
• Increased efficiency with more consistent processes

Northeast Georgia Health System (NGHS) is a not-for-profit community health provider based in the United States. It cares for more than one million patients across 5 hospitals and numerous outpatient locations in northeast Georgia. It’s one of the largest healthcare systems in the state and growing fast.

While clinicians are at the front-line of delivering care, the overall patient experience relies on a variety of IT systems, ranging from the typical infrastructure in use at any large enterprise, to applications tailored for healthcare. One of the most critical applications is the cloud-based electronic patient record system, Epic.

Epic is used by hospitals the world over, but it’s complex to manage. Scheduled downtime for maintenance is seriously disruptive in a 24/7 care setting, and unplanned downtime can be catastrophic. Epic is just one of many systems that the IT team at NGHS needs to maintain.

“The biggest challenge we face as system administrators is lack of time. A lot of work goes into maintaining critical health systems, and we’re always under pressure to find ways to save time,” said Jameson Pugh, System Administrator, NGHS.

Epic needs minor updates and patching monthly, and major upgrades twice a year. The team also needs to patch database servers and manage storage, among other tasks. This was handled manually, which was complex and time-consuming. The team needed to find a new approach to speed up day-to-day maintenance and safeguard system stability.

NGHS has been using Red Hat technology for more than 10 years. Epic runs on the Red Hat Enterprise Linux operating system. “As the industry standard for stability and reliability, Linux serves as the platform for our most mission-critical data,” said Pugh.

For strategic automation, the enterprise uses Red Hat Ansible Automation Platform – the foundation for building and operating automation services at scale, providing a composable, collaborative, and trusted execution environment. Also in its stack is Red Hat Satellite, an infrastructure and system management software that’s used to keep apps and systems running on Red Hat Enterprise Linux secure, efficient, and compliant with industry regulations.

“Red Hat is my go-to partner. We have a great, trusted relationship, and the Red Hat team provides excellent support,” said Pugh. “We use AWX to support development and testing and decided to adopt Ansible Automation Platform to manage production. It includes technical support, security patches, and updates, as well as having a more polished, user-friendly interface than AWX.”

While the team started using Ansible Automation Platform to patch and manage Epic, they quickly found other processes to automate. For example, they use it to manage Dell EMC storage and configure Integrated Dell Remote Access Control (iDRAC) and for VMware host configurations. They also use playbooks from Red Hat Ansible Certified Content Collection and automated vCenter host configuration. “The certified collections ensure supportability, and assist with compliance with vendor standards,” said Pugh.

“One of the great things about Ansible Automation Platform is that it’s vendor-agnostic. We can use it to update VMware that’s hosted on Dell hardware without impacting performance,” said Doug Taliaferro, Systems Administrator, NGHS.

Protected patient records against unplanned outages with zero downtime

Since automating patch management for Epic, the system has experienced no unplanned downtime, which means clinicians have been able to rely on uninterrupted access to one of the organization’s most critical systems.

“Epic is the technological heart of our organization. Everything our clinicians do relies on Epic, and uptime is extremely important,” said Pugh. “Achieving zero unplanned downtime with Ansible Automation Platform is a huge win.”

Saved time for the IT team

By automating time-consuming, manual maintenance tasks, NGHS has more time and resources to devote to higher value activities. Baseline configuration for one RHEL virtual machine, for example, takes around 50 minutes to complete manually. Automating the process and running the playbook across 70 virtual machines takes less than 30 minutes and can be repeated every hour. “Monitoring configuration drift manually would require 70 skilled people working 24/7 – or 295 FTE hours per year. With Ansible Automation Platform, it runs automatically every hour, making it much easier to proactively monitor our environment,” said Pugh.

In another example, a task runs overnight to check the log files for jobs that have run on Epic’s operational database. If it finds errors, Ansible Automation Platform attempts to re-run the job, and if it fails again, it opens a ServiceNow ticket automatically, which saves a huge amount of time. It also simplifies infrastructure management, for example, optimizing VMware host maintenance and management.

“The biggest issue for us was always having a high workload and not enough time. Ansible Automation is helping us with the daily activities that keep the lights on and critical systems running,” said Pugh.

NGHS also has assurance that its systems are stable and reliable—and if it runs into any issues, it has fast access to Red Hat support.

Increased efficiency with more consistent processes

One of the biggest benefits of automation is that playbooks execute tasks consistently every time. This isn’t just more efficient, it ensures uniformity across servers, which in turn keeps systems more stable.

“Ansible Automation Platform makes sure our operational database servers match, that local accounts have the same user or group ID, and that file permissions are consistent. That makes for a much more stable and secure environment,” said Pugh.

IT professionals working in the healthcare sector have a daunting task. Unplanned outages or poor performance risk impacting not just the organization’s reputation and operating costs, but also the ability of clinicians and physicians to care for patients.

With Ansible Automation Platform, NGHS has greater assurance that Epic and other systems are highly available, up to date, and that its IT team has time for higher-value activities than simply keeping systems running.

“We’re planning to bring more teams with us on our automation journey. We’ve used Red Hat workshops in the past and we’re looking to do that again,” said Pugh. “We’re also thinking about using Ansible Automation Platform for our cloud migration to AWS and to replace VMware automation in the future. I’d love to use Red Hat OpenShift® too—the user experience and support are just better than other vendors.”