Ireland builds intelligent text analysis solution to protect farmers’ data

The Government of Ireland’s Department of Agriculture, Food, and the Marine (DAFM) processes grant applications submitted by farmers and their agents. However, incorrect document submissions by applicants led to data breaches in violation of the EU’s General Data Protection Regulation (GDPR). DAFM worked with Red Hat and Version 1 Software to create SmartText, a machine learning (ML) text analysis platform that scans documents and images for sensitive information and prevents unauthorized access. With a foundation built using Red Hat container and management technology, DAFM can correctly categorize documents while protecting back-end systems and delivering new features faster.

image container


  • Significantly reduced GDPR breaches with intelligent, AI/ML text analysis solution
  • Reduced development time from weeks to days with automation, APIs, and DevOps
  • Improved security and stability with enterprise technology and support

Preventing GDPR violations and data breaches

The Government of Ireland’s Department of Agriculture, Food, and the Marine (DAFM) leads, develops, and regulates the country’s agrifood sector, including farming, seafood, forestry, plant health, and animal services. To meet its mission of providing innovative, sustainable services, DAFM aligns with the country’s Public Service ICT Strategy to provide digital services to citizens, businesses, and government agencies. 

DAFM’s Basic Payments Scheme (BPS) handles applications and payments for farming funding grants. It is responsible for issuing €1.2 billion yearly to 120,000 farmers, in line with the European Union (EU)’s General Data Protection Regulation (GDPR) that protects the personal data and privacy of EU citizens. Each year, BPS receives 30,000-40,000 grant application documents submitted to its website by farmers or their agents. These documents often include Personally Sensitive Information (PSI), such as birth certificates, legal documentation, or medical information. However, farmers often failed to correctly classify documents as containing PSI. Agents—some of whom serve dozens of farmers—sometimes uploaded documents to the wrong application. As a result, highly sensitive personal data was exposed to BPS employees or even other farmers.

To mitigate the risk of financial penalties and reputational damage from major GDPR breaches, DAFM sought to replace its legacy data system and processes with an intelligent solution that would help correctly identify PSI and automate detection of breaches.

“Their previous system matched PSI with document numbers, but there was no way to search within documents,” said Stephen Griffin, Architect and Software Development Manager at Version 1 Software, DAFM’s local partner. “Processes for reporting and correction of possible PSI exposure were manual, but with the sheer volume of documents received, constant human review was impossible.”

Building a foundation for machine learning text analysis with Red Hat

BPS worked with Version 1 Software to outline requirements for an intelligent, automated approach to grant application processing. After evaluating several vendors, DAFM and Version 1 decided to use Red Hat OpenShift, supported by Red Hat 3scale API Management and Red Hat’s single sign-on (SSO) technology. 

“Our research showed that we needed an enterprise Kubernetes platform to create our machine learning text analysis solution. Red Hat OpenShift was the clear leader in that market. Red Hat also provides the security and control we need as a government organization,” said Gareth Sheerin, Enterprise Architect, Department of Agriculture, Food, and the Marine. 

With guidance from Version 1 and Red Hat Consulting, DAFM created SmartText, an ML text analytics solution that can extract metadata from submitted documents to correctly identify PSI, in just a few weeks. To meet EU requirements for data ownership, this solution runs on Red Hat OpenShift on premise in DAFM’s datacenter. 3scale API Management, part of Red Hat Integration, uses application programming interfaces (APIs) to connect the platform with DAFM’s citizen-facing application. Red Hat’s SSO technology supports protected mobile access for applicants using OpenID Connect and OAuth authentication.

“SmartText is now a key part of our Smart Action Suite, a set of enterprise productivity applications focused on addressing common service and technology challenges,” said Sheerin.

Its success using open source technology to create the SmartText solution led to DAFM’s recognition with a 2021 Red Hat Innovation Award.

Automating and simplifying compliance to protect farmer data

Reduced GDPR breaches with intelligent text analysis

SmartText, hosted on Red Hat OpenShift, uses real-time artificial intelligence (AI) and ML capabilities to extract metadata and other contextual information from unstructured grant application documents. These documents, including scans of handwritten or typed letters, are analyzed for sentiment, semantically similar words, topics, and entity names. 

“In both written text and images, we look for information pertaining to minors, medical conditions, even credit card numbers or what we call a herd number, which is personally identifiable to an individual,” said Griffin.

Access to the SmartText service itself is protected through 3scale API Management. As a result, DAFM can drastically reduce the number of GDPR breaches—and the associated reputational damage and financial penalties—each year.

Improved productivity with automation, APIs, and DevOps

A DevOps work approach has also helped DAFM significantly reduce manual administration and management. Collaborating on a single platform, teams have better visibility into project statuses—even while working remotely. 3scale API Management lets DAFM share and reuse services and applications as APIs among teams through a user-friendly interface. Each component API is deployed as a container with open source libraries included for simple reuse and redeployment. 

“Red Hat OpenShift lets us develop features from test to production within days, rather than weeks,” said Griffin. “We can pull applications and solutions from the open source community, then tailor them to our needs, compared to various cloud providers that had only one form of input and output available. For example, we now have a Python application image for OpenShift.”

Improved security and stability with supported enterprise technology

Working with Red Hat’s tested, certified technology, backed by experienced technology experts, helps DAFM secure both its back-end systems and public-facing resources. The API gateway created with 3scale API Management and Red Hat’s SSO technology enhances the security capabilities of Red Hat OpenShift. More granular security and access control is possible, without building those capabilities into each application, service, or container. 

“We can provide controlled areas for developers to experiment while protecting production applications,” said Sheerin. “We can launch applications with zero downtime and quickly roll back to a stable version if there are any issues. The tools that we get with Red Hat, like a catalog of certified container images, provide us a lot of assurance.”

Expanding intelligent text analysis capabilities to new payment processes

The SmartText project demonstrated DAFM’s commitment to innovative use of natural language processing and open source technology in alignment with Ireland’s Public Service ICT Strategy. The department has already expanded SmartText to more than 60 other payment schemes and grant systems to help mitigate GDPR breach risks. 

DAFM plans to keep looking for ways to improve its services while remaining compliant with regulations, using its stable yet flexible Red Hat technology foundation.

“The true value in Red Hat’s subscription is that in-depth product knowledge that we don’t have ourselves, particularly for Kubernetes container technology,” said Griffin. “We can rely on Red Hat to help us bring the best possible projects to production in the safest, most stable way.”

About the Government of Ireland Department of Agriculture, Food, and the Marine

The Department of Agriculture, Food and the Marine (DAFM) is responsible for leading, developing and regulating the agri-food sector, protecting public health, and optimizing social, economic and environmental benefits for the government and people of Ireland. One of its responsibilities is handling the application and payments of grants for more than 125,000 farmers, with a total value of 1.2 billion euros, in line with EU regulations.

Innovation is the core of open source. Red Hat customers use open source technologies to change not only their own organizations, but also entire industries and markets. Red Hat Innovators in the Open proudly showcases how our customers use enterprise open source solutions to solve their toughest business challenges. Want to share your story? Learn more.