Top considerations for building a foundation for generative AI

May 20, 2026•
Resource type: E-book
Download PDF

Chapter 1: New possibilities for business innovation …

Generative AI (gen AI) is a powerful tool for organizations that want to create innovative products, optimize processes, and gain competitive advantages in rapidly changing markets. Based on advancements in neural networks and deep learning—teaching computers to process data using an algorithm inspired by the human brain—gen AI extends beyond predictive AI capabilities by not only processing data, but also creating new, original content. Gen AI is reshaping human-machine collaboration, inspiring new approaches to problem solving, and delivering significant business outcomes across industries.

Worldwide, organizations are building new, innovative applications using gen AI technologies. These use cases require further alignment to enterprise requirements—often achieved by customization based on enterprise data.

But even in early stages, adopting gen AI has been shown to deliver key benefits to:

  • Operational efficiency. Handling time-consuming work—such as writing code, creating documentation, or managing supply chains—can help businesses produce more in less time and at lower cost.
  • Revenue growth. Gen AI can help increase revenue streams by identifying personalized upselling opportunities, creating tailored marketing content for specific customer segments, and supporting faster launch of new products and services.
  • Customer satisfaction. Providing instant, accurate support and personalized experiences helps customers get the information and services they need, when needed—building long-term trust and loyalty.
  • Employee productivity. Managing routine tasks with AI—such as summarizing meetings, drafting emails, or finding data in internal systems—lets employees focus on high-value work that requires human creativity and judgment.

… but also new concerns.

Although the benefits and drawbacks of gen AI are still emerging, many organizations want to invest in these new technologies now. Understanding the challenges of gen AI can help businesses establish clear ethics guidelines and development frameworks, comply with government and industry regulations, and detect and correct potential issues.

  • Data privacy. Privacy concerns arise when gen AI models are trained using sensitive or personal data, leading to questions about the protection of individuals’ privacy. Data storage and sovereignty regulations, such as the European Union (EU) General Data Protection Regulation (GDPR), must be met.
  • Data ownership. Using proprietary models—or models pretrained using proprietary data—introduces data ownership issues that may lead to litigation.
  • Bias and fairness. Responses from gen AI tools have been shown to reflect human biases, including harmful stereotypes and hate speech.
  • Ethical use. Synthetic content created by gen AI, such as deepfakes, may be used for malicious activities like privacy infringement and misinformation campaigns. A lack of transparency into gen AI models or tools makes it difficult to understand, interpret, and explain outputs and assign accountability for incorrect or made-up information.

Recent benchmarks illustrate

  • Retrieval-augmented generation (RAG): Integrates facts from external sources—like internal databases, corporate intranets, or the internet—to provide gen AI models with the most accurate, up-to-date information.
  • Prompt tuning: AI models receive cues or front-end prompts—including extra words or AI-generated numbers—guiding them to a desired decision, letting organizations with limited data tailor foundation models to a specific task.
  • Fine-tuning: Enhances a pretrained model further with a smaller, more targeted data set to perform domain-specific tasks more effectively. This additional training data is embedded into the model’s architecture.
  • Unintended consequences. The autonomous nature of gen AI can lead to unintended consequences that can cause real harm to organizations.
  • Regulatory challenges. Rapid advancements in gen AI technologies may outpace regulatory frameworks, making it difficult to create and enforce guidelines that ensure responsible and ethical use.
  • Energy consumption. AI model training is compute-intensive with high energy demand, raising concerns about the effects on environmental sustainability.

 A trusted infrastructure foundation can help organizations achieve the benefits and address the challenges of gen AI initiatives.

Chapter 2: The right foundation for gen AI: Key considerations

The technology foundation you choose for your gen AI initiatives can greatly affect your overall adoption efficiency and success. Organizational leaders should remember these 8 key considerations when deciding on the appropriate gen AI foundation.

Consideration 1: Build faster with proven, trusted toolsets

Developing applications based on gen AI models can be a complex task. The right toolset—reliable commercial or open source community languages, frameworks, and runtimes—can speed model tuning and simplify application development and deployment.

Choose an AI development platform that includes or supports familiar, preferred toolsets to help develop innovative AI solutions more quickly and efficiently. Exploratory data science, training, and tuning capabilities accessed through interactive user interfaces (UIs) can simplify collaboration. Built-in or integrated toolsets, paired with self-service capabilities, can help optimize IT operations (ITOps) while maintaining portability and consistency.

Consideration 2: Standardize to fine tune models rapidly

Because training gen AI models is an expensive, time-consuming process, most organizations build AI solutions using foundation models that are pretrained on general purpose data. Organizations can then use diverse, domain-specific data to adjust foundation models to perform specialized tasks. There are multiple ways to customize foundation models—some computationally intensive, requiring powerful processors and distributed hybrid cloud infrastructure. 

Look for AI platforms that provide tools for preparing and ingesting private enterprise data, support widely adopted techniques for model customization and alignment, and offer access to tools that support repeatable customization patterns. These platforms should also support distributed workload management and orchestration capabilities that deploy training runs—of any model size, data volume, or duration—across hybrid cloud environments. 

Containers for gen AI

Container technologies like Kubernetes provide agile deployment, management, and scalability for efficient cloud-native development of gen AI solutions.

  • Provision environments on demand across on-site datacenters, public clouds, and edge devices.
  • Automatically create, deploy, scale, and manage container instances on physical and virtual infrastructure.
  • Integrate components and data stores from a robust ecosystem of open source and commercial suppliers into gen AI solutions.

Explore additional benefits of containers for AI1

Consideration 3: Serve and run models efficiently with inference

Delivering exceptional user experiences from gen AI solutions with varying application demand can be challenging for ITOps teams. Efficient model inference requires scalable infrastructure and automated management capabilities to ensure high throughput, low latency, and optimal compute utilization. And because AI solutions process vast amounts of data, enforcement of strict security standards across environments is crucial.

Consider platforms that provide inference servers compatible with multiple model types—such as gen AI, large language, and multimodal—to achieve efficient results across multiple environments. High-performance inference servers accelerate gen AI application output by optimizing graphic processing unit (GPU) memory use and helping models perform calculations more efficiently at scale.

Consideration 4: Automate lifecycle management

Continuous integration and continuous delivery (CI/CD) pipelines can automatically deploy and manage gen AI solutions, but scaling these solutions requires the integration of machine learning operations (MLOps) and GenAIOps. While MLOps automates the rigorous data extraction, training, and validation cycles necessary for model stability, GenAIOps addresses the unique needs of large language models (LLMs), such as prompt versioning and vector database management. These ML and gen AI pipelines are more complex than standard workflows, requiring rapid, incremental changes to fine tune performance and mitigate hallucinations—illogical or false outputs—through automated evaluation.

Choose a foundation that lets you create and integrate these specialized AI pipelines—using tools like Kubeflow for model orchestration and Jenkins for broader automation—into existing DevOps workflows. Adopting GitOps with tools like ArgoCD can help you define complex AI deployments as code, ensuring that foundation models and retrieval-augmented generation (RAG) components remain consistent, auditable, and reproducible for enterprises.

Consideration 5: Prepare for agentic AI with transparency and control

Gen AI is excellent at creating content and answering questions. Agentic AI is the next evolution of these capabilities, marking a shift from models that provide simple chat functions to autonomous agents. AI agents don’t just follow a script, but rather they use reasoning tools to break complex goals—for example, onboarding a new client—into smaller tasks, such as checking documents, sending emails, and updating databases. As these systems must independently navigate unexpected changes, transitioning your approach from simple prompt-response patterns to managed, multistep workflows is essential.

Look for an AI foundation that prioritizes integration and security. Agentic AI needs an infrastructure that can connect to external tools and internal data sources while a human-in-the-loop approach is maintained for high-stakes decisions and actions. Focus the search on platforms that offer clear visibility into how an agent thinks and makes choices to make certain that every action and output is auditable. By starting the AI journey with a foundation built for governance and support for external tool use, IT teams can ensure their AI evolves from a creative assistant into a digital twin and remains a reliable and controlled extension of business operations.

Key gen AI model concepts

  • Bias is the presence of patterns in model behavior that affect the fairness, inclusivity, and ethics of outputs, such as favoring certain groups or producing discriminatory responses.
  • Data drift occurs when the statistical properties of training data change over time, affecting model performance with the generation of less accurate or relevant responses.
  • Anomaly detection is the process of identifying and reporting model behaviors that are uncommon or divergent from examples seen during training. 
  • Per-point explainability is the ability to understand why models generate specific outputs, providing visibility for applications where transparency is critical.

Consideration 6: Monitor models consistently to mitigate bias and inaccuracy

The outcomes provided by gen AI models can have a substantial influence on organizational processes. By proactively tracking model behavior, teams can analyze decisions and justifications to identify poor performance and report problematic behaviors immediately. Effective model governance based on this information helps ensure that AI models will respond with unbiased, fair, and correct information in production environments.

Explore AI foundations that provide centralized monitoring capabilities—including bias and data drift metrics, anomaly detection, and per-point explainability— to help teams maintain and correct their gen AI models. Continuous, automatic monitoring in production environments will also increase compliance with the organization’s corporate model governance standards. User-friendly tool interfaces and human-readable, nontechnical reports sharing the resulting information encourage responsible model use and maintenance.

Consideration 7: Take advantage of partner ecosystems for comprehensive capabilities

Gen AI solutions require multiple integrated components to successfully deliver innovative results. The right combination of technologies from a collaborative ecosystem of trusted vendors can speed application development, address bias and data drift, and ensure consistent, reliable performance for your AI deployments.

Seek platform vendors with extensive, certified partner ecosystems that offer complete solutions for developing and deploying gen AI models and applications. Working with a broad selection of components—from data integration and preparation to model training and serving—can help IT teams develop and deploy AI solutions more efficiently. By choosing vendor-certified solutions with proven interoperability, organizations can further improve productivity by reducing IT support requests.

Consideration 8: Work with platform experts to solve issues and learn best practices

Effectively deploying and managing gen AI solutions requires specialized knowledge and experience. Challenges with scalability, reliability, and integration with existing systems can complicate production deployments. Inefficient use of compute resources can result in unnecessary costs. And noncompliance with security standards, privacy policies, and AI regulatory frameworks can lead to legal, financial, or corporate brand image consequences.

Select vendors with teams of experts who provide comprehensive support and guidance for building gen AI solutions. For example, dedicated engineers may support the entire platform with the tools, resources, and experience to speed AI project delivery. Expert consultants can help solve deployment challenges, optimize infrastructure efficiency, and ensure interoperability across your AI solution. And professional training services can help organization teams gain the knowledge and expertise to start new gen AI projects in less time.

Overcoming the challenges of delivering AI 

AI is reshaping how software is built and delivered. The IDC perspective delivers recommendations for the technology buyer.2

  • Establish robust governance frameworks for AI adoption, ensuring compliance with regulatory requirements and internal policies, and implement phased, board-level oversight to manage risk exposure and value realization.
  • Modernize application and data infrastructure to cloud-native architectures, prioritizing Kubernetes-native environments and aligning data, cloud, and AI road maps and engineering agendas.
  • Integrate security across the entire AI data application delivery lifecycle, including by enhancing security expertise within data and AI engineering teams.
  • Manage vendor relationships with a strategic view on platform and environment readiness, midterm dependencies, and demonstrable return on investment.

Chapter 3: Flexible, open technology balances AI innovation with operational stability

Red Hat® AI accelerates the development and deployment of enterprise AI solutions across hybrid cloud environments. It is a comprehensive platform that meets customers where they are—whether starting out or scaling to a full enterprise architecture—while supporting the deployment of any model on any hardware accelerator. By combining the power of open source technologies with state-of-the-art models, Red Hat AI helps organizations accelerate their pace of discovery and democratize access to the latest AI tools.

Red Hat AI includes Red Hat AI Enterprise, for enterprises looking to deploy and scale efficiently and anywhere, Red Hat AI Inference Server, for optimized inference of large language models (LLMs), Red Hat OpenShift® AI, for distributed Kubernetes platform environments, Red Hat Enterprise Linux AI, for individual Linux server environments, and Red Hat AI Accelerator, for supporting hardware acceleration capabilities in Red Hat OpenShift. These solutions combine the power of open source technologies with state of the art open source models, helping organizations accelerate the pace of discovery and democratize access to cutting-edge tools and technologies. Access to the latest innovations is complemented by Red Hat’s partner ecosystem, which offers an array of partner products and services that are tested, supported, and certified to perform with our technologies and help customers solve their business and technical challenges.

Together, these products, services, and technologies deliver: 

Flexible, efficient inferencing

Red Hat AI optimizes model inference across hybrid cloud environments to help organizations deploy their preferred models more cost-effectively and in less time. Its runtime, vLLM, maximizes throughput and minimizes latency, and the LLM Compressor capability uses advanced quantization techniques to improve inference speeds without sacrificing prediction accuracy. The result is a high-performance environment for serving models across a wide range of applications.

Connecting models to data

Red Hat AI simplifies the integration of private enterprise data with AI models, providing standardized tools for preparing and ingesting information to ground AI outputs in specific, relevant business knowledge. Organizations can boost model performance with customization techniques, including fine-tuning, retrieval-augmented generation (RAG), and prompt tuning. This approach ensures that AI models align with enterprise scenarios to deliver measurable business outcomes.

Agentic AI innovation

Red Hat AI provides a trusted foundation for organizational leaders that focuses on security across the entire platform from metal to agents, ensuring policies stay enforced and risks are contained. The platform provides high-performance inference, access to accurate models, and a simplified experience to connect data to agents. With AgentOps capabilities, Red Hat AI delivers security-focused governance and identity-based access control while providing the flexibility for teams to bring their own agent frameworks. This ensures that as AI deployments mature, they remain protected, observable, and fully governed.

AI at scale across hybrid clouds

Red Hat AI helps organizations manage and monitor the entire lifecycles of predictive and gen AI models at scale, from single-server deployments to broad, distributed platforms. Organizations can establish and maintain operational consistency and risk mitigation with a combination of tested, supported AI platforms powered by Red Hat OpenShift. Through centralized monitoring, teams can track bias, data drift, and anomaly detection to ensure models remain reliable, fair, and compliant with corporate governance standards.

Chapter 4: Get started with gen AI

Gen AI is a powerful tool for creating content and changing the way the world interacts with applications and technology. Through AI platforms and tools backed by expertise and partnerships, Red Hat AI offers a central foundation for your organization’s teams to build and deploy generative, predictive, and agentic AI with traceability, observability and control. In fact, Red Hat even uses its own AI tools and platforms3 to improve the use of other open source software.

Explore the path to enterprise AI

Discover how Red Hat AI helps organizations adopt and scale their AI models and tools. Visit redhat.com/ai. 

  1. Red Hat e-book. “Top considerations for building a production-ready AI environment,” 26 Jan. 2026.

  2. IDC Perspective. “Key Software Delivery Challenges and Pain Points in 2026: AI Further Snarls Existing Complexity of Cloud-Native Landscape.” #US54277526, 19 Feb. 2026.

  3. Red Hat overview. "Red Hat Consulting: AI Platform Foundation," 21 Oct. 2025.

  4. Red Hat overview. "Red Hat Consulting: Operationalize AI," 15 Dec. 2025.

  5. Red Hat case study. "Red Hat saves $5 million in IT support costs with AI augmentation," 17 Dec. 2024.

Start your AI journey with confidence and Red Hat Consulting

Work with Red Hat experts to make certain your AI/ML projects begin with ongoing success and innovation in mind. To get started, explore Red Hat’s portfolio of AI/ML services, and: 

Tags:Artificial intelligence