5 considerations for end-to-end MLOps at the edge

Edge AI moves inference out of the datacenter and onto edge devices that are constrained, distributed, and often disconnected. Evaluate your organization’s machine learning operations (MLOps) strategy across these 5 areas that typically determine whether an edge AI deployment succeeds.

Get the right model version onto large fleets over limited-bandwidth networks with processes that are repeatable and verifiable. At fleet scale, the packaging and delivery choices an organization’s IT leaders make early decide how painful every later update will be. Work through these distribution decisions:

• Choose a packaging approach. Options include: embedded in the container, delivered as an external file, decoupled as a ModelCar, or embedded in a bootc image.
• Validate before deployment. Add continuous integration and continuous delivery (CI/CD) pipeline checks that package, test, scan, and promote each model version before rollout. Include hardware validation tests on target device classes, since edge deployments often reveal runtime, driver, accelerator, or resource constraints that cloud MLOps pipelines do not catch.
• Optimize delivery. Use Open Container Initiative (OCI) layer deduplication, and tiered registries (central, regional, and local).
• Reconcile to a declared state. Pull-based GitOps keeps each device aligned to the version intended.
• Sign and verify supply chain. Add signed images, software bills of materials (SBOMs), and trusted registries.

Run models on devices with limited compute, memory, and power to do tasks that have real-time deadlines. The runtime and model format teams choose have to match the hardware they are trained to use, not hardware located at the datacenter. Shared resources, such as graphics processing unit (GPU) memory, can affect both model performance and device availability when they run short or fail. Optimize execution across these areas:

• Match the runtime to the device. Match the runtime to the device. Use Podman or Red Hat® Device Edge on highly constrained hardware. Where full Kubernetes and high availability are required with a minimized footprint, use single-node, 2-node (with Arbiter or Fencing), or compact 3-node Red Hat OpenShift® clusters depending on the edge site’s physical resource constraints.
• Reduce model size. Apply quantization, pruning, and distillation, then convert to a hardware-aware format such as ONNX, OpenVINO, or TensorRT.
• Use hardware accelerators. Reach GPUs and neural processing units through the Container Device Interface (CDI) and the GPU operator.
• Plan for real-time and mixed fleets. Account for real-time execution and mixed-architecture devices using PREEMPT_RT and central processing unit (CPU) pinning.

Keep a distributed fleet of models healthy, current, and auditable after deployment. Once models are in the field, teams need to roll out changes without a site visit. Manage the fleet with these practices:

• Manage and observe centrally. Use Red Hat Edge Manager, Red Hat OpenShift, and Red Hat AI with tiered observability through OpenTelemetry.
• Watch for model drift. Track both data and concept drift alongside latency and confidence metrics.
• Automate repeatable pipelines. Use Tekton and Kubeflow Pipelines for deployment, testing, and updates.
• Roll out changes safely. Use canary promotion and layered rollback. Model this through GitOps with the operating system (OS) in bootc.

Design for sites that lose connectivity or never had it, so inference and updates keep working. Systems need to keep running and recovering on their own because connectivity at the edge is a convenience, not a guarantee. Build for disconnection across these areas:

• Keep inference local. Make sure there is no cloud dependency for time-sensitive decisions.
• Deliver air-gapped updates. Use Open Container Initiative (OCI) archives, local registry mirrors, or USB bootstrap.
• Reconcile against a declared state. Make use of pull-based agents such as Red Hat Edge Manager and Argo CD to keep sites current.
• Build in autonomous recovery. Add liveness probes and restart policies, and deliver updates incrementally with bootc image mode and delta layers.

Keep improving models compliantly by efficiently and safely moving the correct data off edge devices. Bandwidth limits and data sovereignty rules mean organizations cannot send everything back to the datacenter, so capture and routing decisions matter as much as the model. Close the data loop with these steps:

• Capture data selectively. Confidence-threshold triggers and event-driven uplinks reduce bandwidth costs.
• Move and normalize data. Run edge-to-core pipelines with automated quality gates and labeling.
• Automate the retraining loop. Connect drift, retraining, validation, and redeployment.
• Adapt models locally. Use federated and transfer learning, such as Low-Rank adaptation (LoRA), while respecting data sovereignty.