Red Hat Satellite Ask Me Anything Q&A from October 31, 2019

This post covers the questions and answers during the October 2019 Satellite Ask Me Anything (AMA) calls. 

For anyone not familiar, the Satellite AMAs are an "ask me anything" (AMA) style event where we invite Red Hat customers to bring all of their questions about Red Hat Satellite, drop them in the chat, and members of the Satellite product team answers as many of them live as we can during the AMA and we then follow up with a blog post detailing the questions and answers.

The ground rules of the AMA are:

• In the interest of making everyone feel like they can truly ask any question, the Satellite AMA sessions are not recorded. 

• The Satellite AMA is not the appropriate place to ask questions about specific support cases or specific sales issues. While we may be able to give generic feedback about certain areas we cannot use this time to troubleshoot or dig into logs. For your support cases please continue to work with Red Hat support, and for any sales issues please work with your Red Hat or Partner sales rep.

• The AMA is presented using Blue Jeans Prime. All questions are asked via the Q&A panel which allows other attendees to vote on questions that are asked. Questions are read by an event moderator based on the popularity of the questions and answered live and in real time.

As we kicked off the Satellite AMAs we pointed out a few important items happening in the Satellite area:

• Satellite 6.6 was released on October 31, 2019. See more information about the details in the Smart Management October 2019 Release Blog and in the Satellite 6.6 Release Notes. 

• Satellite 5.8 EOL is May 31, 2020. At that point all Satellite 5 versions will be EOL and you will need to transition to Satellite 6. 

• Satellite 6.3 went EOL on October 31, 2019

• For more information about Satellite check out the latest version of the Satellite Frequently Asked Questions.

The next Red Hat Satellite AMAs will run at two times:

• January 15, 2020 at 9am EST/ 2pm GMT / 3pm CET

• January 15, 2020 at 1pm EST / Noon CST / 11am MST / 10am PST

Register for the AMA on events.redhat.com.

Here are questions and answers (lightly edited for readability, grammar, spelling, etc.) from the October 31, 2019 Red Hat Satellite Ask Me Anything.

Asia Pacific event:

Question: What's the best way to create and manage content views? A CV with all repositories (RHEL, HA, resilient storage, etc.) or different CVs for RHEL, RHEL with HA, RHEL with JBoss, etc.) - this is from a performance and manageability perspective.

Answer: Generally speaking, think of CVs as a group of repos that are roughly on the same lifecycle. One content view might be RHEL, HA, and EPEL. These all depend on one another and as mentioned are roughly on the same lifecycle.

When you associate a CV with a system you do not need to apply all repos to all systems.

For JBoss, If you are using JBoss packages that are shipped from Red Hat via the CDN, then you might also want those in the same CV, but since they are an application you need to ask yourself if you manage JBoss on the same lifecycle as RHEL.

Question: What are the major differences between Satellite 6.5 and Satellite 6.6? Can you specifically include any information regarding Puppet?

Answer: Red Hat Satellite 6.6 focuses on updates across reporting, automation and supportability. Refer to the Smart Management release blog for basic details. 

Specific to Puppet, one of these features is the ability to deploy OpenSCAP via Ansible. This was the last major item that required Puppet to be installed on clients.

Question: I heard a rumor that Puppet was being removed from Satellite. Can you comment?

Answer: There is a lot of discussion around Puppet and our customers needs. We are applying more effort on Ansible Integration but we don't want to disrupt our customers that are Puppet shops. It will be determined in the future what this will look like as we are actively working through the options. Stay tuned for future announcements.

Question: Recently publishing / promotion of content views on my Satellite 6.4 has slowed tremendously and hangs. Could this be related to the MongoDB maintenance or lack thereof?

Answer: It is possible that database maintenance is needed, but an authoritative answer can only be found by opening a support case. 

Question: In Satellite 5.x, I was able to reboot clients. Now, in Satellite 6.X, this capability has been taken away. What's the direction—pushing all of this capability to Ansible?

Answer: This is incorrect for supported versions of Satellite 6. This has been available since Satellite 6.2. In Satellite 6.4 we brought in Ansible as well for remote execution. Any of the remote execution functionality in Satellite 6.x enables you to reboot a client. 

Question: Satellite 6.5 & 6.6 support for RHEL 5 and RHEL 6—you can download the repos but there doesn't seem to be any option for Satellite Tools for RHEL 5.

Answer: In regards to Satellite client support, we support versions of RHEL that are supported when that version of Satellite is released. The only supported version of RHEL 5 requires the ELS add-on. As a result you will only see Satellite Tools for RHEL 5 in the RHEL 5 ELS repo

Question: Will we see a publicly available reference architecture of Satellite 6.x with HA?

Answer: With regard to Satellite High Availability (HA), most of the components are HA today. If you look at the Satellite docs in 6.4 we released a guide on making the Capsules load balanced which will continue to deliver content even if the Satellite is unavailable. Also using a hypervisor with HA capability can give you 3-5 nines capability without any major configuration needed. As a result there is no reference arch for Satellite HA, but there is a Solutions article titled: High Availability with Red Hat Satellite 6.3, 6.4, 6.5, and 6.6 that you can refer to for additional information.

Question: Is Satellite integration into an external Puppet Server possible? Is it supported? I want to run Puppet code on a Satellite Server driven by my Puppet Master—not Satellite's

Answer: This could possibly work, but is not tested in our typical testing environment. Please contact your account team to discuss further.

Question: Is Satellite on the roadmap for consumption on the public clouds as a PaaS/SaaS (Marketplace image)?

Answer: With regards to Software-as-a-service (SaaS), we do have a SaaS offering available on cloud.redhat.com that is available at no charge to Smart Management customers. This is not a replacement for Satellite, but there are specific use cases addressed by Smart Management such as Vulnerability, Compliance, and Drift Analysis.

Question: Can Red Hat Satellite be integrated with Chef ?

Answer: Red Hat does not ship a Chef integration plug-in. However you can use Satellite to deliver the Chef RPMs and be sure they are configured properly.

Question: Content Views followup—is it possible to restart the Content View published Version number or make the Version Number a customisable string?

Answer: No—these are not options that are possible. You could file an RFE with a good business case explaining why this would be needed and how it would be used.

Question: We found logs related to Satellite failure not clear or unknown most of the time. For instance Candlepin issues are not giving full details… since “foreman-maintain service status -b“ gives 12+ additional services interrelated, please explain how to easily troubleshoot instead of calling and opening a support case.

Answer: We have roughly 60 Red Hat Insights rules for Satellite—be sure to use Insights on your hosts including your Satellite host for easy troubleshooting before calling or opening a support case. (You will need to view the results at cloud.redhat.com—Results for the Satellite Server are not viewable inside of Satellite.)

When it comes to customer cases we may make Insights rules or KCS articles on common issues. The monitoring guide may also help here.

Question: I'm fairly new to Satellite—I have workloads in on-prem & Azure/GCP ; what Satellite model should I be deploying—(1) RHUI (2) dedicated DIY Satellite (on-prem) (3) Smart Mgt (4) Azure/GCP marketplace images (if applicable) ?

Answer: We would need more detail on the desired use case to advise properly. We’ll explain some of these options:

1: Red Hat Update Infrastructure (RHUI)—makes content available in the cloud. RHUI is a highly available yum server—doesn't really give you all the things you get with Satellite, but you can yum update everything.

2: Satellite makes sense when you reach a certain level of maturity and you want to do more than just run yum update such as using lifecycle environments, Content Views (CV) or provisioning to bare metal, virtual machines, or directly into the public clouds. 

3: Smart Management capabilities are another option via cloud.redhat.com. This consists of cloud management services for Red Hat Enterprise Linux (RHEL) which includes Vulnerability, Compliance, and Drift Analysis (System Comparison). 

4: Azure/GCP Marketplace images—Satellite itself isn’t available as a marketplace image, but we support RHEL marketplace images from most major cloud providers. 

When considering a RHEL marketplace image, you need to consider where you get your subscription from. If you are just starting up a marketplace image you would still need a Smart Management subscription to manage the system with Satellite. Talk to your account team for full information.

In general, a standard build of RHEL in the cloud would probably be best served by being managed by Satellite. This is true if you are paying for Red Hat subscription and using them in the cloud or if you are using the marketplace images.

Question: Content Views: the content we primarily share via Satellite is RHEL. Our Content views (CVs) are different EUS releases (7.6/7.7). Our Versions are baselines at different points of times created using a filter (.e.g once every 3 months) And we use lifecycles as Baselines. Is this is a sound approach?

Answer: Generally speaking, yes. Think of CVs as a group of repositories that are roughly on the same lifecycle. 7.6 / 7.7 are different life cycles, so you are following this model. Filters are a means to further restrict the info in the CV (likely date based). Life cycles helps gate who gets access to what when.

Question:Is there a way to find registered hosts that have not checked in within, say, 6 months or more, and delete them?

Answer: Go to Content Hosts UI and search the following and delete them: 

last_checkin < "180 days ago"

If you use it often, this search can also be saved as a bookmark so it doesn't need to be retyped. This can also be done via the API.

Question: How do the two databases work in Satellite? What forms of data are held in each database? What's the best way to care for those databases?

Answer: There are two databases used in Satellite 6. Katello, Foreman, and Candlepin use PostgreSQL. Pulp uses Mongo. For Mongo—run `mongo repair` every month. For PostgreSQL—Follow Satellite task cleanup recommendations and cron job cleanups.

Question: Talk about content views (CVs)—how best to republish them, and how often should they be republished? How best to delete old ones? (We create/destroy content views weekly. Is this needed?)

Answer: Generally speaking, think of CVs as a group of repos that are roughly on the same lifecycle. Generally speaking CVs will be used for each workload (RHEL5 / RHEL6 / RHEL7). You will probably regenerate CVs as you need to - when you patch systems or when a new version of RHEL comes out. You may also need to update the errata to address a major security issue like Shellshock or Heartbleed.

How best to delete a CV is a side effect of how often you generate them. If you don’t do it often (so you have only a few CVs) then it is probably easiest to do it through the Satellite GUI. If you are using Continuous integration and continuous delivery (CI/CD) you may want to orchestrate the deletion of old CVs. You should also know that if you are churning through a lot of CVs it will be more critical to run the MongoDB maintenance.

Question: I have heard a lot about Insights. How do I get access to it? My organization is conservative about what data we allow to leave our organization.

Answer: Insights is a Software-as-a-Service (SaaS) based offering that started in support to help support teams help customers. Eventually we opened it as a tool to enable customers to help themselves. Insights uses a rule based engine to review issues that might be present in your environment.

Most sev1 issues are caused by known issues—Insights can help you address these issues before you hit them, enabling a proactive approach.

In terms of the data that Insights sends, it is important to know that you have FULL control of what data Insights collects and sends. You can review the collection and see all information before it is sent, then redact any information you do not want to send. Insights does not target any GDPR information. Review the Insights Security Information article for more information.

All supported RHEL versions, typically RHEL 6.4 and above, include a subscription to Insights. Learn more and get started with Insights by visiting the Insights Getting Started page.

Question: The 6.6 upgrade guide refers to the WiredTiger upgrade—is that required? If I already did the WiredTiger upgrade when I upgraded from 6.4 to 6.5 do I need to do it again?

Answer: WiredTiger changes the on-disk format for how MongoDB stored its data. The on-disk format used to be nmap, now it is WiredTiger. 

There are some performance advantages of WiredTiger and we do recommend this change. The WiredTiger upgrade may take some time, so it isn't included as part of the installer and you do need to manually run it. If you already upgraded to WiredTiger in 6.5, you don’t need to do that again on 6.6. The storage engine upgrade should be done just once.

Question: Can I use Satellite Server for OpenShift container platform v4.2 disconnected installation?

Answer: Yes, but this can be complicated. Use Satellite to mirror the containers. This is not a simple export like RPM.

Americas / Europe, Middle East & Africa AMA:

Question: Do you have Satellite 7 in pipeline? If yes, what's the one major feature which will stand it out against satellite 6?

Answer: We are planning a Satellite 7 in the future. This is planned to use the same upstream products as Satellite 6, so the upgrade process will be more like upgrading a Satellite 6 version (not a Satellite 5 to 6 migration). Satellite 7 should also include Pulp 3 which would also get rid of the MongoDB, standardizing on PostgreSQL. We will also likely be running Satellite on only RHEL 8.

Question: Will it be possible at some point to use SSH certificates rather than SSH public keys for remote execution in Satellite?

Answer: Using SSH certs should be possible today, we rely on OpenSSH for authentication.

Question: Currently running Satellite 6.4.4. Can the puppet-agent be removed when moving to Ansible?

Answer: If you are not using the Puppet agent for client management, then you may not need the Puppet agent. The Puppet client is used for endpoint configuration and setting up some functionality such as Insights and OpenSCAP.

As of Satellite 6.5 we supported deploying the Insights agent via Ansible and in Satellite 6.6 we support deploying OpenSCAP via Ansible. As of Satellite 6.6 this means that the Puppet agent isn't needed if you aren't using Puppet on the end points.

Question: I see that Satellite 6.6 is now GA, Is there an easy path to upgrade from Satellite 6.3 to Satellite 6.6. and can you comment on how hard that upgrade is.

Answer: Upgrades are straightforward and we recommend that you use the built-in satellite-maintain / foreman-maintain tools per the upgrade guide. These are incremental upgrades where you need to go 6.3 > 6.4 > 6.5 > 6.6. You can do these one right after the other so the upgrade is completed in a single outage window.

Question: I've been told establishing activation keys strictly using the RHEL version is a bad idea. It's been suggested using multiple activation keys to register a single host is the better path but I'm not sure I follow that suggestion. Can you enlighten me on that idea?

Answer: Activation keys are fundamentally registration tokens that allow you to register a system and address various options. It attaches a subscription and configures a system (correct CV, lifecycle, etc.). You also want to associate the host with host collections. Because of diversity in subscriptions or diversity in your environment you may need to use multiple activation keys. Each activation key should do one thing very well and can be combined via the CLI. We recommend that you read and follow Subscription-manager for the former Red Hat Network User: Part 9 - A Case Study with activation keys.

Question: Currently we patch via an Ansible playbook that patches on the host. Are there advantages of using remote execution, the katello-agent, and initiating patching from the Satellite Server?

Answer: Yes - With regards to Katello-agent, it only handles content related aspects and traditional content stuff. Remote Execution (REX) does content stuff and also handles SCAP, reboots, etc. It is more capable than the katello-agent.

Katello is also a daemon that runs on the client and requires additional ports to be open oon each client.

REX uses SSH and is far more lightweight. Just need to have SSH and sudo set up. REX also has scheduling, so if you want to patch a group of servers you can do it immediately, schedule it for later, or schedule recurring. REX also has the benefit of being able to use capsules to scale the environment.

Question: How can we register servers running AIX OS into Satellite and patch them from Satellite?

If there is not solution for integrating AIX into Satellite, then when it will be implemented? Answer: Currently Satellite only supports Red Hat Enterprise Linux (RHEL) operating systems. If you believe a specific OS should be supported please open a support case and we can evaluate the request.

Customer demand is the biggest impact to making product change. If you have a demand for another operating system to be supported please open a support case. We do look closely at the number of requests we get of this type and use that information to influence the roadmap.

Question: Why is it so fundamentally hard to patch servers at a set time? Was Satellite never designed to truly patch servers? I find patching with Satellite 6 a very time consuming task that is way harder than the Satellite 5.8 version.

Answer: Remote Execution (REX) in Satellite 6 also has scheduling, so if you want to patch a group of servers you can do it immediately, schedule it for later, or schedule recurring.

REX also has the benefit of being able to use Capsules to scale the environment.

Question: Why does the Satellite Server upgrade of a minor version have such a huge impact on the availability of the Satellite services? During the upgrade all services are unavailable until ALL the systems are up to date and a mismatch of Capsule/Satellite versions makes the Capsule not able to serve any clients!

Answer: The primary features that is offline during a Satellite upgrade is the ability to register new hosts. Content and patching from Capsule Servers or Puppet runs will still occur. We do recommend connecting your client to a Capsule and your Capsules to the Satellite for this reason.

Question: Is it possible to query VMware vCenter inventory via `virt-who` for newly provisioned virtual servers faster than every 60 minutes?

Answer: Yes, but this is not recommended.

If you feel as if you need to query the VMware environment more frequently, then your activation keys may be incorrectly configured. Newly registered systems not reported by virt-who will get a temporary subscription. In lieu of setting the virt-who settings low (which puts load on the hypervisor) look at Subscription-manager for the former Red Hat Network User: Part 9 - A Case Study with activation keys to take advantage of the temporary subscription (which are good for 7 days). We often recommend only running virt-who once daily with this temporary subscription model.

Question: It's 2019, why is a fresh Satellite install still using Postgresql 9.2, especially considering parallelism was introduced in Postgresql 9.6?

Answer: Upgrading databases in a product like Satellite must be handled with care to make sure there is no impact to the product or your operational environment. We are looking at an upgrade in a future version of Satellite, but we are conservative when it comes to upgrading underlying database versions.

Question: We’re on Satellite 6.5.2.1. We have not updated to WiredTiger (in part due to the requirement to have /var/tmp twice as big /var/lib/mongodb). Given that MongoDB will be removed in a future release: is there any requirement to upgrade WiredTiger? Can we skip this, and is that advisable?

Answer: WiredTiger changes the on-disk format for how MongoDB stored its data. The on-disk format used to be nmap, now it is WiredTiger. 

There are some performance advantages of WiredTiger and we do recommend this change.

The WiredTiger upgrade may take some time, so it isn't included as part of the installer and you do need to manually run it. It is ok to delay this update if you are not having any performance issues. 

Question: It will be good to have an option to view and export hosts list based on operating system. 

Answer: One of the things we have gotten feedback on it that Satellite has good info about the environment, but it is hard to get the info out. To help address this, in 6.5 we introduced the reporting engine which includes a built-in registered systems report that can be run to generate a CSV that contains this info. In 6.6 you can also schedule and email this report. In versions prior to Satellite 6.5 you can use a hammer command with the csv option to get this information.

Question: Is there any way to get a different High Availability (HA) approach than through a Load Balancer? Why can't we leverage the Satellite to act as a fallback in case of Capsule unavailability?

Answer: The Answer to this is a combination of certificates, consistency, and features. 

For failback the general approach is to have a second identically configured capsule. That is one of the reasons we gave all customers the Satellite infrastructure subscriptions which give up by default any combination of 50 Satellites and Capsules. 

We also haven't had as much demand. Satellite 6.4 introduced the load balanced capsules.

If HA is needed at the Satellite level we recommend putting the Satellite on an enterprise level hypervisor and using the hypervisor level HA. For more information refer to the solutions article titled: High Availability with Red Hat Satellite 6.3, 6.4, 6.5, and 6.6.

Question: We are just getting past the initial configuration of Satellite 6. What are the first couple of things to start on with Satellite? What should we leave until we've gotten more experience with Satellite?

Answer: Most reason companies buy a Smart Management subscription is to use Satellite to manage a bunch of RHEL systems that they need to provision and patch at scale. 

Once you have Satellite installed, getting patch management in place is probably the first thing to do.

The second thing would probably be to set up Insights. Insights will help optimize your usage of RHEL. This allows you to find issues in your environment that might affect your system stability before the impact occurs.

The third thing would probably be provisioning. Provisioning can be complex to get setup when interacting with networking and VM teams. The next items might be things like OpenSCAP or advanced functionality.

Question: "Remote Execution is preferred." Is there a document to guide us as to how to migrate to REX and remove katello-agent?

Answer: We do have a step-by-step guide for this. Review Satellite 6: Goferless Infrastructure Clean-up & How To and Red Hat Satellite 6.2.11 Feature Overview: Goferless Host Management.

Question: How can one set preferences for Remote Execution (REX), either SSH or Ansible, to use available capsules as first choice and the Satellite as last choice?

Answer: Capsule servers pretty much do this automatically. If you use the infrastructure page and look at the subnets you can specify the capsule to use for the subnet.

There is also certain fallback behaviors. We'll look if we are using capsules in other ways (such as for OpenSCAP) and lastly the Satellite is the global fallback.

When you register clients get them associated with the correct subnets and everything should work.

Question: Satellite 5.8 is EOL May 31, 2020. 

What will happen after it goes EOL? Will I still be able to patch or does it just no longer work? What functions stop working? Are there still Security updates?

Answer: Satellite 5.8 goes EOL May 31, 2020.

On June 1 a few things will happen:

• Red Hat will no longer ship bug fixes or enhancements for Satellite 5.

• If you call for support, anything other than upgrading to Satellite 6 will be closed. 

• For content, any repositories already synced, these will continue to sync.

• Any net new repositories (such as RHEL 8.3) will not be available for Satellite 5.

Question: What new feature in 6.6 are you most excited about?

Answer: All of them! When we think of what is core to Satellite, the features that stick out are the ability to leverage Ansible variables. These give the sysadmin a way to customize the behavior of Ansible roles. One of our Technical Account Managers (TAM) recently wrote a blog on this topic titled: Advanced Ansible variables in Satellite.

Now the OpenSCAP client can also be configured via Ansible System roles, so all workflows can now be performed via Ansible and are no longer dependent on Puppet.

Join us for the next AMAs

The next Red Hat Satellite AMAs will run at two times:

• January 15, 2020 at 9am EST/ 2pm GMT / 3pm CET

• January 15, 2020 at 1pm EST / Noon CST / 11am MST / 10am PST

Register for the AMA here.

Please join us and bring any Questions about Satellite that you might have. We look forward to hearing from you!