As anyone in the technology field will tell you, even the most redundant systems experience outages. A valuable part of learning what happened and how to prevent it from happening again can come from memory and crash dumps, which most operating systems have been able to produce for some time. These dumps can show you what was running at the time of the crash and allow greater insight into the issue.
The virtctl memory-dump option within OpenShift Virtualization can help you do that. It allows you to grab the virtual memory directly from the system experiencing trouble, even if it's not accessible through conventional means. A huge advantage of OpenShift Virtualization tools is that they are OS-independent, and you don't need to wait for a physical or virtual machine to become accessible again to grab what was running in memory.
This article demonstrates how to get a crashed virtual machine's memory and a couple of ways to check to see what may have happened.
Generating a virtual machine memory dump using virtctl from the CLI
Creating the memory dump and downloading it for use is a relatively simple process, but it requires you to have the virtctl command line tool available. This tool can be acquired directly from the OpenShift UI by selecting the "Command line tools" link in the help menu.
This demonstration uses Windows Server 2019 and RHEL 9 virtual machines, but this process works the same for any OS that OpenShift Virtualization supports. Look at the VM to understand what I'm working with in this example. Since I'm only concerned with the size of the memory allocated to it, which will affect the time involved in creating and downloading the dump, I can grep just for the memory in the YAML output. The command used is below:
$ oc get vm beehive-win19 -o yaml | grep memory
memory: 4Gi
I'm working with 4Gi, so the process will complete quickly. Completion times will vary depending on machine configuration. Next, I create and export the memory dump with the following command:
$ virtctl memory-dump get beehive-win19 \
> --create-claim --claim-name=beehive-win19-mem \
> --output=windows19-mem.dump.tar.gz
Above, I created a memory dump of the beehive-win19 virtual machine and made a new PVC to store the dump, naming it beehive-win19-mem. A VM export job is created, and the memory dump called windows19-mem.dump.tar.gz is downloaded to the machine from which I ran the command. The VM export job is then removed. Alternatively, if you already have a PVC available, you can simply omit --create-claim and enter the desired storage instead of creating a new one. Below is a truncated example of the output:
PVC beehive-virtual-machines/beehive-win19-mem created
Successfully submitted memory dump request of VM beehive-win19
Waiting for memorydump to complete, current phase: Associating...
Waiting for memorydump to complete, current phase: InProgress...
Memory dump completed successfully
VirtualMachineExport 'beehive-virtual-machines/export-beehive-win19-beehive-win19-mem' created successfully
waiting for VM Export export-beehive-win19-beehive-win19-mem status to be ready...
Downloading file: 1.47 GiB [==================>] 3.27 MiB p/s
Download finished successfully
VirtualMachineExport 'beehive-virtual-machines/export-beehive-win19-beehive-win19-mem' deleted successfully
So now that I have the memory dump, what do I do with it? I'll explore a couple of examples.
Obtaining Information from an OpenShift VM memory dump
After decompressing the tar files, I'll run some commands using the volatility3 tool on the memory dumps. The output below shows a truncated list of the processes the Windows server was running, its parent processes, and some additional info.
$ python3 vol.py -f beehive-win19-beehive-win19-mem-20230808-135236.memory.dump windows.pslist
Volatility 3 Framework 2.5.0
Progress: 100.00 PDB scanning finished
PID PPID ImageFileName Offset(V) Threads Handles SessionId Wow64 CreateTime ExitTime File output
4 0 System 0x880a18c78040 86 - N/A False 2023-07-31 23:36:55.000000 N/A Disabled
88 4 Registry 0x880a18d99040 4 - N/A False 2023-07-31 23:36:52.000000 N/A Disabled
292 4 smss.exe 0x880a1c175040 2 - N/A False 2023-07-31 23:36:55.000000 N/A Disabled
400 388 csrss.exe 0x880a1d799140 10 - 0 False 2023-07-31 23:37:03.000000 N/A Disabled
476 468 csrss.exe 0x880a1c136080 9 - 1 False 2023-07-31 23:37:04.000000 N/A Disabled
496 388 wininit.exe 0x880a1d9ac080 1 - 0 False 2023-07-31 23:37:04.000000 N/A Disabled
Now, I'll look at a RHEL memory dump using a similar command. Below is the truncated output of the RHEL 9 virtual machine's process list.
Please note that to run these commands on a Linux memory dump successfully, you will need to follow the guide the volatility team provides to gather the correct debug symbols for your kernel.
python3 vol.py -v -f cnv-galera-db01-beehive-galeradb01-mem-20230829-184446.memory.dump linux.pslist
Volatility 3 Framework 2.5.0
Progress: 100.00 Stacking attempts finished
OFFSET (V) PID TID PPID COMM
0x8b27c0251c80 1 1 0 systemd
0x8b27c0255580 2 2 0 kthreadd
0x8b27c0253900 3 3 2 rcu_gp
0x8b27c0250000 4 4 2 rcu_par_gp
0x8b27c027b900 5 5 2 slub_flushwq
0x8b27c0278000 6 6 2 netns
Connecting the digital dots
Whether you require the ability to find out why something went awry, track down malicious software, or analyze what was happening on a system when it was compromised, the memory dump option that comes with the virtctl tools can be an invaluable accessory at your disposal with OpenShift virtual machines.
À propos de l'auteur
Contenu similaire
Parcourir par canal
Automatisation
Les dernières nouveautés en matière d'automatisation informatique pour les technologies, les équipes et les environnements
Intelligence artificielle
Actualité sur les plateformes qui permettent aux clients d'exécuter des charges de travail d'IA sur tout type d'environnement
Cloud hybride ouvert
Découvrez comment créer un avenir flexible grâce au cloud hybride
Sécurité
Les dernières actualités sur la façon dont nous réduisons les risques dans tous les environnements et technologies
Edge computing
Actualité sur les plateformes qui simplifient les opérations en périphérie
Infrastructure
Les dernières nouveautés sur la plateforme Linux d'entreprise leader au monde
Applications
À l’intérieur de nos solutions aux défis d’application les plus difficiles
Programmes originaux
Histoires passionnantes de créateurs et de leaders de technologies d'entreprise
Produits
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Services cloud
- Voir tous les produits
Outils
- Formation et certification
- Mon compte
- Assistance client
- Ressources développeurs
- Rechercher un partenaire
- Red Hat Ecosystem Catalog
- Calculateur de valeur Red Hat
- Documentation
Essayer, acheter et vendre
Communication
- Contacter le service commercial
- Contactez notre service clientèle
- Contacter le service de formation
- Réseaux sociaux
À propos de Red Hat
Premier éditeur mondial de solutions Open Source pour les entreprises, nous fournissons des technologies Linux, cloud, de conteneurs et Kubernetes. Nous proposons des solutions stables qui aident les entreprises à jongler avec les divers environnements et plateformes, du cœur du datacenter à la périphérie du réseau.
Sélectionner une langue
Red Hat legal and privacy links
- À propos de Red Hat
- Carrières
- Événements
- Bureaux
- Contacter Red Hat
- Lire le blog Red Hat
- Diversité, équité et inclusion
- Cool Stuff Store
- Red Hat Summit