Red Hat SummitIn Part 1 of this series Federico Iezzi, EMEA Cloud Architect with Red Hat covered the architecture and planning requirements to begin the journey into achieving zero packet loss in Red Hat OpenStack Platform 10 for NFV deployments. In Part 2 he went into the details around the specific tuning and parameters required. Now, in Part 3, Federico concludes the series with an example of how all this planning and tuning comes together!
Putting it all together …
So, what happens when you use the cpu tuning features?
Well, it depends on the hardware choice of course. But to see some examples we can use Linux perf events to see what is going on. Let’s look at two examples.
Virtual Machine
On a KVM VM, you will have the ideal results because you don't have all of the interrupts from the real hardware:
$ perf record -g -C 1 -- sleep 2h $ perf report --stdio -n # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 100 of event 'cpu-clock' # Event count (approx.): 25000000 # # Children Self Command Shared Object Symbol # ........ ........ ....... ................. ..................... # 100.00% 0.00% swapper [kernel.kallsyms] [k] default_idle | ---default_idle native_safe_halt 100.00% 0.00% swapper [kernel.kallsyms] [k] arch_cpu_idle | ---arch_cpu_idle default_idle native_safe_halt 100.00% 0.00% swapper [kernel.kallsyms] [k] cpu_startup_entry | ---cpu_startup_entry arch_cpu_idle default_idle native_safe_halt 100.00% 100.00% swapper [kernel.kallsyms] [k] native_safe_halt | ---start_secondary cpu_startup_entry arch_cpu_idle default_idle native_safe_halt 100.00% 0.00% swapper [kernel.kallsyms] [k] start_secondary | ---start_secondary cpu_startup_entry arch_cpu_idle default_idle native_safe_halt
Physical Machine
On physical hardware, it's quite different. The best results involved backlighting a bunch of ipmi and watchdog kernel modules:
$ modprobe -r iTCO_wdt iTCO_vendor_support $ modprobe -r i2c_i801 $ modprobe -r ipmi_si ipmi_ssif ipmi_msghandler
Note: If you have a different watchdog than the example above, (iTCO is for Supermicro motherboards), check out the kernel modules folder where you can find the whole list: /lib/modules/*/kernel/drivers/watchdog/
Here’s the perf command and output for physical:
$ perf record -F 99 -g -C 2 -- sleep 2h $ perf report --stdio -n # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 4 of event 'cycles:ppp' # Event count (approx.): 255373 # # Children Self Samples Command Shared Object Symbol # ........ ........ ............ ....... ................. .............................................. # 99.83% 0.00% 0 swapper [kernel.kallsyms] [k] generic_smp_call_function_single_interrupt | ---generic_smp_call_function_single_interrupt | --99.83%--nmi_restore 99.83% 0.00% 0 swapper [kernel.kallsyms] [k] smp_call_function_single_interrupt | ---smp_call_function_single_interrupt generic_smp_call_function_single_interrupt | --99.83%--nmi_restore 99.83% 0.00% 0 swapper [kernel.kallsyms] [k] call_function_single_interrupt | ---call_function_single_interrupt smp_call_function_single_interrupt generic_smp_call_function_single_interrupt | --99.83%--nmi_restore 99.83% 0.00% 0 swapper [kernel.kallsyms] [k] cpuidle_idle_call | ---cpuidle_idle_call call_function_single_interrupt smp_call_function_single_interrupt generic_smp_call_function_single_interrupt | --99.83%--nmi_restore 99.83% 0.00% 0 swapper [kernel.kallsyms] [k] arch_cpu_idle | ---arch_cpu_idle cpuidle_idle_call call_function_single_interrupt smp_call_function_single_interrupt generic_smp_call_function_single_interrupt | --99.83%--nmi_restore
Using mpstat, and excluding the hardware interrupts, the results are as follows:
Please note: one CPU core per socket has been excluded - in this case using two Xeon E5-2640 V4.
$ mpstat -P 1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,31,32,32,34,35,36,37,38,39 3600 Linux 3.10.0-514.16.1.el7.x86_64 (ws1.localdomain) 04/20/2017 _x86_64_ (40 CPU) 03:05:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Cool, right? Want to know more about Linux perf events? Check out the following links:
- Linux perf Examples - Read both the web page and the presentation, also listen to the youtube video
- SCALE13x: Linux Profiling at Netflix
- Tracing and Profiling - Yocto Project
- Tutorial - Perf Wiki
Zero Packet Loss, achieved …
As you can see, using tuned and the cpu-partitioning profile is exceptional in that it exposes a lot of deep Linux tuning which usually only a few people know about.
And with a combination of tuning, service settings, and plenty of interrupt isolations (over 50% of the total settings are about interrupt isolations!) things really start to fly.
Finally, once you make sure PMD threads and VNF vCPUs do not get interrupted by other threads allowing for proper CPU core allocation, the zero packet loss game is achieved.
Of course, there are other considerations such as the hardware chosen, the VNF quality, and the number of PMD threads, but, generally speaking, those are the main requirements.
Further Reading ...
Red Hat Enterprise Linux Performance Tuning Guide
Network Functions Virtualization Configuration Guide (Red Hat OpenStack Platform 10)
Check out the Red Hat Services Webinar Don't fail at scale: How to plan, build, and operate a successful OpenStack cloud today!
Sull'autore
Federico Iezzi is an open-source evangelist who has witnessed the Telco NFV transformation. Over his career, Iezzi achieved a number of international firsts in the public cloud space and also has about a decade of experience with OpenStack. He has been following the Telco NFV transformation since 2014. At Red Hat, Federico is member of the EMEA Telco practice as a Principle Architect.
Altri risultati simili a questo
Ricerca per canale
Automazione
Novità sull'automazione IT di tecnologie, team e ambienti
Intelligenza artificiale
Aggiornamenti sulle piattaforme che consentono alle aziende di eseguire carichi di lavoro IA ovunque
Hybrid cloud open source
Scopri come affrontare il futuro in modo più agile grazie al cloud ibrido
Sicurezza
Le ultime novità sulle nostre soluzioni per ridurre i rischi nelle tecnologie e negli ambienti
Edge computing
Aggiornamenti sulle piattaforme che semplificano l'operatività edge
Infrastruttura
Le ultime novità sulla piattaforma Linux aziendale leader a livello mondiale
Applicazioni
Approfondimenti sulle nostre soluzioni alle sfide applicative più difficili
Serie originali
Raccontiamo le interessanti storie di leader e creatori di tecnologie pensate per le aziende
Prodotti
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Servizi cloud
- Scopri tutti i prodotti
Strumenti
- Formazione e certificazioni
- Il mio account
- Supporto clienti
- Risorse per sviluppatori
- Trova un partner
- Red Hat Ecosystem Catalog
- Calcola il valore delle soluzioni Red Hat
- Documentazione
Prova, acquista, vendi
Comunica
- Contatta l'ufficio vendite
- Contatta l'assistenza clienti
- Contatta un esperto della formazione
- Social media
Informazioni su Red Hat
Red Hat è leader mondiale nella fornitura di soluzioni open source per le aziende, tra cui Linux, Kubernetes, container e soluzioni cloud. Le nostre soluzioni open source, rese sicure per un uso aziendale, consentono di operare su più piattaforme e ambienti, dal datacenter centrale all'edge della rete.
Seleziona la tua lingua
Red Hat legal and privacy links
- Informazioni su Red Hat
- Opportunità di lavoro
- Eventi
- Sedi
- Contattaci
- Blog di Red Hat
- Diversità, equità e inclusione
- Cool Stuff Store
- Red Hat Summit