BACKGROUND
RIPSAW
Ripsaw is a benchmark operator for OpenShift and Kubernetes that is used to establish a performance baseline of your cluster by deploying common workloads such as: uperf, iperf3, fio, sysbench, YCSB, pgbench, smallfile, fs-drift, and hammerdb. You can also use your own workload if you are not satisfied with the workloads built in. Since this article focuses on uperf, however, we will ignore custom workloads and the other common workloads.
UPERF
Uperf is a network performance tool that uses a high-level language called a “profile” to model real-world applications. Users create “profiles” to generate generic workloads for assessing network statistics, including but not limited to: bandwidth or latency with different network protocols (for example, TCP, UDP, etc.), TCP congestion control algorithms, and connection setup/teardown scalability for different network protocols.
SETTING UP THE RIPSAW OPERATOR AND RUNNING UPERF
PREREQUISITES
This section assumes you already have an OpenShift/Kubernetes cluster running. If you do not have one running, please create one before continuing to the next step. Also, make sure that you have set the KUBECONFIG environment variable to point to your kubeconfig file.
GETTING THE OPERATOR SOURCES
To set up the operator, clone the Ripsaw git repo:
$ export RIPSAW=/tmp/ripsaw
$ git clone https://github.com/cloud-bulldozer/ripsaw.git ${RIPSAW}
CREATE THE RIPSAW NAMESPACE
Create the namespace like so:
$ kubectl apply -f ${RIPSAW}/resources/namespace.yaml
This command will generate a namespace called “my-ripsaw,” and you should see the following output in your terminal: namespace/my-ripsaw created
PREPARE TO DEPLOY THE OPERATOR
It’s almost time to deploy the operator. But before you do this, you must modify the CR for uperf. For simplicity’s sake, you will use most of the defaults in the uperf CR file. To run uperf across two nodes, you need to select two nodes you want to analyze. To list your available nodes:
$ kubectl get nodes -n my-ripsaw
Once you’ve done so, modify the uperf CR file, which can be found here:
https://github.com/cloud-bulldozer/ripsaw/blob/master/resources/crds/ripsaw_v1alpha1_uperf_cr.yaml
Change “node-0” with the name of the first node you want to use. Then change “node1” with the other node you want to use. Finally, set “cleanup” to “false” so that the benchmark pod isn’t deleted after completion.
DEPLOYING THE OPERATOR
Now that you’ve modified the uperf CR yaml file, it’s time to deploy the operator. To deploy:
$ kubectl apply -f ${RIPSAW}/deploy
$ kubectl apply -f ${RIPSAW}/resources/crds/ripsaw_v1alpha1_ripsaw_crd.yaml
$ kubectl apply -f ${RIPSAW}/resources/crds/ripsaw_v1alpha1_uperf_cr.yaml
$ kubectl apply -f ${RIPSAW}/resources/operator.yaml
You should see the following outputs:
serviceaccount/benchmark-operator created
role.rbac.authorization.k8s.io/benchmark-operator created
rolebinding.rbac.authorization.k8s.io/benchmark-operator created
clusterrole.rbac.authorization.k8s.io/benchmark-operator-kube created
customresourcedefinition.apiextensions.k8s.io/benchmarks.ripsaw.cloudbulldozer.io created
deployment.apps/benchmark-operator created
ANALYZING UPERF RESULTS
FIND UPERF CLIENT POD
The uperf results can be found in a “uperf-client-<...>” pod. To find such pods:
$ kubectl get pods -n my-ripsaw | grep uperf-client
To view your results, run the following command (replacing <...> with the hash of your pod):
$ kubectl logs pod/uperf-client-<...> -n my-ripsaw
The output from the above command will show you the “profile.” A description of what “transactions” and “flowops” are will be provided in the following subsections.
HANDSHAKE PHASES
The output below is the debug output for the handshake phases (1 and 2) to test the connection. For example:
Allocating shared memory of size 156624 bytes
Completed handshake phase 1
Starting handshake phase 2
Handshake phase 2 with <server-pod-ip>
Done preprocessing accepts
Sent handshake header
Sending workorder
Sent workorder
Sent transaction
Sent flowop
Sent transaction
Sent flowop
Sent transaction
Sent flowop
TX worklist success Sent workorder
Handshake phase 2 with <server-pod-ip> done
Completed handshake phase 2
In essence, the handshake process is broken down into two “phases.” During phase 1, the client attempts to connect to the remote host, and verification of version numbers, supported protocols, etc., is executed. During phase 2, however, strands are created in prep for the network testing. A “strand” is a lightweight thread that “either [processes] independent packets (belonging to different connections) or [performs] bookkeeping operations.” Also during this phase is the process of sending (“transferring”) a workorder from the first node to the second node, where the workorder contains one or more flowops. Flowops are essentially basic operations such as “connect,” “disconnect,” “send,” “receive,” and “sendfile.” See reference 5 for a complete list of supported ops, including ones you can write yourself.
EXECUTION OF THE FLOWOPS
Following the handshake phases’ debug output, you should see a similar debug output that describes which transaction from the workorder is being called:
Starting 1 threads running profile:stream-tcp-16384-1 ... 0.00 seconds
TX command [UPERF_CMD_NEXT_TXN, 0] to <server-pod-ip>
timestamp_ms:1579010499379.3477 name:Txn1 nr_bytes:0 nr_ops:0
timestamp_ms:1579010500380.2354 name:Txn1 nr_bytes:0 nr_ops:1
TX command [UPERF_CMD_NEXT_TXN, 1] to <server-pod-ip>
timestamp_ms:1579010500380.3958 name:Txn2 nr_bytes:0 nr_ops:0
timestamp_ms:1579010501382.2000 name:Txn2 nr_bytes:219152384 nr_ops:13376
timestamp_ms:1579010502385.6763 name:Txn2 nr_bytes:727973888 nr_ops:44432
timestamp_ms:1579010503387.1470 name:Txn2 nr_bytes:1226309632 nr_ops:74848
timestamp_ms:1579010504387.8291 name:Txn2 nr_bytes:1772355584 nr_ops:108176
timestamp_ms:1579010505400.3938 name:Txn2 nr_bytes:2225078272 nr_ops:135808
</snip>
timestamp_ms:1579045625165.4785 name:Txn2 nr_bytes:12914786304 nr_ops:788256
timestamp_ms:1579045626166.5405 name:Txn2 nr_bytes:13346537472 nr_ops:814608
Sending signal SIGUSR2 to 140690430813952
called out
timestamp_ms:1579010530670.2065 name:Txn2 nr_bytes:13825212416 nr_ops:843824
TX command [UPERF_CMD_NEXT_TXN, 2] to <server-pod-ip>
timestamp_ms:1579010530670.3181 name:Txn3 nr_bytes:0 nr_ops:0
timestamp_ms:1579010530670.3245 name:Txn3 nr_bytes:0 nr_ops:0
The above output has been truncated (hence the “</snip>”), but this output format is what you should expect to see after successfully running uperf with Ripsaw.
Remember that workorder transactions are linked to specific flowops. In the previous subsection, three separate flowops were sent (as transactions!) in the workorder. In the logs above, their transactions are conveniently named “Txn1,” “Txn2,” and “Txn3,” and the flowops are the values defined in the “profile” output of the logs.
FLOWOP AND TRANSACTION (“TXN”) DEFINITIONS AND STATISTICS
Once the flowops have successfully been executed, the “UPERF_CMD_SEND_STATS” command collects and transfers all the statistics, as shown in the sample output below:
---------------------------------------------------------------------------
TX command [UPERF_CMD_SEND_STATS, 0] to <server-pod-ip>
timestamp_ms:1579010530772.1462 name:Total nr_bytes:13825212416 nr_ops:843825
Group Details
---------------------------------------------------------------------------
timestamp_ms:1579010530671.5735 name:Group0 nr_bytes:0 nr_ops:1
Strand Details
---------------------------------------------------------------------------
timestamp_ms:1579010530671.5737 name:Thr0 nr_bytes:13825212416 nr_ops:843827
Txn Count avg cpu max min
---------------------------------------------------------------------------
Txn0 1 853.41us 0.00ns 853.41us 853.41us
Txn1 52740 570.21us 0.00ns 148.09ms 30.09us
Txn2 1 13.15us 0.00ns 13.15us 13.15us
Flowop Count avg cpu max min
---------------------------------------------------------------------------
connect 1 852.71us 0.00ns 852.71us 852.71us
write 843824 35.61us 0.00ns 148.09ms 29.99us
disconnect 1 12.66us 0.00ns 12.66us 12.66us
You can see that there are three transactions and three flowops. From viewing this debug output, you can see that the client (also known as the “uperf-client” pod) connects to the server (also known as the “uperf-server” pod), sends some data (“write”), then disconnects. So in other words, the first transaction involves connecting each process from the client to the server, the second transaction involves sending 1 byte at a time, and the third transaction involves disconnecting each process from the client to the server (also known as cleanup).
Under the “Txn” (transaction) table, there is a “Count” column, which describes the number of times the transaction was called. So in other words, transaction 0 was called 1 time, transaction 1 was called 52,740 times, and transaction 2 was called 1 time. Similarly, for “Flowop” values, the count represents the number of times connected (once), sent 1 byte (843,824 times), and disconnected (once).
Finally, you can see that the “avg,” “max,” and “min” columns represent the average, maximum, and minimum time it took to execute a transaction or a flowop. The “cpu” column represents the CPU time divided by the count.
NETSTAT RESULTS
The next subsection describes the netstat results. For example:
Netstat statistics for this run
-------------------------------------------------------------------------------
Nic opkts/s ipkts/s obits/s ibits/s
eth0 9503 11004 3.42Gb/s 5.81Mb/s
Nic = Network Interface Card
opkts = packets sent (“output”)
ikpts = packets received (“input”)
obits = bits sent (“output”)
ibits = bits received (“input”)
RUN STATISTICS
The final section describes the statistics for the run. For example, you may see an output like:
Run Statistics
Hostname Time Data Throughput Operations Errors
-------------------------------------------------------------------------------
[<server-pod-ip>] 32.39s 12.87GB 3.41Gb/s 843443 0.00
master 32.39s 12.88GB 3.41Gb/s 843827 0.00
-------------------------------------------------------------------------------
Difference(%) -0.00% 0.05% 0.05% 0.05% 0.00%
If you recall from earlier, you set the test type equal to “stream” in the CR. So, the above output reflects the results from the “stream” test type with a message size of 16,384 bytes.
NETWORK TEST TYPE
This describes the underlying way the packets will take to reach its destination. There are three basic ways a client and server can communicate.
- Using HostNetwork
When the hostnetwork argument in the CR is set to true, the uperf pods can directly access the network interface of the node on which they are running. Having a pod access and/or be accessible on the node’s interface is a privileged operation on OpenShift. It is the type where the throughput obtained between client and server is the highest amongst the other types. For using this type, you tweak the CR as follows:
<snip>
workload:
cleanup: false
name: uperf
args:
serviceip: false
hostnetwork: true
</snip>
2. Using SDN
This is default and uses the overlay network. It can be OpenShiftSDN or OVNKubernetes. Here the traffic is flowed through the overlay network the user chooses. As of now, the OpenShiftSDN is the default SDN used by OpenShift. It configures an overlay network using Open vSwitch (OVS). For using this type, set serviceip: false and hostnetwork: false
3. Using Service
Service is an abstract way to expose an application running on a set of pods as a network service. Pods are expendable resources and are deleted and restarted time and again. Therefore, they get assigned to any node in the cluster and get a new pod IP address. This can be a problem for an application that is using this pod's resources through the pod's IP address. Here the service is helpful as it is not deleted, and in this case, the client and server are connected through a service. For using this type, set serviceip: true and hostnetwork: false
For further fine tuning of the Ripsaw CR and network performance results, please checkout Benchmarking OpenShift Network Performance Part 2: Deep Dive.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.