This post is the sequel to the object storage performance testing we did two years back based on Red Hat Ceph Storage 2.0 FileStore OSD backend and Civetweb RGW frontend. In this post, we will compare the performance of the latest available (at the time of writing) Ceph Storage i.e. version 3.3 (BlueStore OSD backend & Beast RGW frontend) with Ceph Storage 2.0 version (mid-2017) (FileStore OSD backend & Civetweb RGW frontend).
We are conscious that results from both these performance studies are not scientifically comparable. However, we believe that comparing the two should provide you significant performance insights and enables you to make an informed decision when it comes to architecting your Ceph storage clusters.
As expected, Ceph Storage 3.3 outperformed Ceph Storage 2.0 for all the workloads that we have tested. We believe that Ceph Storage 3.3 performance improvements are attributed to the combination of several things. The BlueStore OSD backend, the Beast web frontend for RGW, the use of Intel Optane SSDs for BlueStore WAL, block.db, and the latest generation Intel Cascade Lake processors.
Let’s take a closer look at the performance comparison of the two Ceph Storage versions.
Test Lab Configuration
Here's a quick rundown of our lab environment. Testing was performed in a Dell EMC lab using their hardware, and hardware contributed by Intel.
Ceph Storage 3.3 Environment
Table 1: Ceph Storage 3.3 Test Lab Configurations
Ceph Storage 2.0 Environment
Table 2: Ceph Storage 2.0 Test Lab Configurations
Small Object Performance
In this section we'll take a look at small object performance.
Small Object 100% write workload
Chart 1 compares the performance of small object 100% write workload for both Ceph Storage 3.3 and Ceph Storage 2.0 versions. As shown in the chart, Ceph Storage 3.3 consistently delivered orders of magnitude higher Operations per second (Ops) performance compared to Ceph Storage 2.0. As such we have observed over 500% higher Ops with Ceph Storage 3.3.
You might be wondering, what could be the reason for 2 x sharp performance degradation points in Ceph Storage 3.3 performance line? This was supposedly caused due by the RGW dynamic bucket re-sharding event, which has been explained in-depth in the previous post.
Chart 1: Small Object Size 100% Write Performance
Small Object 100% read workload
The 100% read workload for Ceph Storage 3.3 showed near-perfect deterministic performance compared to Ceph Storage 2.0, whose performance falls from the cliff at around 18th hour of testing. For Ceph Storage 2.0, this decline in read OPS was caused due to an increase in time taken for filesystem metadata lookup as the cluster object count grew.
When the cluster held fewer objects, a greater percentage of filesystem metadata was cached by the kernel in memory. However, when the cluster grew to millions of objects, a smaller percentage of metadata was cached. Disks were then forced to perform I/O operations specifically for metadata lookups, adding additional disk seeks and resulting in lower read OPS.
Bluestore OSD backends powering Ceph Storage 3.3 and beyond do not rely on Linux page caches during read operations. As such the Bluestore OSDs does their own memory management and hence helps in achieving deterministic performance for read workloads as shown in chart 2.
Chart 2: Small Object Size 100% Read Performance
Large Object Performance
Next we'll turn our attention to large object performance benchmarks.
Large Object 100% write workload
For large object 100% write test, Ceph Storage 3.3 delivered sub-linear performance improvement, while Ceph Storage 2.0 shows retrograde performance as depicted in chart 3. During Ceph Storage 2.0 test, we were lacking additional RGW node while Containerized Storage Daemons were also not an option. As such, we were unable to test beyond 4 RGWs for Ceph Storage 2.0 test.
Ceph Storage 3.3, unless saturated by sub-system resource saturation, delivered ~5.5 GBps of write bandwidth. Ceph Storage 2.0, on the other hand showed declining performance.
Chart 3: Large Object Size 100% Write Performance
Large Object 100% read workload
The large object 100% read workload, showed sub-linear performance for both Ceph Storage 2.0 and 3.3 tests. As explained in the last section, at the time of Ceph Storage 2.0 test, we were not having additional physical nodes for RGWs hence we were limited to 4 and could not scale RGWs further.
Ceph Storage 3.3 showed ~5.5 GBps 100% read bandwidth with no bottlenecks observed with 7 RGWs. Our hypothesis is that, Ceph Storage 3.3 could have delivered more bandwidth, have we added more RGWs. Chart-4 compares the performance of Ceph Storage 3.3 and Ceph Storage 2.2 for 100% Large Object workload.
Chart 4: Large Object Size 100% Read Performance
Summary and up next
We acknowledge that the performance comparison shown above is not entirely “Apples-to-Apples." However, we believe that this study should give you key performance insights for different Ceph Storage versions and help you make informed decisions.
We tried our best to choose the common ground across the two studies. Interestingly, with just half, the number of spindles (110 Ceph Storage 3.3 test vs 210 Ceph Storage 2.0 test) Ceph Storage 3.3 with BlueStore OSD backend and Beast RGW frontend web server delivered better than 5x higher Ops for small object and better than 2x higher bandwidth for large object 100% write workload. In the next post, we will uncover the performance of storing 1 billion-plus objects into the Ceph cluster. Stay Tuned.
저자 소개
유사한 검색 결과
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.