Finding block and file OCP application contents in ODF: Creating a file storage project

This article series helps you better understand how to map application data within ODF clusters by using the Rook toolkit and OpenShift commands. This knowledge and the associated tools can be very useful for troubleshooting as well as a deeper understanding of data storage in ODF. Be sure you've read parts one and two before reading this one (part three of the series).

Create a file storage project

You'll follow the same principles that you've used so far to do something similar to the earlier block storage project with an application that uses CephFS from ODF for shared storage. Create a new project called ocs-file-app:

[alexon@bastion ~]$ oc new-project ocs-file-app
Now using project "ocs-file-app" on server "https://api.example.com:6443".

You can add applications to this project with the new-app command. For example, to build a new example application in Ruby, try:

oc new-app rails-postgresql-example

Or use kubectl to deploy a simple Kubernetes application:

kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

For this example, use an OpenShift PHP File Upload Demo application (which I forked from Christian Hernandez's project—due credits) that will serve your interests well:

[alexon@bastion ~]$ oc new-app openshift/php:7.2-ubi8~https://github.com/AlexonOliveiraRH/openshift-php-upload-demo.git --name=file-uploader
--> Found image 67520c7 (5 weeks old) in image stream "openshift/php" under tag "7.2-ubi8" for "openshift/php:7.2-ubi8"

    Apache 2.4 with PHP 7.2 
   
----------------------- 
    PHP 7.2 available as container is a base platform for building and running various PHP 7.2 applications and frameworks. PHP is an HTML-embedded scripting language. PHP attempts to make it easy for developers to write dynamically generated web pages. PHP also offers built-in database integration for several commercial and non-commercial database management systems, so writing a database-enabled webpage with PHP is fairly simple. The most common use of PHP coding is probably as a replacement for CGI scripts.

    Tags: builder, php, php72, php-72

    * A source build using source code from https://github.com/AlexonOliveiraRH/openshift-php-upload-demo.git will be created
      * The resulting image will be pushed to image stream tag "file-uploader:latest"
      * Use 'oc start-build' to trigger a new build

--> Creating resources ...
   
imagestream.image.openshift.io "file-uploader" created
   
buildconfig.build.openshift.io "file-uploader" created
    deployment.apps "file-uploader" created
    service "file-uploader" created
--> Success
    Build scheduled, use 'oc logs -f buildconfig/file-uploader' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/file-uploader' 
    Run' oc status' to view your app.

[alexon@bastion ~]$ oc expose svc/file-uploader -n ocs-file-app

route.route.openshift.io/file-uploader exposed

[alexon@bastion ~]$ oc scale --replicas=3 deploy/file-uploader -n ocs-file-app

deployment.apps/file-uploader scaled

[alexon@bastion ~]$ oc get pods -n ocs-file-app

NAME READY STATUS     
RESTARTS AGE
file-uploader-1-build 0/1 Completed  
0 79s
file-uploader-764468fb46-6vlxg 1/1    
Running 0 11s
file-uploader-764468fb46-cr4mc 1/1    
Running 0 7s
file-uploader-764468fb46-vtsq5 1/1    
Running 0 15s

This new application uses the ocs-storagecluster-cephfs SC:

[alexon@bastion ~]$ oc set volume deploy/file-uploader --add --name=ocs-file-app \
> -t pvc --claim-mode=ReadWriteMany --claim-size=1Gi \
> --claim-name=ocs-file-app --claim-class=ocs-storagecluster-cephfs \
> --mount-path=/opt/app-root/src/uploaded \
> -n ocs-file-app

deployment.apps/file-uploader volume updated
NAME READY STATUS     
RESTARTS AGE
file-uploader-1-build 0/1 Completed  
0 2m7s
file-uploader-69b547dfd6-gvhfk 1/1    
Running 0 33s
file-uploader-69b547dfd6-nzhl8 1/1    
Running 0 26s
file-uploader-69b547dfd6-qbj28 1/1    
Running 0 15s

[alexon@bastion ~]$ oc get pvc

NAME          
STATUS VOLUME                                    
CAPACITY ACCESS MODES STORAGECLASS AGE
ocs-file-app  
Bound   
pvc-73c1bda0-2256-407d-885d-e5bcfd221b27 1Gi       
RWX           
ocs-storagecluster-cephfs 44m

As a simple test, create a file named testfile.txt with the "Hello world!" content and upload it to your application through the application's UI:

[alexon@bastion ~]$ oc get route file-uploader -n ocs-file-app -o jsonpath --template="http://{.spec.host}{'\n'}"

http://file-uploader-ocs-file-app.apps.example.com

[alolivei@alolivei ~]$ echo 'Hello world!' >> testfile.txt

[alolivei@alolivei ~]$ cat testfile.txt 

Hello world!

Upload the file:

Image

And here's the result:

Image

To map a file object, the procedure is a bit different from the block object. That's because CephFS uses the inode address of the file converted to hexadecimal to name it within the file pool. Therefore, it's necessary to find the object using one of the application's pods in the directory mounted inside the pod by CephFS, find out its inode, and then convert the inode number to hexadecimal, as in the example below:

[alexon@bastion ~]$ oc get pods

NAME READY STATUS RESTARTS  
AGE
file-uploader-1-build 0/1 Completed  
0 8m32s
file-uploader-69b547dfd6-gvhfk 1/1    
Running 0 6m58s
file-uploader-69b547dfd6-nzhl8 1/1    
Running 0 6m51s
file-uploader-69b547dfd6-qbj28 1/1    
Running 0 6m40s

[alexon@bastion ~]$ oc rsh file-uploader-69b547dfd6-gvhfk

sh-4.4$ mount | grep csi-vol

172.30.38.159:6789,172.30.136.12:6789,172.30.73.120:6789:/volumes/csi/csi-vol-8a803889-bcce-11eb-8d22-0a580a81023e/3910620a-e68c-424b-8982-8f2b21c26a8a on /opt/app-root/src/uploaded type ceph (rw,relatime,seclabel,name=csi-cephfs-node,secret=<hidden>,acl,mds_namespace=ocs-storagecluster-cephfilesystem)

sh-4.4$ ls /opt/app-root/src/uploaded

testfile.txt

sh-4.4$ stat -c %i /opt/app-root/src/uploaded/testfile.txt | xargs printf '%x\n'

1000000001a

Now that you have this information, follow the same steps as before, with some minor differences, to find the object inside the ocs-storagecluster-cephfilesystem-data0 pool and also the node linked to it, as follows:

sh-4.4$ ceph df

RAW STORAGE:
    CLASS SIZE       
AVAIL USED RAW USED %RAW USED 
    ssd 1.5 TiB 1.2 TiB    
253 GiB 256 GiB 16.68 
    TOTAL 1.5 TiB    
1.2 TiB 253 GiB 256 GiB 16.68 
 
POOLS:
    POOL                                          
ID STORED OBJECTS USED       
%USED MAX AVAIL 
   
ocs-storagecluster-cephblockpool 1 84 GiB 22.45k    
253 GiB 19.43 350 GiB 
   
ocs-storagecluster-cephfilesystem-metadata 2    
1.5 MiB 26 4.5 MiB 0      
350 GiB 
   
ocs-storagecluster-cephfilesystem-data0 3      
171 B 2 24 KiB 0      
350 GiB 

sh-4.4$ rados -p ocs-storagecluster-cephfilesystem-data0 ls | grep 1000000001a

1000000001a.00000000

sh-4.4$ rados -p ocs-storagecluster-cephfilesystem-data0 stat2 1000000001a.00000000

ocs-storagecluster-cephfilesystem-data0/1000000001a.00000000 mtime 2021-05-24T20:33:51.179464+0000, size 13

It looks like you found your object. A simple and quick way to validate this is to export it from within the pool and check its contents:

sh-4.4$ rados -p ocs-storagecluster-cephfilesystem-data0 get 1000000001a.00000000 

/tmp/output.txt

sh-4.4$ cat /tmp/output.txt 

Hello world!

Now finish the process and map which node the file is in:

sh-4.4$ ceph osd map ocs-storagecluster-cephfilesystem-data0 1000000001a.00000000

osdmap e405 pool 'ocs-storagecluster-cephfilesystem-data0' (3) object '1000000001a.00000000' -> pg 3.a8154e0 (3.0) -> up ([2,1,0], p2) acting ([2,1,0], p2)

sh-4.4$ ceph osd status

+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id |            
host | 
used | avail | wr ops | wr data | rd ops | rd data | state  
|
+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | ip-10-0-171-63.ec2.internal | 85.4G | 426G | 86 | 1221k 
| 0 |    
0 | exists,up |
| 1 | ip-10-0-143-192.ec2.internal | 85.4G | 
426G | 78 | 1678k |   
0 | 0  
| exists,up |
| 2 | ip-10-0-154-20.ec2.internal | 85.4G | 426G | 67 | 643k 
| 2 |  
106 | exists,up |
+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+

sh-4.4$ ceph osd tree

ID CLASS WEIGHT TYPE NAME                                           
STATUS REWEIGHT PRI-AFF 
 -1 1.50000 root default                                                                
 -5 1.50000 region us-east-1                                                         
 -4 0.50000 zone us-east-1a                                                     
 -3 0.50000 host ocs-deviceset-gp2-csi-1-data-085b8h                         
  1 ssd 0.50000 osd.1 up 1.00000 1.00000 
-10      
0.50000 zone us-east-1b                                                     
 -9 0.50000 host ocs-deviceset-gp2-csi-2-data-0n9lkb                         
  2 ssd 0.50000 osd.2 up 1.00000 1.00000 
-14      
0.50000 zone us-east-1c                                                     
-13      
0.50000 host ocs-deviceset-gp2-csi-0-data-0gvt22                         
  0 ssd 0.50000 osd.0 up 1.00000 1.00000 

Again, you know that the object is in a PG with OSD ID 2 as its primary, with its replicas in OSDs ID 1 and 0, that this OSD is using the device host ocs-deviceset-gp2-csi-2-data-0n9lkb on the host ip-10-0-154-20.ec2.internal. If you want to be sure that this is the correct node, check the device with lsblk or dmsetup:

[alexon@bastion ~]$ oc debug node/ip-10-0-154-20.ec2.internal
Starting pod/ip-10-0-154-20ec2internal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.154.20
If you don't see a command prompt, try pressing enter.

sh-4.4# lsblk | grep ocs-deviceset-gp2-csi-2-data-0n9lkb
`-ocs-deviceset-gp2-csi-2-data-0n9lkb-block-dmcrypt 253:0 0 512G 
0 crypt 

sh-4.4# dmsetup ls | grep ocs-deviceset-gp2-csi-2-data-0n9lkb
ocs-deviceset-gp2-csi-2-data-0n9lkb-block-dmcrypt (253:0)

Using the name of the PV used by the PVC of the application you saw earlier, grep and see where the paths available for the application are mounted:

sh-4.4# mount | grep pvc-73c1bda0-2256-407d-885d-e5bcfd221b27

172.30.38.159:6789,172.30.136.12:6789,172.30.73.120:6789:/volumes/csi/csi-vol-8a803889-bcce-11eb-8d22-0a580a81023e/3910620a-e68c-424b-8982-8f2b21c26a8a on /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/globalmount type ceph (rw,relatime,seclabel,name=csi-cephfs-node,secret=<hidden>,acl,mds_namespace=ocs-storagecluster-cephfilesystem)
172.30.38.159:6789,172.30.136.12:6789,172.30.73.120:6789:/volumes/csi/csi-vol-8a803889-bcce-11eb-8d22-0a580a81023e/3910620a-e68c-424b-8982-8f2b21c26a8a on /host/var/lib/kubelet/pods/ac40b1fa-a08d-46b5-8bb6-dc55a5638e9e/volumes/kubernetes.io~csi/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/mount type ceph (rw,relatime,seclabel,name=csi-cephfs-node,secret=<hidden>,acl,mds_namespace=ocs-storagecluster-cephfilesystem)

Finally, see the contents of the directories, and you'll find your object inside, ready to be visualized:

sh-4.4# ls /host/var/lib/kubelet/pods/ac40b1fa-a08d-46b5-8bb6-dc55a5638e9e/volumes/kubernetes.io~csi/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/mount

testfile.txt

sh-4.4# cat /host/var/lib/kubelet/pods/ac40b1fa-a08d-46b5-8bb6-dc55a5638e9e/volumes/kubernetes.io~csi/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/mount/estfile.txt 

Hello world!
sh-4.4# ls /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/globalmount

testfile.txt

sh-4.4# cat /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-73c1bda0-2256-407d-885d-e5bcfd221b27/globalmount/testfile.txt 

Hello world!

You now know how to both find and map file object information in the cluster, much like you saw in article two with block information.

[ Get this free book from Red Hat and O'Reilly - Kubernetes Operators: Automating the Container Orchestration Platform. ]

Wrap up

The ODF approach to distributed and scalable storage is very different from other storage solutions. It can be difficult to understand where application objects are stored within the cluster, which presents challenges when troubleshooting (and designing).

This article series is a basic demonstration of how ODF application object mapping works. In part one, you established the environment and necessary utilities. Part two covered block storage while part three examined file storage structures. Now that you know how to do it put it to good use.