Translator 101 Lesson 3: This Time For Real

In the first two parts of this series, we learned how to write a basic translator skeleton that can get through loading, initialization, and option processing. This time we’ll cover how to build that translator, configure a volume to use it, and run the glusterfs daemon in debug mode.

Unfortunately, there’s not much direct support for writing new translators. You can check out a GlusterFS tree and splice in your own translator directory, but that’s a bit painful because you’ll have to update multiple makefiles plus a bunch of autoconf garbage. As part of the HekaFS project, I basically reverse engineered the truly necessary parts of the translator-building process and then pestered one of the Fedora glusterfs package maintainers (thanks daMaestro!) to add a glusterfs-devel package with the required headers. Since then the complexity level in the HekaFS tree has crept back up a bit, but I still remember the simple method and still consider it the easiest way to get started on a new translator. For the sake of those not using Fedora, I’m going to describe a method that doesn’t depend on that header package. What it does depend on is a GlusterFS source tree, much as you might have cloned from GitHub or the Gluster review site. This tree doesn’t have to be fully built, but you do need to run autogen.sh and configure in it. Then you can take the following simple makefile and put it in a directory with your actual source.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# Change these to match your source code.
TARGET  = rot-13.so
OBJECTS = rot-13.o
 
# Change these to match your environment.
GLFS_SRC = /play/glusterfs
GLFS_LIB = /opt/glusterfs/3git/lib64
HOST_OS  = GF_LINUX_HOST_OS
 
# You shouldn't need to change anything below here.
 
CFLAGS  = -fPIC -Wall -O0 -g \
          -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D$(HOST_OS) \
          -I$(GLFS_SRC) -I$(GLFS_SRC)/libglusterfs/src \
          -I$(GLFS_SRC)/contrib/uuid
LDFLAGS = -shared -nostartfiles -L$(GLFS_LIB) -lglusterfs -lpthread
 
$(TARGET): $(OBJECTS)
        $(CC) $(OBJECTS) $(LDFLAGS) -o $(TARGET)

Yes, it’s still Linux-specific. Mea culpa. As you can see, we’re sticking with the rot-13 example, so you can just copy the files from …/xlators/encryption/rot-13/src in your GlusterFS tree to follow on. Type “make” and you should be rewarded with a nice little .so file.

1
2

[jeff@gfs-i8c-01 xlator_example]$ ls -l rot-13.so
-rwxr-xr-x. 1 jeff jeff 40784 Nov 16 16:41 rot-13.so

Notice that we’ve built with optimization level zero and debugging symbols included, which would not typically be the case for a packaged version of GlusterFS. Let’s put our version of rot-13.so into a slightly different file on our system, so that it doesn’t stomp on the installed version (not that you’d ever want to use that anyway).

1
2
3

[root@gfs-i8c-01 xlator_example]# ls /opt/glusterfs/3git/lib64/glusterfs/3git/xlator/encryption/
crypt.so  crypt.so.0  crypt.so.0.0.0  rot-13.so  rot-13.so.0  rot-13.so.0.0.0
[root@gfs-i8c-01 xlator_example]# cp rot-13.so /opt/glusterfs/3git/lib64/glusterfs/3git/xlator/encryption/my-rot-13.so

These paths represent the current Gluster filesystem layout, which is likely to be deprecated in favor of the Fedora layout; your paths may vary. At this point we’re ready to configure a volume using our new translator. To do that, I’m going to suggest something that’s strongly discouraged except during development (the Gluster guys are going to hate me for this): write our own volfile. Here’s just about the simplest volfile you’ll ever see.

1
2
3
4
5
6
7
8
9

volume my-posix
    type storage/posix
    option directory /play/export
end-volume
 
volume my-rot13
    type encryption/my-rot-13
    subvolumes my-posix
end-volume

All we have here is a basic brick using /play/export for its data, and then an instance of our translator layered on top – no client or server is necessary for what we’re doing, and the system will automatically push a mount/fuse translator on top if there’s no server translator. To try this out, all we need is the following command (assuming the directories involved already exist).

1

[jeff@gfs-i8c-01 xlator_example]$ glusterfs --debug -f my.vol /play/import

You should be rewarded with a whole lot of log output, including the text of the volfile (this is very useful for debugging problems in the field). If you go to another window on the same machine, you can see that you have a new filesystem mounted.

1
2
3
4

[jeff@gfs-i8c-01 ~]$ df /play/import
Filesystem           1K-blocks      Used Available Use% Mounted on
/play/xlator_example/my.vol
                     114506240   2706176 105983488   3% /play/import

Just for fun, write something into a file in /play/import, then look at the corresponding file in /play/export to see it all rot-13′ed for you.

1
2
3
4

 
[jeff@gfs-i8c-01 ~]$ echo hello > /play/import/a_file
[jeff@gfs-i8c-01 ~]$ cat /play/export/a_file
uryyb

There you have it – functionality you control, implemented easily, layered on top of local storage. Now you could start adding functionality – real encryption, perhaps – and inevitably having to debug it. You could do that the old-school way, with gf_log (preferred) or even plain old printf, or you could run daemons under gdb instead. Alternatively, you could wait for the next Translator 101 post, where we’ll be doing exactly that.