End-to-end Performance Visualization in Ceph

Background

Ceph is a software-defined storage solution that provides object storage, block storage and a filesystem. It was designed to provide excellent performance, reliability and scalability.

Ceph participated in Google Summer of Code 2016 as a mentoring organization. One of the accepted projects was this one, related to performance visualization of Ceph's components using the LTTng-based blkin library and Twitter Zipkin.

Description

Ceph provides the librbd library to interact with block storage, which uses librados to interact with the storage cluster. Code in librbd and librados needs to be instrumented with blkin to record events of interest to developers, using timestamp and key-value annotations.

This allows the visualization of events in a timeline for a specific request, along with key-value pairs that can provide additional context.

This project builds on previous tracing efforts that use blkin, to extend it to librbd and to fio rbd engine where trace information is initialized and passed to librbd. This project also puts up to date the tools to get the collected traces and send them to a Zipkin server.

Events can be traced and visualized in Zipkin, including timestamps and key-value annotations. The screenshot shows events recorded in OSDs, PGs, the filejournal while performing an aio_write started from fio rbd engine.

Work done

1. Ceph: Trace information was passed along from librbd to librados for aio reads and writes.

   • Add rbd_aio_write_traced and rbd_aio_read_traced functions to librbd that take trace information as one of the parameters.
   • Extend the librados API to take trace information from librbd
   • See commits here,here and the pull request.

2. Babeltrace-zipkin: The previous babeltrace plugins needed to be updated to work with recent versions of Zipkin.

   • Use thriftpy for compatibility with Python3.
   • Fix the Thrift specification used to communicate with Zipkin.
   • Provide scripts to install dependencies with support for Ubuntu and Fedora.
   • See Github commits

3. fio rbd engine: Support was added to initialize trace information and pass it to librbd from fio rbd engine.

   • Add configure support to disable or enable fio rbd engine with blkin.
   • Use rbd_aio_write_traced and rbd_aio_read_traced to pass the trace information to librbd.
   • See Commit

4. blkin: Some minor features were added to blkin.

   • Support for core annotations was added.
   • Initialization for trace information structs (i.e. to initialize trace information in fio rbd engine).
   • See pull requests 1, 2 and 3

Getting started

To see traces initialized from fio rbd engine:

1. Install Zipkin, LTTng, babeltrace-zipkin and related dependencies. You might use one of the provided scripts or refer to this documentation.
2. Build and install Ceph with blkin support (pass -DWITH_BLKIN=ON to cmake configuration).
3. Start a Ceph cluster. You might need to set LD_PRELOAD=liblttng-ust-fork.so
4. Get fio and configure (LDFLAGS="-Lpath/to/build/lib -Wl,-rpath,path/to/build/lib" ./configure --extra-cflags="-I/usr/local/include/") and build it with rbd support. If the blkin dependencies and the right version of librbd is installed in your system during the configure step you should see that blkin rbd tracing is enabled.
5. Make sure Zipkin is running.
6. Enable zipkin events and start an LTTng session lttng create && lttng enable-event zipkin:timestamp && lttng start.
7. Run fio providing the location of your cluster's configuration file in the CEPH_CONF variable. You might need to set LD_LIBRARY_PATH to the location of the blkin libraries.
8. Stop the tracing session and use babeltrace-zipkin to send the traces to Zipkin python3 babeltrace_zipkin.py location/to/traces/ust/uid/1000/64-bit/ -s serverIp -p zipkinPort

This script can be modified to repeat runs after dependencies have been set up correctly.

Authors and Contributors

Victor Araujo (@vears91).

Ali Maredia (@alimaredia) was the mentor from Red Hat that supervised the project. Thank you very much for your support. Special thanks too for the valuable help of Casey Bodley (@cbodley) and Jason Dillaman (@dillaman).

Work to do

• Some work begun on instrumentation for RGW. However, there is a bug with LTTng that causes errors in PUT container operations.

• Due to the high number of I/O requests that can happen, sampling might be of interest.

• More events and key values need to be registered to improve the quality of the information collected by this type of tracing.

• Building the blkin submodule has to be fixed, as linking can cause an issue with duplicate probes

Support or Contact

For any question feel free to contact Victor Araujo at ve.ar91@gmail.com.