I. Introduction

Building a library is good, but you do not know if there are any bugs until you exercice the library by linking it against an executable and running it. While building webrtc (and lib jingle) has been made relatively easy thanks to depot_tools and recipes, testing it has been a real problem. Since in the google system, commits are rolled, tested, then potentially unrolled, you cannot trust the HEAD revision to be stable. Using branch_heads you could compile and test the versions corresponding to a specific release of Chrome, however testing newer libwebrtc / libjingle like those that are used in canary is something wants to do to be ready when it moves to production, but it’s very challenging. This post is going to address two points: the testing system in google (for the curious), and how to achieve almost the same level of testing in a few lines of CMake code.

II. I want my own waterfall !

The official page describe how to see the tests being run, and how to add them, but not how to trigger them, or how to run them locally. It’s not that you can’t, it’s just that it’s quite complicated and demand quite a machinery in place. What one really want would be at minimum to have the capacity to test locally the result of the build before installing it or packaging it (more on that in a later post), and in the best case scenario to have the equivalent of a waterfall that could be run on demand. First, for those in a hurry, there is almost no way to do it yourself, as the infrastructure is own by google and require you to have a committer account with either webrtc or chrome. I do not know if it possible, but i can say that I do not know of anybody in the ecosystem (appart from one intel employee) that enjoy this privilege. For those that are curious or have time, see below what you can do to have the closest equivalent.

1. The missing Testing info

This part will deal with all the info missing from the webrtc.org page (but spread around in different chromium wiki pages) to actually set up testing the google way.

The system used by google for the development of chromium is great. Honestly, it makes a lot of sense, and a lot of effort has been put in it to actually scale to the number of developers working on chromium. The latest “infra” as they call it, has to handle more than 200 commits a day without disruption, and is apparently doing a great job at it. That being said, it is quite an overkill for someone who only wants to work on libwebrtc, and it’s unfortunate that smaller team are being forced into using it or surrender testing all together.

The original test part of the infra is made of several parts: Gtest to write the unit tests, the usual tools to integrate it in the build process (gyp, DEPS, …), buildbot for … well, the build bots, and a new special swarming infrastructure (here and here) coupled with isolated tests (here and here) to even scale better. I will not address the GTest, gyp or DEPS parts here, but deal with what need to be in place to test after a successful build of libwebrtc. To be thorough, i will only touch on the standalone libwebrtc, and not the webrtc-in-chromium and other embedded tests (*it s actually quite different, especially for the renderers, screensharing etc, but hey, we have to take it step by step). Finally, I will only build the webrtc.org version and not the gecko/firefox version (here again, quite different, love the screensharing in FF, and H264 integration, but we have to take it step by step).

The link below will take you step by step through everything you need to set up your own buildbot (with slaves), trybot and other locally. Note that the links to the repositories might be obsolete by the time you read this post (FAST moving target …) and you might want to read the code in the “infra” part of depot_tools on disk to know what’s the latest way of doing things (sic), more specifically, /build which can be directly fetched from https://chromium.googlesource.com/chromium/tools/build.git . The files configuring the standalone webrtc waterfall will appear in master.client.webrtc . Each column of the waterfall correspond to a builedbot (slave) whose information is also in the same folder.

further reading

• https://www.chromium.org/developers/testing/chromium-build-infrastructure

2. Isolated Testing, WTF?

For all the readers working in early stage start up, I want to first make clear that WTF does NOT stand for “Where is the funding”.  After a build of libwebrtc you will find your build folder (/src/out/<CONFIG> by default, <CONFIG> being either Debug or Release) quite crowded. Among all those, you will see some executable with “isolated” and “isolated.state” alongside.  Those files are used by the swarming system to distribute the testing (here and here). In our case, the important information is that they contain the list of the files you need to pass as arguments to the tests to run them. This is still work in progress, but already stable enough to be used. There are 22 tests listed like this. If you then look at the waterfall, you will see that roughly, there are two types of buildbots, the normal ones which run all the unit tests, and the “Large” ones which run performance tests and long tests like [vie|voe]_auto_test, video_capture_tests and the audio_device_tests. Those take quite some time to run, and much longer than the other tests, which justifies running them separately on beefier instances. All the tests need to be run with an extra –test-launcher-bot-mode arguments. Also , for the sake on completeness, both vie_ and voe_auto_test are actually very powerful interactive executable that deserve a post on their own. I encourage everybody to play with it and look at the source code. Inspiring! To be run as a a test, you need to pass them an extra –automated arg.

3. So, how easy it is to support that with CMake?

CMake is part of a trilogy (yeah, another one), CMake/CTest/CDash. CMake is your main configuration and build tool, CTest is, well, your main testing and test suite management tool, but also a client for CDash, the equivalent of the waterfall, i.e. an interactive dashboard to browse an interact with the results of your builds and tests sent by CTest. The advantage here is that it’s all integrated.

NOTE: the best way would be to parse the isolated files, and extract the exact command line, as well as the output files. In our case, for the sake of simplicity, and time, we will manually add the 22 tests one-by-one, and we will ignore the output as long as the test passes. This is thus improvable, but gives you 99% of what you want, in 30mn work ….

to enable testing in CMake, you just need to add the two following lines of code in a CMake script:

1. enable_testing()
2. include(CTest)

In the absence of those two lines, the rest of the code will not crash or raise any error, but no test will be generated, be careful about it.

For each test you want to add, you can use the add_test() command. Here is an example below that handles both windows and mac for adding a normal test, while checking that the name you passed correspond to an existing file (remember, FAST moving target …):

1. set( my_test_binary_name “common_audio_unittests” )
2. set( my_test_binary ${my_test_binary_name} )
3. if( WIN32 )
4.   set( my_test_binary ${my_test_binary}.exe )
5. endif()
6. if( EXISTS ${MY_BUILD_DIR}/${my_test_binary )
7.   add_test(
8.     NAME                                      imp_${my_test_binary_name}
9.     COMMAND                           ${my_test_binary} –test-launcher-bot-mode
10.     WORKING_DIRECTORY  ${MY_BUILD_DIR}
11.   )
12. else()
13.   message( WARNING “${my_test_binary} – NOT FOUND.” )
14. endif()

You can rince and repeat. The final code can be seen here (and the macro add_webrtc_test is defined here).

Once you’re done building the project, you can check which tests have been added by running this command in the build directory “ctest -N“. It will not run the tests, just list them. “ctest” or “make test” (under mac/linux) are equivalent and run all the tests one after the other. For webrtc, it is better not to use the -jX options and run tests in parallel as the tests access the hardware and could interfere with one another. To make sure that people do not make this mistake, you could add a dependency between each test and the previous one in the list. We did not implement that (even though it’s only one more line). If you want to see the output of the tests, you can add a -V to the command line. Here you go, you can test the library as thoroughly as google tests it (well, almost, memory leaks and other thread sanitizer are missing, but hey, it s great already. How to add memory leak checks and coverage computation will be explained in a following posts). Now we can party like it’s 1999. Or can we?

NOTE: webrtc_perf_test needs to access the network, and so is voe_auto_test and thus if you re testing under window, you either have to configure your firewall to allow it, manually click-n-allow when prompted or skip that test if you want the test suite to run.

4. Ok I can run all the test locally, but I still don’t have my waterfall!

True. AT this stage you have enough to build and test the output on a single machine. That a good base for packaging the lib, a topic we will address in a following post. However, you don’t have visibility on the result of the build of the same code base on different systems and compiler, with different compiler options, and you also do not have the nice visual dashboard online.

That is where CDash comes in handy. CDash is the server component of the waterfall, and CTest can be very easily configured to send the result of a build and tests to a CDash server. KitWare, one of the main company behind cmake/ctest/cdash and the working place of some of th most impressive engineers I had to work with in my life, is proposing free hosting for open source project, and reasonably inexpensive hosting options. Of course, the code is free and open source and you can install your own CDash server internally if you prefer. I chose to have my own small account, and here is the very simple content of the CTestConfig.cmake file that you MUST keep in the root of the source directory.

1. set(CTEST_PROJECT_NAME “libwebRTC”)                        # this is linked to a project you must have created in the server beforehand
2. set(CTEST_NIGHTLY_START_TIME “00:00:00 EST”)    # Whatever you want, usually a time at which the computer is not used.
3. set(CTEST_DROP_METHOD “http”)
4. set(CTEST_DROP_SITE “my.cdash.org”)                               # will change depending on your server install
5. set(CTEST_DROP_LOCATION “/submit.php?project=libwebRTC”)
6. set(CTEST_DROP_SITE_CDASH TRUE)

From an empty dir (recommended) you can run ctest -D Experimental to configure, build, run the tests, (optionally check for me leaks, compute coverage, …) and then send the results to the dashboard. For this example, here is what the dashboard looks like (mac, windows). You can see windows and linux build appearing also, since cmake is cross platform it indeed works almost out of the box (except for me command line details and extensions differences) on all platforms. Here each line is a slave (as opposed to a waterfall where a column is a slave), and you can click on the result of each step (configuration, build, test) to get the results as it would have been printed to stdio, giving you approximately the same features as the builedbot waterfall. Eventually for a mature project it can look like this.

CTest also has the capacity to upload files separately. In a following post, I’explain how to use this to automatically drop compiled version of the library or packaged version of those library to a public repository if and only if the tests all passed. Dev can then trust that any binary or package made available is sound, without having to re-compile and re-test themselves (choose your style of victory dance).

further reading

• http://www.cmake.org/Wiki/CMake/Testing_With_CTest
• http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest (Advanced! How to set up automated build instances.)

III. Conclusion

The google system is certainly more scalable. It also allows to trigger builds on demand as part of the validation of commits, which is great. While the CMake/CTest/CDash has been improved in the past 5 years to do this, it is still not a feature that is as easy to deploy (AFAIK, I’m not current on CMake). In the case of a very large project (200 commits a day ….) with only a few companies involved, it’s great.

What is great with the C^3 system, is that anybody can volunteer a build host and contribute to dashboard. You have a fancy configuration (early version of a compiler not out, old version of compiler [borlandCC 5.6], …) that nobody can test on, you can still build and report the errors for the Devs to see. It makes reporting errors as simple as “ctest -D Experimental”. If you configure your computer to be a nightly build host, then dev can hack the code and check if the result solves your problem without having access to the machine at all (yes, you can point the bot to a git branch, not to pollute the main dev or master branches during the trial and error process). So for simpler projects, with a lot of smaller teams contributing (like an open source community around a library ….), it is great. The barrier of entry is low, while being quite powerful if you want it to be. It has been around for years, and used in multiple projects, so it s easy to find people with knowledge of the tools (who is using depot-tools outside of the chromium project … ?). It has also a big code base out there for people to inspire themselves from. It supports ninja….

Here, in a few hours, we have been able to learn how to test the libwebrtc locally to the same level than google does, and to report to an hosted Dashboard that can be shared with others. In following posts, we will see how to add memory leak analysis, code coverage computation (compiling is a good start, testing is better, but how much do we test exactly?), how to create packages to install libwebrtc on computers, and how to then automatically import installed versions in projects that use Cmake to configure their build.

This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.