Why are you so mean? WebRTC NV / ORTC APIs are too hard!

I. Introduction

I saw a lot of reactions to the ORTC announcement by (small) webRTC solution vendors that the new API, wether the webRTC NV or the ORTC one (they share a common inspiration after all) was too complicated. People to start wondering about the reasons why the standard committee / MS was doing that, and why the Australian Government was renaming their research centers “Data61” (here), would that be because area 61 actually exists but is in Australia? Was NSA involved? … and I stop reading the conspiracy theories at that point 🙂

II. Standard Committees: Cathedral or Bazaar?

(For those too young to know which book I’m referring to, I recommend finding yourself a copy and read it.)

The W3C and the IETF are open consortia. Anybody can join and participate. Joining an IETF mailing list actually makes you a member! As far as the W3C is concerned, a membership is involved, but small start up only pay a couple of thousand US Dollars for the first two years. Those two are the two entities involved in the specification of the core of webRTC. There are others that are also worth mentioning like IMTC or 3GPP that are interesting depending on your use of webRTC (interoperability between VoIP and webrtc and Mobile, respectively). In the case of W3C and IETF, the mailing lists are public and not limited to members, so anybody can go there and ask questions, provide feedback, and interact in any way with the members that will eventually make the decision. That feedback from users of the technology is very important for us to make the right decision, and I encourage everybody to go there and exchange. 

III. Use case and feedback

Like any other software, defining a JS API for the browser is about defining the right use case. Whatever you define it will in turn impact the API surface. In the case of webRTC, the original use case is a 1:1 call with audio and video, and that use case was implemented as appRTC. For a very long time appRTC was the reference for bug reports, tests, interoperability between browsers, etc. In turn the Peer Connection API has been tailored to make that use case dead simple. Most of the underlying machinery (all of ICE, the encryption, the codecs, ….)  was hidden within the browsers, and most of the parameters were hidden in SDP. It made writing a webpage that could do a video call a 10 lines homework a student could do.

Use cases evolve as one understanding of a technology improve, as reflected by the corresponding document that is used as an informational reference for both webRTC and RTCweb. Looking at the document tracker, you can see that no less than 16 revisions exist before it was stabilized early this year. If your use case is not in this list, it is very likely that webrtc 1.0 (due sometime around Xmass 2015, if we’re all good boys/girls) will not support it. However, you can voice your need and try to have your use case taken into account for the next version of webRTC (no, not webrtc 1.1, no, not webrtc 2.0, no, not ORTC, just ……. webRTC NV for next version).

Some thought that a 1:1 use case was too simple: peer connection would be too big a black box, and shoehorning all parameters in an SDP blob was just adding complexity and dependencies. Vote happened, decision was made, peerconnection was here to stay. The ones in disagreement created a Community Group, with no standardization power, named ORTC to prepare what could be the base for specifications the day people would want to do things differently, if ever.

IV. WTF happened with all those new API you’re throwing at us?

As usual in the web, people use API in way they were designed for, and it’s awesome. When they do, things break, and/or we get feedback about things that are not working because people assumed it was working in a different way it actually is (or different browsers implement it in a slightly different way). The bug-or-feature discussions happens next, we take note, and put it in the agenda for the next meeting if enough people are interested in it. This time, we were facing very clear and convergent cases.

1:1 is bo~ring!

First, 1:1 is boring and most people are expecting multiparty calls, simulcast or even smarter simulcast using SVC codecs (264, vp9, …). There are slight differences there and the order in which those have been mentioned is not random.

Supporting multiparty calls is having the capacity to have several people join the same conversation, wether in p2p or not. While you can do that with multiple peer connections the underlying assumption is that you want to do it with a single peer connection, to leverage synchronization of streams, common bandwidth adaptation, port optimizations, ….. The problem here is more about how to signal this case between browsers, and gave birth to the infamous Plan B and Unified plan. The former was implemented in chrome for very long, the later is the official spec, but is only fully implemented in Firefox today. Those media streams can be completely independent, i.e. they can come from different sources.

Simulcast is about sending a media stream from a single source using different resolutions. The main usage here is to choose which resolution you are going to use depending on external factors like the resolutions of the remote peer’s screen, the bandwidth of the remote peer, …. While you can implement simulcast using a multiparty implementation as above, you would be losing the information about the relation between the media streams, namely that they all come from the same source, and that one is a decimated/scaled down version of the other. The multiparty implementation would treat all stream equally and in bad network conditions would reduce the resolution of all the streams, killing the purpose. Simulcast usually comes with smart Bandwidth adaptation algorithm that knows he needs to keep the lower resolution stream untouched, and just adapt the highest resolution stream first when bandwidth goes down. Simulcast is most important in use case that involve a media server. In simulcast, the media streams come from the same source, but are independent in the sense that they each can be decoded and rendered/played separately.

SVC codecs allow for yet another level of greatness. SVC will encode the lowest resolution media stream as a normal stream, that can be decoded on its own, and will then encode only the difference between the higher resolution and the base resolution in subsequent streams. The advantage here are multiple: lower bandwidth (low frequency information is not duplicated across streams), better resilience, ….. SVC codecs are especially useful in cases that involve a media server. In this case, the media streams come from the same source, and are NOT independent, except for the lowest resolution stream. The subsequent streams need to have all the lower resolution streams available to be rendered/played.

People are jumping the fence because they have unanswered needs

People are today modifying the SDP on the fly to be able to have access to properties, or capacities of peer-connection internal objects, or to be able to set those properties, or parameters. Several underlying object were modified this way: the ICE agent, the encryption (DTLS), the codec choice, the bandwidth, …

If the use case is valid (and more often than not they are) adding a JS API that  does what people were doing by manipulating the SDP is the right thing to do. We slowly replace an opaque, not specified, API by a specified, JS API with JSON objects. It does not give more work to the developers, since they were doing it already, even though they will have to take the opportunity to refactor and clean their code.

V. Here is why

It so happen that some of the API proposed by the ORTC group would answer both the multiparty/simulcast/SVC problems and the SDP munging problems. They are being slowly integrated in the webRTC specification when and where they make sense (except for Microsoft, which just implements it all his way and dumps it on an unexpecting audience). The time to bring them in webRTC 1.0 specs was shortened by the fact that those had been though about for quite some time now, and overlapping members had worked on both webrtc and ortc and could bridge the gap.

Most of the new API you have seen coming out of the last meeting were APIs that would just provide a good way to achieve what people where trying to achieve by manipulating the SDP, *AND* could be integrated before the end of the year not to push further webRTC 1.0. The other changes are related to paving the way to simulcast, but I already spoke about that in a previous post.

Because the APIs are more granular instead of being tailored for a 1:1 case, it makes writing the 1:1 case with those API look overly complicated in contrast. I do not believe it to be really a problem, as it is always easy to go from granular to simple. Within a few weeks, you will have webrtc-on-ORTC shims, and your website will work exactly the same (as long as you don t need video), or you can keep ignoring Edge all together. There are quite a few things that are overly complicated to do in webRTC today that will be easily doable with the new APIs. No regression in any case, just possible improvements. I expect the same thing to happen for the latest additions to webRTC 1.0 API set. Eventually webRTC and ORTC should also converge. 

I hope that this post brought some light on the decision process followed by W3C. The core of it is feedback from users, and timeline considerations, so once again, if you have a use case, or a question, voice them on the w3c’s public-webrtc mailing list (not the discuss-webrtc mailing list).

 

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

Patching lib webrtc almost automatically

I. Introduction

Most product will want to modify the lib webrtc to add some features. Some, like tokbox, will want to change the libvpx compilation flags to enable VP8 SVC (only temporal scalability) to use between their mobile SDKs. Some others, like voxeet, might want to add additional audio codec. Many, including pristine.io will want to add h264 support.

Whatever the goal, everybody will need to be able to patch the source code in a consistent way against a fast moving library, and keep the number of patch and how to apply them manageable.

Moreover, since this is a public code, there are some patch I will contribute for all, but users might want to have their own private patches and keep them for themselves. The architecture should allow for that.

Finally, just for the fun of it, I wanted to have a way to quickly get patches from the google review process to be able to have in the lib features before they even appear in chrome (or to test those patches with this system).

In this post we will propose one easy, but efficient way to do just that.

II. Implementation

1. Specific CMake Command used here.

CMake has this nice add_subdirectory() command that makes a lot of thing easy. Basically what the command does is just to iterate into the corresponding folder and act on any CMakeLists.txt file that would be present there.

  1. add_subdirectory(Patches)

By making it conditional you can design a nice layout to keep your patches managed, for example by platform:

  1.  if( APPLE )
  2.   add_subdirectory( mac )
  3. elseif( WIN32 )
  4.   add_subdirectory( win )
  5. endif()

2. Patch creation and specific libwebrtc concerns

libwebrtc code source is a patchwork of separated libraries that are being fetch depending on the DEPS file. While gclient as an option to generate a global patch, we preferred simply using git. Then, each patch you generate is for a specific git tree and you have to remember where to apply it.

We created a CMake macro to help automate that:

  1. set_webrtc_patch_target(
  2.   GIT_APPLY_CMD
  3.   APPLY_DIR
  4.   PATCH
  5.   DEPENDS_ON_TARGET
  6. )

The DEPENDS_ON_TARGET, allow us to make sure the patches are applied after the code is downloaded.

The GIT_APPLY_COMMAND allows for flexibility in which git command you use. Some prefer the “git diff” / “git apply” approach, while other prefer “git format-patch” / “git am”. In our case, we keep it simple:

  1. set(
  2.   GIT_APPLY_CMD
  3.   git apply –ignore-space-change –ignore-whitespace
  4. )

3. How to make a clean/undo command

The problem with patches is that they leave the source tree “dirty”, and a good rule for development or even build bots is that the source code should stay clean (unmodified) as much as possible.

When using git, one way to get back to a clean state is to do a reset: “git reset –hard -q”. This would bring all the tracked files to a clean state, but can leave untracked files behind, e.g. if you add new files, or delete others. “git clean -qfdx” if then needed to make sure the source tree is back to where you want it. The code looks like that:

  1. set( GIT_RESET_CMD git reset –hard -q )
  2. set( GIT_CLEAN_CMD git clean -qfdx ) 

Additionally, you need to know where to apply the commands, so you need to keep track of all the directories where a patch has been applied. For each patch, we’re going to add the directory it s applied to in a list. When times come to “unpatch”, we’ll use that list:

  1. list(REMOVE_DUPLICATES PATCHED_DIRS) # remove duplicates
  2. add_custom_target(
  3.   UNPATCH_ALL
  4.   ${CMAKE_COMMAND} -E touch dummy.phony
  5. )
  6. foreach( dir ${PATCHED_DIRS} )
  7.   add_custom_command( 
  8.     TARGET                               UNPATCH_ALL          POST_BUILD
  9.     COMMAND                         ${GIT_RESET_CMD}
  10.     COMMAND                         ${GIT_CLEAN_CMD}
  11.     WORKING_DIRECTORY  ${WebRTC_SOURCE_DIR}/${dir}
  12.     COMMENT                          “Unpatching ${dir}.”  
  13.   )
  14. endforeach()

4. How to integrate my private/proprietary patches?

With the add_subdirectory() command, things are quite simple. The code below checks if there is a “PvtPatches” subdirectory and if there is, walk into it. You can use git subtrees, or the method of your choice to have in your local copy such a directory, with a CMakeFiles copied (hum … largely inspired) from the one in Patches and everything will be good.

  1. if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/PvtPatches)
  2.   add_subdirectory(pvtPatches)
  3. endif()

Note that extra care has been taken about the variable that contains the list of directories to apply the reset and clean command to, so that you can modify it from within subdirectory and it remains valid. The “CACHE” option make sure it s consistent across the entire project whatever the current source directory is. The “INTERNAL” option is here to make sure this variable does not appear in the graphical user interface that goes along with cmake.

  1. set( PATCHED_DIRS “” CACHE INTERNAL “Internal variable.” )

Of course, you’re likely to re-run the build generation tool after applying the patches:

  1. python /src/webrtc/build/gyp_webrtc.py

III. Conclusion

You have seen in this post how to set up a simple patch system for libwebrtc. Of course, it would not be complete without a few examples, so in following post I will show how to integrate google patches:

  • support for openH264 (patch)
  • support for AVFoundation renderer for mac (patch)

The most difficult is not the patching, but the testing. I need to push some examples and stand alone tests first.

Cheers.

Alex.

 

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

Dashboard “Greenness”, one bug at a time

I. Introduction

Precompiled libraries for stable version of webrtc (those used in chrome) have been requested many times on the mailing list, but so far nobody as put him/herself at making them.  One of the goal of this blog is to provide those to lower the barrier of entry for people that want to build on top of webrtc.

As I was preparing the libraries on linux, i bumped again in the failing test I mentioned in a previous post:

The first error seems to be related to a bad allocation. That’s were you realize that running this on the smallest possible linux instance in AWS was possibly a bad idea. It should disappear when I host the Linux built bot on a bigger instance. The second error is more elusive, and I can’t figure it out just from the logs. Once I will have set up a more powerful linux build host, I will debug there directly.

As far as I am concerned, having even a single test failing is a no-no. So I dug deeper. Here is the build before the changes.

II. Investigating

In the mean time, I moved the build bot to a stronger (c3.2x) instance. Indeed, the first error was a memory allocation problem triggered by an undersized instance, and went away without any special attention.

The second error was related to screen sharing tests, which is not a surprise given that we are running on a virtual machine without display.

The original tests are run through a test driver written in python. The code is separated from libwebrtc and can be found there. The main file is here. Here again, it is code coming from chrome which contains a lot of things not needed to test the standalone version of webrtc (chrome sandbox, …).

It also does a lot of nice things in term of checking that no left over from previous, possibly failing, tests are not on the way. There are a lots of extra steps that improve the stability and robustness of the tests, so it’s not all bad.

To make things simple, you just need to install Xvfb and openbox, 

  1. sudo apt-get install xvfb
  2. sudo apt-get install openbox

then define a display, create it, and run the window manager before you run your test (the code below is written to stay as close as possible to google tests conditions).

  1.  export DISPLAY=:9
  2.   Xvfb :9 -screen 0 1024x768x24 -ac -dpi 96&
  3.  openbox&

Now, all tests pass!

III. Conclusion

The greenness of the dashboard is something that is of utmost importance. If the dashboard dis not free  you are developing blindfolded. Making it green is an everyday challenge,. It can be seen as too much to bother about, but it is actually a developer safety net, and allow you to focus on developing only.

The advantages of cmake here are twofold: lower barrier of entry, and collaborative dashboard. 

Once again, one can see that the chrome build tools, however good and advanced, are an overkill in the case of the standalone libwebrtc. I do believe it is slowing down adoption of and contribution to webrtc, as one needs to become a chromium developer first, and the learning curve is steep.

In any case, you can now download tested, precompiled, libraries and headers for linux, mac or windows on the Tool page. If what you want is just to develop something against libwebrtc that work against the latest stable chrome, you have all you need now. 

Some people request features that are not, or not yet, in webrtc. In a following post, I will explain how to patch libwebrtc effortlessly as part of the process described before.

Enjoy.

 

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

How to set up build bots for libwebrtc

I. Introduction

Following My previous posts, I got a lot of e-mails concerning setting up the build bots. I have to admit that my previous post did not address that in detail, and that the documentation about it is sparse and confusing, as the recommended way to do it in the cmake community changed through the 15 years of the (vtk, itk) projects. So here is a post to describe, step by step, how to set up your own bots in a matter of hours, and manage them remotely through git, without ever having to connect to them again (in theory, in practice, s#$%^ happens, and you might also want to connect from time to time to debug problems directly.)

It is good policy to keep the build bot script separated from the main code, as they might contain sensitive information about your infrastructure. For example, you might put access key to upload the result of the build (packaged libraries) and that had better be private. Moreover, with the current setting, anybody that manage to have access to your build script end up being able to run anything on your build bot, which is also something you don’t want 🙂 In our case, it’s more a tutorial, and all scripts are accessible here.

II. Scripting CTest

So far, I touched on two ways of using ctest:

  • as an extension of CMake, to handle test suite directly from within the CMake files.
  • as an CDash client, to run CMake and automatically send the results of the upgrade, configure, build and test steps to a CDash server. 

There is a third way to use ctest: through scripting. You can write a files, using cmake syntax to prepopulate CTEST_<> variables to run ctest in a controlled way. You then call “ctest -S” with your file as argument to run ctest in script mode.

Some very useful variables are defined for you to use, to set CTEST cache, or environment variables, or hardcode compilers, and any given program or cmake variable before hand. That allow for example to run 32 bits and 64 bits builds, debug and release, on a single machine. Another example is to have multiple versions of compilers on a given machine, and use ctest scripts to use a specific one at a time. One of th best script I saw, written by gaethan lehman, was handling different versions of MSVC and Java on windows. Hat off.

One can get more information about CTest capacity on the old page written when people were still using purify (here, here and here),  and a more recent version here, .

III. What about libwebrtc?

For this example, I used the latest method, developed for ITK v4. A very fast overview is here. This version was focussed on git, and implement some nice tricks to handle different branches, which makes setting up bots for development branches easier.

1. A generic script that does all the heavy lifting

The idea is to have a very generic script that handle most of the problems for you, and to leave only a few variables to be defined to the user. I ported the generic script to be usable for libwebrtc: libwebrtc_common.cmake. Unless you’re a purist, I do not recommend modifying it, or even looking at it. It now allows you to define a set of parameters, some of the usual CMAKE or CTEST variables, but some new dashboard_ variables as well, to control your build.

  • dashboard_model = Nightly | Experimental | Continuous
  • dashboard_track = Optional track to submit dashboard to
  • dashboard_loop = Repeat until N seconds have elapsed
  • dashboard_root_name = Change name of “My Tests” directory
  • dashboard_source_name = Name of source directory (libwebrtc)
  • dashboard_binary_name = Name of binary directory (libwebrtc-build)
  • dashboard_data_name = Name of ExternalData store (ExternalData)
  • dashboard_cache = Initial CMakeCache.txt file content
  • dashboard_do_cache = Always write CMakeCache.txt
  • dashboard_do_coverage = True to enable coverage (ex: gcov)
  • dashboard_do_memcheck = True to enable memcheck (ex: valgrind)
  • dashboard_no_clean = True to skip build tree wipeout
  • dashboard_no_update = True to skip source tree update
  • CTEST_UPDATE_COMMAND = path to git command-line client
  • CTEST_BUILD_FLAGS = build tool arguments (ex: -j2)
  • CTEST_BUILD_TARGET = A specific target to be built (instead of all)
  • CTEST_DASHBOARD_ROOT = Where to put source and build trees
  • CTEST_TEST_CTEST = Whether to run long CTestTest* tests
  • CTEST_TEST_TIMEOUT = Per-test timeout length
  • CTEST_COVERAGE_ARGS = ctest_coverage command args
  • CTEST_TEST_ARGS = ctest_test args (ex: PARALLEL_LEVEL 4)
  • CTEST_MEMCHECK_ARGS = ctest_memcheck args (defaults to CTEST_TEST_ARGS)
  • CMAKE_MAKE_PROGRAM = Path to “make” tool to use
  • Options to configure builds from experimental git repository:
  • dashboard_git_url = Custom git clone url
  • dashboard_git_branch = Custom remote branch to track
  • dashboard_git_crlf = Value of core.autocrlf for repository

If you want to extend the capacity of this core script, some hooks are also provided to keep things clean and compartementalized.

  • dashboard_hook_init = End of initialization, before loop
  • dashboard_hook_start = Start of loop body, before ctest_start
  • dashboard_hook_started = After ctest_start
  • dashboard_hook_build = Before ctest_build
  • dashboard_hook_test = Before ctest_test
  • dashboard_hook_coverage = Before ctest_coverage
  • dashboard_hook_memcheck = Before ctest_memcheck
  • dashboard_hook_submit = Before ctest_submit
  • dashboard_hook_end = End of loop body, after ctest_submit

2. A very simple file to define a bot

Eventually, that makes writing a build script very easy indeed:

  1. set(CTEST_SITE “Bill_._Our_fearless_leader” )
  2. set(CTEST_BUILD_NAME “Ubuntu-12.04-32-Deb” )
  3. set(CTEST_BUILD_FLAGS -j8 )
  4. set(CTEST_DASHBOARD_ROOT “/home/ubuntu/Dashboards” )
  5. set(CTEST_TEST_TIMEOUT 1500 )
  6. set(CTEST_BUILD_CONFIGURATION Debug )
  7. set(CTEST_CMAKE_GENERATOR “Unix Makefiles” )
  8. set(dashboard_model Experimental )
  9. include(libwebrtc_common.cmake)

And … voila! you have a linux build bot all set up! Replace ‘Debug’ by ‘Release’, and you have your release build ready. To change from 32 to 64 bits, since we use ninja, you have to set up the right env variable, but it’s not difficult either:

  1. set( CTEST_ENVIRONMENT
  2.   “GYP_DEFINES=’target_arch=x64′” # or ia32 for 32 bits
  3. )

The corresponding file is here.

A word of warning though, installing the dev environment for libwebrtc is hard. First, it will almost only work under ubuntu, second, the environment install scripts provided do not seem to work, so you will end up having to manually install quite a few libs yourself before being able to compile. The good news is, you will only have to do that once. 

3. How to automate it all?

Now you are armed with several files for each build you want to run. You might very well run many build on the same machine, e.g. 32/64, Debug/Release. For Linux machines, you might want to cross-compile the android binaries as well (more on the mobile target in another post). 

However, you still need to have access to the machine, and manually launch

  1. ctest -S My_Build_script.cmake

For it to work.

One way around this is to define a (shell) script that run those commands for you. However, whenever you make a modification to the script, you have to connect to the machine again, and manually update the local script to the new version, grrrrr

That’s where cron (on linux or mac) and Scheduled tasks (on windows) comes in handy, as it can run a command at a given time for a given user. Here you have two schools: the original cmake members designed everything so that people could configure their always-on desktops to be used during sleeping hours. More recent developers will want to set up either a dedicated build bot, or a hosted build bot, and might want to reduce the cost by switching the machine off when the job is done. I will illustrate the later for linux (and mac) and the files for windows will be provided in the github account for those interested. Note that for windows build bot specifically, it has been shown that you’d better reboot the machine once a day in any case if you want it to work ….

Starting devices remotely is easy, all the cloud providers provide command line API, and you can maintain a build master (very tiny instance) whose sole job will be to wake up the bots once  day for them to fetch the latest code, configure, build test, and submit to the dashboard. In AWS EC2, that means playing with IAM, but nothing too hard there, and it’s very well documented. On linux, the cron daemon accept the ‘reboot’ keyword, and will run the corresponding task whenever the device is started.

  1. @reboot /home/ubuntu/Dashboards/Scripts/Bill-runall.sh

Finally, you can just use the shutdown command in your script to stop the instance when they’re done.

  1. shutdown -h now

We’re only left with automated updating of the scripts. 

The trick used by the ITK community is to keep the scripts (and the cron table) in a git repository, and to update this repository first. In this example, you can see from the shell script that we expect a ~/Dashboard/Scripts directory to contain the build scripts, a ~/Dashboard/Logs directory to be present, and that the crontable also get updated on the fly. Now I can just commit to my git repository, and the build bot will auto update itself. Sweet.

  1. # Go to the working directory
  2. cd /home/ubuntu/Dashboards/Scripts
  3. # get the latest scripts
  4. git pull –rebase -q
  5. # update the crontable
  6. crontab ./Bill-crontab
  7. # Run the builds
  8. ctest -S ./Bill-32-Debug.cmake
  9. ctest -S ./Bill-32-Release.cmake
  10. # done, let’s shutdown the instance to avoid paying too much
    sudo shutdown -h now

The full script with some additional features is here.

IV. Conclusion

It should now be pretty clear that setting up a build bot for libwebrtc is actually quite easy. The code provided in github should actually make it even easier. Feel free to set up your own build bot, hopefully with settings that are to yet present in any of the bots contributing to the dashboard today, and contribute to the fun. I should update to a bigger CDash server that will allow for more than 10 builds a day very soon. I would love to see people contributing for arm, android, iOS, …..

If you find this useful, let other know, and nice comments are also appreciated. 😉

Installing libwebrtc

I.Introduction

Once you have compiled a library for development, the can use it directly from the build tree for any other project that depends on it. It is usually not a good idea if you have several members in your dev team, if you use different systems, and/or if you want to distribute your work. That’s where the notion of install and versioning makes sense (1). Packaging (2) then allow you to install on a computer different than the one you built the project on. After you have installed a library (and corresponding headers and other needed files) it would be also nice to be able to import (3) it in a project easily.

The good news is that CMake handles all of that with (again!) a very few lines of code. Using the install() command you can define what to install, where to install it, and to some extend couple files per components for interactive installers. You might also remember that I told you in a previous post that CMake was kind of a trilogy (CMake/CTest/CDash). Well, there is a sequel called CPack. It’s not as good as the first ones (sequels rarely are), but it gets its own cmake variable prefix so I guess it’s cool 🙂 CPack handles the packaging part, which is build on top of the installation part. Now let’s dig in.

II.Installing targets, or files locally.

In our use of CMake, we do not have targets for each library or executable. Moreover, the tests are not relocatable easily, so it’s better not to try. Libraries can be installed in a flat directory structure, but headers need to follow a certain directory structure to be usable, so we will have to follow two different strategies there.Finally,  the install() command has a lot of signature, so let’s focus on those which would be of use for us.

1. Versioning

The versioning follows the CMake convention (to be compatible with the other tools and command) as explained here.

  1. #———————————————————————————————–
  2. # Versioning
  3. set( WEBRTC_MAJOR_VERSION 0 ) # not fully tested yet
  4. set( WEBRTC_MINOR_VERSION 1 )  # really not fully tested, not fully implemented
  5. set( WEBRTC_BUILD_VERSION 1 )    # should be the SVN rev, but it s hard to get it automatically from the git commit msg.
  6. set( WEBRTC_VERSION
  7.   ${WEBRTC_MAJOR_VERSION}.${WEBRTC_MINOR_VERSION}.${WEBRTC_BUILD_VERSION}
  8.   )
  9. set( WEBRTC_API_VERSION
  10.   # This is the ITK/VTK style where SOVERSION is two numbers…
  11.   “${WEBRTC_MAJOR_VERSION}.${WEBRTC_MINOR_VERSION}”
  12.   )
  13. set( WEBRTC_LIBRARY_PROPERTIES ${WEBRTC_LIBRARY_PROPERTIES}
  14.   VERSION       “${WEBRTC_VERSION}”
  15.   SOVERSION “${WEBRTC_API_VERSION}”
  16.   )

further reading:

2. set up destination folders per component types

Here again, nothing fancy, just following the CMake convention so that find_package can be used later (see find_package() documentation about the expected paths ).

  1. # ————————————————————————–
  2. # Configure the export configuration
  3. # WEBRTC_INSTALL_BIN_DIR          – binary dir (executables)
  4. # WEBRTC_INSTALL_LIB_DIR          – library dir (libs)
  5. # WEBRTC_INSTALL_DATA_DIR         – share dir (say, examples, data, etc)
  6. # WEBRTC_INSTALL_INCLUDE_DIR      – include dir (headers)
  7. # WEBRTC_INSTALL_CMAKE_DIR        – cmake files (cmake)
  8. if( NOT WEBRTC_INSTALL_BIN_DIR )
  9.   set( WEBRTC_INSTALL_BIN_DIR “bin” )
  10. endif()
  11. if( NOT WEBRTC_INSTALL_LIB_DIR )
  12.   set( WEBRTC_INSTALL_LIB_DIR “lib” )
  13. endif()
  14. if( NOT WEBRTC_INSTALL_DATA_DIR )
  15.   set( WEBRTC_INSTALL_DATA_DIR “share” )
  16. endif()
  17. if( NOT WEBRTC_INSTALL_INCLUDE_DIR )
  18.   set( WEBRTC_INSTALL_INCLUDE_DIR “include” )
  19. endif( )
  20. if( NOT WEBRTC_INSTALL_CMAKE_DIR )
  21.   set( WEBRTC_INSTALL_CMAKE_DIR “lib” )
  22. endif( )

3. Handle libraries

Just like we did for the tests, we will first need to import all the libraries name from the filesystem before we can do anything. Unlike for the test, where we had to worry about different arguments for each test, all libraries are treated equal so we can automate the process. The file( GLOB_RECURSE ) command does just that. Under mac, all the libs are at the root of the ninja build, but under windows, they are created on their respective subdirectory, so we need to use the GLOB_RECURSE and not just GLOB which would work only for mac.

  1. set(WEBRTC_BUILD_ROOT ${WebRTC_SOURCE_DIR}/src/out/${CMAKE_BUILD_TYPE}) # the CMAKE_BUILD_TYPE variable allow consistency with build target
  2. set(WEBRTC_LIB_EXT a) # the default
  3. if(WIN32)
  4.   set(WEBRTC_LIB_EXT lib) # you’re on windows! you know who you are 🙂
  5. endif()
  6. file( GLOB_RECURSE # under windows, the libs are within the subfolders
  7.   WEBRTC_LIBS           # the output variable
  8.   ${WEBRTC_BUILD_ROOT}/*.${WEBRTC_LIB_EXT}  # the pattern, i.e. all files with the right extension under the build root.
  9.   )

Now, we could directly feed this to the install() command:

  1. foreach( lib ${WEBRTC_LIBS}
  2.   install(
  3.     FILES                  ${lib}
  4.     DESTINATION ${WEBRTC_INSTALL_LIB_DIR}
  5.     COMPONENT   Libraries
  6.     )
  7.  endforeach()

However, we want to remove the libraries that were used for the tests, and we have to prepare a list of libraries to populate a configuration file that will be installed along the libraries, and make it easy to use the installed version. The full version looks like that:

  1. set(WEBRTC_LIBRARIES “”) # prepare the config for the build tree
  2. foreach(lib ${WEBRTC_LIBS})
  3.   string(FIND ${lib} “test” IS_TEST)
  4.   if(IS_TEST EQUAL -1)
  5.     get_filename_component(lib_name ${lib} NAME_WE)
  6.     string(REPLACE “lib” “” lib_target_name ${lib_name})
  7.     set(WEBRTC_LIBRARIES ${WEBRTC_LIBRARIES} ${lib_target_name})
  8.     install(
  9.       FILES       ${WEBRTC_BUILD_ROOT}/${lib}
  10.       DESTINATION ${WEBRTC_INSTALL_LIB_DIR}
  11.       COMPONENT   Libraries
  12.     )
  13.   endif()
  14. endforeach()

4. Handle headers files

The delicate part of handling headers, is that a specific root directory and subdirectories layout is expected by the files including them. The DEPS file give you some hints about which dir you should use for the includes:

  1. # Define rules for which include paths are allowed in our source.
  2. include_rules = [
  3.   # Base is only used to build Android APK tests and may not be referenced by
  4.   # WebRTC production code.
  5.   ‘-base’,
  6.   ‘-chromium’,
  7.   ‘+gflags’,
  8.   ‘+net’,
  9.   ‘+talk’,
  10.   ‘+testing’,
  11.   ‘+third_party’,
  12.   ‘+webrtc’,
  13. ]

Apart from the missing flags, those are all top level directories of the WebRTC source. A quick sanity check (grep -R -h \#include * | sort -u > log) confirms that it seems to be the layout expected by the #include lines.

so for each of /net, /talk, /testing, /third_party, /webrtc we need to walk the subdirectory layout and use it at install time (that’s the main difference with the libraries handling code). That will justify using the RELATIVE option of the file( GLOB_RECURSE ) command.

  1. file(
  2.   GLOB_RECURSE header_files                                      # output variable
  3.   RELATIVE  ${WebRTC_SOURCE_DIR}/src             # the path will be relative to /src/, as expected by the #includes
  4.   FOLLOW_SYMLINKS                                                      # we need to follow the symlinks to chromium subfolders
  5.   ${WebRTC_SOURCE_DIR}/src/net/*.h               
  6.   ${WebRTC_SOURCE_DIR}/src/talk/*.h
  7.   ${WebRTC_SOURCE_DIR}/src/testing/*.h
  8.   ${WebRTC_SOURCE_DIR}/src/third_party/*.h
  9.   ${WebRTC_SOURCE_DIR}/src/webrtc/*.h
  10. )

Now the install command is easy to write.

  1. foreach( f ${header_files} )
  2.   get_filename_component( RELATIVE_PATH ${f} PATH ) # NOTE ALEX: it seems that newer versions of CMake use DIRECTORY instead of PATH …
  3.   install(
  4.     FILES                  ${WebRTC_SOURCE_DIR}/src/${f}
  5.     DESTINATION ${WEBRTC_INSTALL_INCLUDE_DIR}/${RELATIVE_PATH}  # that’s the tricky part here
  6.     COMPONENT   Headers
  7.     )
  8. endforeach()

5. Are we there yet?

YES! We can now install. You have now an install target in  your build system. Under mac, you can simply type “make install”, and under windows, if you used the default (ninja/MSVC) you will have an “INSTALL” target in the list of target in MSVC. It is not build by default, and yo need to trigger the build manually. Administrator rights will surely be needed. By default, everything is installed under /usr/local on mac and unix, and under “Program Files” for windows (with (x86) for the 32b builds).

In a following post, I will show how to package all those files to be installed on a remote computer.

 

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

Showing some Love to my dashboard.

11046789_1010459968998193_7337377444744452326_o

A Dashboard is like a young baby, fragile and in need of love. While for most people the dashboard is not adding value so much as a new feature to the lib itself (like a nice lib installer, more in a future post), for developer it’s the key to success.

Most developers have a bias toward some OS and some dev tools. I, for one, do most of my main coding on mac nowadays. However, you need to know that the code is tested across OS, compilers, options, etc. If your dashboard is well maintained, it removes the need for you to go and test all theses most of the time. So one need to have quite a few things before a Dashboard is usable to its full strength and be your safety net.

  • All the supported OS, compilers, and options need to appear as a separate build on the dashboard. (In our case, we want as many builds as there is on the official waterfall).
  • The code to be tested as much as possible for each platform, so you need to have visibility on the code coverage on each platform (more in a later post). Removing a failing test to make the dashboard appear green and hide an error is not OK.
  • The dashboard needs to stay Green (without any reported error) all the time, so when when a new error is introduced you see it right away, and it’s not hidden within an already failing test.

So yesterday I spent a little bit of time to test my scripts on linux. As expected it worked almost out the box, but the first results showed 9 failing tests! It’s quite a lot. So i went to double check the official waterfall, and indeed there were linux build, and they were all green, so I was doing something wrong. Or was I?

Looking at the waterfall, there are 47 build or try bots reporting. The base matrix to compute the different kind of builds use the following parameters:

  • OS: win, Mac, iOS, Linux, Android,
  • ARCH: 32 / 64
  • BUILD_TYPE: Debug / Release
  • OPTIONS: Normal / Large-tests / GN / Simulator / (other exotic builds)
  • For android: try bot on real devices: Nexus 5, 7.2, 9

I don’t want to reproduce all, at least not for now, but I would like to cover all the desktop OSes, arch, and build_type as a start. Looking at the linux 32 release builds, I realized that it was running less tests than its windows / mac counterpart. So I started by modifying my test scripts not to include those if the platform is linux. Boom, down to only two errors. The first error seems to be related to a bad allocation. That’s were you realize that running this on the smallest possible linux instance in AWS was possibly a bad idea. It should disappear when I host the Linux built bot on a bigger instance. The second error is more elusive, and I can’t figure it out just from the logs. Once I will have set up a more powerful linux build host, I will debug there directly.

Next blogs should be about packaging, then about adding coverage computation (for gcc and clang builds) and memory leak verification (using valgrind). Stay tuned.

 

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

How to test LibWebRTC

I. Introduction

Building a library is good, but you do not know if there are any bugs until you exercice the library by linking it against an executable and running it. While building webrtc (and lib jingle) has been made relatively easy thanks to depot_tools and recipes, testing it has been a real problem. Since in the google system, commits are rolled, tested, then potentially unrolled, you cannot trust the HEAD revision to be stable. Using branch_heads you could compile and test the versions corresponding to a specific release of Chrome, however testing newer libwebrtc / libjingle like those that are used in canary is something wants to do to be ready when it moves to production, but it’s very challenging. This post is going to address two points: the testing system in google (for the curious), and how to achieve almost the same level of testing in a few lines of CMake code.

II. I want my own waterfall !

The official page describe how to see the tests being run, and how to add them, but not how to trigger them, or how to run them locally. It’s not that you can’t, it’s just that it’s quite complicated and demand quite a machinery in place. What one really want would be at minimum to have the capacity to test locally the result of the build before installing it or packaging it (more on that in a later post), and in the best case scenario to have the equivalent of a waterfall that could be run on demand. First, for those in a hurry, there is almost no way to do it yourself, as the infrastructure is own by google and require you to have a committer account with either webrtc or chrome. I do not know if it possible, but i can say that I do not know of anybody in the ecosystem (appart from one intel employee) that enjoy this privilege. For those that are curious or have time, see below what you can do to have the closest equivalent.

1. The missing Testing info

This part will deal with all the info missing from the webrtc.org page (but spread around in different chromium wiki pages) to actually set up testing the google way.

The system used by google for the development of chromium is great. Honestly, it makes a lot of sense, and a lot of effort has been put in it to actually scale to the number of developers working on chromium. The latest “infra” as they call it, has to handle more than 200 commits a day without disruption, and is apparently doing a great job at it. That being said, it is quite an overkill for someone who only wants to work on libwebrtc, and it’s unfortunate that smaller team are being forced into using it or surrender testing all together.

The original test part of the infra is made of several parts: Gtest to write the unit tests, the usual tools to integrate it in the build process (gyp, DEPS, …), buildbot for … well, the build bots, and a new special swarming infrastructure (here and here) coupled with isolated tests (here and here) to even scale better. I will not address the GTest, gyp or DEPS parts here, but deal with what need to be in place to test after a successful build of libwebrtc. To be thorough, i will only touch on the standalone libwebrtc, and not the webrtc-in-chromium and other embedded tests (*it s actually quite different, especially for the renderers, screensharing etc, but hey, we have to take it step by step). Finally, I will only build the webrtc.org version and not the gecko/firefox version (here again, quite different, love the screensharing in FF, and H264 integration, but we have to take it step by step).

The link below will take you step by step through everything you need to set up your own buildbot (with slaves), trybot and other locally. Note that the links to the repositories might be obsolete by the time you read this post (FAST moving target …) and you might want to read the code in the “infra” part of depot_tools on disk to know what’s the latest way of doing things (sic), more specifically, /build which can be directly fetched from https://chromium.googlesource.com/chromium/tools/build.git . The files configuring the standalone webrtc waterfall will appear in master.client.webrtc . Each column of the waterfall correspond to a builedbot (slave) whose information is also in the same folder.

further reading

  • https://www.chromium.org/developers/testing/chromium-build-infrastructure

2. Isolated Testing, WTF?

For all the readers working in early stage start up, I want to first make clear that WTF does NOT stand for “Where is the funding”.  After a build of libwebrtc you will find your build folder (/src/out/<CONFIG> by default, <CONFIG> being either Debug or Release) quite crowded. Among all those, you will see some executable with “isolated” and “isolated.state” alongside.  Those files are used by the swarming system to distribute the testing (here and here). In our case, the important information is that they contain the list of the files you need to pass as arguments to the tests to run them. This is still work in progress, but already stable enough to be used. There are 22 tests listed like this. If you then look at the waterfall, you will see that roughly, there are two types of buildbots, the normal ones which run all the unit tests, and the “Large” ones which run performance tests and long tests like [vie|voe]_auto_test, video_capture_tests and the audio_device_tests. Those take quite some time to run, and much longer than the other tests, which justifies running them separately on beefier instances. All the tests need to be run with an extra –test-launcher-bot-mode arguments. Also , for the sake on completeness, both vie_ and voe_auto_test are actually very powerful interactive executable that deserve a post on their own. I encourage everybody to play with it and look at the source code. Inspiring! To be run as a a test, you need to pass them an extra –automated arg.

3. So, how easy it is to support that with CMake?

CMake is part of a trilogy (yeah, another one), CMake/CTest/CDash. CMake is your main configuration and build tool, CTest is, well, your main testing and test suite management tool, but also a client for CDash, the equivalent of the waterfall, i.e. an interactive dashboard to browse an interact with the results of your builds and tests sent by CTest. The advantage here is that it’s all integrated.

NOTE: the best way would be to parse the isolated files, and extract the exact command line, as well as the output files. In our case, for the sake of simplicity, and time, we will manually add the 22 tests one-by-one, and we will ignore the output as long as the test passes. This is thus improvable, but gives you 99% of what you want, in 30mn work ….

to enable testing in CMake, you just need to add the two following lines of code in a CMake script:

  1. enable_testing()
  2. include(CTest)

In the absence of those two lines, the rest of the code will not crash or raise any error, but no test will be generated, be careful about it.

For each test you want to add, you can use the add_test() command. Here is an example below that handles both windows and mac for adding a normal test, while checking that the name you passed correspond to an existing file (remember, FAST moving target …):

  1. set( my_test_binary_name “common_audio_unittests” )
  2. set( my_test_binary ${my_test_binary_name} )
  3. if( WIN32 )
  4.   set( my_test_binary ${my_test_binary}.exe )
  5. endif()
  6. if( EXISTS ${MY_BUILD_DIR}/${my_test_binary )
  7.   add_test(
  8.     NAME                                      imp_${my_test_binary_name}
  9.     COMMAND                           ${my_test_binary} –test-launcher-bot-mode
  10.     WORKING_DIRECTORY  ${MY_BUILD_DIR}
  11.   )
  12. else()
  13.   message( WARNING “${my_test_binary} – NOT FOUND.” )
  14. endif()

You can rince and repeat. The final code can be seen here (and the macro add_webrtc_test is defined here).

Once you’re done building the project, you can check which tests have been added by running this command in the build directory “ctest -N“. It will not run the tests, just list them. “ctest” or “make test” (under mac/linux) are equivalent and run all the tests one after the other. For webrtc, it is better not to use the -jX options and run tests in parallel as the tests access the hardware and could interfere with one another. To make sure that people do not make this mistake, you could add a dependency between each test and the previous one in the list. We did not implement that (even though it’s only one more line). If you want to see the output of the tests, you can add a -V to the command line. Here you go, you can test the library as thoroughly as google tests it (well, almost, memory leaks and other thread sanitizer are missing, but hey, it s great already. How to add memory leak checks and coverage computation will be explained in a following posts). Now we can party like it’s 1999. Or can we?

NOTE: webrtc_perf_test needs to access the network, and so is voe_auto_test and thus if you re testing under window, you either have to configure your firewall to allow it, manually click-n-allow when prompted or skip that test if you want the test suite to run.

4. Ok I can run all the test locally, but I still don’t have my waterfall!

True. AT this stage you have enough to build and test the output on a single machine. That a good base for packaging the lib, a topic we will address in a following post. However, you don’t have visibility on the result of the build of the same code base on different systems and compiler, with different compiler options, and you also do not have the nice visual dashboard online.

That is where CDash comes in handy. CDash is the server component of the waterfall, and CTest can be very easily configured to send the result of a build and tests to a CDash server. KitWare, one of the main company behind cmake/ctest/cdash and the working place of some of th most impressive engineers I had to work with in my life, is proposing free hosting for open source project, and reasonably inexpensive hosting options. Of course, the code is free and open source and you can install your own CDash server internally if you prefer. I chose to have my own small account, and here is the very simple content of the CTestConfig.cmake file that you MUST keep in the root of the source directory.

  1. set(CTEST_PROJECT_NAME “libwebRTC”)                        # this is linked to a project you must have created in the server beforehand
  2. set(CTEST_NIGHTLY_START_TIME “00:00:00 EST”)    # Whatever you want, usually a time at which the computer is not used.
  3. set(CTEST_DROP_METHOD “http”)
  4. set(CTEST_DROP_SITE “my.cdash.org”)                               # will change depending on your server install
  5. set(CTEST_DROP_LOCATION “/submit.php?project=libwebRTC”)
  6. set(CTEST_DROP_SITE_CDASH TRUE)

From an empty dir (recommended) you can run ctest -D Experimental to configure, build, run the tests, (optionally check for me leaks, compute coverage, …) and then send the results to the dashboard. For this example, here is what the dashboard looks like (mac, windows). You can see windows and linux build appearing also, since cmake is cross platform it indeed works almost out of the box (except for me command line details and extensions differences) on all platforms. Here each line is a slave (as opposed to a waterfall where a column is a slave), and you can click on the result of each step (configuration, build, test) to get the results as it would have been printed to stdio, giving you approximately the same features as the builedbot waterfall. Eventually for a mature project it can look like this.

CTest also has the capacity to upload files separately. In a following post, I’explain how to use this to automatically drop compiled version of the library or packaged version of those library to a public repository if and only if the tests all passed. Dev can then trust that any binary or package made available is sound, without having to re-compile and re-test themselves (choose your style of victory dance).

further reading

  • http://www.cmake.org/Wiki/CMake/Testing_With_CTest
  • http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest (Advanced! How to set up automated build instances.)

III. Conclusion

The google system is certainly more scalable. It also allows to trigger builds on demand as part of the validation of commits, which is great. While the CMake/CTest/CDash has been improved in the past 5 years to do this, it is still not a feature that is as easy to deploy (AFAIK, I’m not current on CMake). In the case of a very large project (200 commits a day ….) with only a few companies involved, it’s great.

What is great with the C^3 system, is that anybody can volunteer a build host and contribute to dashboard. You have a fancy configuration (early version of a compiler not out, old version of compiler [borlandCC 5.6], …) that nobody can test on, you can still build and report the errors for the Devs to see. It makes reporting errors as simple as “ctest -D Experimental”. If you configure your computer to be a nightly build host, then dev can hack the code and check if the result solves your problem without having access to the machine at all (yes, you can point the bot to a git branch, not to pollute the main dev or master branches during the trial and error process). So for simpler projects, with a lot of smaller teams contributing (like an open source community around a library ….), it is great. The barrier of entry is low, while being quite powerful if you want it to be. It has been around for years, and used in multiple projects, so it s easy to find people with knowledge of the tools (who is using depot-tools outside of the chromium project … ?). It has also a big code base out there for people to inspire themselves from. It supports ninja….

Here, in a few hours, we have been able to learn how to test the libwebrtc locally to the same level than google does, and to report to an hosted Dashboard that can be shared with others. In following posts, we will see how to add memory leak analysis, code coverage computation (compiling is a good start, testing is better, but how much do we test exactly?), how to create packages to install libwebrtc on computers, and how to then automatically import installed versions in projects that use Cmake to configure their build.

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

Automating libwebrtc build with CMake

I.Introduction

libwebrtc is the base of many products. It is of course the base of the webrtc JS API implementation in chrome, firefox and opera, but it is also the base of a lot of mobile SDKs and other products out there. Keeping an updated version, based on a stable code base, tested, has been notoriously difficult if you’re not a google employee. This post will be the first one in a serie of post explaining step-by-step how to set up a system that can automatically compile test and package libwebrtc for you, for several platforms. It follows a previous post that explained how the google build system works (depot tools), and will focus more on the CMake part of the automation.

II. There are already (a few) very good resources out there

There is not a “best way” to compile libraries. As long as the library are built, tested, and integrated automatically, the job is done. Specifically for mobile the pristine scripts (see below) are doing a good job at this. I m biased toward CMake since I worked with it for many years, however, there are reasons why i kept working with it for so long. Not having to deal with compiler differences is a joy, not having to deal with OS differences is also a joy. Being able to add test on the fly with simple command, which in turn will also enable sending result to a dashboard is an overjoy, especially when support for valgrind and coverage comes out of the box, is AWESOME, and finally, installing and packaging is equally made easy (yes, cross platform). Following those blog post, you will be able in a few days to set up a state of the art compilation, testing, and packaging system for libwebrtc, where it would take (from my experience) much longer. Of course, if you have an existing testing infrastructure, and CI solution, that might not play along well, but it’s often best to separate building the webRTC libs and building whatever you build around it, be it a webrtc plugin, or a cordova wrapper (see below) or a mobile SDK.

further reading

  • http://tech.pristine.io/build-ios-apprtc/
  • https://github.com/pristineio/webrtc-build-scripts
  • http://tech.pristine.io/automated-webrtc-building/
  • https://github.com/eface2face/cordova-plugin-iosrtc

III. How to best use CMake

CMake is a cross-platform and cross compiler configuration manager, sometimes called a meta-build system. Its main purpose is to find libraries and program for you on the host computer, set up targets (executable, libraries, tests, ..), whatever the operating system and the compiler you use.

At the bare minimum, you can use it like any other script language and try to encapsulate the webrtc build system in system calls. the “execute_process” command would be use for that purpose. It has the disadvantage of doing everything at configuration time, and linearly. A second call to the script would re-run everything, every time. It is preferable to set up targets and dependencies at configuration time (the only time you would directly run cmake) and then to use the native compiler to handle the rest, in a resumable fashion.

A full usage of cmake would be to use it to replace entirely webrtc build system, and redefine all libraries and executable, the corresponding source files, compilation options, etc. While this might be tractable for single executable, or libraries, doing it for all of webrtc would mean migrating all the .gyp[i] files and keeping them synchrone at every update. Anybody that look at the mess the 6,000 lines src/build/common.gypi is, and still be sane, would shy away from this.

In this case we are going to take the “super build” approach, and let cmake drive the google build system, import and run the tests, add coverage support, prepare packages and so on and so forth. It allows to use the google build system untouched for the basics, and to use CMake where it’s best.

IV. Let’s do it

1. A new CMake module to detect depot tools

CMake has a collection of modules installed alongside to make detecting and using your usual libraries and packages easier, cross platform. Most of those modules are made of CMake code in a file named FindXXX.cmake. From your code you can use it by invoking the “find_package( XXX )“. If the package is mandatory, you can tell find_package to fail right away by adding the REQUIRED argument. For example, CMake comes with a FindGit.cmake pre-installed. In our code we can simply write this:

  1. find_package(Git REQUIRED)

Now, depot tools is not yet recognize by vanilla CMake. However CMake has a mechanism for projects to extend its capacity. You can write your own FindXXXXX.cmake modules and let CMake know about them by setting the CMAKE_MODULE_PATH variable in your code. You should deb careful to extend and not overwrite the variable, in case your code is called from another CMake code (super build style). In our case, we put the scripts in a “Cmake” directory right under the root directory.

  1. set( CMAKE_MODULE_PATH
  2.   ${CMAKE_MODULE_PATH} # for integration in superbuilds
  3.   ${CMAKE_CURRENT_SOURCE_DIR}/Cmake
  4.   )

You can see the corresponding full code here.

Now we still need to code the FindDepotTools.cmake. CMake provides all the primitives to do this. In our case, we just find client and we’re good. The rest is handled by CMake for us, including OS and Paths differences.

  1. find_program(DEPOTTOOLS_GCLIENT_EXECUTABLE
  2.   NAMES gclient gclient.bat # Hints about the name of the exe. client.bat is the windows version.
  3. # below this line is standard CMake stuff for all packages to handle QUIET, REQUIRED, ….
  4. include(${CMAKE_ROOT}/Modules/FindPackageHandleStandardArgs.cmake)
  5. find_package_handle_standard_args(DepotTools
  6.   REQUIRED_VARS DEPOTTOOLS_GCLIENT_EXECUTABLE
  7.   FAIL_MESSAGE “Could not find the gclient executable.”
  8.   )

Now our main script core is relatively easy to write.

  1. project(webrtc)
  2. set(CMAKE_MODULE_PATH
  3.   ${CMAKE_MODULE_PATH} # for integration in superbuilds
  4.   ${CMAKE_CURRENT_SOURCE_DIR}/Cmake
  5.   )
  6. find_package(DepotTools REQUIRED)
  7. find_package(Git               REQUIRED)

You can see the corresponding full code here.

2. Implement “gclient config” as a CMake custom command and custom target.

Now, we want to run the “client config” command line from CMake. This command line will create a .gclient file which should not exist beforehand. If the file exist, the command should be a noop. Moreover, I want the build system to try every time it is run. I want it work both across Operating systems.

let’s first define what the command should be, in a cross-platform manner.

  1. set( gclient_config
  2.   ${DEPOTTOOLS_GCLIENT_EXECUTABLE} config # gclient has been found before automatically
  3.   –name src
  4.   https://chromium.googlesource.com/external/webrtc.git
  5.   )
  6.   if(WIN32)
  7.     set(gclient_config cmd /c ${gclient_config}) # Windows command prompt syntax
  8.   endif()

The first constraints are met with CMake’s add_custom_command(). It allows to create a CMake command that will generate a file as output and will not be triggered if the file already exist.

  1. add_custom_command(
  2.      OUTPUT   ${CMAKE_SOURCE_DIR}/.gclient
  3.      COMMAND  ${gclient_config}
  4.    )

However, nothing tells you in the script when to run this custom command. That’s where the add_custom_target() comes in the picture. It creates a (potentially empty) target for the chosen build system, to which commands can attach themselves. Also, it helps defining dependency with other targets to define build/execution order. In the case of client config, there is no previous step, so there is no dependency. By making the target depend on the file generated by the command you automatically link them. The ALL parameter always include this target in the build.

  1. add_custom_target(
  2.      webrtc_configuration ALL
  3.      DEPENDS ${CMAKE_SOURCE_DIR}/.gclient
  4.    )

You can see the corresponding full code here.

3. Implement “gclient sync” and add dependency

Implementing “gclient sync” is more or less the same: define the command, add a custom command, then add a custom target. This time though, we want this target to only execute AFTER “client config”. So we need to add a dependency. The code is actually self explanatory. Note that we force sync NOT to run the hooks (-n option).

  1. set(gclient_sync ${DEPOTTOOLS_GCLIENT_EXECUTABLE} sync -n -D -r 88a4298)
  2.    if(WIN32)
  3.      set(gclient_sync cmd /c ${gclient_sync})
  4.    endif()
  5.   add_custom_command(
  6.      OUTPUT  ${CMAKE_SOURCE_DIR}/src/all.gyp 
  7.      COMMAND ${gclient_sync}
  8.    )
  9.   add_custom_target(
  10.      webrtc_synchronization ALL
  11.      DEPENDS ${CMAKE_SOURCE_DIR}/src/all.gyp
  12.    )
  13.    add_dependencies( webrtc_synchronization webrtc_configuration )

4. Implement “gclient runhooks”

Gclient runhooks will generate the build files, wo we need to be sure that the most obvious parameters are set beforehand. Ninja handles Release and Debug at build time, so no need to take care of that now, but if you had chosen another type of compiler, you might need to do it now. the switch between 32 and 64 bits architectures (or arm flavors for mobile) are handled through environment variables. CMake has a proxy function to access those:

  1. if(DEFINED ENV{GYP_DEFINES})
  2.   message( WARNING “GYP_DEFINES is already set to ENV{GYP_DEFINES}”.
  3. else()
  4.   if(APPLE)
  5.     set(ENV{GYP_DEFINES} “target_arch=x64”)
  6.   else()
  7.     set(ENV{GYP_DEFINES} “target_arch=ia32”)
  8.   endif()
  9. endif()

Then the rest of the code is pretty much the same as before:

  1.  set(gclient_runhooks ${DEPOTTOOLS_GCLIENT_EXECUTABLE} runhooks)
  2.    if(WIN32)
  3.      set(gclient_runhooks cmd /c ${gclient_runhooks})
  4.    endif()
  5.    add_custom_command(
  6.      OUTPUT  ${CMAKE_SOURCE_DIR}/src/out # it’s arbitrary.
  7.      COMMAND ${gclient_runhooks}
  8.    )
  9.    add_custom_target(
  10.      webrtc_runhooks ALL
  11.      DEPENDS ${CMAKE_SOURCE_DIR}/src/out
  12.    )
  13.    add_dependencies(webrtc_runhooks  webrtc_synchronization)

Full code for sync and unhook can be seen here.

5. Prepare the Build with Ninja target

As mentioned previously, we have to deal with Debug / Release modes here. We follow the CMake syntax, which luckily fits ninja syntax.

  1. if(NOT CMAKE_BUILD_TYPE)      # allow to set it in a superscript and honor it here
  2.   set(CMAKE_BUILD_TYPE Debug) # Debug by default
  3. endif()

We can then run the usual.

  1.    set(webrtc_build ninja -v -C ${CMAKE_SOURCE_DIR}/src/out/${CMAKE_BUILD_TYPE})
  2.    add_custom_command(
  3.      OUTPUT  ${CMAKE_SOURCE_DIR}/src/out/${CMAKE_BUILD_TYPE}/libwebrtc.a # it’s arbitrary.
  4.      COMMAND ${webrtc_build}
  5.    )
  6.    add_custom_target(
  7.      webrtc_build ALL
  8.      DEPENDS ${CMAKE_SOURCE_DIR}/src/out/${CMAKE_BUILD_TYPE}/libwebrtc.a
  9.    )
  10.    add_dependencies(webrtc_build webrtc_runhooks)

6. Finally building!

you can then run cmake once and then launch the build with your native compiler. This code has been tested on mac (use “make”) and windows (open webrtc.sln in MSVC, and build the ALL target, or hit F7).

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

The chromium / webRTC build system

I.Introduction

This post is an introduction to the build system used by Google in many projects, with a specific focus on building WebRTC. It does not pretend to be exhaustive, but should give you an overview of all the steps involved, and all the files involved, so you can dig deeper and debug faster if there is a problem along the way. It will also set the ground for a follow up post about how to automate the build. Finally, it will give some hints about how to proceed with modifications of the code and rebuild incrementally after that. A full post will be written relative to the details of that last point.

II.depot_tools

depot tools is the name of the collection of tools used by the chromium team to handle all of their development process, including code reviews, remote testing, etc. Even if for simple steps like getting the code, everything eventually boils down to git or svn commands, chromium and by extension WebRTC have a dynamic way of handling dependencies and compiler flags. It’s buried within multiple interdependent files (DEPS, .gyp, .gypi, ….) and you should really avoid modifying these until you have a specific need for it.

further read

  • https://www.chromium.org/developers/how-tos/depottools [OLD]
  • http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools.html

III.fetch and the webrtc recipes

The official site will list more requirement in term of libraries and compiler for each system they support and is a must read. They recommend to simply run the “fetch” command line and be done with. For a one shot attempt at building webrtc that might be enough, but for most people it is not.

A look at the fetch source code (fetch.py not to be confused with client’s fetch) you will see it just calls a “recipe”. Recipes are hardcoded values and command designed to make it a one liner. In this case the webrtc recipe just hardcode the values  in a .gclient file, having the same result than a call to “client config” with the right parameters, and execute the equivalent of a call to “client sync”, which download the code. We feel it makes more sense to call explicitly the gclient commands and keep control on what we do.

IV.gclient config and .gclient file

You can achieve the same result than “fetch” by running the following command line:

  1. gclient config –name src https://chromium.googlesource.com/external/webrtc.git

Originally, the webrtc code was handle through SVN, and webrtc was using “trunk” as their base root name. Following the migration to git, and to stay closer to the way chromium way of doing things, that has been changed to “src” and need to be forced. That explains the “–name src” part of the command line.

Now if you look at the .gclient file generated by the recipe or the one generated by the hove command you will notice one line is missing:

  1. ‘with_branch_heads’: True,

That actually adds 1/2 GB to the download of the source,  and is only needed if you want to work with the same version that chrome includes. Most of the time this is not the case. (see this page, the “Working with releases branches” section).

Finally, there are also two recipes for iOS and android, which end up adding a single line to the .gclient file rested to the target_os (in the original recipe, target_os is a pass-through):

  • “target_os = [ios,mac]”
  • “target_os = [android,unix]”

It supposes that you are cross compiling the iOS lib on mac and the android lib on unix, respectively. If you are on mac/unix/win, it is worth adding the target_os line , and “target_os_only = True“, to reduce the amount of code and testing files downloaded to the bare minimum. It’s not mandatory though, and needs to be done through file manipulation, as there is no parameter in gclient config to do that (yet).

further read:

a .gclient file can be very complex indeed and like many things in webrtc/chromium, most of them are hand written. A perfect example is the chromium .gclient file hardcoded in the webrtc code. You can see that custom dependencies are being defined as a way to prune the original chromium dependencies. Chromium developer will tell webrtc developers who complain about the time it takes to fetch webrtc, that it would be twice worse with webkit enabled 🙂 Those who have time to spend optimizing their pure, lean-n-mean webrtc libs could investigate what else they could remove for a production build.

V. gclient sync

gclient sync is an interesting command with lots of options (grep/search for “CMDsync” in the client.py file). Its main role is to get the code from the url provided in the .gclient file, then parse the DEPS file. If the DEPS file contains any hooks, it can also run them directly. In our case, we separated running the hooks from the synchronization. There are many reasons for that, some of them we will describe in further posts, but one simple reason is to be able to illustrate how much time getting webrtc code vs getting chromium code takes.

The result of this parsing of DEPS is the .gclient_entries file which contains pairs of folder : url including first the name (here “src”) and original url, then all of the dependencies. If you are on windows, there will be an additional dependency to winsdk.

There are a lot of options, but we are going to keep things simple. We are going to use -r to fix a revision to make our productions build more stable. We are going to use -D to prune directories no more used. Finally we are going to use -n to avoid running the hooks during the sync step.

  1. gclient sync -n -D -r 88a4298

The revision tag here is a git hashtag in general and the hash to the specific commit before the Google team silently change the threading model, resulting in a lot of spin locks in the code. That also is the reality of working with the HEAD, and why we will treat in a separate post about how to test the compiled libraries before using them in production.

further reading

Later on, after running the hooks, you can out of curiosity take a look at the 78 entries strong file for chromium at /src/chromium/.gclient_entries You will see that some have “none” as url, as set up by the custom_deps we spoke about earlier.

VI.gclient runhooks

The hooks in this case come from the DEPS file.

  • check_root_dir.py    # just checks if you re still using “trunk” or not. Backward compatibility.
  • sync_chromium.py   # the one you re gonna learn to hate. Takes 40mn from Mountain View, CA with 89MB DL.
  • setup_links.py            # see below
  • download_from_google_storage # get the test files
  • src/webrtc/build/gyp_webrtc.py # generate the build files – see below

sync_chromium.py is just an extended wrapper for “gclient sync”. Since the .gclient file is already present in the /chromium folder, sync is the next steps. It should be noted that it modifies the GYP_DEFINES environment variable to remove the NACL tool suite from chromium. As you can see, there are many ways to do things, and they are all entangled in the source code, which makes it quite difficult to handle.

setup_links.py is here to address backward compatibility. SVN has one advantage over git: you can directly retrieve subdirectories, while in git you have to get everything. When both chromium and webrtc where using svn, some of the chromium subdirectories were directly retrieved and inserted in the webrtc layout using that SVN feature. Now that everything moved to GIT, it is not possible anymore. Instead of modifying the code and the includes to reflect the new layout (with all of chromium code under the /chromium dir), the google engineers decided to chose the backward compatible approach of creating symlinks for each of the previously inserted folders. The layout is then the same and there is continuity. This is what setup_links.py does. Unfortunately, it comes at the cost of running this command as an administrator under windows, which is an extra burden.

/src/webrtc/build/gyp_webrtc.py is a very important file. It is the file that will generate the build files based on the gyp and gypi files on one hand, and on some environment variable on the other. by default, ninja files will be created in src/out/Debug and src/out/Release, respectively. If you’re going for an automated build, you should use the default. Now, if you want to be able to debug in an IDE like Xcode or MSVC, there are a few extra steps for you to add to the process. You can follow the official page for that, section “build”.

If you modify any gyp or gypi file to add compiler options, or add projects, you will have to re-run gyp_webrtc. If you change generator, you will also have to run it again. Usually “gclient runhooks” does that for you.

VII. Build

That step depends on what choice you made with respect to your gyp generators. Here we will assume ninja since the main goal is to open the path to automated compilation (oh yes, and also, because google almost only supports ninja …..).

Some practical notes about ninja and webrtc.

more or less each gyp file will generate a .ninja file. If you look on the root directory, say /out/Debug in our case, you will see one main ninja file “build.ninja”and then everything else is in the obj subdirectory whose internal layout mirror the webrtc source layout. Once the build will be done, the resulting libraries and executable will be under the root for mac, and where their corresponding .ninja file resides for windows. That makes cross-platform packaging code … interesting to write. Ninja supports incremental compilation. so if you modify a gyp file to add a source file, add a project, etc, you can regenerate the ninja files and rerun to compile only what s needed. However, if you make some other modifications, like changing the compilation parameters, ninja will not recompile if the binary is still present, and you have to remove it manually. Practically, if time is not a real issue, to be safe, one might want to remove the /out/ directory before regenerating the ninja files and recompiling.

In our case

just run the following line and you should be all set (works with Release instead of Debug).

  1. ninja -C /out/Debug

further reading

  • https://martine.github.io/ninja/manual.html

Creative Commons License
This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.