This post is also available in: Français (French)

I.Introduction

This post is an introduction to the build system used by Google in many projects, with a specific focus on building WebRTC. It does not pretend to be exhaustive, but should give you an overview of all the steps involved, and all the files involved, so you can dig deeper and debug faster if there is a problem along the way. It will also set the ground for a follow up post about how to automate the build. Finally, it will give some hints about how to proceed with modifications of the code and rebuild incrementally after that. A full post will be written relative to the details of that last point.

II.depot_tools

depot tools is the name of the collection of tools used by the chromium team to handle all of their development process, including code reviews, remote testing, etc. Even if for simple steps like getting the code, everything eventually boils down to git or svn commands, chromium and by extension WebRTC have a dynamic way of handling dependencies and compiler flags. It’s buried within multiple interdependent files (DEPS, .gyp, .gypi, ….) and you should really avoid modifying these until you have a specific need for it.

further read

• https://www.chromium.org/developers/how-tos/depottools [OLD]
• http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools.html

III.fetch and the webrtc recipes

The official site will list more requirement in term of libraries and compiler for each system they support and is a must read. They recommend to simply run the “fetch” command line and be done with. For a one shot attempt at building webrtc that might be enough, but for most people it is not.

A look at the fetch source code (fetch.py not to be confused with client’s fetch) you will see it just calls a “recipe”. Recipes are hardcoded values and command designed to make it a one liner. In this case the webrtc recipe just hardcode the values  in a .gclient file, having the same result than a call to “client config” with the right parameters, and execute the equivalent of a call to “client sync”, which download the code. We feel it makes more sense to call explicitly the gclient commands and keep control on what we do.

IV.gclient config and .gclient file

You can achieve the same result than “fetch” by running the following command line:

1. gclient config –name src https://chromium.googlesource.com/external/webrtc.git

Originally, the webrtc code was handle through SVN, and webrtc was using “trunk” as their base root name. Following the migration to git, and to stay closer to the way chromium way of doing things, that has been changed to “src” and need to be forced. That explains the “–name src” part of the command line.

Now if you look at the .gclient file generated by the recipe or the one generated by the hove command you will notice one line is missing:

1. ‘with_branch_heads’: True,

That actually adds 1/2 GB to the download of the source,  and is only needed if you want to work with the same version that chrome includes. Most of the time this is not the case. (see this page, the “Working with releases branches” section).

Finally, there are also two recipes for iOS and android, which end up adding a single line to the .gclient file rested to the target_os (in the original recipe, target_os is a pass-through):

• “target_os = [ios,mac]”
• “target_os = [android,unix]”

It supposes that you are cross compiling the iOS lib on mac and the android lib on unix, respectively. If you are on mac/unix/win, it is worth adding the target_os line , and “target_os_only = True“, to reduce the amount of code and testing files downloaded to the bare minimum. It’s not mandatory though, and needs to be done through file manipulation, as there is no parameter in gclient config to do that (yet).

further read:

a .gclient file can be very complex indeed and like many things in webrtc/chromium, most of them are hand written. A perfect example is the chromium .gclient file hardcoded in the webrtc code. You can see that custom dependencies are being defined as a way to prune the original chromium dependencies. Chromium developer will tell webrtc developers who complain about the time it takes to fetch webrtc, that it would be twice worse with webkit enabled 🙂 Those who have time to spend optimizing their pure, lean-n-mean webrtc libs could investigate what else they could remove for a production build.

V. gclient sync

gclient sync is an interesting command with lots of options (grep/search for “CMDsync” in the client.py file). Its main role is to get the code from the url provided in the .gclient file, then parse the DEPS file. If the DEPS file contains any hooks, it can also run them directly. In our case, we separated running the hooks from the synchronization. There are many reasons for that, some of them we will describe in further posts, but one simple reason is to be able to illustrate how much time getting webrtc code vs getting chromium code takes.

The result of this parsing of DEPS is the .gclient_entries file which contains pairs of folder : url including first the name (here “src”) and original url, then all of the dependencies. If you are on windows, there will be an additional dependency to winsdk.

There are a lot of options, but we are going to keep things simple. We are going to use -r to fix a revision to make our productions build more stable. We are going to use -D to prune directories no more used. Finally we are going to use -n to avoid running the hooks during the sync step.

1. gclient sync -n -D -r 88a4298

The revision tag here is a git hashtag in general and the hash to the specific commit before the Google team silently change the threading model, resulting in a lot of spin locks in the code. That also is the reality of working with the HEAD, and why we will treat in a separate post about how to test the compiled libraries before using them in production.

further reading

Later on, after running the hooks, you can out of curiosity take a look at the 78 entries strong file for chromium at /src/chromium/.gclient_entries You will see that some have “none” as url, as set up by the custom_deps we spoke about earlier.

VI.gclient runhooks

The hooks in this case come from the DEPS file.

• check_root_dir.py    # just checks if you re still using “trunk” or not. Backward compatibility.
• sync_chromium.py   # the one you re gonna learn to hate. Takes 40mn from Mountain View, CA with 89MB DL.
• setup_links.py            # see below
• download_from_google_storage # get the test files
• src/webrtc/build/gyp_webrtc.py # generate the build files – see below

sync_chromium.py is just an extended wrapper for “gclient sync”. Since the .gclient file is already present in the /chromium folder, sync is the next steps. It should be noted that it modifies the GYP_DEFINES environment variable to remove the NACL tool suite from chromium. As you can see, there are many ways to do things, and they are all entangled in the source code, which makes it quite difficult to handle.

setup_links.py is here to address backward compatibility. SVN has one advantage over git: you can directly retrieve subdirectories, while in git you have to get everything. When both chromium and webrtc where using svn, some of the chromium subdirectories were directly retrieved and inserted in the webrtc layout using that SVN feature. Now that everything moved to GIT, it is not possible anymore. Instead of modifying the code and the includes to reflect the new layout (with all of chromium code under the /chromium dir), the google engineers decided to chose the backward compatible approach of creating symlinks for each of the previously inserted folders. The layout is then the same and there is continuity. This is what setup_links.py does. Unfortunately, it comes at the cost of running this command as an administrator under windows, which is an extra burden.

/src/webrtc/build/gyp_webrtc.py is a very important file. It is the file that will generate the build files based on the gyp and gypi files on one hand, and on some environment variable on the other. by default, ninja files will be created in src/out/Debug and src/out/Release, respectively. If you’re going for an automated build, you should use the default. Now, if you want to be able to debug in an IDE like Xcode or MSVC, there are a few extra steps for you to add to the process. You can follow the official page for that, section “build”.

If you modify any gyp or gypi file to add compiler options, or add projects, you will have to re-run gyp_webrtc. If you change generator, you will also have to run it again. Usually “gclient runhooks” does that for you.

VII. Build

That step depends on what choice you made with respect to your gyp generators. Here we will assume ninja since the main goal is to open the path to automated compilation (oh yes, and also, because google almost only supports ninja …..).

Some practical notes about ninja and webrtc.

more or less each gyp file will generate a .ninja file. If you look on the root directory, say /out/Debug in our case, you will see one main ninja file “build.ninja”and then everything else is in the obj subdirectory whose internal layout mirror the webrtc source layout. Once the build will be done, the resulting libraries and executable will be under the root for mac, and where their corresponding .ninja file resides for windows. That makes cross-platform packaging code … interesting to write. Ninja supports incremental compilation. so if you modify a gyp file to add a source file, add a project, etc, you can regenerate the ninja files and rerun to compile only what s needed. However, if you make some other modifications, like changing the compilation parameters, ninja will not recompile if the binary is still present, and you have to remove it manually. Practically, if time is not a real issue, to be safe, one might want to remove the /out/ directory before regenerating the ninja files and recompiling.

In our case

just run the following line and you should be all set (works with Release instead of Debug).

1. ninja -C /out/Debug

further reading

• https://martine.github.io/ninja/manual.html

This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.