Many questions on the discuss-webrtc mailing list nowadays are about specific configuration flags. It’s frequent enough to deserve a blog post to provide the basic and try to reduce the number of stuck people.

The copy-paste attitude

Nowadays, a lot of people don’t really care about understand, or at least less than doing. They copy-paste code from the web, if it “works”, job is done. If it does not, then they ask more questions on mailing lists or on stack-overflow until someone give them the answer, often without making any effort themselves. It is regrettable in general, and don’t even get me started on the fact that often, credit is not given to those who invested the time and gave the answer. I can give you several examples of companies that received tens to hundred of millions of dollars based on free Open-Source projects or other people free support. Business as usual some would say.

Sometimes, you don’t get an alternative to ask

However, in the case of libwebrtc, I have to admit that it is often legitimate, as there is really no other documentation than the code itself, millions of lines of code, ever changing with no backward compatibility. There is also few if any incentive for the original coders to document, answer questions or help in general. That’s where the community, you, me, everybody can jump in and try to help. Quite a few are active on discuss-webrtc including fippo, myselves and others, and we are trying to document different aspects. God knows that it is not enough, and by all means, if you have some info that was extremely painful to find, share it in a google searchable manner 🙂

Today, let’s speak about GN.

GN, or Generate Ninja, is the part of the chrome toolset that handles the build configuration of chrome and dependent libraries like libwebrtc. It is a replacement of earlier GYP, even though one can still find some remnant of the former here and there. It manages the build configuration files (recognisable by their .gni or .gn extension) and generate build conductor files (.ninja) and toolchains (also .ninja).

A note on ninja, why it s fast, and the link to distributed builds

NINJA, I call ninja a Build “conductor”, because they are just that. In a similar fashion as old makefiles on linux, every possible atomic action (compilation of C++ files, of C files, linking, time stamping …..) is mapped to an alias in a toolchain file. Then a ninja file just describe the list of all the actions needed, in order. Files are compiled one at a time, and everything is made atomic so you can very easily distribute the compilation over cores on a single machine, or over several machines in a trivial one-to-one fashion. If you’re a googler, you can add the GOMA back-end and get distributed build. If you’re not, you can contact cosmo and see about their distributed compilation and testing tool.

Ninja is NOT a compiler, and it also does not handle logic about configuration depending on flags, or compiler, or host/target machines, … that s GN’s job.

Chrome wants to simplify things for all platforms its support.

By default, chrome is logically moving toward a unified toolset across platforms. The rest of the vendors (MSVC, Xcode, QtCreator, … ) are equally moving toward using all of LLVM tooling (clang, lld, …), google is just doing it faster, creating sometimes some problems for people who wants to use libwebrtc with the “native” compiler in their default way, while google ships a more recent clang, stdlibs, or linkers. Being able to (cross-) compile binaries for any platform on a linux host would also greatly simplify the infrastructure, when and if it happens.

All the google code nowadays should be using GN and ninja. Hybrid builds leveraging native compilers IDEs, but launching ninja under the hood are provided but not supported.

Meta-build	build	compiler	standard lib	linker
GYP	MS Project/Solution or MSBUILD or NMAKE	cl.exe	msvc’s	link.exe
GYP	Xcode projects	gcc	libstd	ld
GYP	make	gcc	libstd	ld
GYP	ninja	all of the above	all of the above	all of the above
GN	ninja	goog modified clang	google modified	lld

Google defines several level of support:

• google supported options: there is a corresponding build bot which automatically checks that any submitted code compiles, run, and pass the unit tests with the given options/flags. The code does not get merged unless it passes this build.
  
• google options of interest: (my wording), there is a corresponding build bot that does all of the above, but it belongs to the FYI group, and the code can get merged if this bot fails but all the main bots succeed. It is used for example to clean the code base before adding a new option or a new compiler, or as a transition before phasing out an option.
  
• Community supported option: I call this part the “antechamber of death”. When Google lost interest in supporting a feature, it puts it under this section. There is no build bot, and no commitment from google that they will not break this special configuration. There is an underlying unsaid commitment to remove said feature in the future if no-one stand-up and submit tickets and patches to maintain it when it gets broken. After all, if it gets broken and nobody complains, it must mean nobody uses it, right? This is logical to the extend that
  • nobody really announces (for libwebrtc, this is not true for chrome), when something moves to “Community-supported”.
  • google coders are blind to whether they break it or not, asking for the community to potentially race the 100+ webrtc engineers at google (1000+ chrome engineers), to keep the thing working. Worse than that, when ask to fix their commit, or to review a patch, they have n incentive to do it, as their are not going to be evaluated on this.

I don’t want by any mean to sound ungrateful. Buying the code and putting it public, open-source, with a very permissive license, then maintaining it in some shape of form, even if that is not always what I need, is already a huge effort that I would love to see more companies replicate. At the same time, I’m happy Microsoft jumped in a few years back, first for their UWP build, and now for Edgium, otherwise I believe the MSVC builds would have been dropped. Pure MSVC support has been dropped for chrome for example.

The GN Flags, how many, how to learn about them, and check them

Time/Ease tradeoff. Most of the libwebrtc code and applications are all build in one go, from source code. Electron for example, takes a different approach and pre-build almost everything to save time. Following chrome way, all configuration details are in memory and all projects are configured at the same time, preventing any mismatch that would allow source code to build, but link to fail, or executable to fail at run time. Following Electron way is faster, but only tractable if you can deterministically determine configurations options, and only have a few of them.

Chrome as a myriad of options, and it always had been a problem. Logically, when creating GN, features were added to manage those options, the GN “args”, better for developers, under the GN args command.

A first question could be: just how many flags is there that I can specify when calling GN gen to generate my ninja files. This is easily answered by listing the flags (without description) and counting them:

gn args <build_dir> --list --short | wc -l

the answer is 198 with my simple mac build ….. . Usually at this point on people have an “Oh S%$%$” moment. There is no way that one learn about 198 flags, and imaging how big that configuration matrix with all permutation would be! Luckily, there are several things that make dealing with Flags a tractable problem.

• Gn is an extremely well documented tool, and the variable are usually equally well documented. Remove the --short options from the GN args <build_dir> --list command to get the description of each flag. Still, 198 flags is a lot.
• libwebrtc specific flags are usually prefixed by “rtc_”, so you could jut focussed on those. While a good idea, in practice it is not very helpful with respect to the questions we see on discuss-webrtc.
• if you are only interested in the description of a specific flag, its default value and what to know what it does, just pass its name to the command: GN args <build_dir> --list=use_rtti
  When I see unknown flags being used on discuss-webrtc, I use this command to check not only what they do, but also what their default is. More often than none, people use those flags with the default value, and could completely remove them from their command line, focussing on those who actually change the default behavior.
• even more powerful,
  • gn args <build_dir> --list --short --overrides-only will list only the args used whose value is NOT the default one on that specific build. Basically, this provides you with the minimum flag set (and values) you need to specify to reproduce the given build.
  • gn args --list --args="os=\"android\"" This checks what the values of all flags would be provided a given arg or list of args. Note that it does not need a build directory. It is the only way to check dependencies between flags without creating a build or looking at the source code.

There is also a very convenient power tool written in C++ and Qt that allows to navigate GN files. It complements very well the command line GN tool, without completely replacing it.

Important GN flags, especially under windows

• use_rtti: Google supported, default: false. Long story short, if you see missing vtable symbols for derived classes at linking time, you likely need to set this to True. It happens for example when you create your own video capturer. It is off by default because in chrome all the code is compiled in one go and you never need to extend later, and because it allows to reduce the binary size.
  
• is_clang: Google Supported, default: True. This is the main flag for switching between MSVC toolchain, and google provided toolchain. It gets a little bit more complicated with recent versions of MSVC which ship clang-cl.exe in addition to cl.exe, but let’s keep it simple for now.
  • use_lld: Community supported, default: False. This flag is only taken into account if is_clang is true. If set to false, it tells ninja to use the MSVC linker in its toolchain, improving the chances that generated binaries are compatible with MSVC project thereafter.
  • use_custom_libcxx: community supported, default: True. Google contributed patches to upstream libcxx. Not to wait for the patches to go in, they carry in depot tools a fully patched version of libcxx. This flag allows you to use clang, but link against the MSVC standard lib instead.

flag	Compiler	cxx std lib	linker
is_clang=false	MSVC cl.exe	MSVC lib	MSVC link.exe
<default> is_clang=True use_custom_libcxx=True use_lld=True	goog clang	goog custom libcxx	lld
is_clang=True use_custom_libcxx=False use_lld=True	goog clang	MSVC lib	lld
is_clang=True use_lld=False use_custom_libcxx=False	goog clang	MSVC lib	link.exe

• rtc_use_protobuff, google supported (within reason), Default: True. In debug mode, some examples and tools are generating errors under some condition (long story below). While not directly related, switching this flag to False controls whether the faulty code is being compiled. This is not a problem in release, and likely not a problem in general.
  Examples and tools dealing with RTP packets filtering use clang specific features (macro offset: define offset of a member in a class instance memory mapping to be able to point to that member independently of what class it is a member of, as long as it class is of standard layout, which can be checked by std::is_standard_layout, which returns different result depending on which libcxx you use.)

Conclusion

Hopefully this blog post will provide some guidance, tools and processes to debug linking and runtime problems with libwebrtc. Of course, it only scratches the surface, and it is rendered obsolete at the right of 200 commits a week. That’s why cosmo software proposes pre-compiled and pre-tested libwebrtc packages, and corresponding professional services.