Back in 2013, I did a full review, including benchmarks, of all the open source webRTC media servers. Defining use cases and corresponding tests and benchmarks was hard, and while I was happy with the results, which I presented a couple of times in conferences, the results are now grossly outdated. A year and a half ago, I tried to trigger a gathering of all the actors, as they all belong to IETF. It is very difficult for a potential user to know which project to use, and answers on the webrtc-discuss mailing list were confusing at best. We needed something more descriptive, easier to understand. I got traction from most of the players, but couldn’t make it happen. This year, hopefully, things will be different.

The Genesis

While starting to work on webRTC, I was still running some public research lab, and was affiliated with an Image and Video Processing lab called IPAL. While working on the topology and geometry of orientable 2-manifolds (well, surfaces of real objects …) with a PhD student, Stephane Rigaud, to better understand brain cells, I met a bright young man, Francois regnoult, who was assisting for coding tasks, especially for Image processing algorithm and GPU coding (CUDA). Since his Intern visa was about to expire, and the company i was working in needed resource, we hired him and started the work on an MCU/SFU. A little bit later, he was joined by Ludovic Roux, a more senior researcher from IPAL, with a PhD in Image/Video processing, and a solid knowledge of algorithm development practices in C/C++.

The benchmarking process

the usual suspects

As any good project on an unknown technology, we started with an exhaustive research of all the existing project, and then proceed to benchmark them. We did actually a complete sweep of the ecosystem, and some synthetic results can be found here, or there (slide 7).

Of course, some projects have died, and some new project came to replace them, or to extend the list. So this is what the list would look like today (no, it is not just Kurento and Jitsi). I might miss some, but those are, AFAIK the usual suspects. Feel free to point me to those I missed, I love to learn.

• lynckia (licode) – open source
• meetecho (janus) – open source
• kurento – open source
• atlasian/jitsi (meetme) – open source
• meedoze – open source
• media soup – open source
• ————————————————————————-
• pexip
• dialogic (power media XMS)
• tokbox (mantis) – not available separately from the platform
• Radisys 

Note that more traditional VoIP servers (asterisk, freeswitch, kamailio, ….) have now enough support for webRTC protocols that they can be used in some cases as media server for webRTC solutions. We have not included them in the original study as it was not the case early 2014.

Which Questions to answer (methodology)?

The goal was not only to do a performance benchmark, but also to assess the architecture, which impact directly the cost of integration in our existing platform , and the maintenance effort going forward.

• Do we have expertise in the programming language used?
• how many lines of code to understand and maintain?
• is there an existing test suite?

were some of the questions we were trying to answer that were at least as important as the brute performance of a stand alone media server.

Results

The results, one table with all infos for each project/product, were presented publicly at webRTC Conference and Expo in June 2014: Scaling webRTC Video Infrastructure. Those who wants to skip the conceptual details about underlying technologies can go directly to slide #22.

Discussion and Conclusion

In our study, the case was obvious and implicit: video-conferencing. When most people come to the discuss-webrtc mailing list, their use case is equally obvious … to them. However, there are so many use cases, that it is very difficult to provide a generic answer (the generic question being of course: which media server is the best, and each vendor to answer “mine”.). It’s an ill-defined problem, and would the use case be defined, there is no publicly available benchmark that would allow you to check each project claim.

It’s actually telling that the experts in the field cannot even agree on a name. If you are a reader of webrtchacks.com, you have seen most of the vendors listed above writing about their product, using a different name for them. See also that tweet feed between the main coders of half of the list mentioned earlier. Here is a (non-exhaustive) list of names I could find to describe what I called here a webrtc media server.

• switch
• gateway
• application server
• media server
• conference server
• MCU
• SFU
• video bridge
• video router
• webrtc server

At one point, when the question came one more time on the mailing list, and before everybody stand up to say “my solution is better”, as a rough, weak, consensus was building in favor of the “it depends on the use case, really” answer, I proposed that we all meet on the side of an IETF meeting, since we were all attending those meetings anyway (Beware: Long thread, but very instructive). The goal was to discuss and come up with a list of the most common use case, to which the users could relate and suggestions about which tests would best characterize each use case. With so many open projects, lead by brilliants engineer, it was somewhat a shame that the only white paper on the subject was a dialogic sponsored piece written by someone who was external to all projects, and not the most technical of all. That work we would do all together would make for the basis of a benchmark, would simplify answering questions, and would allow each project to illustrate their focus and their strong point. No place for opinions anymore, there would be hard data. 

Unfortunately, for many reasons, the meeting did not happen.

While I have been maintaining contact with some, and have been touring the Asian vendors (intel, shiguredo) about whose products I will make individual posts in the following weeks, I have not been in touch with the european guys.

The opportunity to be in Europe for a whole month, the happiness to find an excuse not to spend it with my parents (there is an age beyond which you should not live with your parent anymore, however good home-cooked food can be), the gentle but firm pressure from my wife to visit more of Europe, and the availability of low cost flights, all came together to convince me that it was the time to undertake the trip of a lifetime: an European Tour of webRTC media Servers !*

 

* … or MCU, ….. or maybe SFU, …. gateway perhaps? …. well, they’ll teach me, at that’s the goal of a trip, learn more 🙂

This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.