Since our “Real-time communication testing evolution with WebRTC 1.0” article was accepted for publication by IEEE, not many new scientific papers have been published on WebRTC testing apart from [1] and [2] ([3] is also available, but without indication wether it has been peer-reviewed and accepted for publication or not), but a lot has happen with KITE.
Beyond being the official testing tool for webrtc.org, and adding tests for multistream and simulcast to better cover the last pieces of the WebRTC 1.0 spec, support for mobile browsers and mobile app testing, as well as for Electron apps testing have been added. KITE can now support up to 20 clients configurations, making it the most complete and most versatile #webrtc testing tool known to date.
Beyond the Interoperability Mode, a Load Testing mode has been added to KITE, specifically to stress test webrtc servers and infrastructures, with a capacity of 1 millions users. It has been tested in production by several customers in a one-to-many streaming environment.
      Of course, you can have CoSMo run KITE for you (hosted and managed), or you can run it yourself, on premises. More details in this Post.

I. Help maturing the webrtc 1.0 Spec

As google is moving to support unified plan, multi stream and simulcast, there was a specific need on tests to be able to support that effort. When those test were written, only Firefox was supporting those, and in a not fully compliant manner. Moreover, while multistream can be tested in p2p, browser-to-browser mode, simulcast can only be tested against an SFU, so a new kind of test and test infrastructure needed to be developed.

You can check the simplified dashboard at webrtc.org with 8 configuration made visible, or the cosmo hosted dashboard, with all configurations, and all tests on: CoSMo’s Dashboard.

II. add support for Mobile and Electron

With electron being so hot, and electron WebRTC implementations almost on Fire it was obvious that any serious WebRTC testing service had to have electron support. It was equally clear from everybody that we met, that, while desktop browser support was a nice to have, the difficulty, and the added value was most often than not on testing teh mobile apps. Both cases had the same challenges: it does not work out of the box.

Unlike chrome, Firefox is not recognised as browser on android and required quite a lot of modifications to work. Thanks to collaboration with Mozilla, the Appium community, Selenium HQ, and the webdriver JAVA bindings community, we patched all of the above projects to be able to support Firefox on Android. The results will appear soon on the webrtc.org dashboard for everybody to follow.

Thanks to collaboration with ElectronHQ, and feedback from Slack, we managed to modify Electron and its webdriver implementation to bring it to parity with it’s chromium parent. You can see a video of the test here. 

In both cases, we are pushing the patches to the different upstream repositories, but in the mean time, it’s likely that we are part of the happy (very) few capable of running webrtc tests against android firefox clients and Electron.

III. KITE Load Testing Mode

What we missed in our original article, is that half of the testing projects where not (client-centric) interoperability tests, but server or infrastructure-centric testing project. Load multiplier, Jattack, jitsi hammer, all were load testing projects.

Load testing of video conferences is very different in practice than load testing of streaming infrastructure.

In video conferences, the number of streams grows as the square of the number of users. It means that if you get 30 users, you get approximately 900 streams, and you get close to the limit a single server can handle. As illustrated by Jitsi in one of their post (in which they push up to 28 users), the main bottleneck of media servers is the bandwidth. With an usual 1Gbps NIC, and supposing that a VGA@30fps stream is around 1Mbps, you can handle around 1,000 streams. With 30 clients connecting to a server, you saturate a server, with around 300 clients you can start really stressing a Video Conference infrastructure, if your clients can handle receiving all those streams back.

In the case of one-to-many streaming, you do not have the same problem on the client side, each client only receives one stream. However, you need many more clients. For social sites, have thousands of viewers if not unheard of, for live event, you’d better scale to millions. We modified the KITE architecture to handle multiple hubs, in multiple locations (e.g. AWS availability zone), which, beyond improving the scalability to the millions of viewers, allowed for geolocation testing as well.

Of course, it does not work that easily with apple devices, mobile devices and Edge. Mobile devices need physical devices. Apple only allows it’s operating systems to run on apple hardware. Both Edge and Safari limits the number of browsers under automation to 1 per operating system. Finally, Edge does not enumerate (recognise) fake webcam like many cams or other preventing anyone from testing webrtc on VMs.

Thankfully, the KITE architecture was made with hybrid SE grids in mind, and we have now, in addition to our AWS and GCE based hubs, a physical grid made of roughly 20 mac mini, 20 android devices, a few iPhone, and 20 Dell Optiflex micros (both 3050 and older 9020) to help. It does not reach millions, but allows some flexibility, and we can always buy more if need be. In most cases, when testing with hundreds of thousand or couple of millions, customers were happy with using only Chrome or Firefox only.

Conclusion

KITE has gained quite a few features in the past months. Some of them specific to webrtc, some of them more generic and exclusive to KITE like mobile testing, and electron apps testing.

A grid manager has been added to facilitate the automatic set up of selenium grids, and in Load Testing mode, KITE can now spawn testing instances on-demand.

Right now, the team is hard at work adding network instrumentation as a capacity to KITE, and make writing test easier. In most of the case we’ve met so far, most of the existing selenium tests could dbe imported very easily and directly run on the many configurations KITE supports where the clients were limiting themselves to chrome, or chrome and firefox at best. 

If you’re interested in using KITE, or want to collaborate on the project, do not hesitate to contact us.