While intense discussions happen on the W3C Webrtc mailing list (someone asked everybody what they want to see in Webrtc NV ….), it’s nice to take a step back and look at the first results of KITE, a new end-to-end testing tool designed to make Webrtc 1.0 and all applications using it better.

Compilation of WebRTC (my other hobby when I’m not doing hallo Kitty PJ contests with the kids),  is a lot of fun, but end-to-end testing of WebRTC is in another league all together.

For a long time, many bugs went unseen by lack of proper testing. It was nobody’s fault really, the web did not have any p2p API before WebRTC, so it did not have any tool at disposal to test that kind of behaviour.

Since the beginning of WebRTC, you can read blog posts here and there, that could have lead you to believe that all it takes is a single extremely gifted engineer to find those bugs and report them. Practically, nothing could be more wrong.

It takes full testing infrastructures, with thorough tests and extensive test matrices to be able to catch a bug! Google itself have a complete “waterfall” that runs thousands of tests for each commit to chrome (the webrtc subset of chrome has his own waterfall) . Most of the blog posts you might have read came from individual working in companies like tokbox, appear.in, callstats.io, that have at minima an extensive monitoring infrastructure in place (we’re seeking about hundreds of collection points per call), and most likely a testing infrastructure as well. Most of the work comes from the infrastructure developers, maintainers and operators, that provide the rest of the team with the capacity to analyse and debug. Hats off to them, the shadow workers, whatever company they work in.

If you cannot find enough bugs to write blog posts and shine, but still need the attention, the new trend is to write post-mortem analysis of other peoples bugs, and other people fix, showing that you could have done it. But I diverge here.

One of the original goal for KITE from the main sponsor Google was to make a new category of tests easy to deploy and operate, with the hope that it would in turn give visibility on a category of bugs that was elusive so far. None of the existing solutions seemed to provide this  (web and native, desktop and mobile, hosted and on-premises, …).

While we are in a phase of KITE development where the focus is more on the deployment and operation side of things, i.e. adding support for more browsers, more OSes, mobile, native (CEF, electron, Qt, …), and not yet in the phase where we write more tests, it is refreshing to see that we are already catching complex bugs.

I will keep the Edge bugs for a later post, because there are quite specific, but an interesting bug we just had closure on was happening between Safari (or STP) and Firefox. It was only happening one way, it was only happening when certain codecs combinations where involved during the offer-answer, and so on and so forth. People interested in the gory detail, and the hunt through the crash dumps for the bugs can look at the corresponding ticket (here). Long story short, after a couple of days of intense exchanges, the bug was found and fixed by niels (a.k.a. the usual webrtc QA suspect @ mozilla.com ).

In brief, as all the good story, KITE and bug crushing is about team work. Without the great work of Hoang VU, Sajid HUSSAIN, and Jose RECIO, KITE wouldn’t exist. Without the early and ongoing support of Google, KITE would not be as advanced as it is today and without the great responsiveness of the Mozilla team, that bug would still be there. I can only claim responsibility for writing about it (and still, in tainted english) 🙂 Nothing to be terribly proud of.

We’re looking forward to find more bugs with KITE, and make WebRTC 1.0 better as we go. We’re also looking forward to more adoption of KITE by companies that want the same level of testing chrome (voluntary) uses, and MS, Apple and Mozilla …, hum, … enjoy (?). KITE is open source, and free, and allow you to test your app / back-end / infra like only the biggest players could achieve so far. Of course, we are always available to help you taylor it to your needs, would you need help in that matter.

Enjoy.