Every week or so, I get the same questions:

• do we have to chose between webrtc and ortc?
• Is that <feature> supported in this <browser>?
• Is interoperability between this <browser> and that <browser> possible?
• Which <version> of that <browser> should I use?
• How do I test <feature> in <browser>?
• …

What people really want is a scorecard, one way to see if a specific Api is supported for a given browser under a specific OS. For webRTC, it is more complicated, since many APIs depend on each other, and also because availability of features for interoperability on the wire are important but are not APIs per say (availability of H264 as a codec for example).

The &yet score card, coupled to talky is a good example of a product manager interest. However, at a certain level, one wants to be able to test by him/herself, and integrate those tests in a build system.

Today I’m going to speak more about automated testing for webRTC APIs across browsers. 

Test the web forward.

Promoting the idea that the web belong to everybody and should be seen as a platform, the W3C has started an initiative to make the web more testable. The Web Driver Working Group [here] has been created to help the development of tools that would “drive” browsers (think selenium). A more global effort, test the web forward, has also started. Testing frameworks have been developed in JS to test each implementation of the specs. Automation of IDL testing, and all other W3C specs keywords, is already provided for you. All the code is open source, available in github [here], and one can go ahead and start testing in a matter of minutes, without extensive knowledge of specs or web development. 

When it comes to webrtc, there are two test suite in the web platform tests that are of interest: “mediacapture-streams” which tests everything related to GetUserMedia, MediaStream, and MediaStreamTracks, (i.e. the Media Capture and Streams Specification) and “webrtc” which tests the corresponding specification, WebRTC 1.0.

Adapter.js

During the life of webrtc, the specifications varied significantly. Depending on when they implemented it, browsers can end up with same APIs names, but with different behavior. To smooth those differences, Google and Mozilla originally created “adapter.js“, which was small JS shim to smooth those small details out, and allow for interoperability and common testing. Those tests are higher level than the web-platform tests, i.e. they tests scenarios, and not only APIs. 

Thanks to some work by Philippe Hancke last year, Edge and ORTC are also supported, as much as possible (the lack of common video codec for the moment prevent video interoperability, even though there is no such limit with respective APIs.).

Thanks to work from Damon O, the developer behind the rtc.io library, travis is reused to run those tests across browsers as part of automated testing. In term of host OS, it is limited to linux though.

Using adapter.js (or an equivalent) is almost mandatory today for a web app to run across browsers as it also adds shims for not yet available APIs, like the promise-based APIs in chrome.

Practically, how do I test?

Here I am only going to address manual testing using the web-platform test suite (without CSS).

You should start by cloning the web-platform-tests from github, and follow the readme. Three easy steps (modify host, run the web and web socket servers, and open the test runner in our browsers) should be enough for you to start testing. In the following examples, I used the test runner, with “/mediacapture-streams” as a filter. 

If you want to have real results, you should modify the web-platform-tests to include adapter.js as explained in their corresponding readme.

If you’re testing in chrome version 47 or above, you might want to use “localhost” instead of “web-platform.test” for your server name, so that GetUserMedia calls do not time out 🙂

Note that the number of tests run depends on the number of failures. Failures of the first test in a meta-test will greatly reduce the number of tests run. E.G. mediastreamtrack-init has some 63 tests internally that only appear in the result if the first one is successful. if the first one fail, it reports only one test, as failed.

what results should I expect?

Well, the w3c tests are not all up-to-date, even though we’re working on it. Adapter.js is also not always up-to date.

Specifically for Safari, safari is recognized by adapter.js as chrome, and uses chrome shim, which is likely inadequate. The security prompt might be implemented but is not available. It’s understandable as the security prompt is in the safari part, which is not updated in safari nighty. Safari nightly uses whatever Safari version you have installed, and update some internal libraries only. That prevents a lot of test from running. We could expect many if not all the 22 tests that timed out, to pass. I’m still investigating if a DataChannel only call can be made, since DCs do not require a prompt.

I did not test edge this time ( I love my macbook airs 🙂 ).

Here are some raw results without deeper investigation on the failures. I separated the MST-init tests results in chrome to have a comparable baseline. It’s interesting to see that Firefox is the most W3C spec compliant browser, according to the tests.

Conclusion

This gives an insight on how one could tests JS APIs across browsers, leveraging existing tools, sharing the maintenance cost with all W3C and the entire community. I will be working int he following weeks with all stakeholder to update those tests suite and tools, and get a better view of wether webrtc is ready yet. 

This does not touch on the automation of testing, nor testing across OSes. While the adapter.js repository automates the testing using travis, it is limited to a linux host, and some parties are working right now on a solution that would allow testing across mobile and desktop browsers, as well as across OSes. Interoperability with native applications / framework (appRTC, cordova, web view, ….) is also investigated. If you’re interested, contact me.

Finally, this post also does not address the problem of end-to-end testing, wether between same browsers, or different browsers. That’s another quite challenging topic, and browser vendors have today different approaches.

This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.

This work by Dr. Alexandre Gouaillard is licensed under a Creative Commons Attribution 4.0 International License.

This blog is not about any commercial product or company, even if some might be mentioned or be the object of a post in the context of their usage of the technology. Most of the opinions expressed here are those of the author, and not of any corporate or organizational affiliation.