Recent discussions with several parties, make me realise that the first steps to master webrtc are not yet documented enough. Once, as a challenge, I asked a master student doing his graduation project with us to try to recompile and example provided with libwebrtc separately from the compilation of libwebrtc itself. Basically to make a project with the same example source code but that would link against the pre-compiled library. 5 months later, it was still not successful. Thanks to our internal tool that I shared then, we eventually did it in two weeks. This post is about the usual ordeal people have to go through to understand the state of affair, and of course how CoSMo can shield you from that.
How to make sure the quality of a [webrtc] video call, or video streaming is good? One can take all possible metrics from the statistic API, is still be nowhere closer to the answer. The reasons are simple. First most of the statistic reported are about network, and not about video quality. Then, it is known, and people who have try also know, that while those influence the perceived quality of the call, they do not correlate directly, which means you cannot guess or compute the video quality based on those metrics. Finally, the quality of a call is a very subjective matter, and those are difficult for computers to compute directly.
In a controlled environment, e.g. in the lab, or while doing unit testing, people can use reference metrics for video quality assessment, i.e. you tag a frame with an ID on sender side, you capture the frames on the receiving side, match the ID (to compensate for jitter, delay or other network induced problems) and you measure some kind of difference between the two images. Google has “full stack tests” that do just that for many variations of codecs and network impairments, to be run as part of their unit tests suite.
But how to do it in production and in real-time?
For most of the WebRTC PaaS use cases (Use Case A), the reference frame is not available (it would be illegal for the service provider to access the customer content in any way). Granted, the user of the service could record the stream on the sender side and on the receiving side, and compute a Quality Score offline. However, this would not allow to act on or react to sudden drops in quality. It would only help for post-mortem analysis. How to do it in such a way that quality drops can be detected and act on in real-time without extra recording, upload, download, ….?
Which webrtc PaaS provides the best Video Quality in my case, or in some specific case? For most, this is a question that can’t be answered. How can I achieve this 4×4 comparison, or this zoom versus webrtc, in real time, automatically, while instrumenting the network?
CoSMo R&D came up with a new AI-based Video Assessment Tool to achieve such a feat in conjunction with its KITE testing engine, and corresponding Network Instrumentation Module. The rest of this blog post is unusually sciency so reader beware.
We are really laser focussed on delivering our projects and preparing for the upcoming standard committee meeting. With all of that going on, it took several message from linkedin to congratulate me for my work anniversary to realise the truth: CoSMo was 2 years old. I’d like to take a moment to look back at what we have achieved.