Testing your Video Player with Bad Streams
Finding bad streams to test your player error handling is hard. It takes time to find or generate the streams you need.The Eyevinn stream corruptor service can be used to apply corruptions and faults to MPEG-DASH manifests and segment requests in a controlled manner and on the fly.
When testing the implementation of a video player, good streams are usually easy to come by. You can find test streams from many sources. For example at http://testassets.dashif.org you can find a library of streams separated by feature that you can use to test different formats and use cases of your dash player.
What is harder to find is a source of video streams with some specific kind of known fault in them. Testing how your player behaves in case of a poor network conditions or errors in the stream is often as important as testing the good stream. How will my player react of there is a problem with the stream, or the network?
Hunting for bad streams
We know that problems with streams do occur, but it is difficult to test this in a test environment because it is difficult to model and replicate errors that only occur under certain conditions.
The result is that we don’t test with bad streams a great deal, because bad streams are hard to find. When problems occur, it usually occurs in production, when its already too late. We may have rolled out a client application to a large amount of customers, making updates and patching troublesome.
It may also be difficult to reproduce errors that occur as a result of bad streams. As they may be limited to a certain region, or a certain conditions that only apply for a subset of your users.
Trouble shooting and reproduction of the error is time consuming and difficult. Involving Analysing logs, searching the needle in the hay stack, finding the conditions that cause the behavior and trying to replicate them.
Corrupting the stream
Thinking about this problem we realized that there are various use cases where we use manifest manipulation as means to solve a technical use case. Like Server-side ad insertion or VOD-to live solutions.
We realized that we can use a similar approach to corrupt your stream in a controlled way. You can inject stream errors by manipulating the manifest and intercepting segment requests. This is what the Eyevinn stream corruptor is all about.
Rather than hosting a library of corrupted streams, the Eyevinn stream corruptor is a hosted cloud service that will break the streams we already have access to, and it will do so in a predictable way and on the fly. Giving us the possibility to do controlled player tests with erroneous streams.
How it works
We developed a simple solution with a service backend that works as a proxy sitting between the client player and the CDN or Origin where the manifest and segments of the stream are stored.
To test the player using a corrupted stream, the player manifest requests needs to be done towards the stream corruptor proxy service with a set of parameters, including the url to the manifest of the stream that you want to corrupt and a set of corruption parameters.
The stream corruptor service will fetch the manifest and manipulate it to inject corruptions in the stream and return it to the player.
The stream corruptor service will simply do a redirect to the original segment, apply a delay or respond with some error defined in the request. One example of how a manifest request could look like when using the stream corruptor service can be seen below.
The url above will play back the test stream https://irtdashreference-i.akamaihd.net/dash/live/901161/bfs/manifestBR.mpd (available at http://testassets.dashif.org) and respond with 404 status on 10% of the segment-requests. Right now it will do this randomly. If you want to test this, you can simply copy the url above and load it in your dash-player of preference. Open the Networks-tab in the dev tools of the browser and see how the player handles the random 404-responses.
Examples of stream corruptions
The Stream Corruptor service currently can manipulate dash-manifests to apply corruptions. Some examples of possible corruptions are listed below.
404 errors on segments
The stream corruptor will return 404-error response on a number of segments. The proportion of failed requests can be set by a parameter segmentCorruptionPercentage between 0–100 %.
In fact, it does not need to be 404-errors, you can send any statusCode that you want to be set to the response of the segment requests.
Timeout error on segment
The stream corruptor service will simply not respond to certain segment requests, causing the request to timeout in the client.
Random delay on segments
In this case, the stream corruptor will add a random delay between 0–2 s to the segment requests.
Segment lost between encoder and origin
This corruption can be useful in a live stream scenario with SegmentTimeline mpd-format. It is a simulation of the error case that a segment has been lost between the encoder and origin, causing a gap in the manifest timeline.
Finding bad streams to test your player error handling is hard. It takes time to find or generate the streams you need.
The Eyevinn stream corruptor service can be used to apply corruptions and faults to dash manifests and segment requests in a controlled manner and on the fly.
The corruptions that are listed in this post are some use cases that we have come up with, that we think are quite common and likely to occur in the field at some point or another that we have implemented in this first version of the service.
If you have ideas or questions about this, your welcome to get in touch with feedback, questions or ideas.
This blog was written by an Eyevinn Special Video-Dev Team. Bring in an Eyevinn Special Video-Dev Team when you need a unique combined skillset with the experience from the entire video development domain, to add a feature to your software but lacks the right competence in-house or don’t have enough resources available to do it.
Eyevinn Technology is an independent consultant firm specialized in video and streaming. Independent in a way that we are not commercially tied to any platform or technology vendor.