Transcribe with Open AI Whisper and Open Source Cloud

We have in previous blog posts described how we can transcode and create a streaming package with the open source projects we have made available in Open Source Cloud. In this post we will describe how we can automatically transcribe and create subtitle files from the video using open source and Open AI Whisper.

So let us recap a bit and start by transcoding and packaging a video file, in this example a promotion video for Streaming Tech Sweden.

AWS_ACCESS_KEY_ID="{{secrets.awsaccesskeyid}}" \
AWS_SECRET_ACCESS_KEY="{{secrets.awssecretaccesskey}}" \
osc transcode -d 60 \
https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4 \
s3://lab-testcontent-store/birme/ \
s3://lab-testcontent-store/birme/autosubs/ 

We store the transcoded ABR bundle in a temporary bucket and the output will be available in a subfolder called autosubs. We choose to transcode only 60 seconds of the video as it will be sufficient for this example. When this process is completed we will have an HLS stream available at https://testcontent.eyevinn.technology/birme/autosubs/index.m3u8 and we can try playing it in an HLS video player.

To improve the availability of this content we want to add subtitles, and one automatic way of doing that is by using the open source subtitle generator available in Open Source Cloud. This generator uses Open AI Whisper for the automatic transcription and prepares the input to a media format that Open AI Whisper can handle.

With this service we can pass both video and audio for transcription, and in this case we will even use the HLS stream we just created as input. As this service uses the API for Open AI Whisper we need to provide an API key. The recommended way is to store this API key and other sensitive data as a secret. You create a secret by going to the tab Service Secrets on the service page in the Open Source Cloud web user interface.

We will create a secret that we call openaikey and store value of the API key we have acquired. To refer to a secret we enter the variable substitution {{secrets.openaikey}} as the environment variable instead of the actual value.

To transcribe we use the OSC command line interface.

OPENAI_API_KEY="{{secrets.openaikey}}" \
osc transcribe https://testcontent.eyevinn.technology/birme/autosubs/index.m3u8

When the transcription is completed the command returns with subtitles in VTT format.

WEBVTT

00:00:00.000 --> 00:00:03.300
Probably the most interesting challenge

00:00:03.300 --> 00:00:04.900
of the 21st century.

00:00:04.900 --> 00:00:07.320
Streaming-Tech TV Plus.

00:00:07.320 --> 00:00:08.540
Money.

00:00:08.540 --> 00:00:10.160
Money, money, money, money.

00:00:10.160 --> 00:00:14.440
Great Streaming-Tech talks curated from all over the world.

00:00:14.440 --> 00:00:18.260
They said they would prefer blur if I was going like this.

00:00:18.260 --> 00:00:20.660
Somewhere, somehow, there is a limit.

00:00:20.660 --> 00:00:24.160
Paul Ivins Streaming-Tech Sweden events.

...

If we instead wanted the output in another subtitle format such as SRT we can specify that as an option.

OPENAI_API_KEY="{{secrets.openaikey}}" \
osc transcribe -f srt \
https://testcontent.eyevinn.technology/birme/autosubs/index.m3u8

and the output will be:

1
00:00:00,000 --> 00:00:05,120
Probably the most interesting challenge of the 21st century.

2
00:00:05,120 --> 00:00:06,520
The Streaming Tech TV Plus

3
00:00:06,520 --> 00:00:08,860
Money.

4
00:00:08,860 --> 00:00:10,020
Money money money money.

5
00:00:10,020 --> 00:00:13,540
Great streaming tech talks curated from all over the world.

6
00:00:13,540 --> 00:00:18,379
They said they would prefer Blur if I was going like this

...

By default it will assume that the spoken language is english and if we know that it is another language we can specify that for the subtitle generator using the option -l.

We can modify and correct the subtitle before we package it together with the HLS stream. How we do that is another blog post so stay tuned for more examples of what you can do with Open Source Cloud!

What is Open Source Cloud

A software as a service based on open source with a unique transparent model where revenue is shared with the open source authors. Open Source Cloud offers media industry companies a quick way to incorporate open source in their solutions and the option to run the same software in-house as the source code is publicly available.

• API and CLI documentation: https://api.osaas.io
• Javascript Client Library reference documentation: https://js.docs.osaas.io/