Building a VOD preparation pipeline with SVT Encore in Open Source Cloud
In this blog post I will give an example on how you with Open Source Cloud client libraries can build a VOD preparation pipeline with SVT Encore in Open Source Cloud.
One of the first steps in a media supply chain for VOD distribution is to transcode the source media file into multiple files with different sizes and resolutions. We will refer to this set of files as an ABR bundle in this text.
Next steps are to create the streaming files from the ABR bundle and then transfer the streaming files to a CDN origin.
[source media] => (SVT Encore) => [ABR bundle] => (Packager) => [HLS/MPD]The scope of this blog is focused on creating the ABR bundle and converting it into streaming files are excluded.
Prerequisites
It is assumed that you already have signed up to an account on Open Source Cloud and that you are on a subscription plan where you can have more than one active service at a time. In this example we will also transfer the ABR bundle to an AWS S3 bucket so in order to try it out yourself that is also needed.
In this example we will use the client libraries for NodeJS so you need to have a working installation of Node as well.
Initiate a new project
Create a new Node project using npm.
npm initWhat you enter here is not important for the purpose of this example. Then we will install the client library.
npm install --save @osaas/client-coreCreate a file called transcode.js with the following content.
const { Context } = require('@osaas/client-core');
async function main() {
try {
const ctx = new Context();
} catch (err) {
console.log(err.message);
}
}
main();Setup an Open Source Cloud context
What this code will do is to initiate and setup an Open Source Cloud context. A context holds the configuration for accessing Open Source Cloud and it reads the personal access token from the environment variable OSC_ACCESS_TOKEN . The Personal Access Token is found under the Settings menu in the Open Source Cloud web user interface.
We then set this environment variable with this command.
export OSC_ACCESS_TOKEN=<personal-access-token>It is very important that you never have this token stored in code or a file in your repository. Once this is set we can try our script.
node transcode.jsNothing will actually happen but we should not see any errors if everything is correct. For example if the access token is not found in the environment the result would be this.
Personal access token is required to create a context. Please provide it
in the config or set the OSC_ACCESS_TOKEN environment variable.Launch an Encore instance
Now it is time to add some code to launch an Encore instance in Open Source Cloud. To be able to do that you first need to create a temporary Service Access Token.
const sat = await ctx.getServiceAccessToken('encore');Then we can create the instance.
const instance = await createInstance(
ctx,
'encore',
sat,
{
name: 'example'
}
);
await delay(5000);Before we continue we add a 5 second delay to ensure that the instance has been created. The delay() function is implemented as below.
const delay = (ms) => new Promise((res) => setTimeout(res, ms));Create a job
When we have an instance running we can add a job to the Encore queue. We will transcode the source file stswe-tvplus-promo.mp4
const source =
new URL('https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4');
const jobId = Math.random().toString(36).substring(7);
const encoreJobUrl = new URL('/encoreJobs', instance.url);
const data = await createFetch(encoreJobUrl, {
method: 'POST',
body: JSON.stringify({
profile: 'program',
outputFolder: `/usercontent/${jobId}`,
baseName: jobId,
inputs: [
{
type: 'AudioVideo',
uri: source.toString()
}
]
}),
headers: {
'x-jwt': `Bearer ${sat}`,
'Content-Type': 'application/json'
}
});
const encoreJob = JSON.parse(data);Wait for the job to be completed
Now we have a job put on the queue and once it is picked up it will be processed. In this example we will wait for the job to be completed by polling the job resource endpoint.
Let us develop a simple function that does this.
const MAX_ITER = 10000;
async function waitForEncoreJobToComplete(jobUrl, token) {
for (const _ of Array(MAX_ITER)) {
const data = await createFetch(jobUrl, {
method: 'GET',
headers: {
'x-jwt': `Bearer ${token}`,
'Content-Type': 'application/json'
}
});
const job = JSON.parse(data);
if (job.status === 'SUCCESSFUL') {
return job;
}
await delay(10000);
}
}Then we add that we will wait for the job to be completed before we continue.
const job = await waitForEncoreJobToComplete(
new URL(encoreJobUrl.toString() + '/' + encoreJob.id), sat
);
console.log(job);And the output of this script would be for example:
...
output: [
{
file: '/usercontent/p0brqt/p0brqt_x264_3100.mp4',
fileSize: 1993937,
format: 'MPEG-4',
overallBitrate: 3190299,
duration: 5,
videoStreams: [Array],
audioStreams: [Array],
type: 'VideoFile'
},
{
file: '/usercontent/p0brqt/p0brqt_x264_2069.mp4',
fileSize: 1323219,
format: 'MPEG-4',
overallBitrate: 2117150,
duration: 5,
videoStreams: [Array],
audioStreams: [Array],
type: 'VideoFile'
},
{
file: '/usercontent/p0brqt/p0brqt_x264_1312.mp4',
fileSize: 840253,
format: 'MPEG-4',
overallBitrate: 1344404,
duration: 5,
videoStreams: [Array],
audioStreams: [Array],
type: 'VideoFile'
},
{
file: '/usercontent/p0brqt/p0brqt_x264_806.mp4',
fileSize: 541488,
format: 'MPEG-4',
overallBitrate: 866380,
duration: 5,
videoStreams: [Array],
audioStreams: [Array],
type: 'VideoFile'
},
{
file: '/usercontent/p0brqt/p0brqt_x264_324.mp4',
fileSize: 265367,
format: 'MPEG-4',
overallBitrate: 424587,
duration: 5,
videoStreams: [Array],
audioStreams: [Array],
type: 'VideoFile'
},
{
file: '/usercontent/p0brqt/p0brqt_STEREO.mp4',
fileSize: 84651,
format: 'MPEG-4',
overallBitrate: 135441,
duration: 5,
audioStreams: [Array],
type: 'AudioFile'
},
...
]
...This contains a list of all the created files and this constitutes our ABR bundle. You can try to download a file with the following command.
curl -H 'x-jwt: <sat>' <instance.url>/usercontent/p0brqt/p0brqt_x264_324.mp4The <instance.url> is the URL to the Encore instance and <sat> is the service access token that is required.
Putting it all together
Adding this all together we would end up with this file.
const { Context, createInstance, createFetch } = require('@osaas/client-core');
const delay = (ms) => new Promise((res) => setTimeout(res, ms));
const MAX_ITER = 10000;
async function waitForEncoreJobToComplete(jobUrl, token) {
for (const _ of Array(MAX_ITER)) {
const data = await createFetch(jobUrl, {
method: 'GET',
headers: {
'x-jwt': `Bearer ${token}`,
'Content-Type': 'application/json'
}
});
const job = JSON.parse(data);
if (job.status === 'SUCCESSFUL') {
return job;
}
await delay(10000);
}
}
async function main() {
try {
const ctx = new Context();
const sat = await ctx.getServiceAccessToken('encore');
const instance = await createInstance(
ctx,
'encore',
sat,
{
name: 'example'
}
);
await delay(5000);
const source = new URL('https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4');
const jobId = Math.random().toString(36).substring(7);
const encoreJobUrl = new URL('/encoreJobs', instance.url);
const data = await createFetch(encoreJobUrl, {
method: 'POST',
body: JSON.stringify({
profile: 'program',
outputFolder: `/usercontent/${jobId}`,
baseName: jobId,
inputs: [
{
type: 'AudioVideo',
uri: source.toString()
}
]
}),
headers: {
'x-jwt': `Bearer ${sat}`,
'Content-Type': 'application/json'
}
});
const encoreJob = JSON.parse(data);
const job = await waitForEncoreJobToComplete(
new URL(encoreJobUrl.toString() + '/' + encoreJob.id),
sat
);
console.log(job);
} catch (err) {
console.log(err.message);
}
}
main();Transfer ABR bundle to an S3 bucket
The generated ABR bundle are now temporarily stored in the Encore instance but as soon as the instance is removed the bundle will be lost. What we want to do is to transfer this bundle to a more persistent place, for example an S3 bucket. We will now extend our code with this functionality.
To accomplish that we will use another service available in Open Source Cloud that is called Retransfer. With this service you can create a job that transfer a file from one location to another location.
Let us create a function that creates a Retransfer job in Open Source Cloud.
const { waitForJobToComplete, createJob } = require('@osaas/client-core');
async function transferFile(ctx, name, source, destination, token) {
const serviceAccessToken = await ctx.getServiceAccessToken(
'eyevinn-docker-retransfer'
);
const job = await createJob(
ctx,
'eyevinn-docker-retransfer',
serviceAccessToken,
{
name,
awsAccessKeyId: process.env.AWS_ACCESS_KEY_ID,
awsSecretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
cmdLineArgs: `--sat ${token} ${source.toString()} ${destination.toString()}`
}
);
await waitForJobToComplete(
ctx,
'eyevinn-docker-retransfer',
job.name,
serviceAccessToken
);
}Now we can call this function once the transcode job is completed for all video and audio files in the bundle.
const outputFiles = job.output.filter(
(file) =>
file.type === 'VideoFile' || file.type === 'AudioFile'
);
const transferPromises = outputFiles.map((file) =>
transferFile(
ctx,
path.basename(file.file),
new URL(file.file, instance.url),
destination,
sat
)
);
await Promise.all(transferPromises); Transcode Client Library
This hopefully gives you an idea and inspiration of what you can do with the client libraries to Open Source Cloud. This particular use case is also provided in an open source client library for transcoding with Open Source Cloud. Following example written in Typescript.
import { Context, Log } from '@osaas/client-core';
import { QueuePool } from '@osaas/client-transcode';
async function main() {
const ctx = new Context();
try {
const pool = new QueuePool({ context: ctx });
await pool.init();
await pool.transcode(
new URL(
'https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4'
),
new URL('s3://lab-testcontent-store/birme/'),
{
duration: 5
}
);
await pool.destroy();
} catch (err) {
Log().error(err);
}
}
main();Thank you for your time!
/Jonas Birmé VP R&D Eyevinn Technology
What is Open Source Cloud
A software as a service based on open source with a unique transparent model where revenue is shared with the open source authors. Open Source Cloud offers media industry companies a quick way to incorporate open source in their solutions and the option to run the same software in-house as the source code is publicly available.
