Completing the VOD pipeline with Shaka Packager in Open Source Cloud

4 min readJun 2, 2024

In last blog post we had a detailed walk-through on how you with SVT Encore in Open Source Cloud can build first part of a VOD pipeline. In this blog post we will use the recent addition to Open Source Cloud, the open source packager Shaka Packager, to complete this pipeline.

Using the Command Line Interface

Based on the Open Source Cloud Javascript library that we used in the last blog post we also provide a Command Line Interface that you can use to manage service instances and jobs in Open Source Cloud. The CLI is a Node script and can be installed with for example NPM.

npm install -g @osaas/cli

With this tool you can for example list your running instances and given that you have set the environment variable OSC_ACCESS_TOKEN with your personal access token you can run this command.

osc list eyevinn-test-adserver

This will output a list of your running test-adserver instances.

To create an SVT Encore instance you could then run:

osc create encore queue1

This will create an SVT Encore instance with the name queue1

And to remove it run the command

osc remove encore queue1

Transcode a file and place the ABR bundle on an S3 bucket

You can use this CLI to achieve the same thing as we accomplished in the last blog post by running the command.

osc transcode \
  https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4 \
  s3://abr-bucket/myvideo/

This will transcode the file stswe-tvplus-promo.mp4 and place the created ABR bundle on the bucket s3://abr-bucket/myvideo/ which is basically what the code we wrote in the last blog post did.

import { Context, Log } from '@osaas/client-core';
import { QueuePool } from '@osaas/client-transcode';

async function main() {
  const ctx = new Context();

  try {
    const pool = new QueuePool({ context: ctx });
    await pool.init();
    await pool.transcode(
      new URL(
        'https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4'
      ),
      new URL('s3://abr-bucket/myvideo/'),
      { }
    );
    await pool.destroy();
  } catch (err) {
    Log().error(err);
  }
}

main();

Transcode and create a streaming package

To add the completing step to create a streaming package of the transcoded ABR files you run the command.

osc transcode \
  https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4 \
  s3://abr-bucket/myvideo/ \
  s3://vod-bucket/myvideo/

Then the ABR bundle is placed on the S3 bucket s3://abr-bucket/myvideo and the HLS and MPEG-DASH manifest, and media segments on s3://vod-bucket/myvideo/

Using the Client Library

If we want to perform the same thing but with code we can use the Client library to accomplish the same result. Let us start with the following code that we developed in the last blog post.

const { Context, createInstance, createFetch } = require('@osaas/client-core');

const delay = (ms) => new Promise((res) => setTimeout(res, ms));
const MAX_ITER = 10000;

async function waitForEncoreJobToComplete(jobUrl, token) {
  for (const _ of Array(MAX_ITER)) {
    const data = await createFetch(jobUrl, {
      method: 'GET',
      headers: {
        'x-jwt': `Bearer ${token}`,
        'Content-Type': 'application/json'
      }
    });
    const job = JSON.parse(data);
    if (job.status === 'SUCCESSFUL') {
      return job;
    }
    await delay(10000);
  }
}

async function main() {
  try {
    const ctx = new Context();

    const sat = await ctx.getServiceAccessToken('encore');
    const instance = await createInstance(
      ctx,
      'encore',
      sat,
      {
        name: 'example'
      }
    );
    await delay(5000);

    const source = new URL('https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4');
    const jobId = Math.random().toString(36).substring(7);
    const encoreJobUrl = new URL('/encoreJobs', instance.url);
    const data = await createFetch(encoreJobUrl, {
      method: 'POST',
      body: JSON.stringify({
        profile: 'program',
        outputFolder: `/usercontent/${jobId}`,
        baseName: jobId,
        inputs: [
          {
            type: 'AudioVideo',
            uri: source.toString()
          }
        ]
      }),
      headers: {
        'x-jwt': `Bearer ${sat}`,
        'Content-Type': 'application/json'
      }
    });
    const encoreJob = JSON.parse(data);
    const job = await waitForEncoreJobToComplete(
      new URL(encoreJobUrl.toString() + '/' + encoreJob.id),
      sat
    );
    console.log(job);
    const outputFiles = job.output.filter(
      (file) =>
        file.type === 'VideoFile' || file.type === 'AudioFile'
    );
    const transferPromises = outputFiles.map((file) =>
      transferFile(
        ctx,
        path.basename(file.file),
        new URL(file.file, instance.url),
        destination,
        sat
      )
    );
    await Promise.all(transferPromises);    
  } catch (err) {
    console.log(err.message);
  }
}

main();

The above code will start an SVT Encore instance, enqueue a transcode job and transfer the result to an S3 bucket. We will then add that after the transfer job is completed we will start a packager job.

import { createStreamingPackage } from '@osaas/client-transcode';
// ...
// Get the list of video files in the ABR bundle that was created
const videos = outputFiles
  .filter((file) => file.type === 'VideoFile')
  .map((file) => path.basename(file.file));
// Get the audio file in the ABR bundle that was created
const audio = outputFiles.find((file) => file.type === 'AudioFile')?.file;
if (videos && audio) {
  await createStreamingPackage(
    ctx,
    destination,
    videos,
    path.basename(audio),
    new URL('s3://vod-bucket/myvideo/')
  );
}

The option to create a streaming package has also been added to the transcode() function in the Transcode Client Library so you can simply do this instead.

import { Context, Log } from '@osaas/client-core';
import { QueuePool } from '@osaas/client-transcode';

async function main() {
  const ctx = new Context();

  try {
    const pool = new QueuePool({ context: ctx });
    await pool.init();
    await pool.transcode(
      new URL(
        'https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4'
      ),
      new URL('s3://abr-bucket/myvideo/'),
      { packageDestination: new URL('s3://vod-bucket/myvideo/') }
    );
    await pool.destroy();
  } catch (err) {
    Log().error(err);
  }
}

main();

That’s it for this time but stay tuned for more examples of what you can do with Open Source Cloud!

/Jonas Birmé VP R&D Eyevinn Technology

What is Open Source Cloud

A software as a service based on open source with a unique transparent model where revenue is shared with the open source authors. Open Source Cloud offers media industry companies a quick way to incorporate open source in their solutions and the option to run the same software in-house as the source code is publicly available.

API and CLI documentation: https://api.osaas.io
Javascript Client Library reference documentation: https://js.docs.osaas.io/