Create VOD package from a video file on an S3 bucket with Open Source Cloud

In this blog we will describe how you can transcode and create a VOD package when a file is uploaded to an AWS S3 bucket using open source made available as services in Eyevinn Open Source Cloud. For creating the VOD package we will use the open source transcoder SVT Encore (based on ffmpeg) and Encore Packager (based on Shaka Packager) available in Open Source Cloud.

What is Eyevinn Open Source Cloud?

We developed and launched Open Source Cloud to reduce the barrier to getting started with open source and at the same time contribute to a sustainable model for open source by giving back a share of the revenue to the creator.

Setup services in Open Source Cloud

In Open Source Cloud we will setup and configure the transcoding service SVT Encore, the Encore Packager service, the Encore Callback Listener service and the packaging queue. A job is created using the REST API endpoint provided by SVT Encore and job progress is posted to the Encore Callback Listener endpoint. When a transcoding job is finished the Callback Listener will place an item on the packaging queue (Valkey key/value store) that the Encore Packager will pick up packaging jobs from. The created VOD package is then uploaded to another S3 bucket. How to setup this pipeline is described in detail on the Open Source Cloud blog.

Lambda Function to Submit Jobs

When the transcoding and packaging services are configured in Open Source Cloud we will develop an AWS Lambda function that is triggered when a file is created on the S3 bucket.

Create this Lambda function in the AWS console that will parse the received event and generate a pre-signed URL to the new file on the S3 bucket.

import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";

const client = new S3Client();

export const handler = async (event, context) => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));

  try {
    const command = new GetObjectCommand({
      Bucket: bucket,
      Key: key,
    });
    const presignedUrl = await getSignedUrl(client, command, { expiresIn: 60 * 60 * 4 });
    console.log('The presigned URL is:', presignedUrl);
  } catch (err) {
      console.log(err);
      const message = `Error getting object ${key} from bucket ${bucket}. Make sure they exist and your bucket is in the same region as this function.`;
      console.log(message);
      throw new Error(message);
  }
};

Add a function for obtaining an access token for the SVT Encore service in Open Source Cloud. The OSC_ACCESS_TOKEN is the API key you find in the Open Source Cloud web console.

async function getServiceAccessToken() {
  const url = new URL('https://token.svc.prod.osaas.io/servicetoken');
  const response = await fetch(url, {
    method: 'POST',
    body: JSON.stringify({ serviceId: 'encore' }),
    headers: {
      'x-pat-jwt': `Bearer ${process.env.OSC_ACCESS_TOKEN}`,
      'Content-Type': 'application/json'
    }
  });
  if (response.ok) {
    const serviceAccessToken = await response.json();    
    console.log(serviceAccessToken);
    return serviceAccessToken.token;
  }
}

Next thing is to add a function for submitting the job to SVT Encore in OSC. Replace the URL to your SVT Encore instance and the URL for the callback listener you created.

import { basename, extname } from 'node:path';

async function submitJob(sat, url) {
  const encoreUrl = new URL('/encoreJobs', 'https://eyevinn-vod.encore.prod.osaas.io');
  const baseName = basename(url.pathname, extname(url.pathname)).replace(/\s+/, '')
  const response = await fetch(encoreUrl, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${sat}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      externalId: baseName,
      profile: 'program',
      outputFolder: '/usercontent/',
      baseName,
      progressCallbackUri: 'https://eyevinn-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback',
      inputs: [
        {
          uri: url.toString(),
          copyTs: true,
          type: 'AudioVideo'
        }
      ]
    })
  });
  if (response.ok) {
    const job = await response.json();
    console.log(job);
    return `https://lab.cdn.eyevinn.technology/osc/${baseName}/${job.id}/index.m3u8`;
  }
  return undefined;
}

This function returns the CDN URL where the player can access the VOD file and something you want to change to reflect your setup.

Now we can put this together and update the handler.

export const handler = async (event, context) => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));

  try {
    const command = new GetObjectCommand({
      Bucket: bucket,
      Key: key,
    });
    const presignedUrl = await getSignedUrl(client, command, { expiresIn: 60 * 60 * 4 });
    console.log('The presigned URL is:', presignedUrl);
    const sat = await getServiceAccessToken();
    console.log(sat);
    if (sat) {
      const vodUrl = await submitJob(sat, new URL(presignedUrl));
      console.log(vodUrl);
      return vodUrl;
    } else {
      console.log('Unable to obtain service access token');
    }
  } catch (err) {
      console.log(err);
      const message = `Error getting object ${key} from bucket ${bucket}. Make sure they exist and your bucket is in the same region as this function.`;
      console.log(message);
      throw new Error(message);
  }
};

Conclusion

Now you have a setup for creating VOD packages from files on an S3 bucket that utilizes the open source transcoder SVT Encore available as a service in Eyevinn Open Source Cloud.

For more information and to create an account on Open Source Cloud visit www.osaas.io