Giter Club home page Giter Club logo

video-on-demand-on-aws-foundation's Introduction

Video on Demand on AWS Foundation

Deploy a customizable architecture to build a video-on-demand workflow.


About this solution

Video on Demand on AWS Foundation is a reference implementation that automatically provisions the Amazon Web Services (AWS) services necessary to build a scalable, distributed video-on-demand workflow.

Solution overview

We designed this solution to help you start encoding video files with AWS Elemental MediaConvert. You can customize and use this solution as the starting point to create a more complex workflow.

Out of the box, this solution helps you to accomplish the following:

  • Automatically transcode videos uploaded to Amazon Simple Storage Service (Amazon S3) into formats suitable for playback on a wide range of devices.
  • Customize MediaConvert job settings by uploading your own file and using different job settings for different inputs.
  • Store transcoded files in a destination bucket and use Amazon CloudFront to deliver to end viewers.
  • Manage costs, view logs, implement patching, and run automation runbooks for this solution from a central location.

In addition to the transcoded video, the outputs include input file metadata, job settings, and output details. These outputs are stored in a separate JSON file that can be used for further processing.

Benefits

Reference implementation - Leverage this solution as a reference implementation to automatically provision the AWS services necessary to build a scalable, distributed video-on-demand workflow.

Customization - Customize this solution and then use it as the starting point to create a more complex workflow.

Use cases

Streaming media - As consumer demand for video streaming increases, media and entertainment companies are looking for secure and reliable web-based video streaming alternatives to traditional television. This solution automatically provisions the services necessary to build a scalable, distributed architecture that ingests, stores, processes, and delivers video content. Using this solution, you can avoid inefficient trial-and-error approaches, and save on time and costs for your streaming media projects.

Educational content delivery - Professional development and educational initiatives create incentives and can be important revenue generators for nonprofit organizations. This solution can help you create modern, scalable content delivery and learning management systems to support your membership and programming offerings. The solution streamlines the processes for delivering online training and learning content.

Architecture overview

Architecture reference diagram

Deploying this solution with the default parameters deploys the following components in your AWS account. Architecture

Prerequisites

How to deploy the solution

  1. Sign in to the AWS Management Console and launch the video-on-demand-on-aws-foundation.template CloudFormation template, which is available on the solution home page: Video on Demand on AWS.
  2. The template launches in the US East (N. Virginia) Region by default. To launch the solution in a different AWS Region, use the Region selector in the console navigation bar.

For more detailed instructions, see the solution implementation guide.

Solution resources, post deployment

  • Source S3 bucket to store the source video files. The solution uploads a job-settings.json file, used to define the encoding settings for MediaConvert, to the source S3 bucket.
  • Destination S3 bucket to store the outputs from MediaConvert.
  • Job submit AWS Lambda function to create the encoding jobs in MediaConvert.
  • Job complete Lambda function to process the outputs.
  • Amazon CloudWatch to track encoding jobs in MediaConvert and invoke the Lambda job complete function.
  • An Amazon SNS topic to send notifications of completed jobs.
  • Amazon CloudFront configured with the destination S3 bucket as the origin for global distribution of the transcoded video content.

Creating a custom build

We developed this solution using the AWS Cloud Development Kit (CDK) and leveraging three of the AWS Solutions Constructs. To make changes to the solution:

  1. Download or clone this repo.
  2. Update the source code.
  3. Either deploy the solution using the CDK or run the deployment/build-s3-dist.sh script. The build script:
    1. Generates the CloudFormation template from the CDK source code using cdk synth.
    2. Runs deployment/cdk-solution-helper to update the template so that it pulls the Lambda source code from Amazon S3.
    3. Packages the Lambda code ready to be deployed to an Amazon S3 bucket in your account.

For details on deploying the solution using the CDK see the CDK Getting Started guide.

1. Run unit tests for customization

Run unit tests to ensure that your added customization passes the tests:

cd deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh

2. Create an S3 bucket

We configured the CloudFormation template to pull the Lambda deployment packages from an S3 bucket in the Region the template is being launched in. Create a bucket in the desired Region and append the Region name to the bucket name (for example, my-bucket-us-east-1).

aws s3 mb s3://my-bucket-us-east-1

3. Create the deployment packages

Build the distributable:

chmod +x ./build-s3-dist.sh
./build-s3-dist.sh my-bucket video-on-demand-on-aws-foundation v1.2.0

Note: The _build-s3-dist_ script expects the bucket name as one of its parameters, and this value should not include the Region suffix.

Deploy the distributable to the S3 bucket in your account:

aws s3 cp ./regional-s3-assets/ s3://my-bucket-us-east-1/video-on-demand-on-aws-foundation/v1.2.0/ \
    --recursive --acl bucket-owner-full-control

4. Launch the CloudFormation template

Deploy the CloudFormation template from deployment/global-assets/video-on-demand-on-aws-foundation.template into the same Region as your newly created S3 bucket.

Troubleshooting

The email address you provided when deploying this solution receives notifications both when MediaConvert jobs complete successfully and when they fail. The email address also receives notifications about errors that might have occurred while trying to submit a job or process the output from a job.

If you’re notified about a MediaConvert job failure, complete the following steps.

  1. From the main account where the solution is deployed, sign in to the AWS Elemental MediaConvert console.
  2. In the navigation pane, select Jobs.
  3. Select the job ID of the job that failed.
  4. On the Job Summary page, review the Overview section for an error message with more information on why the job failed. On this page, you can also find MediaConvert error codes for details on how to address the issue.

If the error is not a MediaConvert job failure, possibly one of the two Lambda functions, job_submit or job_complete, encountered an error. The email you received has an ErrorDetails link that takes you directly to the CloudWatch logs generated by the failed function. The logs have additional details on why it failed.

Note: When overriding the sample job-settings.json, we recommend exporting job settings from a MediaConvert job that’s successfully completed. Incorrect encoding settings will result in the job_submit Lambda function to fail.

How to uninstall the solution

You can uninstall this solution from the AWS Management Console or by using the AWS CLI. You must manually delete the S3 buckets and CloudWatch logs created by this solution. AWS Solutions do not automatically delete these resources in case you have stored data to retain.

For more detailed instructions, see the solution implementation guide.

Collection of operational metrics

This solution collects anonymized operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the implementation guide.

License information

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

video-on-demand-on-aws-foundation's People

Contributors

amazon-auto avatar colinazn avatar dch90 avatar eggoynes avatar ekielley avatar jimtharioamazon avatar sandimciin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

video-on-demand-on-aws-foundation's Issues

I don't have access to the cloudfront url

I have completely and without error implemented the cloudformation template from the guide indicated at: https://aws.amazon.com/es/solutions/implementations/video-on-demand-on-aws/, everything is correct, but at When I need to access the video with the cloudfront URL generated by the template, I get an access denied notice, so I consume the videos already transcoded in the S3 bucket, because I don't have access to the cloudfront distribution URL, Please if someone could help me. Leave any screnshots for a better understanding.
image
image
image

Can't make cloudfront accessible for public (as a test)

Now that I have figured out my video viewing error, I can't figure out why i can't view my cloudfront. I keep getting:

Error
CodeAccessDenied/Code
RequestID gibberish here /RequestId
Hostid long gibberish /HostId
/Error

I'm coping the various urls out of cloud front, and trying them through http: and https: (Had to take the <> out because it inserted it as code)

TIA

JobComplete lambda cannot logs:PutEvents

Describe the bug
While the JobComplete Lambda does include several helpful console.log() calls, there are no logs in Cloudwatch.

To Reproduce
Upload video, wait for job to complete, look up logs in CloudWatch

Expected behavior
Lambda output is available in Cloudwatch

Please complete the following information about the solution:

  • Version: 1.3.0
  • Region: euc1
  • Was the solution modified from the version published on this repository? Yes: disabled AnonymizedUsage, removed emailAddress parameter & SNS subscription, added several CFn Outputs (i.e. nothing that should affect this issue)
  • If the answer to the previous question was yes, are the changes available on GitHub? No
  • Have you checked your service quotas for the sevices this solution uses? n/a
  • Were there any errors in the CloudWatch Logs? No 🙃

Screenshots
n/a

Additional context
The Lambda execution IAM Role does not include permission to logs:PutEvents (via AWSLambdaBasicExecutionRole or otherwise). Adding that permission fixes the issue.

Test script fails on vanilla clone

If I clone the workflow and run the testscript I receive the following error message:

  console.log
    TypeError: Cannot read property '0' of undefined
        at Object.sendMetrics (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/lib/utils.js:193:28)
        at Object.<anonymous> (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/lib/utils.spec.js:180:17)
        at Object.asyncJestTest (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:106:37)
        at /home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:45:12
        at new Promise (<anonymous>)
        at mapper (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:28:19)
        at /home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:75:41
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

      at Object.sendMetrics (lib/utils.js:232:17)

  console.log
    TypeError: Cannot read property '0' of undefined
        at Object.sendMetrics (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/lib/utils.js:193:28)
        at Object.<anonymous> (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/lib/utils.spec.js:184:17)
        at Object.asyncJestTest (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:106:37)
        at /home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:45:12
        at new Promise (<anonymous>)
        at mapper (/home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:28:19)
        at /home/christoph/Desktop/cf_vod_workflow.git/source/job-complete/node_modules/jest-jasmine2/build/queueRunner.js:75:41
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

      at Object.sendMetrics (lib/utils.js:232:17)

Furthermore, regardless of what I do (I explicitly deleted job-settings.json) I always see the settings from that job-settings.json in the console output.

What is the problem here?

Best,
Ck

SNS notification missing InputFile and InputDetails – sometimes (race condition)

Describe the bug
Sometimes, the SNS notification is missing details on the job input (InputFile and InputDetails) – e.g.:

{
 "Id": "foo",
 "InputDetails": {},
 "Outputs": { ... }
}

To Reproduce
Upload video files repeatedly; due to the nature of the race condition (see below), it should trigger easily with very short video files.

Expected behavior
Notifications always contain input data.

Please complete the following information about the solution:

  • Version: 1.3.0
  • Region: euc1
  • Was the solution modified from the version published on this repository? Yes (cf. #29)
  • If the answer to the previous question was yes, are the changes available on GitHub? No
  • Have you checked your service quotas for the sevices this solution uses? n/a
  • Were there any errors in the CloudWatch Logs? No

Screenshots
n/a

Additional context
The whole logic around jobs-manifest.json (JM) seems to invite different kinds of race condition; most relevant for this issue:

  • event with status INPUT_INFORMATION triggers
    • read JM
    • append data
    • write JM
  • event with status COMPLETE triggers
    • read JM
    • use information from JM for input details

If both events fire in quick succession (more likely for short/fast jobs), the COMPLETE may read the JM before input data was written to it, triggering the "no entry found" if block, which sets InputDetails to an empty object and does not set InputFile at all:

const index = manifest.Jobs.findIndex(job => job.Id === jobDetails.Id);
if (index === -1) {
console.log(`no entry found for jobId: ${jobDetails.Id}, creating new entry`);
jobDetails.InputDetails = {};
manifest.Jobs.push(jobDetails);
results = jobDetails;
} else {

The missing data could be filled in from a mediaconvert:GetJob call (that is happening anyway).

Suspected additional issue
I have not tested this, but from reading the code, it seems highly likely that concurrent processing of jobs leads to missing entries in the JM (data written by one Lambda may be overwritten by a second Lambda that has read JM before the write).

The CDK deployment method fails with `'%%VERSION%%' is not a valid value for TagValue`

Describe the bug

The README.md describes two possible ways to deploy the solution:

3. Either deploy the solution using the CDK or run the `deployment/build-s3-dist.sh` script. The

However, since %%VERSION%% was moved directly into the source code in ca3c9c1, creating stack with CDK fails with this message:

 Resource handler returned message: "'%%VERSION%%' is not a valid value for TagValue - it contains illegal characters (Service: ServiceCatalogAppRegistry, Status Code: 400"

The %%VERSION%% variable is only replaced by the build-s3-dist.sh script, which doesn't run if one chooses the CDK deploy method, hence the error.

To Reproduce
Clone the repo, build the app, deploy with cdk deploy from the ./source/cdk folder.

Version:
I only tested with 1.3.2, however, if I determined the cause correctly, the version introduced the bug is 1.2.1.

Support for Israel(Tel Aviv) Region "il-central-1"

The solution deployment is failing in Israel(Tel Aviv) Region "il-central-1"
Version: 1.3.2
Released: 11/2023

Additional context
Resource handler returned message: "Access denied for operation 'AWS::CloudFront::Distribution: You don't have permission to access the S3 bucket for CloudFront logs: tlv-video-on-demand2-logs6819bb44-hi0kc3kp3sgr.s3.il-central-1.amazonaws.com If you're using IAM, you need s3:GetBucketAcl and s3:PutBucketAcl permissions to create a distribution or to update log settings for an existing distribution. In addition, the S3 ACL for the bucket must grant you FULL_CONTROL. (Service: CloudFront, Status Code: 403, Request ID: 9055e875-32a7-47b2-8bb6-3c2755592765)'." (RequestToken: 6edcdc85-1361-d6a9-8dd3-6fd77c8752a4, HandlerErrorCode: AccessDenied)

Screenshot 2023-12-24 at 13 56 49

Error when trying to submit job

Describe the bug
A clear and concise description of what the bug is.

2022-10-10T05:29:05.307Z	4cc35dea-e300-4ec3-9008-40971534614c	ERROR	Invoke Error 	
{
    "errorType": "UnexpectedParameter",
    "errorMessage": "Unexpected key 'Name' found in params",
    "code": "UnexpectedParameter",
    "message": "Unexpected key 'Name' found in params",
    "time": "2022-10-10T05:29:04.809Z",
    "stack": [
        "UnexpectedParameter: Unexpected key 'Name' found in params",
        "    at ParamValidator.fail (/var/runtime/node_modules/aws-sdk/lib/param_validator.js:50:37)",
        "    at ParamValidator.validateStructure (/var/runtime/node_modules/aws-sdk/lib/param_validator.js:77:14)",
        "    at ParamValidator.validateMember (/var/runtime/node_modules/aws-sdk/lib/param_validator.js:88:21)",
        "    at ParamValidator.validate (/var/runtime/node_modules/aws-sdk/lib/param_validator.js:34:10)",
        "    at Request.VALIDATE_PARAMETERS (/var/runtime/node_modules/aws-sdk/lib/event_listeners.js:132:42)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at callNextListener (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:96:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/event_listeners.js:86:9",
        "    at finish (/var/runtime/node_modules/aws-sdk/lib/config.js:396:7)",
        "    at /var/runtime/node_modules/aws-sdk/lib/config.js:414:9"
    ]
}

To Reproduce

  • spin up CF stack
  • upload this Job Settings file (unsure if any impact)

job-settings.txt

  • upload a mov to source-bucket/assets01/
  • no file created, no job submitted, stack trace above

Expected behavior
A clear and concise description of what you expected to happen.

Job is submitted and video files created

Please complete the following information about the solution:

  • [1.1.0 ] Version: [e.g. v1.0.0]
  • [ap-southeast-2] Region: [e.g. us-east-1]
  • [Yes] Was the solution modified from the version published on this repository?
  • [Yes] If the answer to the previous question was yes, are the changes available on GitHub?
  • [Yes] Have you checked your service quotas for the sevices this solution uses?
  • [Yes] Were there any errors in the CloudWatch Logs?

Additional context
Not sure if there was any API changes recently that might have broken the default source code. Any help appreciated.

SNS notifications do not include input details for some or all job inputs

VOD Foundations relies on MediaConvert's INPUT_INFORMATION cloudwatch event to return details for all the inputs used in a job. However, In MediaConvert, INPUT_INFORMATION is not guaranteed to return before COMPLETED events from cloudwatch for quick jobs. This means that in some cases the job may complete and SNS notification be sent without including job input information.

https://app.asana.com/0/1200382082298559/1200470174838695/f

Failed to download and validate the job-settings.json file

Currently job-submit lambda expects a <uploaded_video>/job-settings.json file. But the solution uploads a default job-settings.json inside the assets01 directory.

This is the error that I am getting

{
  "Details": "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=/aws/lambda/vod-jobSubmitB391E42F-1O4CG56DZQQ8Z",
  "Error": {
    "Message": "Failed to download and validate the job-settings.json file. Please check its contents and location. Details  on using custom settings: https://github.com/awslabs/video-on-demand-on-aws-foundations",
    "Error": "AccessDenied: Access Denied"
  }
}

Add WMV file extension

I would want to launch the workflow with WMV files.
We just need to add .wmv as an event notification of the source AWS S3 bucket

`jobSubmit` alters `Destination`, regardless of what is defined in `job-settings.json`

Describe the bug
A solution documentation states that the jobSubmit lambda function applies settings defined in job-settings.json and submits a job to AWS MediaConvert.
Reality is different, and the jobSubmit lambda function alters settings and overrides the destination of the resulting files.

To Reproduce

  1. Set up the solution via AWS CloudFormation template
  2. Alter job-settings.json file to change Destination setting (for example, to destinationbucket/videos/$fn$). $fn$ here stands for a defined variable.
  3. Upload video and see that the processed files were uploaded to destinationbucket/guid/AppleHLS1/ regardless of what was defined above.

Expected behavior
Resulting files uploaded to destinationbucket/videos/$fn$

Please complete the following information about the solution:

  • Version: v1.3.0
  • Region: eu-central-1
  • Was the solution modified from the version published on this repository? // Modified job-settings.json
  • If the answer to the previous question was yes, are the changes available on GitHub? // Not available, but the example is defined right above
  • Have you checked your service quotas for the sevices this solution uses? // No issues with quotas
  • Were there any errors in the CloudWatch Logs? // No errors in the logs

Is this file "jobs-manifest.json" really needed in production?

I found that each job ends up writing the results to the same file "jobs-manifest.json".

Is this likely to cause conflicts when there are a large number of jobs or to overstretch the file size over time?
Is there any other solution to record the output or can I simply remove this part of the code in production?

Error validate the Job.setting.json

I have problems when uploading the video in the Input Bucket, I receive a notification in my email indicating that the download of the file failed, I leave the specific message:

"Mistake": {
"Message": "Failed to download and validate the job-settings.json file. Please check its contents and location. Details on using custom settings: https://nam10.safelinks.protection.outlook.com/?url=https% 3D%7C3000%7C%7C%7C&sdata=QFshicn4hdr3fy9JCYHt5GjrxDzkM7mfInJ7KrcbZfY%3D&reserved=0",
"Error": "AccessDenied: Access Denied"

Request is blocked by CORS policy No 'Access-Control-Allow-Origin' header is present on the requested resource.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Pick up output name of video play list.. and

Access to XMLHttpRequest at 'https://xxxx.cloudfront.net/distribution/AppleHLS1/playlist.0Mbps_qvbr.m3u8' from origin 'http://127.0.0.1:5500' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Chrome Browser

Expected behavior
A clear and concise description of what you expected to happen.
Expected
Please complete the following information about the solution:

  • [ 1.1.0] Version: [e.g. v1.0.0]
  • [us-east-1 ] Region: [e.g. us-east-1]
  • [N ] Was the solution modified from the version published on this repository?
  • If the answer to the previous question was yes, are the changes available on GitHub?
  • [ N] Have you checked your service quotas for the sevices this solution uses?
  • [N ] Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context
Add any other context about the problem here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.