Skip to content

Commit

Permalink
Polls stream action. Cleanup lambda. SDK upgrades
Browse files Browse the repository at this point in the history
  • Loading branch information
mboulin committed Jul 21, 2023
1 parent db628f8 commit 4909433
Show file tree
Hide file tree
Showing 95 changed files with 6,130 additions and 800 deletions.
70 changes: 68 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,12 @@ Authenticated viewers are able to read and send messages. Unauthenticated users

#### Stream overlays

Streamers can trigger various stream overlays: host a quiz, feature a product, feature an Amazon product, show a notice and trigger a celebration. More information on how overlays are triggered by streamers is available here: [Trigger overlays](#stream-overlay-configuration).
Streamers can trigger various stream overlays: host a quiz, host a poll, feature a product, feature an Amazon product, show a notice and trigger a celebration. More information on how overlays are triggered by streamers is available here: [Trigger overlays](#stream-overlay-configuration).

![Quiz action](screenshots/features/action-quiz.png)

![Poll action](screenshots/features/action-poll.png)

![Product action](screenshots/features/action-product.png)

![Amazon Product action](screenshots/features/action-amazon-product.png)
Expand All @@ -94,10 +96,16 @@ Streamers can trigger various stream overlays: host a quiz, feature a product, f

![Celebration action](screenshots/features/action-celebration.png)

The stream actions are received by the viewers through the IVS Player using [Timed Metadata](https://docs.aws.amazon.com/ivs/latest/userguide/metadata.html).
All stream actions (with the exception of hosting a poll) are received by the viewers through the IVS Player using [Timed Metadata](https://docs.aws.amazon.com/ivs/latest/userguide/metadata.html).

![Viewer stream actions architecture](screenshots/architecture/receive-stream-actions.png)

The poll stream overlay action on the otherhand leverages the IVS Chat Messaging [SDK](https://aws.github.io/amazon-ivs-chat-messaging-sdk-js/1.0.2/) to receive and emit different poll action events.

Note: The processing of votes happens on the streamer's side while they are on the stream manager page. Navigating away from this page will effectively skip votes.

![Poll stream action architecture](screenshots/architecture//poll.png)

### Stream health monitoring

The Stream Health page is only accessible to authenticated users, from the `/health` URL. It enables streamers to monitor live and past stream sessions. For each session, the page will show the stream events, the video bitrate and frame rate in the form of charts and a summary of the encoder configuration at the time of go-live. [Learn more](https://docs.aws.amazon.com/ivs/latest/userguide/stream-health.html)
Expand Down Expand Up @@ -465,6 +473,64 @@ Testing is automated using two GitHub Actions workflows: one for running the bac

See [Api Rates](https://webservices.amazon.com/paapi5/documentation/troubleshooting/api-rates.html) for more information.

## Services Used

Below is a list of the all the services used for the UGC Demo.

- [API Gateway](https://aws.amazon.com/api-gateway/pricing/)
- [CloudFront](https://aws.amazon.com/cloudfront/pricing/)
- [CloudWatch Logs](https://aws.amazon.com/cloudwatch/pricing/)
- [Cognito](https://aws.amazon.com/cognito/pricing/)
- [DynamoDB](https://aws.amazon.com/dynamodb/pricing/on-demand/)
- [Elastic Container Registry](https://aws.amazon.com/ecr/pricing/)
- [Elastic Container Service](https://aws.amazon.com/fargate/pricing/)
- [EventBridge](https://aws.amazon.com/eventbridge/pricing/)
- [Lambda](https://aws.amazon.com/lambda/pricing/)
- [Secrets Manager](https://aws.amazon.com/secrets-manager/pricing/)
- [Interactive Video Service](https://aws.amazon.com/ivs/pricing/)

<br>

The following is a detailed usage-based summary. Use it as a guide to estimate project costs.

<details>
<summary>Click here to view details</summary>

### Overall Usage

| Service | 1 user | 10 users | 100 users |
| -------------------------------------------------------------------- | -----: | -------: | --------: |
| Total number of requests in a month (average request size 5kB): | 1 request/second | 10 request/second | 100 request/second |
| [API Gateway](https://aws.amazon.com/api-gateway/pricing/) | 2,592,000 | 25,920,000 | 259,200,000 |
| Homepage size is 317B. Total of GB downloaded from visits to the homepage: | 1 request/second (GB) | 10 request/second (GB) | 100 request/second (GB)
| [CloudFront](https://aws.amazon.com/cloudfront/pricing/) | 0.82 | 8.22 | 82.17 |
| 1KB of log per request. Total GB of logs generated by traffic: | 1 request/second (GB) | 10 request/second (GB) | 100 request/second (GB) |
| [CloudWatch Logs](https://aws.amazon.com/cloudwatch/pricing/) | 2.59 | 25.92 | 259.20 |
| Monthly Active Users (MAU). With no advanced features. No SAML or OIDC Auth: | Number of MAU | Number of MAU | Number of MAU |
| [Cognito](https://aws.amazon.com/cognito/pricing/) | 50,000 | 100,000 | 1,000,000 |
| Average item size 105 Bytes. Asumming Monthly Active Users. Each user going live once a day: | MAU 50,000 (GB) | MAU 100,000 (GB) | MAU 1,000,000 (GB) |
| [DynamoDB](https://aws.amazon.com/dynamodb/pricing/on-demand/) | 0.01 | 0.01 | 0.11 |
| Average build size 103.01 MB: | 1 deployment a day for a month (GB) | 10 deployment a day for a month (GB) | 100 deployment a day for a month (GB) |
| [Elastic Container Registry](https://aws.amazon.com/ecr/pricing/) | 0.10 | 1.03 | 10.30 |
| Number of x86 pods with 0.25vCPU and 512 RAM. 20GB ephemeral storage: | 1 request/second | 10 request/second | 100 request/second |
| [Elastic Container Service](https://aws.amazon.com/fargate/pricing/) | 1 | 1 | 2 |
| Total number of requests in a month that go to API destinations: | 1 request/second | 10 request/second | 100 request/second |
| [EventBridge](https://aws.amazon.com/eventbridge/pricing/) | 2,592,000 | 25,920,000 | 259,200,000 |
| Total number of requests in a month ( 0.125GB memory allocated and 0.5 GB ephemeral storage allocated and 699ms average billable time: | 1 request/second | 10 request/second | 100 request/second |
| [Lambda](https://aws.amazon.com/lambda/pricing/) | 2,592,000 | 25,920,000 | 259,200,000 |
| Total number of requests in a month. We have 3 secrets in the manager: | 1 request/second | 10 request/second | 100 request/second |
| [Secrets Manager](https://aws.amazon.com/secrets-manager/pricing/) | 2,592,000 | 25,920,000 | 259,200,000 |

### Amazon IVS usage

| Service | Based on | Based on | Based on |
| -------------------------------------------------------------------- | -----: | -------: | --------: |
| [Interactive Video Service](https://aws.amazon.com/ivs/pricing/) Low-latency streaming input | Hours streamed | Hours streamed | Hours streamed |
| [Interactive Video Service](https://aws.amazon.com/ivs/pricing/) Low-latency streaming output | Hours watched | Hours watched | Hours watched |
| [Interactive Video Service](https://aws.amazon.com/ivs/pricing/) Chat | Chat usage | Chat usage | Chat usage |


</details>

## About Amazon IVS

Expand Down
2 changes: 1 addition & 1 deletion THIRD-PARTY-LICENSES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -103,4 +103,4 @@ You may add Your own copyright statement to Your modifications and may provide a

9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS
END OF TERMS AND CONDITIONS
9 changes: 6 additions & 3 deletions cdk/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ AWS_PROFILE_FLAG = --profile $(AWS_PROFILE)
STAGE ?= dev
PUBLISH ?= false
STACK ?= UGC-$(STAGE)
CDK_OPTIONS = $(if $(AWS_PROFILE),$(AWS_PROFILE_FLAG)) -c stage=$(STAGE) -c publish=$(PUBLISH) -c stackName=$(STACK)
SCHEDULE ?= rate(48 hours)
CDK_OPTIONS = $(if $(AWS_PROFILE),$(AWS_PROFILE_FLAG)) -c stage=$(STAGE) -c publish=$(PUBLISH) -c stackName=$(STACK) -c scheduleExp="$(strip $(SCHEDULE))"
FE_DEPLOYMENT_STACK = UGC-Frontend-Deployment-$(STAGE)
SEED_COUNT ?= 50
OFFLINE_SESSION_COUNT ?= 1
Expand All @@ -25,8 +26,10 @@ help: ## Shows this help message
@echo " Option 1: export AWS_PROFILE=user1\n"
@echo " Option 2: make <target> AWS_PROFILE=user1\n"
@echo "2. Set the STAGE value to \"dev\" or \"prod\" to use the corresponding configuration. The default value is \"dev\". \n" | fold -s
@echo "3. AWS CLI is required to run the seed command (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). \n"
@echo "4. Add JSON=<file_path> as an argument for the seed command to use a JSON file to create data. Furthermore, the script will create randomly generated data to match the SEED_COUNT. By default the script will create 1 offline session count out of the seed count. To change this please use the OFFLINE_SESSION_COUNT attribute. \n"
@echo "3. Set the SCHEDULE value to either a cron or rate expression based on the UTC time zone. By default, a rate expression is used to run the cleanupUnverifiedUsers Lambda every 48 hours. \n" | fold -s"
@echo "Read more about schedule expressions for EventBridge rules here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html \n" | fold -s
@echo "4. AWS CLI is required to run the seed command (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). \n"
@echo "5. Add JSON=<file_path> as an argument for the seed command to use a JSON file to create data. Furthermore, the script will create randomly generated data to match the SEED_COUNT. By default the script will create 1 offline session count out of the seed count. To change this please use the OFFLINE_SESSION_COUNT attribute. \n"

app: install bootstrap deploy ## Installs NPM dependencies, bootstraps, and deploys the stack

Expand Down
4 changes: 3 additions & 1 deletion cdk/bin/cdk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const app = new App();
const stage = app.node.tryGetContext('stage');
const stackName = app.node.tryGetContext('stackName');
const shouldPublish = app.node.tryGetContext('publish') === 'true';
const scheduleExp = app.node.tryGetContext('scheduleExp');
// Get the config for the current stage
const { resourceConfig }: { resourceConfig: UGCResourceWithChannelsConfig } =
app.node.tryGetContext(stage);
Expand All @@ -22,7 +23,8 @@ new UGCStack(app, stackName, {
env: { account, region },
tags: { stage, project: 'ugc' },
resourceConfig,
shouldPublish
shouldPublish,
scheduleExp
});

new UGCFrontendDeploymentStack(app, `UGC-Frontend-Deployment-${stage}`, {
Expand Down
131 changes: 131 additions & 0 deletions cdk/lambdas/cleanupUnverifiedUsers.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
import {
AdminDeleteUserCommand,
ListUsersCommand,
UserType
} from '@aws-sdk/client-cognito-identity-provider';
import { WriteRequest } from '@aws-sdk/client-dynamodb';
import { marshall } from '@aws-sdk/util-dynamodb';

import {
batchDeleteItemsWithRetry,
cognitoClient,
convertToChunks
} from './helpers';

const { CHANNELS_TABLE_NAME: channelsTableName, USER_POOL_ID: userPoolId } =
process.env;

export const handler = async () => {
try {
if (!channelsTableName || !userPoolId)
throw new Error(
'Missing required variables: channelsTableName or userPoolId are not defined.'
);

const deletedCognitoUserSubs: string[] = [];
let listUnconfirmedUsers: UserType[] = [];
let paginationToken: string | undefined;

const getUnconfirmedUsers = async () => {
const listUnconfirmedUsersCommand = new ListUsersCommand({
UserPoolId: userPoolId,
Filter: 'cognito:user_status ="UNCONFIRMED"',
PaginationToken: paginationToken,
AttributesToGet: ['sub']
});

const listUnconfirmedUsersResponse = await cognitoClient.send(
listUnconfirmedUsersCommand
);

if (listUnconfirmedUsersResponse?.Users) {
listUnconfirmedUsers = [
...listUnconfirmedUsers,
...listUnconfirmedUsersResponse.Users
];
}
paginationToken =
listUnconfirmedUsersResponse.PaginationToken || undefined;

if (paginationToken) await getUnconfirmedUsers();
};

await getUnconfirmedUsers();

if (listUnconfirmedUsers.length === 0) return;

// Filter users created at least 24 hours ago
const expiredUnconfirmedCognitoUsers =
listUnconfirmedUsers.filter((cognitoUser) => {
const { UserCreateDate = '' } = cognitoUser;
if (!UserCreateDate) return false;
const millisecondsInOneDay = 60 * 60 * 24 * 1000;

const timeElapsedSinceCreation =
new Date().getTime() - new Date(UserCreateDate).getTime();

return Math.abs(timeElapsedSinceCreation) > millisecondsInOneDay;
}) || [];

if (expiredUnconfirmedCognitoUsers.length === 0) return;

// Delete unverified Cognito users created 24 hours or more ago in parallel
const deleteCognitoUserPromises = expiredUnconfirmedCognitoUsers.map(
({ Username, Attributes }) => {
const subAttribute = Attributes?.find(
(attribute) => attribute.Name === 'sub'
);

return new Promise(async (resolve, rejects) => {
try {
if (!Username || !subAttribute || !subAttribute.Value) return;

const deleteUserCommand = new AdminDeleteUserCommand({
UserPoolId: userPoolId,
Username
});
const response = await cognitoClient.send(deleteUserCommand);
deletedCognitoUserSubs.push(subAttribute.Value);
resolve(response);
} catch (err) {
console.error(err);
rejects({});
}
});
}
);
await Promise.allSettled(deleteCognitoUserPromises);

if (deletedCognitoUserSubs.length) {
// Batch delete a maximum of 25 items at a time from DynamoDB.
const deleteRequests = deletedCognitoUserSubs.reduce((acc, userSubs) => {
return [
...acc,
{
DeleteRequest: {
Key: marshall({
id: userSubs
})
}
}
];
}, [] as WriteRequest[]);

const deleteRequestChunks = convertToChunks(deleteRequests, 25);

for (const chunkIndex in deleteRequestChunks) {
await batchDeleteItemsWithRetry({
[channelsTableName]: deleteRequestChunks[chunkIndex]
});
}
}
} catch (error) {
console.error(error);

throw new Error(
'Failed to remove unverified users due to unexpected error'
);
}
};

export default handler;
48 changes: 47 additions & 1 deletion cdk/lambdas/helpers.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
import { convertToAttr } from '@aws-sdk/util-dynamodb';
import { DynamoDBClient, QueryCommand } from '@aws-sdk/client-dynamodb';
import {
BatchWriteItemCommand,
DynamoDBClient,
QueryCommand,
WriteRequest
} from '@aws-sdk/client-dynamodb';
import { CognitoIdentityProviderClient } from '@aws-sdk/client-cognito-identity-provider';

export const cognitoClient = new CognitoIdentityProviderClient({});
export const dynamoDbClient = new DynamoDBClient({});

export const getChannelByChannelAssetId = (channelAssetId: string) => {
Expand All @@ -23,3 +30,42 @@ export const isRejected = (
export const isFulfilled = <T>(
input: PromiseSettledResult<T>
): input is PromiseFulfilledResult<T> => input.status === 'fulfilled';

export const convertToChunks = (
array: object[],
chunkSize: number
): { [key: number]: typeof array } => {
const result: { [key: number]: typeof array } = {};
let keyIndex = 0;

for (let i = 0; i < array.length; i += chunkSize) {
const chunk: object[] = array.slice(i, i + chunkSize);
result[keyIndex] = chunk;
keyIndex += 1;
}

return result;
};

export const batchDeleteItemsWithRetry = async (
requestItems: Record<string, WriteRequest[]> | undefined,
retryCount = 0,
maxRetries = 4
): Promise<void> => {
const batchWriteCommandInput = {
RequestItems: requestItems
};
const batchWriteCommand = new BatchWriteItemCommand(batchWriteCommandInput);
const response = await dynamoDbClient.send(batchWriteCommand);

if (response.UnprocessedItems && response.UnprocessedItems.length) {
if (retryCount > maxRetries)
throw new Error(
`Failed to batch delete AWS DynamoDB items: ${response.UnprocessedItems}`
);

await new Promise((resolve) => setTimeout(resolve, 2 ** retryCount * 10));

return batchDeleteItemsWithRetry(response.UnprocessedItems, retryCount + 1);
}
};
Loading

0 comments on commit 4909433

Please sign in to comment.