AWS Blog

Deploying Multi-Region S3 Replication with 01 command

By

Amazon Simple Storage Service (also known as Amazon S3) is a well-known Amazon Web Services (AWS) service that customers can use to store data securely and reliably. Using Amazon S3, businesses will be able to build a low-cost, yet highly available storage solution. To deserve 11 numbers 9 (99.9999999999%) in terms of SLAs, Amazon S3 offers features available for replication to another region, which allows for easy automatic backups of various regions (Multi-region Replication).

Overview

The article below references an AWS CDK (Cloud Development Kit) project using TypeScript. Using AWS CDK together with AWS CloudFormation StackSets, customers can deploy the following resources:

  • Amazon S3 Bucket on the primary region with custom KMS key.
  • AWS CloudFormation StackSet replicates copies to different regions.
  • AWS IAM Role with access to the primary region and copies.

Interestingly, you can deploy these resources on the desired Regions, WITH ONLY 01 SINGLE COMMAND !!!

Steps to proceed deploying Multi-Region S3 Replication

Step 1: Prerequisite

To use Self-managed StackSets, you first need to create two IAM roles. You can create them manually using the AWS Management Console or use the official CloudFormation templates provided by AWS, click to download:

iam roles for cloudformation

This project is taken from a Repository containing pre-packaged templates on Github, link here. Just adjust some parameters for CloudFormation Stack to be usable.

// Configured ./aws/index.ts

const option = {
  env: {
    region: 'ap-southeast-1'
  },
  replications: [
    'ap-northeast-1',
    'ap-southeast-2',
    '...'
  ]
}

Then, run using the command below to deploy everything.

npx cdk deploy

Step 2: Edit parameters of Primary Region and Data Source

Changes to data inside amazon S3 buckets in primary regions are replicated to other AWS regions, for example here the main region Viet-AWS is making ap-southeast-1 (Singapore) and other regions ap-northeast-1 (Tokyo) and ap-southeast-2 (Sydney).

In primary region, you need Amazon S3 bucket with custom KMS (Key Management System) key for encryption.

Learn more about AWS KMS here: https://aws.amazon.com/kms/.

import * as s3 from '@aws-cdk/aws-s3'
import * as kms from '@aws-cdk/aws-kms'

const key = new kms.Key(this, 'Key')
const alias = key.addAlias('archive')

const bucket = new s3.Bucket(this, 'Bucket', {
  bucketName: `${props.prefix}-archive`,
  encryption: s3.BucketEncryption.KMS,
  encryptionKey: alias,
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
  bucketKeyEnabled: true,
  versioned: true,
  removalPolicy: cdk.RemovalPolicy.RETAIN
})

To use S3 bucket replication, you need to create an IAM role with permission to access data in Amazon S3 and use kms key:

import * as iam from '@aws-cdk/aws-iam'

const role = new iam.Role(this, 'ReplicationRole', {
  assumedBy: new iam.ServicePrincipal('s3.amazonaws.com'),
  path: '/service-role/'
});


role.addToPolicy(
  new iam.PolicyStatement({
    resources: [
      bucket.bucketArn
    ],
    actions: [
      's3:GetReplicationConfiguration',
      's3:ListBucket'
    ]
  })
);

role.addToPolicy(
  new iam.PolicyStatement({
    resources: [
      bucket.arnForObjects('*')
    ],
    actions: [
      's3:GetObjectVersion',
      's3:GetObjectVersionAcl',
      's3:GetObjectVersionForReplication',
      's3:GetObjectVersionTagging'
    ]
  })
);

role.addToPolicy(
  new iam.PolicyStatement({
    resources: [
      key.keyArn
    ],
    actions: [
      'kms:Decrypt'
    ]
  })
);

After completing the above steps, the next step is to create an Amazon S3 bucket with a KMS key that can be used in any region you want to replicate, here we configure the KMS key in the regionap-northeast-1 (Tokyo) and ap-southeast-2 (Sydney).

Step 3: Creat CloudFormation StackSet for Multi-Region S3 Replication

To avoid having to create each CloudFormation Stack in each region you want to replicate amazon S3 bucket data, AWS CloudFormation StackSet is used to automate deployment from the region. Currently, AWS CDK only supports low-level access to CloudFormation StackSet resources:

import * as cdk from '@aws-cdk/core'

new cdk.CfnStackSet(this, "StackSet", {
  stackSetName: `${props.prefix}-archive-replication`,
  permissionModel: "SELF_MANAGED",
  parameters: [
    {
      parameterKey: 'Prefix',
      parameterValue: props.prefix
    },
    {
      parameterKey: 'ReplicationRole',
      parameterValue: role.roleArn
    }
  ],
  stackInstancesGroup: [
    {
      regions: props.replications,
      deploymentTargets: {
        accounts: [AccountId],
      },
    },
  ],
  templateBody:templateReplicationData,  #đây là template chứa Amazon S3 và KMS cho mọi region
});

The parameterReplicationRole is required to grant access to KMS keys by region that IAM role uses to perform replication.

Parameters:
  Prefix:
    Type: String
  ReplicationRole:
    Type: String

Resources:
  Key:
    Type: AWS::KMS::Key
    Properties:
      KeyPolicy:
        Version: 2012-10-17
        Id: access-account
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
            Action: kms:*
            Resource: '*'
          - Sid: Replication
            Effect: Allow
            Principal:
              AWS: !Ref ReplicationRole
            Action:
              - kms:Encrypt
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:DescribeKey
            Resource: '*'

  KeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/archive/replication
      TargetKeyId: !Ref Key

  Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Sub ${Prefix}-archive-replication-${AWS::Region}
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls: Yes
        BlockPublicPolicy: Yes
        IgnorePublicAcls: Yes
        RestrictPublicBuckets: Yes
      VersioningConfiguration:
        Status: Enabled
      BucketEncryption:
        ServerSideEncryptionConfiguration:
        - BucketKeyEnabled: Yes
          ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms
            KMSMasterKeyID: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:${KeyAlias}

AWS resources in the region are replicated using their own name patterns to differentiate, we will example is ap-northeast-1 in the following configuration:

  • KMS Key
    arn:aws:kms:ap-northeast-1:11223344:alias/archive/replication

  • S3 Bucket
    arn:aws:s3:::prefix-archive-replication-ap-northeast-1

Using this name pattern, you can extend the IAM role used for region replication from the primary region:

role.addToPolicy(
  new iam.PolicyStatement({
    resources: props.replications.map(
      region => `arn:aws:kms:${region}:$(AccountId):alias/archive/replication`
    ),
    actions: [
      'kms:Encrypt'
    ]
  })
);

role.addToPolicy(
  new iam.PolicyStatement({
    resources: props.replications.map(
      region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}/*`
    ),
    actions: [
      's3:ReplicateDelete',
      's3:ReplicateObject',
      's3:ReplicateTags'
    ]
  })
);

role.addToPolicy(
  new iam.PolicyStatement({
    resources: props.replications.map(
      region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}`
    ),
    actions: [
      's3:List*',
      's3:GetBucketVersioning',
      's3:PutBucketVersioning'
    ]
  })
);

Step 4: Configure to deploy Multi-Region S3 Replication

Set up S3 Replication Configuration at Amazon S3 bucket in the primary region. In this step, AWS CDK can still use low-level objects to configure replication:

const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;

cfnBucket.replicationConfiguration = {
  role: role.roleArn,
  rules: props.replications.map(
    (region, index) => (
      {
        id: region,
        destination: {
          bucket: `arn:aws:s3:::${props.prefix}-archive-replication-${region}`,
          encryptionConfiguration: {
            replicaKmsKeyId: `arn:aws:kms:${region}:${AccountID}:alias/archive/replication`
          }
        },
        priority: index,
        deleteMarkerReplication: {
          status: 'Enabled'
        },
        filter: {
          prefix: ''
        },
        sourceSelectionCriteria: {
          sseKmsEncryptedObjects: {
            status: 'Enabled'
          }
        },
        status: 'Enabled'
      }
    )
  )
}

Step 5: Deploy

As said above, after running using the command below, everything will be implemented with this command:

npx cdk deploy

Conclusion

With only the AWS Cloud Development Kit, you can create the AWS CloudFormation template.

From this template, you can create the AWS CloudFormation Stack. This stack will help you deploy services such as Amazon S3 bucket, AWS Identity & Access Management role, an AWS Key Management Service key, and 01 AWS CloudFormation StackSet.

AWS CloudFormation StackSet then uses the template above to create AWS CloudFormation Stack for different regions, and this Stack continues to be used to create resources such as S3 bucket, IAM role, and KMS key, where Amazon S3 bucket features S3 Replication Configuration.

Share
August 16, 2021
August 16, 2021