John Tipper
Navigate back to the homepage

AWS CDK cross-account deployments with CDK Pipelines and cdk-assume-role-credential-plugin

John Tipper
December 4th, 2020 · 7 min read

Cross-account deployments with AWS CDK and CDK Pipelines and cdk-assume-role-credential-plugin

AWS CDK is really very nice for the speed with which you can create lots of infrastructure in a reusable fashion. However, for a long time, performing cross-account deployments was rather painful when some stacks had to go to one account and others to a different account, because getting the right credentials to CDK for the different accounts was difficult. Woe betide you if you wanted to do lots of deployments to very many accounts. This post will demonstrate how to use an AWS plugin for CDK called cdk-assume-role-credential-plugin to make life easy.

What we want to achieve: a fully automated build pipeline

We’ll assume that you have 2 accounts into which you would like to deploy some infrastructure, where that infrastructure has been defined using CDK. We’ll also assume that the project where that CDK infrastructure exists is based on more than just CDK: maybe you have some other compilation steps required as part of the deployment process. An example might be where you are compiling some Lambdas (Rust or GoLang, perhaps) and you then want to use those compiled binaries in a CDK deployment. We’ll do the deploy in a generic CodeBuild project which could perform other steps if you wish, in addition to doing the deployment.

We want our build and deployment of our project to be fully automated. When we decide we want to modify the steps inside our build and deployment pipeline, we’d like these changes to the pipeline to be automated too.

We’ll also assume that the account where your CI/CD pipeline is running is in a different account to the accounts where you want to deploy your infrastructure. This means 3 sets of credentials we need to deal with, but this could easily be many more if you have lots of accounts where you wish to deploy to.

We don’t want to have to tinker around with project settings and dependencies: we’d like this to be done for us with minimal setup.

Prerequisites

In order to use CDK, we need to have bootstrapped the accounts and regions to which we want to deploy stuff. Bootstrapping is defined here, but there is also some useful information in the CDK design documention on GitHub which is not in the AWS documentation.

The act of bootstrapping creates some infrastructure in the account and region that is targeted. There are 2 styles of bootstrapping: legacy and new. The legacy way is still the default and creates just an S3 bucket into which assets are published when deploying, but the new way creates some additional resources, such as an ECR repository (for storing Docker images that are the result of building Docker assets) and IAM roles which may be assumed by CDK when synthesizing and deploying resources. We need the new way of bootstrapping, and because this is not yet the default for CDK, there are a few extra arguments to use.

For each account/region pair into which we wish to deploy, we need to run the following command:

1cdk bootstrap --trust <trusted account id>[,<trusted account id>...] --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://<target account id>/<region>

What this command is doing is saying that each <trusted account id> in the list will be allowed to assume particular IAM roles within the target account (<target account id>), called the Publishing and Deployment Action Roles, when writing assets to S3 or ECR or executing changesets. Those roles will have some permissions associated with uploading assets to CDK buckets and creating and starting changesets, but they won’t, in and of themselves, be able to do very much. When CloudFormation runs the changeset, it needs to create and mutate infrastructure, so needs a fairly broad set of permissions: quite how broad depends on what you want to be able to manage with CDK. In the command above, we are giving an Access-All-Areas pass to CloudFormation (the AWS service, not the identity calling CDK), and you may wish to de-scope this if you don’t want CDK/CloudFormation to be able to do everything in the target account.

There are 2 stages to a CDK deployment: synthesis followed by deployment. As part of the synthesis of a Cloud Assembly, the user may specify context lookups. An example might be to query Route53 HostedZone details by way of HostedZone.fromLookup(), for instance. In order to execute the lookup, the user running CDK synth requires AWS credentials and these credentials need to be scoped to the target account. Calling cdk deploy will also cause a synthesis to happen first, before deployment occurs, unless the user passes a path to an already synthesized Cloud Assembly by means of the --app /path/to/cdk.out, so it may also require AWS credentials. We will use a CDK plugin called cdk-assume-role-credential-plugin to retrieve credentials for us, but we need to tell this plugin what role to assume when retrieving STS credentials by way of sts:AssumeRole. By default, the plugin will look for a role called cdk-readOnlyRole to fetch context. That role does not exist, so we need to either create it, or provide another role which has sufficient read privileges in order to satisfy any CDK context lookups we wish to permit.

For the purposes of this example, we will assume a role called cdk-readOnlyRole exists in each of our 2 target accounts, where those accounts trust our CI/CD account, i.e. an appropriately permissioned user inside our CI/CD account may call sts:AssumeRole on that role in the target accounts. How you create this is up to you: there is an example inside the cdk-assume-role-credential-plugin repository on GitHub, look for the required-resources.ts files. That example uses CDK to create a stack which defines the role which is given an AWS managed policy called ReadOnlyAccess. If you are automatically creating accounts into which you wish your CI/CD account to be able to deploy, you’ll probably create these roles at this point right after creating the account. Remember, there is a role called OrganizationAccountAccessRole in each sub-account which is assumable by the master/admin account of the AWS Organization which has admin permissions, so you might use this to create your read-only roles if you wish.

Project setup

We will use Projen to create and manage our project. It is not a templating tool, where the generated templates then immediately start to rot. Projen generates your project definition files for you, but all management of these is done through Projen. Re-running projen regenerates the files for you.

Install projen:

1npm install -g projen

We will create a monorepo with 2 subprojects: one for our build pipeline and another for our project itself (i.e. the infrastructure we wish to actually deploy to the other accounts).

1mkdir example
2cd example
3git init
4
5# directory for our build pipeline
6mkdir pipeline
7pushd pipeline
8projen new awscdk-app-ts
9popd
10
11# directory for our project
12mkdir infra
13pushd infra
14projen new awscdk-app-ts
15popd

Build Pipeline

We now have a lot of boilerplate created for us that would otherwise take ages to do (or at the very least, it would take me ages). Let’s configure our build pipeline:

1// within pipeline/.projenrc.js
2const project = new AwsCdkTypeScriptApp({
3//...
4cdkDependencies: [
5 '@aws-cdk/pipelines',
6 '@aws-cdk/aws-codepipeline',
7 '@aws-cdk/aws-codepipeline-actions',
8 '@aws-cdk/aws-codebuild',
9 '@aws-cdk/aws-iam',
10 '@aws-cdk/aws-s3',
11 '@aws-cdk/aws-logs',
12 ],
13context: {
14 '@aws-cdk/core:newStyleStackSynthesis': true,
15 },
16devDeps: [
17 'cdk-assume-role-credential-plugin@^1.2.1',
18 ],
19//...
20}
21project.cdkConfig.plugin = ["cdk-assume-role-credential-plugin"];
22
23project.synth();

Note that we are adding in some CDK dependencies which we will use to define our build pipeline and the cdk-assume-role-credential-plugin as a dev dependency. We want a minimum version of 1.2.1 as there was a race condition I came across in 1.2.0 which caused the plugin to misbehave under some circumstances. We apply that plugin by means of the call to project.cdkConfig.plugin=.

Also note that we set a context value: this will be added to the cdk.json file when we run projen. We are telling CDK that we are using the new-style bootstrapping.

Now let’s install our dependencies:

1pushd pipeline
2npx projen

Now let’s define our build pipeline. Firstly, let’s define a policy for a role which our pipeline will assume:

1// src/lib/PipelinePolicyDocument
2import { Effect, PolicyDocument, PolicyStatement } from '@aws-cdk/aws-iam';
3
4export interface PipelinePolicyProps {
5 account: string;
6 region: string;
7}
8
9export class PipelinePolicyDocument extends PolicyDocument {
10
11 constructor(props: PipelinePolicyProps) {
12 super({
13 statements: [
14 new PolicyStatement({
15 sid: 'CloudWatchLogsPolicy',
16 effect: Effect.ALLOW,
17 actions: [
18 'logs:CreateLogGroup',
19 'logs:CreateLogStream',
20 'logs:PutLogEvents',
21 ],
22 resources: ['*'],
23 }),
24 new PolicyStatement({
25 sid: 'S3GetObjectPolicy',
26 effect: Effect.ALLOW,
27 actions: [
28 's3:GetObject',
29 's3:GetObjectVersion',
30 ],
31 resources: ['*'],
32 }),
33 new PolicyStatement({
34 sid: 'S3ListBucketPolicy',
35 effect: Effect.ALLOW,
36 actions: [
37 's3:ListBucket',
38 ],
39 resources: ['*'],
40 }),
41 new PolicyStatement({
42 sid: 'S3PutObjectPolicy',
43 effect: Effect.ALLOW,
44 actions: [
45 's3:PutObject',
46 ],
47 resources: ['*'],
48 }),
49 new PolicyStatement({
50 sid: 'S3BucketIdentity',
51 effect: Effect.ALLOW,
52 actions: [
53 's3:GetBucketAcl',
54 's3:GetBucketLocation',
55 ],
56 resources: ['*'],
57 }),
58 new PolicyStatement({
59 sid: 'AccessGitHubPublishSecret',
60 effect: Effect.ALLOW,
61 actions: [
62 'ssm:GetParameter',
63 'ssm:GetParameters',
64 ],
65 resources: [
66 `arn:aws:ssm:${props.region}:${props.account}:parameter/path/to/my/token`,
67 ],
68 }),
69 new PolicyStatement({
70 sid: 'DecryptGitHubSecrets',
71 effect: Effect.ALLOW,
72 actions: [
73 'kms:Decrypt',
74 ],
75 resources: [
76 '*',
77 ],
78 }),
79 new PolicyStatement({
80 sid: 'AssumeCDKReadonlyRole',
81 effect: Effect.ALLOW,
82 actions: [
83 'sts:AssumeRole',
84 ],
85 resources: [
86 'arn:aws:iam::*:role/cdk-readOnlyRole',
87 'arn:aws:iam::*:role/cdk-hnb659fds-deploy-role-*',
88 'arn:aws:iam::*:role/cdk-hnb659fds-file-publishing-*',
89 ],
90 }),
91 ],
92 });
93 }
94}

The above is just a lot of boilerplate and is basically copied from any example you’ll find in the AWS docs for CDK Pipelines. The last stanza is important though, and doesn’t appear fully in the docs. Remember, our pipeline needs to be able to perform context lookups, publish assets to S3 prior to deploying and then actually execute the CDK deployment. It’s really important you add these.

Now let’s define the Pipeline stack for the master branch of our repo (replace OWNER and REPO as required):

1// src/lib/PipelineStack
2import { Artifact } from '@aws-cdk/aws-codepipeline';
3import { GitHubSourceAction } from '@aws-cdk/aws-codepipeline-actions';
4import { Role, ServicePrincipal } from '@aws-cdk/aws-iam';
5import { Construct, SecretValue, Stack, StackProps } from '@aws-cdk/core';
6import { CdkPipeline, SimpleSynthAction } from '@aws-cdk/pipelines';
7import { PipelinePolicyDocument } from './PipelinePolicyDocument';
8
9export class PipelineStack extends Stack {
10
11 pipeline: CdkPipeline;
12
13 sourceArtifact: Artifact;
14 cloudAssemblyArtifact: Artifact;
15 buildRole: Role;
16
17 sourceAction: GitHubSourceAction;
18
19 constructor(scope: Construct, id: string, props?: StackProps) {
20 super(scope, id, props);
21
22 this.sourceArtifact = new Artifact();
23 this.cloudAssemblyArtifact = new Artifact();
24
25 this.sourceAction = new GitHubSourceAction({
26 actionName: 'GitHub',
27 output: this.sourceArtifact,
28 oauthToken: SecretValue.secretsManager('/path/to/my/token', {
29 jsonField: 'token',
30 }),
31 owner: 'OWNER',
32 repo: 'REPO',
33 branch: 'master',
34 variablesNamespace: 'SourceVariables',
35 });
36
37 this.buildRole = new Role(this, 'CodeBuildRole', {
38 roleName: id + '-role',
39 assumedBy: new ServicePrincipal('codebuild.amazonaws.com'),
40 inlinePolicies: {
41 'codebuild-policy': new PipelinePolicyDocument({
42 account: (!props || !props.env || !props.env.account) ? '' : props.env.account,
43 region: (!props || !props.env || !props.env.region) ? '' : props.env.region,
44 }),
45 },
46 });
47
48 this.pipeline = new CdkPipeline(this, 'Pipeline', {
49 cloudAssemblyArtifact: this.cloudAssemblyArtifact,
50
51 sourceAction: this.sourceAction,
52
53 synthAction: SimpleSynthAction.standardYarnSynth({
54 sourceArtifact: this.sourceArtifact,
55 cloudAssemblyArtifact: this.cloudAssemblyArtifact,
56 subdirectory: 'pipeline',
57
58 installCommand: 'yarn --cwd pipeline install --frozen-lockfile && yarn --cwd pipeline projen',
59 buildCommand: 'yarn --cwd pipeline run build',
60 synthCommand: 'yarn --cwd pipeline cdk synth',
61 }),
62 });
63 }
64}

In this stack, we define a basic CDK pipeline. It will build a CodePipeline with 2 stages: a source stage which links to GitHub and will be triggered automatically by webhooks whenever a push occurs. This source stage assumes that there is a pre-provisioned secret in the Secrets Manager under the path /path/to/my/token. Note that the permissions for the Role allow this token to be retrieved. We also define a build stage, which will call CDK synth and deploy to build our pipeline.

So far, if we deployed this pipeline, it would build and redeploy itself and whilst this is cool, it’s not that useful, so let’s add another stage where we actually do some work.

1// src/lib/MyCodeBuild.ts
2import { BuildSpec, ComputeType, LinuxBuildImage, PipelineProject } from '@aws-cdk/aws-codebuild';
3import { Artifact } from '@aws-cdk/aws-codepipeline';
4import { CodeBuildAction, GitHubSourceAction } from '@aws-cdk/aws-codepipeline-actions';
5import { Role } from '@aws-cdk/aws-iam';
6import { LogGroup, RetentionDays } from '@aws-cdk/aws-logs';
7import { Construct, Duration } from '@aws-cdk/core';
8
9export interface MyCodeBuildProps {
10 id: string;
11 parent: Construct;
12 input: Artifact;
13 buildRole: Role;
14}
15
16const buildSpec = {
17 version: '0.2',
18 phases: {
19 install: {
20 'runtime-versions': {
21 nodejs: '10',
22 },
23 'commands': [
24 'npm install -g aws-cdk cdk-assume-role-credential-plugin',
25 ],
26 },
27 build: {
28 commands: [
29 // add whatever build command you want here
30 'yarn --cwd infra run build',
31 'pushd infra && cdk deploy --app cdk.out/ --require-approval never "*" && popd',
32 ],
33 },
34 },
35};
36
37
38export class MyCodeBuild {
39
40 codeBuildAction: CodeBuildAction;
41
42 constructor(props: MyCodeBuildProps) {
43
44 // some log group - name it as you see fit and retain the logs for as long as needed
45 const logGroup = new LogGroup(props.parent, props.id + '-loggroup', {
46 logGroupName: `/aws/codebuild/${props.id}`,
47 retention: RetentionDays.ONE_WEEK,
48 });
49
50 // change this as required
51 this.codeBuildAction = new CodeBuildAction({
52 actionName: 'ReleaseMyInfra',
53 input: props.input,
54 project: new PipelineProject(props.parent, props.id, {
55 role: props.buildRole,
56 environment: {
57 buildImage: LinuxBuildImage.AMAZON_LINUX_2_3,
58 computeType: ComputeType.SMALL,
59 privileged: true,
60 },
61 buildSpec: BuildSpec.fromObject(buildSpec),
62 logging: {
63 cloudWatch: {
64 logGroup: logGroup,
65 enabled: true,
66 },
67 },
68 }),
69 });
70 }
71}

In the above, we define a CodeBuild stage which uses the same permissions as we defined earlier. We accept as input some artifact, which we will define later (we’ll use the output from the source stage). When the build runs, it installs onto the Nodejs environment the aws-cdk and cdk-assume-role-credential-plugin.

We now need to integrate this latter stage into our pipeline.

1import { App } from '@aws-cdk/core';
2import { PipelineStack } from './lib/PipelineStack';
3import { MyCodeBuild } from './lib/MyCodeBuild';
4
5const app = new App();
6
7// define where our CI/CD environment will run
8const account = '123456789012';
9const region = 'eu-west-1';
10
11const pipelineStack = new PipelineStack(app, 'MyPipeline', {
12 env: {
13 account: account,
14 region: region,
15 },
16});
17
18const releaseMyInfra = new MyCodeBuild({
19 id: 'ReleaseMyInfra',
20 parent: pipelineStack.pipeline,
21 input: pipelineStack.sourceArtifact,
22 buildRole: pipelineStack.buildRole,
23});
24
25// note that we add a normal CodeBuild stage here, but we can use addApplicationStage if we just want to build and deploy a pure CDK application
26// we can pass different build artifacts to the latter stages if we wish, there's not always a need to pass the entire source code
27// checkout to this stage as an input
28releaseMyInfra.addActions(releaseMyInfra.codeBuildAction);
29
30app.synth();

In the above, we add our newly defined CodeBuild stage into our pipeline, which will execute after the pipeline has built itself. This concludes the definition of our pipeline.

Our CDK Project

We now need to look at our infrastructure project, which is the project we want to build and deploy. I’m not going to go into a great deal of detail here: you can create anything you want in the same way that you created the CDK pipeline. I’m just going to cover the minimum to set it up.

1// within infra/.projenrc.js
2const project = new AwsCdkTypeScriptApp({
3//...
4cdkDependencies: [
5 // some CDK dependencies here, whatever you need for your project
6 ],
7context: {
8 '@aws-cdk/core:newStyleStackSynthesis': true,
9 },
10devDeps: [
11 'cdk-assume-role-credential-plugin@^1.2.1',
12 ],
13//...
14}
15project.cdkConfig.plugin = ["cdk-assume-role-credential-plugin"];
16
17project.synth();

Again, let’s install our dependencies:

1pushd pipeline
2npx projen

Let’s create two stacks that exist in two different accounts:

1const app = new App();
2
3const stack1 = new ImaginaryStack(app, 'MyStackInAccount1', {
4 env: {
5 account: '111111111111',
6 region: 'eu-west-1',
7 },
8});
9
10const stack2 = new ImaginaryStack(app, 'MyStackInAccount2', {
11 env: {
12 account: '222222222222',
13 region: 'us-east-1',
14 },
15});
16
17app.synth();

You can now iterate in the standard fashion: calling yarn run build will build and run your tests, then perform a cdk synth. You will need AWS credentials if you perform context lookups as part of your synth.

Deployment

When we are ready, we can commit and push to Git.

1git add .
2git commit -m "Initial commit"
3git push origin master

Note that this will have no effect inside AWS yet, as we have not deployed anything. We now need to perform an initial seed deployment to AWS. Assuming that the build of the pipeline completes, successfully, our pipeline will becomes self aware and even if the latter stages fail, whenever we push changes to our Git repository, our pipeline will rebuild, redeploy itself, then perform whatever build stages we have defined, which for our example, is the deployment of more CDK infrastructure.

The initial deployment of our pipeline needs to be done with credentials that permit the user to deploy the pipeline to the CI/CD account:

1pushd pipeline
2cdk deploy --profile my-profile-allowing-context-lookups-and-deployments

This onetime call to CDK deploy is something that you will need to riff on: it’s highly dependent on the permissions you have defined within your CI/CD account.

What you don’t see in the normal logs is the heavy lifting that is being done for you by cdk-assume-role-credential-plugin: for each stack, it will retrieve credentials if the standard ones won’t suffice for the target accounts (111111111111 and 222222222222) by assuming the arn:aws:iam::*:role/cdk-hnb659fds-deploy-role-* and arn:aws:iam::*:role/cdk-hnb659fds-file-publishing-* roles in the target accounts to publish CDK assets as required then create and execute the changesets. CloudFormation will assume the powerful execution roles that were defined when bootstrapping.

Troubleshooting

If you run into issues, then they are likely to be associated with incorrect permissions. You can turn on additional logging by mutating your pipeline (just make the changes, then push: the pipeline will take care of rebuilding itself) to add logging to either the pipeline or infra project buildscripts (or both):

1// within the pipeline buildscript
2 'yarn --cwd pipeline run build --debug -v -v -v',
3
4// within the infra project buildscript
5 'yarn --cwd infra run build --debug -v -v -v',

You’ll now have pretty verbose logs which should assist with tracking down any issues.

More articles from John Tipper

Integrating AWS CDK into GitHub Actions

Integrating AWS CDK into GitHub Actions

September 13th, 2020 · 2 min read

A static website with API backend using AWS CDK and Java

An example of creating a static website using AWS CDK and Java

September 12th, 2020 · 5 min read
© 2018–2021 John Tipper
Link to $https://twitter.com/john_tipperLink to $https://github.com/john-tipperLink to $https://www.linkedin.com/in/john-tipper-5076395