AWS Infrastructure provisioning using Pulumi
Overview
AWS is the leading cloud services provider — and I personally also endorse that.
Coming to the point and context — No enterprise would prefer to use the web console to setup the infras when it comes to scalability, maintainability, and other similar parameters.
Obviously the only thing that needs HUMAN attention is the root user setup(profile, billing details, etc.), setting up a new non-user ops account for the CLI, SDKs etc.
IAC (infrastructure as code) is in trend & is heavily used to provision the infrastructure in cloud, be it simple IAM role provisioning, DNS entries via route53, EC2 instances, VPCs, Serverless based tech, databases using RDS, anything is now configurable via code(or config).
AWS has a great number of developer tools to help with infrastructure provisioning. AWS’s IAC base is AWS SDK(running on top of AWS core APIs) at most of the places. While they have custom services and tools on top of it(the AWS SDK)— like AWS cloud formation (quite famous & adopted widely) and AWS CDK (again quite famous, but very recent)
Pulumi is one of the important tech in the area of the IAC and they are quite confidently defining it —as a universal IAC manager/ provider.
I again feel that Pulumi does a great job with infrastructure provisioning, though I tried and experienced only the AWS based integration and setup while writing this blog post.
The code of the POC done as part of the blog post here is going to set up a serverless API backed by AWS lambda function and that is accessible via API gateway along with proper DNS provisioned at AWS route53 and certificate provisioned through AWS ACM.
Quick pointers of Pulumi.
- Pulumi is an universal IAC tool.
- maintains the state of the infra.
- compatible with AWS, AZURE, GCP, K8S, etc.
- very programmatic & developer friendly.
- preview the changes before provisioning.
- easy to debug and hard stopping if there are any errors during preview of the stack.
- test the changes using unit/ integration tests. (not included below)
- adds reusability and extensibility for the infra/ resources, can create chain of resources dependent on each other.
- it also supports Policy as Code
— Pulumi’s main components —
- Pulumi CLI/Engine — Developer/ CICD runner or agent machines need the pulumi CLI, which comes along Pulumi Engine.
- Language Host — based on the chosen language(could be variety of languages) the Language Host(the runtime) manages the provisioning(in easy words deployment of resources).
- Storage — Pulumi stores the state of the stack in a backend.
A backend API + storage endpoint used by CLI(Engine) to coordinate the updates of the stack whenever appropriate. These backends can be:-
- Service — a managed cloud experience using Pulumi’s online application or a self-hosted Pulumi Service.
- Self-Managed — a self managed object store, could be AWS S3, Azure & GCP or a plain local filesystem. - Resource Providers — This contains two blocks:-
- Resource plugin — binary used by deployment engine to manage a resource
- SDK — Pulumi SDK on top of the specific cloud providers. It provide bindings to different type of resources, that the provider can manage. Example @pulumi-aws
Example Code & the directory structure
The repository for the POC is located at github here :— https://github.com/neuw/pulumi-aws-lambda
For the repository shared above, prerequisites are:-
- NodeJS (14+)
- AWS CLI v2
- AWS account (setup aws credentials)
- PULUMI account (setup using pulumi.com)
- any IDE / editor of your choice
This POC Project uses typescript, otherwise available options are JAVA, GO, PYTHON, etc.
The folder structure for POC is :-
./ - The root of the code
-/apis - the lambda function & related pulumi config
-/hello-world - the folder of hello-world lambda function
-/global-infra - global pulumi config for route53, certs
-/infra - environment/ stage specific pulumi configThe commands used are very easy:-
pulumi up - on every root level of the stack~/global-infra % pulumi up
~/infra % pulumi up
~/apis % pulumi uporder has to be like above only because they are changed like this.
Top level intention diagram:-
The stack reference and flow diagram
The details of all stacks(mentioned by order of execution):-
global-infra — it is meant for most common provisioning of the route53 hosted public zone & acm public certs. The resources provisioned are:-
- a new hosted zone for api.*
- certs corresponding to the same
- DNS CNAME record entry(entries) in the hosted zone for the validation required by the ACM against a certificate
- similarly a cert for a previously manually created hosted zone for app.*
infra — it helps in provisioning of the environment specific resources, resource which are common to a particular environment, the resources are:-
- AWS api gateway custom domain.
- The reference to the global-infra is also required to know the hosted zone details — actually we need the zoneID of the zone.
- A & AAAA type DNS record entries in route53 pointing to the custom API gateway.
The resources that are provisioned are:-
apis — a basic hello-world API.
- Lambda function, the function code is inside the hello-world folder
- IAM roles/ permissions
- API Gateway mapping, route, stage & a HTTP API i.e API gateway v2
- Bounded to a custom domain referred from the infra stack.
Happy Puluming
I loved provisioning the infra while having a pure developer experience with Pulumi.
The above very common requirement of the AWS serverless world i.e API backed by Lambda and API gateway with proper pointing out of DNS at Route53 was quite easy with Pulumi.
Hope & wish to explore pulumi more with further more dynamic stacks done with proper CICD and hooks for notifications etc.
A highly suggested framework — awesome work team Pulumi
Feel free to connect with me over LinkedIn here
