Introduction

Hi there, fellow developers, DevOps, or anybody else interested in AWS!

Here at Deegloo, we’ve been building projects using AWS for some time now and we’ve reached the stage where we feel confident enough to make a contribution to the community by sharing what we’ve learned in the process.

What you can learn by reading these posts

You can benefit by going through this series of blog posts whether you are a total newcomer to AWS or you just want to find out how AWS Lambda, SNS, CloudWatch, CloudFormation, and S3 services can be used to collect, process, and store data. In reality, these services can be used as building blocks of a vast number of scenarios with only slight modifications and tweaks. E.g. SNS service can be used to trigger Lambda execution but it can also be used to send SMS or an e-mail. CloudWatch can be used to monitor components of your infrastructure or event scheduler. S3, a very popular and cheap storage service does the job of keeping datasets of any size without increasing the complexity of the solution.

We thought: there is no better way of conveying what we learned in our AWS journey than by presenting our real-life scenario completely detailed and stripped out, bringing actual value to the reader!

Our value-based, real-life scenario proposition can be simplified and depicted with the following diagram:

If you’re anxious to see the end result infrastructure or you just want to have a sneak peek, you can access the completed infrastructure template here. Otherwise, just follow this three-part series to build this template yourself.

For the newcomers to the AWS it might look complex at this moment, but, have no fear – our plan is to guide you step by step, making sure you are equipped with the necessary information for building such or similar infrastructure yourself. Best of all – we made this blog post series in a way that you can take any piece of the infrastructure and have it running individually. After all, your scenario is likely to be different than what we present here and you will want to be flexible to take any part you need independently in order to be able to tinker with it.

Schedule, notify and trigger it!

In short, these are the mechanisms we would like to present to you. Let me expand on this a bit.

First, we would like to show you how to schedule AWS Lambda execution (e.g. daily or hourly) and a nice trick of triggering multiple instances of another Lambda using AWS SNS (Simple Notification Service) service. This might come in handy when a single instance of Lambda cannot finish the job within given AWS time limits requiring you to think about chunking work into multiple parallel executable segments. The last piece will show you how a file upload to the S3 bucket can trigger Lambda execution and how you might benefit from such an event.

CloudFormation – a way of defining infrastructure you might fall in love with

In AWS there are three ways you can create a resource: using AWS Console, Command Line Interface (CLI), and through CloudFormation.

All of these approaches are perfectly fine and selecting the one you will use depends on your scenario or preference. E.g. if you’d like to create a resource quickly, with UI instructing you every step of the way – you might choose to use AWS Console. It is perhaps the easiest and the best way to start experimenting if you’re completely new to AWS. In case you wanted to have slightly more programmatic access, AWS CLI might be a more suitable approach. We often use CLI in our CI pipelines to create a resource or check resource property “on the fly” during the build process.

However, what if you wanted to build a somewhat more complex infrastructure of AWS components and you wanted to describe those components through a designated, readable language? What if you wanted to be able to rebuild such infrastructure from scratch whenever you want, or, perhaps, create an infrastructure for another environment with slightly different settings? Well, this is where CloudFormation comes into play. CloudFormation provides you with the ability to treat your infrastructure as a code – store it in YAML or JSON file and commit it to a repo, just as if it was any other source code file.

Being into CloudFormation as we are, we hope you won’t mind us using CloudFormation YAML templates to describe the infrastructure for the purpose of this blog series.

Prerequisites

If you are new to AWS, go ahead and create yourself a free account – https://aws.amazon.com/free/. It will be more than enough to do anything from this blog post series and it won’t hurt your wallet. A disclaimer though – be responsible and read what is included for free and what is not. E.g. if you decide to spin up dozens of EC2 instances and you forget to shut them down – you may end up being charged with the amount you never really intended to pay :).

Now, we are sure you are a responsible person, so go and try out AWS Console yourself – e.g. play with creating AWS Lambda and see what configuration parameters exist. It should be easier to follow our next posts on this topic.

That’s it – these are the only prerequisites for you to be able to create any piece of AWS infrastructure described in this blog post series!

Where do you go from here?

Please follow the links to the blog posts belonging to this series. We’ll add them as we go so stay tuned.