DevOps on AWS - Tips and Tricks

This is my list of hints and tips for this course. It’s markdown so you can save it, access it or store it anywhere. I might also give you other links that are course specific. I’ll add specific answers to questions I get during the course. I’ll share it with everyone.

Your Instructors

• Ian Falconer https://www.linkedin.com/in/leftbrainstuff/

Administrivia

We need to jump through some hoops to get access to the labs, notes and my hints and tips. Be consistent with the email address you use for all sites. There are three seperate sites you need to access and one bitly link which is this page:

• Join or login to https://www.aws.training/ to ensure your training and certifications are captured. No we don’t spam you or sell your details.
• Access Qwiklab (yes it is spelt INCORRECTLY)
  • aws.qwiklabs.com for the labs in this class
  • run.qwiklabs.com for outside of the class or to do other labs at your own pace.
  • NOTE: Some are free others require course credits. Also check out the AWS Professional Developer Series of MOOCs on edX https://www.edx.org/aws-developer-professional-series and there are MOOCs on Coursera too.
• Access the course notes and slides. You’ll receive two emails. One confirming your attendance at this course and with the following links. The download link seems broken. You can download apps for phones, tablets and laptops. Or use your browser.
• www.vitalsource.com look for a signup link and download link. Or just go to https://evantage.gilmoreglobal.com/#/user/signin
• Once you’ve logged into Vitalsource (aka Bookshelf, Gilmore, eVantage) you can redeem your unique course materials code (in a seperate email) and update your book list. You should see a lab guide and student guide for DevOps on AWS, version 2.3 and 2.4. Just pick the latest version as they’ve bundled together by the vitalsource folks.
• The student guide is the powerpoint decks and notes and the lab guide is the step by step instructions for the labs. The lab guide is included in the labs so this document is somewhat redundant. You can download the Vitalsource Bookshelf app for Windows, Mac, IoS and Android at https://support.vitalsource.com/hc/en-us/articles/201344733-Bookshelf-Download-Page
• You can probably print the student and lab guides to pdf from the app. You’ll need to confirm this as the vitalsource folks update the app and website each week.

Group and Individual Exercises

What is DevOps?

• split into groups and agree on a definition of DevOps. Also describe key attributes of a DevOps culture. Also describe issues or challenges with DevOps. What DevOps metrics do you track?
• Time 20 - 30 minutes max
• Nominate a spokesperson and spend 2 or 3 minutes describing your findings to the class

Pair Exercise

Visit TODO this link is broken http://scorekeep-env.dpua9j3gcs.us-west-2.elasticbeanstalk.com/ and find a partner or play tictactoe against a random generated player. The instructor will display the AWS Xray traces on the screen. NOTE read the instructions to setup and play the game

Cool links

• James Hamilton, AWS SVP, talks about our infrastructure. This should be the first AWS video you watch. Here’s the youtube video. https://www.youtube.com/watch?v=JIQETrFC_SQ There are Youtube videos from more recent ReInvents with some updates too. Here is James in 2016. It’s titled as Global Innovation at Scale. https://www.youtube.com/watch?v=uj7Ting6Ckk James Hamilton also publishes blog posts on AWS Infrastructure regularly. Here’s one on number of data centers titled How Many Data Centers Needed World-Wide
• AWS re:Invent 2017: Scaling Up to Your First 10 Million Users (ARC201). This is like the Tech Essentials course in a single video. Well worth a watch. https://www.youtube.com/watch?v=w95murBkYmU
• Amazon EC2 Instance Types explained in neat tabular comparisons. https://aws.amazon.com/ec2/instance-types/ . Also here’s a third party site that has a table that lets you sort on memory, network performance, cost and instance type. You can also quickly compare costs here too. https://ec2instances.info/ . Here’s a stackoverflow thread on non AWS benchmarks of different instance types. https://stackoverflow.com/questions/20663619/what-does-amazon-aws-mean-by-network-performance
• 9 Metrics DevOps Teams Should be Tracking: http://www.datical.com/blog/9-metrics-devops-teams-tracking/
• https://github.com/jmespath/jmespath.terminal), a Linux command line application, which can be installed on an instance using Pip.
• Implementing DevSecOps Using AWS CodePipeline: https://aws.amazon.com/blogs/devops/implementing-devsecops-using-aws-codepipeline/
• AWS DevOps Whitepaper from 2014 http://d0.awsstatic.com/whitepapers/AWS_DevOps.pdf
• I visit the AWS Service Limits page frequently. It pays to be aware of soft and hard limits of a service. Soft limits can be increased easily through a support request. Hard limit removal either indicates you’re trying to do something in a non optimal way or that you have a specific use case not yet supported. AWS product teams are happy to work with customers on feature requests and the like. https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
• What is DevOps?
  • Agile manifesto https://agilemanifesto.org/ 4 behaviours and 12 principles
  • Modern summary of agile and DevOps https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91
  • DevOps Topologies https://web.devopstopologies.com/
• DevOps tools
  • Summary of the major CICD tools and frameworks https://hostadvice.com/blog/devops-toolbox-jenkins-ansible-chef-puppet-vagrant-saltstack/ including the The SUSE DevOps Framework (aka DevOps Tooling Quagmire)
  • Orchestration tooling in lieu of DevOps CICD tooling https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-puppet-ansible-saltstack-or-cloudformation-7989dad2865c
• Here is a community effort to summarize AWS documentation for a number of services and concepts. It’s called the AWS Open Guide on Github. Some information may be stale so confirm in the documentation links https://github.com/open-guides/og-aws#cloudformation
• Also check out the AWS Glossary https://docs.aws.amazon.com/general/latest/gr/glos-chap.html and the General Reference https://docs.aws.amazon.com/general/latest/gr/Welcome.html
• Acloud Guru born in the cloud https://acloud.guru/our-story is an effective user of the AWS Free Tier and Serverless
• Amazon Elastic Beanstalk is fully integrated compute environment for deploying, operating and maintaining production grade applications running on Amazon EC2.
  • AWS Elastic Beanstalk Worker Environments which work well with SQS (not Windows IIS) to build queue chaining workloads https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
• One page link to all the AWS Service documentation pages https://docs.aws.amazon.com/index.html#lang/en_us
• DotStep is a framework for creating AWS Step Functions using a code first approach. https://github.com/paulfryer/dotstep.git
• Here is a full serverless backend and front end app in github, https://github.com/aws-samples/aws-serverless-airline-booking and the full build on Twitch https://pages.awscloud.com/GLOBAL-devstrategy-OE-BuildOnServerless-2019-reg-event.html
• How does Amazon and AWS do DevOps?
  • The Amazon Builders’ Library is a collection of papers written by domain and distributed systems experts from within Amazon and AWS. https://aws.amazon.com/builders-library
  • AWS DevOps entry page https://aws.amazon.com/devops

Best Practice

• http://www.opendatacenteralliance.org/docs/DevCloudCapApp.pdf (This pdf has some good information about developing cloud applications)
• https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/ 􏰀
• http://www.theserverside.com/feature/Developing-for-the-Cloud-How-Developing-in-the-Cloud-is-Different
• The AWS Blog has some great posts on DevOps. https://aws.amazon.com/blogs/devops/
• Centralized Logging (aka log everything) is a recommended best practice cloud pattern. You can access the deployment guide and cloudformation templates at https://aws.amazon.com/solutions/centralized-logging/
• AWS Secrets Manager vs. Hashicorp Vault vs. AWS Parameter Store https://hackernoon.com/aws-secrets-manager-vs-hashicorp-vault-vs-aws-parameter-store-bcbf60b0c0d1
• DevOps and CICD require an existing environment to operate in. Often that environment needs to scale across accounts and teams. We can use the same IaaC methods to build these foundational elements with a centralized management theme. AWS services like AWS Organizations, AWS Service Catalog and AWS Budgets can all be used to create software defined and managed orgs. Here are some interesting links that will get you thinking big about DevSecOps:
  • Automate account creation, and resource provisioning using AWS Service Catalog, AWS Organizations, and AWS Lambda https://aws.amazon.com/blogs/mt/automate-account-creation-and-resource-provisioning-using-aws-service-catalog-aws-organizations-and-aws-lambda/
  • How to update AWS Service Catalog provisioned products to new product versions and report changes using AWS Step Functions, AWS Lambda, and Amazon Athena https://aws.amazon.com/blogs/mt/how-to-update-aws-service-catalog-provisioned-products-to-new-product-versions-and-report-changes-using-aws-step-functions-aws-lambda-and-amazon-athena/
  • Create a security partition for your applications using AWS Service Catalog and AWS Lambda https://aws.amazon.com/blogs/mt/create-a-security-partition-for-your-applications-using-aws-service-catalog-and-aws-lambda/
  • Don’t forget to see what our AWS partners are doing. This view will give you a feel for what customers are doing to integrate AWS into their existing enviroments. The Most Viewed APN Blog Posts in 2018 https://aws.amazon.com/blogs/apn/the-most-viewed-apn-blog-posts-in-2018/
  • How to GitOps Your Infrastructure https://aws.amazon.com/blogs/startups/how-to-gitops-your-infrastructure/
• Who’s monitoring the monitors? AWS solutions like the limit monitor help you be predictive or detective about limits. https://aws.amazon.com/solutions/limit-monitor/

Testing

• Taskcat was developed by the AWS Quickstart team (quickstarts are AWS reference architectures) https://github.com/aws-quickstart/taskcat
  • A deep dive into testing with TaskCat https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/
  • and the earlier blog post titled Up your AWS CloudFormation testing game using TaskCat https://aws.amazon.com/blogs/infrastructure-and-automation/up-your-aws-cloudformation-testing-game-using-taskcat/
• Testing accuracy and regression with Amazon Connect and Amazon Lex https://aws.amazon.com/blogs/contact-center/testing-accuracy-and-regression-with-amazon-connect-and-amazon-lex/
• Using codedeploy-local in How to Test and Debug AWS CodeDeploy Locally Before You Ship Your Code https://aws.amazon.com/blogs/devops/how-to-test-and-debug-aws-codedeploy-locally-before-you-ship-your-code/
• UI Testing at Scale with AWS Lambda https://aws.amazon.com/blogs/devops/ui-testing-at-scale-with-aws-lambda/
• Testing AWS GameDay with AWS Well-Architected Framework – Remediation https://aws.amazon.com/blogs/apn/testing-aws-gameday-with-the-aws-well-architected-framework-remediation/
• An intro to Lambda testing in Improved Testing on the AWS Lambda Console https://aws.amazon.com/blogs/compute/improved-testing-on-the-aws-lambda-console/
• Using AWS CodePipeline, AWS CodeBuild, and AWS Lambda for Serverless Automated UI Testing https://aws.amazon.com/blogs/devops/using-aws-codepipeline-aws-codebuild-and-aws-lambda-for-serverless-automated-ui-testing/
• The AWS SaaS Factory is a set of AWS partner tools. Read about Testing SaaS Solutions on AWS https://aws.amazon.com/blogs/apn/testing-saas-solutions-on-aws/ which describes SaaS and third party API specific testing that should be part of your integration tests, site reliability approach and failover testing
• Distributed Load Testing Using Fargate https://github.com/aws-samples/distributed-load-testing-using-aws-fargate Solution to setup AWS Fargate to run multi-region distributed performance testing. Runs Distributed Load Tests using AWS Fargate and Taurus. You can use it to test your services under high stress scenarios and understand it’s behavior and scalability.
• Some useful IaaC patterns using CodePipeline, Cloudformation and Step Functions
  • Self paced workshop demonstrating CICD best practice using the AWS Code* tool suite and third party frameworks. Includes unit and integration testing cloudformation built environments. This example has you manually build in the management console. This would be more efficient using some method of IaaC. https://github.com/aws-samples/aws-serverless-workshops/tree/master/DevOps
  • A Cloudformation and ChangeSet Codepipeline approach to CICD https://github.com/aws-samples/codepipeline-nested-cfn.git
  • aws-codepipeline-stepfunctions https://github.com/aws-samples/aws-codepipeline-stepfunctions.git Integrate AWS CodePipeline and AWS Step Functions state machines. The integration enables developers to build much simpler CodePipeline actions that perform a single task and to delegate the complexity of dealing with workflow-driven behavior associated with that task to a proper state machine engine. As such, developers will be able to build more intuitive pipelines and still being able to visualize and troubleshoot their pipeline actions in detail by examining the state machine execution logs.
  • Building a Kubernetes CI/CD pipeline on AWS with CodePipeline & CodeBuild @ Shopgun https://itnext.io/building-a-kubernetes-ci-cd-pipeline-on-aws-with-codepipeline-codebuild-shopgun-43ccf76277b5
• Amazon Aurora Performance Assessment and IaaC to setup your test environment https://d1.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf and How Aurora Serverless Works https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html

Compute Links

• aws cli wait command will time out after 120 checks. They’re labled as failed checks in the documentation but they aren’t strictly a failure. Whatever the wait command is waiting on never reaches the wait state if it times out. The timeout period can vary so check the documentation for the service and wait state you’re interested in. Here’s EBS https://docs.aws.amazon.com/cli/latest/reference/ec2/wait/snapshot-completed.html
• Diagnose failed status checks
  • Status Checks for Your Instances https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html
  • Troubleshooting Instances with Failed Status Checks https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html
• When choosing EC2 instances you need to consider not just compute but also networking. Get familiar with the following EC2 networking documentation.
  • The Floodgates Are Open – Increased Network Bandwidth for EC2 Instances (circa Jan 2019), https://aws.amazon.com/blogs/aws/the-floodgates-are-open-increased-network-bandwidth-for-ec2-instances/
  • Elastic Network Adapter – High Performance Network Interface for Amazon EC2 (circa Jun 2016) https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/
  • Placement Groups https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html
  • Enhanced Networking on Linux https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html
  • Enhanced Networking on Windows https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking.html
• Rails
  • Deploy and Manage Production Rails 5 App with AWS Elastic Beanstalk https://medium.com/@tranduchanh.ms/deploy-and-manage-production-rails-5-app-with-aws-elastic-beanstalk-3efb0dfe021a
  • Using AWS Secrets Manager in Rails https://syndicode.com/2019/01/17/using-aws-secrets-manager-in-rails/
  • Best Practices for Managing AWS Access Keys https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html
• Running Windows on AWS. Here are some interesting IaaC Microsoft examples:
  • AWS Quickstarts with Microsoft resources https://aws.amazon.com/quickstart/?quickstart-all.sort-by=item.additionalFields.updateDate&quickstart-all.sort-order=desc&awsf.quickstart-homepage-filter=categories%23microsoft

Containers

• A deep dive on Fargate. Lot’s of feature updates around container orchestration so watch the AWS what’s new for the latest. Here’s a slide share that deep dives on Fargate. https://de.slideshare.net/AmazonWebServices/deep-dive-into-aws-fargate
• Configuring Cloudwatch logs with containers. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html
• ECS and Fargate blue green cloudformation templates (from the blog post) https://github.com/aws-samples/ecs-blue-green-deployment
• Kubernetes
  • Making Cluster Updates Easy with Amazon EKS https://aws.amazon.com/blogs/compute/making-cluster-updates-easy-with-amazon-eks/ describes new API calls and the rapid evolution of EKS so consider frequent upgrade paths
  • KubeCon Seattle 2018 Recap https://aws.amazon.com/blogs/opensource/kubecon-seattle-2018-recap/ notice AWS has shared our EKS product roadmap in Github
  • Another nice EKS workshop: https://eksworkshop.com/introduction/ .This workshop covers k8 basics and some cool things such as the k8 dashboard, helm etc.. :-) https://ecsworkshop.com/ and https://eksworkshop.com/
  • Updates to Amazon EKS Version Lifecycle (EKS v1.1.10 will be deprecated on 22 July 2019 in line with K8S community support) https://aws.amazon.com/blogs/compute/updates-to-amazon-eks-version-lifecycle/
• ECS Vs. EKS Vs. Fargate is a simple tabular comparison of these 3 AWS Container services https://blog.totalcloud.io/ecs-vs-eks-vs-fargate-good-bad-ugly/
• EFS support by ECS? According to the container roadmap (public) it’s being worked on Yes EFS is on the roadmap https://github.com/aws/containers-roadmap/projects/1?card_filter_query=efs and there are some third party support for sharing state between containers using EFS as the common data store, ala Amazon ECS and Docker volume drivers, part 2: Amazon EFS https://aws.amazon.com/blogs/compute/amazon-ecs-and-docker-volume-drivers-amazon-efs/
• Instrumenting Kubernetes for Observability using AWS X-Ray and Amazon CloudWatch https://github.com/aws-samples/reinvent2018-dev303-code
• Self paced 10 hr (2+4+4) Kubernetes workshop. AWS Workshop for Kubernetes https://github.com/aws-samples/aws-workshop-for-kubernetes
• Firecracker microvm that underpins AWS Lambda and AWS Fargate
  • Firecracker Announcement (circa Nov 2018) Firecracker – Lightweight Virtualization for Serverless Computing (Secure and fast microVMs for serverless computing and containers) https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/
  • and here’s the github page https://firecracker-microvm.github.io/
  • and here is the May 2019 Firecracker Open Source Update May, 2019 https://aws.amazon.com/blogs/opensource/firecracker-open-source-update-may-2019/
  • Firecracker design document deep dives on the technology and how Firecracker is designed to be highly secure, performant, consistent and suited from small low power devices to large multitenant deployments https://github.com/firecracker-microvm/firecracker/blob/master/docs/design.md
• EKS Windows Preview is available from early 2019 https://github.com/aws/containers-roadmap/tree/master/preview-programs/eks-windows-preview

Storage links

• S3 Transfer Acceleration Speed Checker
• http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html uses a multi part upload to check the speed difference when using S3 transfer acceleration between regions.
• S3 Deep Dive Mar 2017 https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-s3-march-2017-aws-online-tech-talks
• Run a Static Website on Amazon S3
  • Here’s the AWS Static Website builder 30min in 7 steps https://aws.amazon.com/getting-started/projects/host-static-website/
  • Want to watch a video AWS Quick Start - Hosting a Static Website on AWS (Demo) https://www.youtube.com/watch?v=BpFKnPae1oY
  • Try using Hugo to build a simple deployment tool to update your static website. Using Hugo and AWS to build a fast, static, easily managed and deployed website. https://dev.to/tom_geraghty/using-hugo-and-aws-to-build-a-fast-static-easily-managed-and-deployed-website-2fme
  • For a more sophisticated CICD pipeline for your static website at https://docs.aws.amazon.com/codestar/latest/userguide/templates.html

Networking Links

• List of CIDR ranges of AWS regions http://ec2-reachability.amazonaws.com/
• Check out the AWS Global Accelerator service which gives you non DNS (layer 4) options for multi region connectivity. https://aws.amazon.com/global-accelerator/
• For those looking for a Network refresh (100 level) checkout the free AWS digital training. Start with Understanding CIDR Notation https://www.aws.training/learningobject/video?id=16480 then Introduction to Virtual Private Cloud (VPC) https://www.aws.training/learningobject/video?id=15884 and Subnets, Gateways and Route Tables explained https://www.aws.training/learningobject/video?id=16490
• A solid reference for AWS networking is the https://www.amazon.com/Certified-Advanced-Networking-Official-Study/dp/1119439833/ref=sr_1_1?s=books&ie=UTF8&qid=1519925473&sr=1-1&keywords=advanced+networking https://aws.amazon.com/certification/certified-advanced-networking-specialty/ Note that some of the more recent updates (ReInvent 2018) are not included. VPC Transit Gateway and Global Accelerator. But you can check out their FAQ pages.

IAM

• Policy summaries. Making sense of complicated permission chains. https://aws.amazon.com/blogs/security/move-over-json-policy-summaries-make-understanding-iam-policies-easier/#more-2875

Scripting

• Big collection of mighty useful bash scripts for AWS admin tasks. https://github.com/swoodford/aws
• AWS documentation contains useful CLI commands like this https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html#finding-an-ami-cli. 6 tips and tricks for AWS command-line ninjas https://cloudonaut.io/6-tips-and-tricks-for-aws-command-line-ninjas/
• Lots of API code examples from AWS documentation https://github.com/awsdocs/aws-doc-sdk-examples and also check out the Deep Dive: AWS Command Line Interface https://www.youtube.com/watch?v=ZbgvG7yFoQI if you prefer a video guide
• There are numerous CLI quirks for different output formats and OS’s. Check out Controlling Command Output from the AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html#cli-usage-output-filter
• JMESPath is the JSON query engine for the AWS CLI. Here’s a tutorial, Useful JMESPath in AWS CLI, https://blog.ashiny.cloud/2017/09/25/useful-jmespath-in-awscli/ and also check out the Deep Dive: AWS Command Line Interface https://www.youtube.com/watch?v=ZbgvG7yFoQI. Here’s an install tutorial for the jmespath-terminal titled Interactive AWS CLI Query Filtering with JSONPath https://random.ac/cess/2017/10/31/interactive-aws-cli-query-filtering/
• Google Chrome and Firefox Console Recorder for AWS https://github.com/iann0036/AWSConsoleRecorder/ Records actions made in the AWS Management Console and outputs the equivalent CLI/SDK commands and CloudFormation/Terraform templates.
• Things you need to know about the AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html
• AWS CLI Queries and jq (an alternative to JMESPath) https://theagileadmin.com/2017/05/26/aws-cli-queries-and-jq/. Also check out a bash script example that is dissected and explained titled AWS CLI Queries and jq https://theagileadmin.com/2017/05/26/aws-cli-queries-and-jq/
• AWS Shell is productivity booster for the AWS CLI https://github.com/awslabs/aws-shell
• More JMESPath examples
  • Useful JMESPath in AWS CLI https://blog.ashiny.cloud/2017/09/25/useful-jmespath-in-awscli/
  • Advanced AWS CLI JMESPath Query Tricks https://opensourceconnections.com/blog/2015/07/27/advanced-aws-cli-jmespath-query/ including finding AMI ids to feed into cloudformation rather than embedding ami ids.
• Use the AWS Systems Manager Parameter Store to Query for AWS Regions, Endpoints at https://aws.amazon.com/blogs/aws/new-query-for-aws-regions-endpoints-and-more-using-aws-systems-manager-parameter-store/
• CloudFormation Helper Scripts Reference and the child links to each helper script https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html
  • cfn-init https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html
  • cfn-hup https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-hup.html
  • cfn-signal https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-signal.html
  • cfn-metadata https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-get-metadata.html

Infrastructure as Code (IaaC)

• AWS Cloudformation samples and reference material https://aws.amazon.com/cloudformation/aws-cloudformation-templates/
• AWS Cloudformation supported AWS resources https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html This json file provides direct links to documentation to which I find very helpful
• A great way to understand how Cloudformation can build, update and delete immutable or mutable environments is to reverse engineer AWS Quickstarts (gold standard reference architectures). Check out https://aws.amazon.com/quickstart/saas/identity-with-cognito/ for the deployment guide and https://github.com/aws-quickstart/saas-identity-cognito for all the Cloudformation templates.
• You can validate your Cloudformation templates using https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-validate-template.html or by using a JSON or YAML linter. You can also build and run a pipeline for validating your cloudformation templates https://docs.aws.amazon.com/solutions/latest/aws-cloudformation-validation-pipeline/deployment.html
• Cloudformation to Terraform and vice versa https://www.npmjs.com/package/@humanmade/cf-to-tf cf-to-tf is a node CLI tool. from tf to cf requires cloudformer which only supports some AWS services https://stackoverflow.com/questions/47065279/convert-terraform-templates-to-cloudformation-templates
• Cloudformation Template Reference Guide contains all the scaffolding and reference material you need to build cloudformation templates from scratch. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html
• Choosing a deployment tool with so many options. Not to mention now considering containers and servers
  • Here’s a long but bulleted summary of deployment tooling. Called Choosing a deployment tool, https://gist.github.com/jaceklaskowski/bd3d06489ec004af6ed9 , - ansible vs puppet vs chef vs salt but it also ties in Docker containers. Would be nice to see Kubernetes also included.
  • Using Chef, Puppet, and Ansible to Manage Kubernetes https://logdna.com/blog/using-chef-puppet-and-ansible-to-manage-kubernetes/
• How do you find out which CloudFormation changes cause what type of interruptions? That’ll vary by each resource and property type. Have a look at some of the ones noted in here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html
• AWS Cloud Development Kit is an IaaC SDK https://github.com/awslabs/aws-cdk has more than 120 contributors, more than 1200 commits and is in developer preview on github.
  • Here are some CDK examples you can try. TypeScript, Java, Python and Javascript supported. https://docs.aws.amazon.com/cdk/latest/guide/about_examples.html
  • The CDK Workshop https://cdkworkshop.com/
  • Patterns prebuilt using the AWS CDK https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-patterns-readme.html %TODO add SLOC comparison between CDK and Cloudformation
  • AWS CDK vs Terraform vs CloudFormation https://medium.com/@kavukcu.tolga/aws-cdk-vs-terraform-vs-cloudformation-621421d1dad4
• Understanding Chef components https://docs.chef.io/chef_overview.html

Multiple Account Best Practice and Examples

• How about multi region CICD. Using AWS CodePipeline to Perform Multi-Region Deployments https://aws.amazon.com/blogs/devops/using-aws-codepipeline-to-perform-multi-region-deployments/
• Multi cloud guardrails and IaaC using Turbot. https://turbot.com/features/ This team grew from the Johnson and Johnson xbot HPC orchestration and guardrail service. The original xbot technology is circa 2015 https://www.youtube.com/watch?v=za1EysyUVS0
• Distributed Load Testing Using Fargate https://github.com/aws-samples/distributed-load-testing-using-aws-fargate Solution to setup AWS Fargate to run multi-region distributed performance testing. Runs Distributed Load Tests using AWS Fargate and Taurus. You can use it to test your services under high stress scenarios and understand it’s behavior and scalability.
• AWS Landing Zone is designed to quickly set up a secure, multi-account AWS environment based on AWS best practiceshttps://aws.amazon.com/solutions/aws-landing-zone/?did=sl_card&trk=sl_card
• Enabling self-service provisioning of AWS resources with AWS Control Tower https://aws.amazon.com/blogs/mt/enabling-self-service-provisioning-of-aws-resources-with-aws-control-tower/ . This solution uses the following AWS services:
  • AWS Control Tower
  • AWS Service Catalog
  • AWS CloudFormation
  • Amazon CloudWatch
  • Amazon RDS
  • AWS Organizations
• AWS Service Catalog Integration with AWS Budgets. This integration lets you manage cost and manage centralized deployment assets. https://aws.amazon.com/blogs/aws-cost-management/launch-aws-service-catalog-integration-with-aws-budgets/
• Automate account creation, and resource provisioning using AWS Service Catalog, AWS Organizations, and AWS Lambda https://aws.amazon.com/blogs/mt/automate-account-creation-and-resource-provisioning-using-aws-service-catalog-aws-organizations-and-aws-lambda/

CICD

• Set Up a CI/CD Pipeline on AWSAutomate your software delivery process using continuous integration and delivery (CI/CD) pipelines Then go to the implementation guide link for 9 build templates. https://aws.amazon.com/getting-started/projects/set-up-ci-cd-pipeline/faq/ and in github at https://github.com/awsdocs/aws-codepipeline-user-guide/blob/master/doc_source/tutorials.md
• AWS Quickstarts (gold standard reference deployments) like Blue-Green Deployments to AWS Elastic Beanstalk on the AWS Cloud https://aws-quickstart.s3.amazonaws.com/quickstart-codepipeline-bluegreen-deployment/doc/blue-green-deployments-to-aws-elastic-beanstalk-on-the-aws-cloud.pdf can be reverse engineered for reuse. Note the clean nomenclature and informative documentation provided.
• AWS Codepipeline integrations are quite diverse. Visit https://aws.amazon.com/codepipeline/product-integrations/ for a sample, and more detail at https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations-action-type.html and then grab a list of blog posts by third parties on how they integrate with Codepipeline to build customer CICD environments. https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations.html and more specifically https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations-action-type.html
• Codebuild and Maven demo project step by step https://docs.aws.amazon.com/codebuild/latest/userguide/getting-started.html and from the Java sdk https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/codebuild/AWSCodeBuild.html
• Choosing what deployment and orchestration tools to use can be a challenge. Here are just two discussions on this topic.
  • When to use Elastic Beanstalk?…OpsWorks?…CloudFormation? https://aritexit.com/post/use-elastic-beanstalk-opsworks-cloudformation/
  • Opsworks vs Puppet https://www.upguard.com/articles/opsworks-vs-puppet
• Safe Lambda Deployments includes rolling deployment demos https://github.com/awslabs/serverless-application-model/blob/master/docs/safe_lambda_deployments.rst
• Jenkins
  • From the AWS DevOps Blog Simplify Your Jenkins Builds with AWS CodeBuild https://aws.amazon.com/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/ for managing Jenkins build environments at scale.
  • Here’s a companion serverless build presentation, from ReInvent 2017, titled DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline and Codestar (3 methods in one talk) https://www.slideshare.net/AmazonWebServices/dev315automating-lambda-deployments-with-github-jenkins-aws-codepipeline-and-codestar Here’s the youtube video https://www.youtube.com/watch?v=lYYLGBdFXqM.
  • AWS CodeBuild Jenkins Plugin https://github.com/awslabs/aws-codebuild-jenkins-plugin
  • Building Continuous Deployment on AWS with AWS CodePipeline, Jenkins and AWS Elastic Beanstalk https://aws.amazon.com/blogs/devops/building-continuous-deployment-on-aws-with-aws-codepipeline-jenkins-and-aws-elastic-beanstalk/
  • How about serverless instead of using Jenkins. This pattern moves the more complex CICD logic out of CodePipeline into AWS Step Functions where much more complex and efficient testing logic can be implemented. Using AWS Step Functions State Machines to Handle Workflow-Driven AWS CodePipeline Actions https://aws.amazon.com/blogs/devops/using-aws-step-functions-state-machines-to-handle-workflow-driven-aws-codepipeline-actions/
• Don’t forget to look at what our AWS partners and their tooling can help with. Here’s just one interesting example I found on the AWS Blogs Dome9 on static code analysis of your infrastructure as code https://aws.amazon.com/blogs/startups/dome9-on-static-code-analysis-of-your-infrastructure-as-code/
• aws-codepipeline-stepfunctions - enables developers to build much simpler CodePipeline actions that perform a single task and to delegate the complexity of dealing with workflow-driven behavior associated with that task to a proper state machine engine. As such, developers will be able to build more intuitive pipelines and still being able to visualize and troubleshoot their pipeline actions in detail by examining the state machine execution logs. https://github.com/aws-samples/aws-codepipeline-stepfunctions/blob/master/README.md
• Using Slack to trigger pipeline approvals. Use Slack ChatOps to Deploy Your Code – How to Integrate Your Pipeline in AWS CodePipeline with Your Slack Channel https://aws.amazon.com/blogs/devops/use-slack-chatops-to-deploy-your-code-how-to-integrate-your-pipeline-in-aws-codepipeline-with-your-slack-channel/
• Another serverless CICD example from AWS Solutions. It has quite a busy architecture diagram and uses Secrets Manager as part of the CICD pipeline. It’s titled Serverless CI/CD for the Enterprise on AWS and can be found at https://aws.amazon.com/quickstart/architecture/serverless-cicd-for-enterprise/ . Beware that this Quickstart currently uses python2.7 and older version of boto3. These will need updating.
• A deep dive on deploying serverless applications using AWS Lambda functions into production. It’s titled Safe Lambda deployments https://github.com/awslabs/serverless-application-model/blob/master/docs/safe_lambda_deployments.rst

Logging and Monitoring

• Is the limitation for insights still one log group at a time? Cloudwatch service limits https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html A maximum of 4 concurrent CloudWatch Logs Insights queries, including queries that have been added to dashboards. You can request a limit increase. CloudWatch Logs Insights can discover a maximum of 1000 log event fields in a log group. This limit cannot be changed.

Streaming

• Announcing Amazon Kinesis SubscribeToShard API Support in the AWS SDK for Ruby https://aws.amazon.com/blogs/developer/announcing-amazon-kinesis-subscribetoshard-api-support-in-the-aws-sdk-for-ruby/
• Several Kinesis posts in The most-viewed AWS Database Blog posts in 2018 https://aws.amazon.com/blogs/database/the-most-viewed-aws-database-blog-posts-in-2018/
• Build More Reliable and Secure Windows Services Using Amazon Kinesis Agent for Microsoft Windows https://aws.amazon.com/blogs/opensource/amazon-kinesis-agent-microsoft-windows/
• AWS Labs on Github for Kinesis https://github.com/awslabs?utf8=%E2%9C%93&q=Kinesis&type=&language=
• AWS Samples on Github for Kinesis https://github.com/aws-samples?utf8=%E2%9C%93&q=Kinesis&type=&language=
• aws-lambda-fanout - eplicate data from an Amazon Kinesis Stream to another account or another region for processing, or to another environment such as development https://github.com/aws-samples/aws-lambda-fanout
• Several architectures for ingestion, transformation and querying of streaming data
  • https://aws.amazon.com/blogs/big-data/optimizing-downstream-data-processing-with-amazon-kinesis-data-firehose-and-amazon-emr-running-apache-spark/
  • https://aws.amazon.com/blogs/big-data/build-and-run-streaming-applications-with-apache-flink-and-amazon-kinesis-data-analytics-for-java-applications/
  • https://aws.amazon.com/blogs/compute/building-a-scalable-log-solution-aggregator-with-aws-fargate-fluentd-and-amazon-kinesis-data-firehose/
  • https://aws.amazon.com/blogs/big-data/create-real-time-clickstream-sessions-and-run-analytics-with-amazon-kinesis-data-analytics-aws-glue-and-amazon-athena/
  • New – Compute, Database, Messaging, Analytics, and Machine Learning Integration for AWS Step Functions https://aws.amazon.com/blogs/aws/new-compute-database-messaging-analytics-and-machine-learning-integration-for-aws-step-functions/

Serverless

• Lambda videos to watch. Sometimes it’s hellful to start from the beginning before you dive deep into more sophisticated uses of serverless and Lambda.
  • AWS re:Invent 2014 | (MBL202) NEW LAUNCH: Getting Started with AWS Lambda https://www.youtube.com/watch?v=UFj27laTWQA
  • AWS re:Invent 2015 | (CMP301) AWS Lambda and the Serverless Cloud https://www.youtube.com/watch?v=pBLdMCksM3A
  • AWS Lambda: Event-driven Code in the Cloud (circa 2015) https://www.youtube.com/watch?v=copO_JQQsBs
• Amplify self paced builds
  • Building Serverless Web Applications with React and AWS Amplify https://github.com/dabit3/aws-amplify-workshop-react
  • Building Mobile Applications with React Native & AWS Amplify https://github.com/dabit3/aws-amplify-workshop-react-native
  • AWS Awesome Days (or self paced deep dives) are a great way to build and dive deep.
    • AWS Amplify Awesome https://github.com/dabit3/awesome-aws-amplify
    • AWS AppSync Awesome https://github.com/dabit3/awesome-aws-appsync

Caching

• Loss of caching nodes could be due to underlying hardware failure, reboots or restarts during a maintenance window or even from the loss of an AZ. https://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/FaultTolerance.html

Security

• Neat graphic and detailed description of signature version 4 signing of authenticating requests. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html information about signature version 2 (generally deprecated and less preferred than signature version 4) . Here are some query examples https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
• Secrets Manager comparison to SSM Parameter Store. Here I start with what’s new in 2018 on AWS and search for the word secrets. Then keep exploring the read more links and finally the FAQs for Secrets Manager. Starting with https://aws.amazon.com/about-aws/whats-new/2018/ and ending up at Q: What is the difference between Secrets Manager and Parameter Store? in https://aws.amazon.com/systems-manager/faq/
• Creating a Golden AMI Pipeline Integrated with Qualys for Vulnerability Assessments https://aws.amazon.com/blogs/apn/creating-a-golden-ami-pipeline-integrated-with-qualys-for-vulnerability-assessments/
• AWS Config RDK: Multi-account and multi-Region deployment https://aws.amazon.com/blogs/mt/aws-config-rdk-multi-account-and-multi-region-deployment/
• Amazon Macie uses Support Vector Machine–Based Classifier to discover, classify a range of data types https://docs.aws.amazon.com/macie/latest/userguide/macie-classify-objects-classifier.html
• Security Hub and Control Tower https://aws.amazon.com/blogs/enterprise-strategy/aws-control-tower-and-aws-security-hub-powerful-enterprise-twins/

• The Full List of the Security, Compliance, and Identity Sessions, Workshops, and Chalk Talks Being Offered at AWS re:Invent 2017 (these links are great for a deep dive on security) https://aws.amazon.com/blogs/security/a-full-list-of-the-security-compliance-and-identity-sessions-workshops-and-chalk-talks-available-at-aws-reinvent-2017/

Amazon Connect

• Diagnosing contact flows. You can access the contact flow logs in near realtime using Cloudwatch or Kinesis streams. Refer to https://docs.aws.amazon.com/connect/latest/adminguide/connect-ag.pdf#amazon-connect-service-limits in the section Contact Flow Logs.
• Building a state-aware workflow with Amazon Connect and AWS Step Functions https://aws.amazon.com/blogs/contact-center/building-a-state-aware-workflow-with-amazon-connect-and-aws-step-functions/

Self paced Learning and Building

• AWS Certification roadmap https://aws.amazon.com/certification/ Check out the learning paths link at the bottom of the page.
• Read the service FAQ pages, http://aws.amazon.com/faqs/, and documentation for each of the services. Just search for AWS + + documentation in any search engine. You can keep the documentation as pdf, html online or even in your Kindle. You can also git clone the documentation for most services.
• Find and build interesting AWS and partner solutions you find the in AWS Blog https://aws.amazon.com/blogs/ . Any post you find with a yellow launch button will build that solution using Cloudformation.
• AWS free digital training is mostly 100 level but we also have over 40 hours of Machine Learning training available for free. You can search by topic, role or level. https://www.aws.training/LearningLibrary?src=courses You’ll find specialist deep dives from level 100 through 300 like this video describing the differences between NACLs and Security groups. https://www.aws.training/Details/Video?id=16486 NOTE: You’ll need to enroll and allow popups in your browser.
• You can also take AWS Qwiklabs Labs for free at https://aws.amazon.com/training/self-paced-labs/
• Get a sandbox or personal account. There are free tiers for many services. https://aws.amazon.com/free/
• http://run.qwiklabs.com and complete quests and labs. These enhance your familiarity with AWS services without you having to use your own account. Some labs are free. Others will require you to redeem Qwiklab credits. Reach out to your training manager or AWS account manager. Also check out the Exam guides for SA, SysOps and Advanced Networking https://www.amazon.com/Certified-Advanced-Networking-Official-Study/dp/1119439833/ref=sr_1_1?s=books&ie=UTF8&qid=1519925473&sr=1-1&keywords=advanced+networking
• Search github, https://github.com/aws , and the AWS blogs, https://aws.amazon.com/blogs/ , for solutions that interest you. Look for posts with a launch button. These will build a complete environment using Cloudformation. Retrieve the Cloudformation templates either from the built environment in your account or from Github. You can reverse engineer or use these templates as scaffolds for your own use.
• Visit Stackoverflow and the AWS discussion forum to pose questions or to contribute to answers about AWS
• You can also take a number of AWS MOOCs (Massive Open Online Courses) on EDx and Coursera including:
  • EDx https://www.edx.org/school/aws
  • Coursera https://www.coursera.org/aws
• There are many other self paced labs and solutions you can build on AWS. Try:
  • Build a Serverless Web Application https://aws.amazon.com/getting-started/projects/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/
  • How about AWS Developer Center https://aws.amazon.com/developer/ where you can build the Mythical Misfits app in your choice of programming language.
• The AWS Podcast has a monthly update which is a great way to keep up with the latest changes, releases and interviews with domain experts https://aws.amazon.com/podcasts/aws-podcast/
• AWS has released a number of webinars and now has a monthly cadence https://aws.amazon.com/about-aws/events/monthlywebinarseries/
• AWS Answers is now available to the public. It contains some interesting links. https://aws.amazon.com/answers/
• Get to know your AWS Solution Architects and your Technical Account Manager (TAM). The SAs help you to architect and understand best practice. The TAMs provide support for your applications running on AWS. They can help you prepare for major events like testing and scaling. They can also help troubleshoot and provide visibility into AWS infrastructure metrics for troubleshooting. https://aws.amazon.com/premiumsupport/faqs/
• AWS Glossary contains service names and nomenclature https://docs.aws.amazon.com/general/latest/gr/glos-chap.html
• Now go build stuff…

Continue reading articles in my Amazon Web Services series

• Data Warehousing on AWS
• Migrating to AWS
• AWS Business Essentials
• IAM Demo
• Architecting on AWS
• SysOps on AWS
• S3 Demo
• Predict the Future
• AWS Tech Essentials
• Developing on AWS
• DevOps on AWS
• Advanced Architecting on AWS
• Big Data on AWS
• AWS Deep Dive Toolbox
• Security Engineering on AWS
• Deep Learning on AWS
• AWS List of Services
• Networking on Aws
• AWS Data and Analytics
• Microsoft Immersion Day
• Adelaide Deep Racer Hints and Tips
• Deep Racer Awards
• Windows on AWS
• AWS Ask Me Anything
• Cloudwatch and Systems Manager Workshop
• Containers Immersion Day
• Redshift Immersion Day
• Innovation in Ambiguity
• AWS Contingency Planning
• AWS CLI Examples
• Migrating to Cloud in 2023
• Chaos Engineering Workshop
• Chatgpt Friend or Foe