AWS for Partners - Data and Analytics

Introduction and foundations for partners to build capability around Data and Analytics using AWS

Administrivia

Duration - 1/1/2 hours Mixed Mode - Virtual over chime and in person

What you need to know before using AWS

This section contains links about key aspects and differentiators of AWS.

AWS Global Infrastructure

AWS Global Infrastructure. Here are several videos that ‘open the kimono’ on how AWS is designed and built to support millions of customers across the globe.
- James Hamilton, AWS SVP and Distinguished Engineer, talks about the design decisions and inner workings of the AWS global infrastructure. James also provides the history behind major technological innovations like we’re seeing now in Cloud Computing. This deck is over 4 years old but still a good summary. This should be the first AWS video you watch. https://www.slideshare.net/AmazonWebServices/spot301-aws-innovation-at-scale-aws-reinvent-2014 . Here’s the youtube video. https://www.youtube.com/watch?v=JIQETrFC_SQ There are Youtube videos from more recent ReInvents with some updates too. Here is James in 2016. It’s titled as AWS re:Invent 2016: Amazon Global Network Overview with James Hamilton https://www.youtube.com/watch?v=uj7Ting6Ckk
- Here is a 4 min snippet from 2016 titled AWS re:Invent 2016: Introduction to Amazon Global Network and CloudFront PoPs with James Hamilton https://www.youtube.com/watch?v=FjHBGjLnou0&feature=youtu.be
- AWS re:Invent 2017 Keynote - Tuesday Night Live with Peter DeSantis, VP AWS Global Infrastructure talks about the AWS global infrastructure. Up to 15:46 minutes is about the infrastructure. https://www.youtube.com/watch?v=dfEcd3zqPOA&feature=youtu.be&t=1h17m0s
- https://www.infrastructure.aws/ now has an interactive map and animations describing the AWS Global Infrastructure. 100 GBps intercontinental network.
- This one is self explanatory AWS re:Invent 2018: Amazon VPC: Security at the Speed Of Light (NET313) https://www.youtube.com/watch?v=UP7wDBjZ37o&feature=youtu.be
- James Hamilton also publishes blog posts on AWS Infrastructure regularly. Here’s one on number of data centers titled How Many Data Centers Needed World-Wide at https://perspectives.mvdirona.com/2017/04/how-many-data-centers-needed-world-wide/

Frameworks, Patterns and Anti Patterns

Being Well Architected is the key to building cost optimized, secure, performant, reliable, maintainable, supportable and operable solutions.

The cloud design principles articulate how you need to think about build cloud solutions; here is the first (circa 2011) AWS Whitepaper on designing cloud solutions https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf
Well Architected is a methodology, tool, api, set of good practices, system of inquiry, snapshot and codified lingua franca (a common language) https://aws.amazon.com/architecture/well-architected/ Shared Responsibility Model describes the partnership and responsibilities that you (the user of) and AWS have in delivering trusted and compliant cloud solutions
Cloud Adoption Framework provides guidance on planning any cloud adoption journey https://aws.amazon.com/professional-services/CAF/
Stages of Adoption
- Stephen Orban’s original (circa 2016) blog post titled ‘The Journey Toward Cloud-First & the Stages of Adoption’ https://aws.amazon.com/blogs/enterprise-strategy/the-journey-toward-cloud-first-the-stages-of-adoption/
- AWS blog posts on Stages of Adoption https://aws.amazon.com/blogs/enterprise-strategy/tag/stages-of-adoption/
- Workshop on the Stages of Adoption https://getstarted.awsworkshop.io/00-intro/03-stages-of-adoption.html
Working Backwards - how Amazon, and AWS, innovate by starting with the customer and working backwards https://www.youtube.com/watch?v=aFdpBqmDpzM
the Great Stall
- Eric Tachibana’s 2015 talk on how AWS Professional Services used data about cloud migrations to identify and remediate what Eric termed the ‘Great Stall’. Hint Eric is a great speaker and one of my early mentors in AWS.
- The Great Stall and Working Backwards are also described in the whitepaper titled ‘Building a Cloud Operating Model’ https://d1.awsstatic.com/whitepapers/building-a-cloud-operating-model.pdf

Additional Reference Material

AWS Services

AWS has over 200 services and more than 280 Apis for those services.

Overview of Amazon Web Services (circa 21Apr2021) describes AWS and the services. A great place to start with questions like ‘what is service X and what does it do?’ https://d1.awsstatic.com/whitepapers/aws-overview.pdf
To learn more about a particular service; web search for ‘service name’ and ‘FAQ’. For example:
- for Virtual Private Cloud (VPC) visit https://aws.amazon.com/vpc/faqs/
- also check out the pricing page https://aws.amazon.com/vpc/pricing/ HINT: use pricing as part of your early architectural decision making to avoid Well Architected ANTI patterns There are many great videos on AWS on Youtube including:
AWS re:Invent 2017: Scaling Up to Your First 10 Million Users (ARC201). This video walks you through the architecture to serve from 1 to 10M users using AWS. https://www.youtube.com/watch?v=w95murBkYmU HINT If you only watch one video about AWS, then watch this one…

Key AWS Service Metrics

Amazon CloudWatch monitors more than 1 quadrillion (or 1000 trillion) metric observations, triggers more than 3.9 trillion events, and ingests more than 100 petabytes of logs per month.
Amazon S3 holds trillions of objects and regularly peaks at millions of requests per second.
Amazon SQS handles more than 25 billion requests per hour
At just 3 years after general availability (circa 2017), AWS Lambda processes trillions of executions every month.
More than 350,000 databases have been migrated using AWS Database Migration Service
On September 30, 2019, Amazon’s Consumer business turned off its final Oracle database after migrating nearly 7,500 databases and 75 petabytes of data across hundred of items to AWS database services
The AWS global infrastructure is comprised of 80 Availability Zones within 25 geographic Regions with announced plans for 15 more Availability Zones and 5 more AWS Regions in Indonesia, India, Spain, Switzerland and Melbourne.
Just before Ian joined AWS in 2014, Amazon was deploying more than 50 Million code changes every year. https://www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html
AWS Nitro System Introduced 2017 https://aws.amazon.com/ec2/nitro/ This is a new, performant, secure and cost optimized approach to hardware virtualization from AWS. Nitro enclaves now extend that capability. https://aws.amazon.com/ec2/nitro/nitro-enclaves/
Firecracker https://aws.amazon.com/about-aws/whats-new/2018/11/firecracker-lightweight-virtualization-for-serverless-computing/ An open source approach to virtualization for serverless and containers that is secure, performant, lightweight and dynamic.

AWS Global Infrastructure

AWS own and operate their entire global infrastructure including multiple redundant 100 Gbps links

Get an overview of AWS global infrastructure https://aws.amazon.com/about-aws/global-infrastructure/
For details on AWS Infrastructure in Australia see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
AWS Data Centre Controls see https://aws.amazon.com/compliance/data-center/controls/
Take a virtual tour of an AWS data centre https://aws.amazon.com/compliance/data-center/

James Hamilton, AWS VP and Distinguished Engineer, has many deep dive videos and blog posts on the AWS infrastructure.

AWS Innovation at Scale from 2014 https://www.youtube.com/watch?v=JIQETrFC_SQ
Amazon Global Network Overview 2016 with James Hamilton https://www.youtube.com/watch?v=uj7Ting6Ckk
Peter DeSantis, VP AWS Global Infrastructure talks about the AWS global infrastructure at ReInvent 2017. Up to 15:46 minutes is about the infrastructure. https://www.youtube.com/watch?v=dfEcd3zqPOA&feature=youtu.be&t=1h17m0s

To learn more about AWS Regions and Availability Zones watch https://www.youtube.com/watch?v=JIQETrFC_SQ

for AWS Regions watch from 16:10 to 18:44
for AWS Availability Zones watch from 18:45 to 25:17

AWS Risk Management Frameworks

AWS Shared Responsibility Model https://aws.amazon.com/compliance/shared-responsibility-model/
AWS Cloud Adoption Framework and AWS Professional Services (ProServe) https://aws.amazon.com/professional-services/ . Find more information about the Solution, Technology and Industry Practices and access all the Whitepapers, Online Tech Talks, Blog Posts, Podcasts and all Annoucements
AWS Well-Architected https://aws.amazon.com/architecture/well-architected is a ’lingua franca’ ( common language) for how to do things well in the AWS Cloud
Access hundreds of AWS Whitepapers at https://aws.amazon.com/whitepapers
Amazon has internal mechanisms that are used for risk management, specifically by starting with the customer and working backwards. This open source blog posts describes how AWS used this mechanism to create the CDK. Working backwards: The story behind the AWS Cloud Development Kit https://aws.amazon.com/blogs/opensource/working-backwards-the-story-behind-the-aws-cloud-development-kit/

AWS Core Services

AWS Compute

Choosing Compute Options https://aws.amazon.com/startups/start-building/how-to-choose-compute-option/ AWS Storage
- Cloud Storage Options https://aws.amazon.com/products/storage/ AWS Database
AWS Database Options https://aws.amazon.com/products/databases/ AWS Networking
Networking and Content Delivery on AWS https://aws.amazon.com/products/networking/

AWS Data and Analytics Services

Analytics on AWS, start here https://aws.amazon.com/big-data/datalakes-and-analytics/
Analytics Services on AWS with brief descriptions https://docs.aws.amazon.com/whitepapers/latest/aws-overview/analytics.html
AWS Documentation on all services, sdks, tutorials and references available in pdfs, html, markdown on github and as ebooks. https://docs.aws.amazon.com/
Deep Dive on NoSQL by Rick Houlihan (great example of AWS and Amazon SMEs sharing their thoughts and advice freely)
- Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) https://www.youtube.com/watch?v=HaEPXoXVf2k
- Rick on Why the PIE theorem is more relevant than the CAP theorem https://www.alexdebrie.com/posts/choosing-a-database-with-pie/

AWS Reference Architectures

AWS and Partner published

AWS Solutions - Vetted Technology Solutions for the AWS Cloud https://aws.amazon.com/solutions/
AWS Solution Constructs - Vetted architecture patterns that are extensions of the AWS Cloud Development Kit (cdk)
AWS Solution Consulting Offers - vetted solutions to common business and technical problems
AWS Quickstarts - Automated reference deployments using IaaC https://aws.amazon.com/quickstart

Power BI on AWS

Third party tooling like Power BI are well supported on AWS https://d1.awsstatic.com/whitepapers/using-microsoft-power-bi-with-aws.pdf?did=wp_card&trk=wp_card

AWS Training - Free, Classroom based and Online

AWS Delivered training from foundational to speciality can be found at https://bit.ly/3gz2r89

Training Types

Digital - Free, online, bite sized, role based and self paced - https://bit.ly/3dWquzv
Instructor led (classroom or virtual) - Paid, multi day - https://bit.ly/2R4Zu86
MOOCs - from eDX, Coursera, Udacity and Udemy - from basics to deep dives to DeepRacer and ML
- eDX - https://bit.ly/3vsgynE
- Coursera - https://www.coursera.org/courses?query=aws
- Udacity - https://bit.ly/3xvG50P
- Udemy - https://bit.ly/3dUZ6SK
Certifications - 11 certifications from foundation to speciality - https://amzn.to/3exlJvw

Self paced Learning and Building

AWS Certification roadmap https://aws.amazon.com/certification/ Check out the learning paths link at the bottom of the page.
Read the service FAQ pages, http://aws.amazon.com/faqs/, and documentation for each of the services. Just search for AWS + + documentation in any search engine. You can keep the documentation as pdf, html online or even in your Kindle. You can also git clone the documentation for most services.
Find and build interesting AWS and partner solutions you find the in AWS Blog https://aws.amazon.com/blogs/ . Any post you find with a yellow launch button will build that solution using Cloudformation.
AWS free digital training is mostly 100 level but we also have over 40 hours of Machine Learning training available for free. You can search by topic, role or level. https://www.aws.training/LearningLibrary?src=courses You’ll find specialist deep dives from level 100 through 300 like this video describing the differences between NACLs and Security groups. https://www.aws.training/Details/Video?id=16486 NOTE: You’ll need to enroll and allow popups in your browser.
You can also take AWS Qwiklabs Labs for free at https://aws.amazon.com/training/self-paced-labs/
Get a sandbox or personal account. There are free tiers for many services. https://aws.amazon.com/free/
http://run.qwiklabs.com and complete quests and labs. These enhance your familiarity with AWS services without you having to use your own account. Some labs are free. Others will require you to redeem Qwiklab credits. Reach out to your training manager or AWS account manager. Also check out the Exam guides for SA, SysOps and Advanced Networking https://www.amazon.com/Certified-Advanced-Networking-Official-Study/dp/1119439833/ref=sr_1_1?s=books&ie=UTF8&qid=1519925473&sr=1-1&keywords=advanced+networking
Search github, https://github.com/aws , and the AWS blogs, https://aws.amazon.com/blogs/ , for solutions that interest you. Look for posts with a launch button. These will build a complete environment using Cloudformation. Retrieve the Cloudformation templates either from the built environment in your account or from Github. You can reverse engineer or use these templates as scaffolds for your own use.
Visit Stackoverflow and the AWS discussion forum to pose questions or to contribute to answers about AWS
You can also take a number of AWS MOOCs (Massive Open Online Courses) on EDx and Coursera including:
- EDx https://www.edx.org/school/aws
- Coursera https://www.coursera.org/aws
There are many other self paced labs and solutions you can build on AWS. Try:
- Build a Serverless Web Application https://aws.amazon.com/getting-started/projects/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/
- How about AWS Developer Center https://aws.amazon.com/developer/ where you can build the Mythical Misfits app in your choice of programming language.
The AWS Podcast has a monthly update which is a great way to keep up with the latest changes, releases and interviews with domain experts https://aws.amazon.com/podcasts/aws-podcast/
AWS has released a number of webinars and now has a monthly cadence https://aws.amazon.com/about-aws/events/monthlywebinarseries/
AWS Answers is now available to the public. It contains some interesting links. https://aws.amazon.com/answers/
Get to know your AWS Solution Architects and your Technical Account Manager (TAM). The SAs help you to architect and understand best practice. The TAMs provide support for your applications running on AWS. They can help you prepare for major events like testing and scaling. They can also help troubleshoot and provide visibility into AWS infrastructure metrics for troubleshooting. https://aws.amazon.com/premiumsupport/faqs/
AWS Glossary contains service names and nomenclature https://docs.aws.amazon.com/general/latest/gr/glos-chap.html
Now go build stuff…

Continue reading articles in my Amazon Web Services series