AWS (Amazon Web Services) 云计算

推荐资源

AWS 官网	https://aws.amazon.com/
AWS Management Console	https://aws.amazon.com/console/
AWS Doc	https://docs.aws.amazon.com/
AWS CLI (Lastest 2.13.20 Command Reference)	https://awscli.amazonaws.com/v2/documentation/api/latest/reference/index.html
AWS SDK	https://aws.amazon.com/developer/tools/ Python:   https://aws.amazon.com/sdk-for-python/ Python API 1.28.53: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html Java: https://docs.aws.amazon.com/sdk-for-java/  Java API 2.20.153: https://sdk.amazonaws.com/java/api/latest/
AWS Github	https://github.com/aws
AWS Book	《Amazon Web Services in Action, Third Edition》By Michael Wittig and Andreas Wittig (中文版叫《AWS云计算实战》)
AWS Video	https://www.udemy.com/course/aws-solutions-architect-professional/ https://www.udemy.com/course/aws-certified-solutions-architect-professional-training/ https://learning.oreilly.com/videos/amazon-web-services/9780137928521/
AWS Glossary	​​​​https://docs.aws.amazon.com/glossary/latest/reference/glos-chap.html​​​​​​
AWS Training	https://aws.amazon.com/training/learn-about/architect/
AWS Certification 认证	https://aws.amazon.com/certification
AWS Certification Practice Test	Examtopic：（注意甄别答案） https://www.examtopics.com/exams/amazon/ https://www.examtopics.com/exams/amazon/aws-certified-solutions-architect-professional-sap-c02/ Udemy： https://www.udemy.com/course/practice-exam-aws-certified-solutions-architect-professional/

附：如有需要，以上链接的网页右上角都可以切换成中文版。

Management Console （管理控制台）

CLI (Command Line Interface)

SDK (Software Development Kit)

Please note that everything in AWS is 100% API driven.

其他推荐阅读：

谷歌云GCP	https://blog.csdn.net/Beth_Chan/article/details/113461721
阿里云（包含阿里云计算、存储、数据处理、Java 微服务等案例）	https://blog.csdn.net/Beth_Chan/article/details/111176779
MOOC网站访问日志分析（阿里云案例）	https://blog.csdn.net/Beth_Chan/article/details/113727493
Infrastructure as Code - Terraform	https://blog.csdn.net/Beth_Chan/article/details/133276479（待整理）
Python	https://blog.csdn.net/Beth_Chan/article/details/133421056（待整理）

安全

IAM (Identity and Access Management)

       身份和访问管理

       IAM policies define the permissions that a principal has. These policies are written in JSON format and specify the actions that the principal is allowed or denied (effect) to perform on specific resources.

       Principal: a principal is an entity that can be granted permissions to access AWS resources. This entity can be a user, a role, a service or even another AWS account.

Identity & Federation Key Concepts

• Account 账号
• IAM Users 用户: long term credentials
• IAM Groups 组 
• IAM Roles 角色: short-term credentials, uses STS
• IAM Permission 权限
• IAM Policies:
  • AWS Managed (AdminstratorAccess, PowerUserAccess); Customer Managed; Inline Policies.
  • Resource Based Policies (S3 bucket, etc.); Identity Based Policies.
• SCP: Service Control Policies
• STS: Security Token Service

IAM Policies Deep Dive

eg: PowerUserAccess

• When you assume a role (user, application or service), you give up your original permissions and take the permissions assigned to the role
• When using a resource-based policy, the principal doesn't have to give up any permissions

IAM Access Analyzer

AWS Organizations

• Organizational Units (OU): Business Unit, Environment liftcycle, Project based. 
• Management Account
• Member Account

AWS Directory Service

• Managed Microsoft AD – standalone or setup trust AD with on-premises, has MFA, seamless join, RDS integration
• AD Connector – proxy requests to on-premises
• Simple AD – standalone & cheap AD-compatible with no MFA, no advanced capabilities

Note:

• AD: Active Directory, found on any Windows server with AD Domain Services; Centralized security management, create account, assign permissions; Database of objects: User, Accounts, Computers, Printers, File Shares, Security Groups; Objects are organized in trees, a group of trees is a forest
• ADFS: Active Directory Federations Services, Identity provider (IdP), ADFS provides Single Sign-On across applications
• MFA: Multi-Factor Authentication 多重身份认证
• SAML 2.0: Security Assertion Markup Language 2.0, SAML across 3rd party: AWS Console, Dropbox, Office365, etc.
• AWS SSO: Single Sign-On Federation is the new managed and simpler way. Connect to multiple AWS Accounts (Organization) and SAML apps.

AWS Resource Access Manager

        Share resources, eg: VPC subnets, Transit Gateway, Route 53 Resolver, etc.

Amazon Cognito

Detection and Incident Response

• Security Hub

• GuardDuty

• Amazon Inspector

• AWS Cloudtrail

• Amazon Detective

• AWS Config

• AWS IoT Device Defender

• CloudEndure Disaster Recovery

Infrastructure Protection

Search "WAF & Shield", you can see "AWS WAF", "AWS Shield" and "AWS Firewall Manager".

WAF：Web Application Firewall

Protects your web applications from common web exploits (Layer 7, HTTP layer), WAF is not for DDoS protection. It's used to define Web ACL rules.

Deployment:

• Deploy on Application Load Balancer (localized rules) 
• Deploy on API Gateway (rules running at the regional or edge level)
• Deploy on CloudFront (rules globally on edge locations)
  • Used to front other solutions: CLB, EC2 instances, custom origins, S3 websites
• Deploy on AppSync (protect your GraphQL APIs)

Definition & Actions:

• Define Web ACL (Web Access Control List):
  • Rules can include IP addresses, HTTP headers, HTTP body, or URI strings
  • Protects from common attack - SQL injection and Cross-Site Scripting (XSS)
  • Size constraints, Geo match
  • Rate-based rules (to count occurrences of events)
• Rule Actions: Count | Allow | Block | CAPTCHA (验证码)

Managed Rules:

Four types of rules:

• Baseline Rule Groups – general protection from common threats

        AWSManagedRulesCommonRuleSet, AWSManagedRulesAdminProtectionRuleSet, …

• Use-case Specific Rule Groups – protection for many AWS WAF use cases

        AWSManagedRulesSQLiRuleSet, AWSManagedRulesWindowsRuleSet, 
        AWSManagedRulesPHPRuleSet, AWSManagedRulesWordPressRuleSet, …

• IP Reputation Rule Groups – block requests based on source (e.g., malicious IPs)

        AWSManagedRulesAmazonIpReputationList, AWSManagedRulesAnonymousIpList

• Bot Control Managed Rule Group – block and manage requests from bots

        AWSManagedRulesBotControlRuleSet

AWS Shield

AWS Shield is a managed Distributed Denial of Service (DDoS, many requests at the same time, your services cannot serve the real users) protection service that safeguards applications running on AWS. AWS Shield provides always-on detection and automatic inline mitigations that minimize application downtime and latency, so there is no need to engage AWS Support to benefit from DDoS protection.

AWS Firewall Manager

Centrialized security management, centrially configure and manage firewall rules across accounts and applications.

Click on "Create policy", choose policy type and then the options will be different.

eg: AWS WAF:

eg: AWS Network Firewall:

• Manage rules in all accounts of an AWS Organization
• Security policy: common set of security rules
  • WAF rules (Application Load Balancer, API Gateways, CloudFront)
  • AWS Shield Advanced (ALB, CLB, NLB, Elastic IP, CloudFront)
  • Security Groups for EC2, Application Load Balancer and ENI resources in VPC
  • AWS Network Firewall (VPC Level)
  • Amazon Route 53 Resolver DNS Firewall
  • Policies are created at the region level
• Rules are applied to new resources as they are created (good for compliance) across all and future accounts in your Organization

Comparison

Data Protection

• KMS (Key Management Service)

        Create Customer Master Key (CMK) for S3 Encryption

• ASM (Secrets Manager)

        Parameter Store

• ACM (AWS Certificate Manager)

        Create TLS certificate

• Amazon Macie

• CloudHSM (Hardware Security Module)

Compliance

• AWS Artifact

• AWS Audit Manager

网络

Overview

Amazon Virtual Private Cloud is a commercial cloud computing service that provides users a virtual private cloud, by "provision[ing] a logically isolated section of Amazon Web Services Cloud". Enterprise customers are able to access the Amazon Elastic Compute Cloud over an IPsec based virtual private network.

You need to provide IPv4 or IPv6 CIDR range while creating the VPC.

AWS Direct Connect and Site to Site VPN are the services which provide the connectivity between AWS and on-premises networks. AWS Direct connect provides the private connectivity via the dedicated network while Site to Site VPN provides the secure (IPSec) connectivity over the internet.

AWS Direct Connect and VPN both provides the private connectivity between AWS and your corporate network. However VPN traffic flows over the internet and hence can not be considered as consistent whereas Direct Connect connection is over the dedicated physical connection and is more consistent and stable.

AWS Lambda requires NAT to connect to the Internet. Public IP addresses cannot be assigned to an AWS Lambda function.

Amazon CloudFront is a content delivery network operated by Amazon Web Services. Content delivery networks provide a globally-distributed network of proxy servers that cache content, such as web videos or other bulky media, more locally to consumers, thus improving access speed for downloading the content.

AWS Network Security

        

VPC (Virtual  Private Cloud)

虚拟私有云

基本概念:

• Region, Availability Zone, VPC
• CIDR (无类别域间路由）
• 子网 Subnet：Public/Private/Hybrid
• 路由表 Route Table
• IP(Internet Protocol，网络协议）v4 / v6 (Private/Public/Elastic IP)
• (Elastic) Network Interfaces 网络接口
• Security Group 安全组
• Network Access Control List (NACL)，Network ACLs (Access Control Lists，访问控制列表）
• NAT Gateway, NAT Instance (Setup up NAT on EC2)
• Ingress/Inbound；Egress/Outbound
• Firewall
• Resource Group 资源组

CIDR

VPC secondary CIDR blocks

1. You can add secondary VPC CIDRs to existing VPC

2. CIDR block must not overlap with existing CIDR or peered VPC CIDR

3. If Primary CIDR is from RFC1918 then you can not add secondary CIDR

from other RFC1918 IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)

4. CIDR block must not be same or larger than the CIDR range of routes in any of the VPC Route tables. For example, if VPC primary CIDR block is 10.0.0.0/16 and you want to associate a secondary CIDR block in the 10.2.0.0/16 range. You already have a route with a destination of 10.2.0.0/24 to a virtual private gateway, therefore you cannot associate a CIDR block of the same range or larger. However, you can associate a CIDR block of 10.2.0.0/25 or smaller.

5. You can have total 5 IPv4 and 1 IPv6 CIDR block for VPC

Subnet, Route Table, IP, IGW  (Internet Gateway，网关)

Private, Public vs Elastic IP

Elastic Network Interfaces (ENI)

More about Elastic Network Interfaces (ENI)

1. You can not detach primary network interface from an instance

2. You associate security groups with network interfaces and not with individual IP addresses.

3. Second ENI allows instance to be multi-homed (subnets) in same AZ

4. ENIs can not be used for NIC teaming which means they can not be used together to increase instance network bandwidth

5. The number of ENIs that you can attach to instance and number of secondary IP addresses per ENI depends on EC2 instance type

Example: c4.xlarge instance may be attached 4 ENIs with each ENI supporting 15 IPv4 private

addresses and 15 IPv6 addresses

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

 

7. Cross-account Network Interface

• You can create the ENI into another account

• The cross-account network permission grants an AWS-authorized provider account

permission to attach a customer network interface to an instance in the provider

account.

Examples:

1. RDS instances reside in AWS managed VPC however the Network interface is

created in customer VPC where customer can control the traffic using Security

Groups. These are also called Requester managed network interfaces.

2.

EKS (Control Plane) master nodes are launched in AWS managed VPC and it creates ENIs into

your VPC so that it can communicate with EKS worker nodes

3.

For AWS Workspaces/Appstream the underlying EC2 instances are launched inside AWS

managed VPC and ENIs are created into your VPC so that those instances can communicate with

applications inside your VPC

Firewalls - Security group, Network ACLs

how to use them to allow , inbound and outbound ports

Internetwork traffic privacy in Amazon VPC - Amazon Virtual Private Cloud

NAT Gateway

如果Web架构是单台EC2的话还算好处理，把EC2设定好固定IP即可；但如果是比较大型或是高流量的架构通常都会用 Auto Scaling，这时候EC2的数量跟IP变成不固定，随时都可能变动，因此我们需要把所有EC2内对外的请求连线，做出一些处理，让这些请求到达第三方服务的时候，IP永远是固定的。

NAT (Network Address Translation， 网络地址转换），字面上的意思就是它可以转换IP。

由AWS官方提供的架构图可知，DB Server是设定在Private Subnet里面，并不能直接连到外网，但透过Route table的设定可以连到Public Subnet的NAT Gateway，再通过NAT Gateway转换IP，并通过Public Subnet的Internet Gateway连到Internet。

Private NAT Gateway...

NAT Gateway Troubleshooting / Rules / Limitations

https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html
https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html

https://repost.aws/knowledge-center/vpc-resolve-port-allocation-error https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html#nat-gateway-troubleshooting-tcp-issues

Amazon Route 53（DNS）

DNS: Domain Name System

Route 53 Scenairos: EC2 instance; EC2 DNS name; ALB; CloudFront distribution; API Gateway; RDS DB instance; S3 bucket; VPC interface endpoint

 

• VPC has a default DNS server AWSProvidedDNS

• VPC DNS settings can be changed using DHCP Options set

• AWSProvidedDNS can resolve the DNS from Route53 Private Hosted Zone

• For hybrid DNS resolution between VPC and on-premises network, use Route53 Resolver endpoints.

• DHCP Option set can not be edited. Create new one and associate it with VPC and you can have only one DHCP option set associated at a time.

• For hostname resolution, we should enable both enableDnsSupport and enableDnsHostname

• AWS Provided DNS server runs at VPC base + 2 IP address. You can also query DNS server at this IP or 169.254.169.253 virtual IP within VPC

DHCP Option Sets

• The options field of a Dynamic Host Configuration Protocol message contains the configuration parameters like domain name, domain name server, and the NetBIOS node type

• AWS automatically creates and associates a DHCP option set for your VPC upon creation and sets following parameters:

• domain-name-servers : This defaults to AmazonProvidedDNS

• domain-name: This defaults to the internal Amazon domain name for your region (e.g <ip>.ap-south-1.compute.internal)

VPC Flow Logs

Flow Log Syntax

• <version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport>

<dstport> <protocol> <packets> <bytes> <start> <end> <action>

<log-status>

• Srcaddr, dstaddr help identify problematic IP

• Srcport, dstport help identity problematic ports

• Action : success or failure of the request due to Security Group / NACL

• Can be used for analytics on usage patterns, or malicious behavior

• Query VPC flow logs using Athena on S3 or CloudWatch Logs Insights

VPC Flow Logs Monitoring

Analyze VPC Flows logs with CloudWatch Insights

Flow Logs limitation

Amazon VPC Flow Logs do not record traffic

• To and from VPC-native DNS services

• Amazon EC2 metadata service

• Dynamic Host Configuration Protocol (DHCP) services

• Windows license activation server

Network monitoring

• Packet Capture (for deep packet inspection)

        Wireshark (Windows/Linux) and tcpdump (Linux) which can be run on EC2 instance

• traceroute

• telnet

• nslookup - Used to resolve the hostnames into IP addresses

• ping

        Ping records network round trips using Internet Control Message Protocol

        ICMP traffic should be allowed through Security Groups, NACL

Network Performance - Basics

• Bandwidth – Maximum rate of transfer over the network

• Latency – Delay between two points in a network

• Delays include propagation delays for signals to travel across medium

• Also includes the processing delays by network devices

• Jitter –Variation in inter-packet delays.

• Throughput – Rate of successful data transfer (measured in bits per sec)

• Bandwidth, Latency and Packet loss directly affects the throughput

• Packet Per Second (PPS) – How many packets processed per seconds

• Maximum Transmission Unit (MTU) – Largest packet that can be sent over the network © Stephane Maarek, Chetan Agraw

 VPC Endpoint (Gateway, Interface) & VPC Endpoint Service

PrivateLink （VPC Interface Endpoint Service）

What is PrivateLink?

https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html

Target Group & Application Load Balancer & Network Load Balancer & VPC Endpoint Service & VPC Endpoint：

https://aws.amazon.com/blogs/networking-and-content-delivery/application-load-balancer-type-target-group-for-network-load-balancer/

直接将ALB注册为NLB目标，无需主动管理不断变化的ALB IP地址。这是通过使用引入的应用程序负载均衡器类型的目标组来实现的。由此可以将NLB的优势（包括PrivateLink和区域静态IP地址）与ALB提供的高级路由结合起来，对应用程序的流量进行负载平衡。

 VPC Peering

• Connect two VPC, privately using AWS’ network

• Make them behave as if they were in the same network

• Peered VPCs can be in same AWS region or across AWS regions

• You can do VPC peering with another AWS account

Caveats:

• VPC CIDRs should be non-overlapping

• You must update route tables in each VPC’s subnets to ensure instances can communicate across VPC

AWS Transit Gateway (TGW)

Why Transit Gateway?

Transit Gateway Route Table

Attachment

Transit Gateway Peering

Hybrid (+ VPN / Direct Connect)

AWS Direct Connect (DX)

What is Direct Connect?

• A dedicated network connection from on-premises to AWS

• AWS <-> DirectConnect Location <-> On-premises Data Center

• Low latency and consistent bandwidth

• Lower data transfer cost

• Access AWS Private Network (VPC) and AWS public services endpoints (e.g S3, DynamoDB)

Direct Connect uses private, public, and transit virtual interfaces (VIF).

Virtual Interfaces –VIF (Logical connectivity)

• In order to use the DX connection you must provision the Virtual

Interfaces

• A VIF is a configuration consisting primarily of an 802.1Q VLAN

• There are 3 types of the VIFs

• Public VIF - Enables the connectivity to all AWS public IP addresses

• Private VIF - Enables the connectivity to VPC via Virtual Private Gateway or

Direct Connect Gateway

• Transit VIF – Enables the connectivity to Transit Gateways via Direct Connect gateway

Direct Connect Gateway

• Global network device – Accessible in all regions
• Direct Connect integrates via a private VIF or a transit VIF
• The Private VIF or Transit VIF and Direct Connect gateway must be owned by same AWS account however VPCs (VGWs) or Transit Gateways can be from same or different AWS accounts

Border Gateway Protocol (BGP)

Bidirectional Forwarding Detection (BFD)

Link Aggregation Group (LAG)

Direct Connect Monitoring

Metrics

AWS VPN

• Site-to-Sute VPN
• AWS Cilent VPN

VPN Basics

• VPN allows hosts to communicate privately over an untrusted intermediary network like internet, in encrypted form

• AWS supports Layer 3 VPN (not Layer 2)

• VPN has 2 forms – Site to Site VPN and Client to Site VPN

• Site to Site VPN connects 2 different networks.

• Client to Site VPN connects the client device like laptop to the private network

• VPN types

• IPSec (IP Security) VPN which is supported by AWS managed VPN

• Other VPNs like GRE and DMVPN are not supported by AWS managed VPN

Load Balancing

Elastic Load Balancers (ELB)

• 1. Application Load Balancer (ALB)

2. Network Load Balancer

3. Gateway Load Balancer

Auto Scaling

Auto Scaling Groups（ASGs，自动扩展组/自动伸缩组）AZ: Availability Zone 可用区

计算

EC2 (Elastic Compute Cloud )

AMI (Amazon Machine Image, Amazon系统映像）：操作系统和预安装软件的组合。不包括操作系统内核。操作系统内核从Amazon Kernel Image (AKI) 加载。

HVM (Hardware Virtual Machine)：最新也是最快的虚拟化类型。

Container Service

ECS

Amazon Elastic Container Service (Amazon ECS)

EKS

Amazon Elastic Kubernetes Service (Amazon EKS)

Kubernetes Architecture

Pod to Pod communication

Security Groups in EKS

• Cluster security groups
• 
• Pod security groups
• 

Exposing services

EKS Summary

• EKS control plane is launched in AWS managed VPC and EKS data plane (worker nodes) is launched in customer VPC.

• EKS provisions ENIs into customer VPC to enable communication between EKS control plane and data plane

• EKS cluster API endpoint is publicly accessible by default but can be configured as a private in which case it can be accessed from customer VPC via the EKS owned ENI

• EKS uses Amazon VPC Container Network Interface (CNI) plugin for Pod networking.

• CNI allocates IPs to each Pod from available Secondary IPs

• Maximum number of Pods per node depends on number of ENIs and number of IP addresses per ENI

• For supported Nitro based instance types, Pod per node limit can be increased using Prefix delegation (/28 for IPv4 and /80 for IPv6)

• Custom Networking enables associating secondary VPC CIDR (100.64.0.0/16) and when combined with SNAT enables much larger IPv4 private IPs for Pods.

• CNI allows Nodes to enable/disable SNAT to allow outbound internet access to Pods through the Internet gateway or NAT gateway respectively.

• By default, ENI security group is assigned to all the Pods which have been allocated secondary IPs for that ENI

• Pods specific security group can be assigned using Trunk & Branch ENI feature for selected Nitro system based instances.

• Pod services can be configured using ClusterIP, NodePort, LoadBalancer and Ingress resources.

• ClusterIP allows accessing services from within the cluster only.• NodePort allows accessing services externally using Node IP and static port

• LoadBalancer service can be configured to use CLB or NLB in instance mode.

• Ingress service can be configured to use ALB in instance or IP mode.

• AWS Load Balancer Controller can be used for LoadBalancer (with NLB IP mode) and Ingress service (with ALB) configurations.

• externalTrafficPolicy=Local allows NLB in instance mode to preserve client IP address by disabling kube-proxy to send traffic to other nodes.

ECR

Amazon Elastic Container Registry (Amazon ECR)

Docker images

Image scanning

AWS Fargate

ECS Fargate (Serverless Docker)

EKS Fargate (Serverless Kubernetes)

AWS Lambda

       Serverless. No need to provison and manage server.

       Implement a function in Python, Java, JavaScript（Node.js）, Go, C# or Ruby, etc.

        在调用AWS lambda函数时，输入可提供一个事件（event）和一个上下文（context）对象。event是函数获得输入参数的一种方法，通常采用JSON格式。

        Python的print和JavaScript的console.log都默认会被重定向到CloudWatch Logs。

        Python JSON dumps & load
            json.dumps(): from JSON object to string
            json.load(): from string to JSON object 
        JavaScript（Node.js） JSON stringify & parse
            JSON.stringify(): from JSON object to string
            JSON.parse(): parse string to JSON object

用code inline，zip或者contain image部署都可以。

例子：

Python:

def handler_name(event, context): 
    //...
    return some_value

Context:

import time

def lambda_handler(event, context):   
    print("Lambda function ARN:", context.invoked_function_arn)
    print("CloudWatch log stream name:", context.log_stream_name)
    print("CloudWatch log group name:",  context.log_group_name)
    print("Lambda Request ID:", context.aws_request_id)
    print("Lambda function memory limits in MB:", context.memory_limit_in_mb)
    # We have added a 1 second delay so you can see the time remaining in get_remaining_time_in_millis.
    time.sleep(1) 
    print("Lambda time remaining in MS:", context.get_remaining_time_in_millis())

        

Node.js:

Context:

exports.handler = async function(event, context) {
  console.log('Remaining time: ', context.getRemainingTimeInMillis())
  console.log('Function name: ', context.functionName)
  return context.logStreamName
}

AWS Step Functions

Elastic Beanstalk

AWS Batch

AWS LightSail

AWS Outposts

AWS App Runner

存储 Storage

S3 (Simple Storage Service)

对象存储服务 Amazon S3 

存储桶 bucket：可提供访问控制，不同存储桶可以有不同的可访问性

数据对象 data object：由内容和元数据组成。元数据：最后修改日期、内容类型、用户自定义。每个对象由键来确定。存储桶位于一个区域内。上传静态文件后得到的是一个URL （https://bucket-name.s3.amazonaws.com/sample+key/name.jpg)

单个文件最大5T。

设计为99.999999999%的可靠性。

EBS (Elastic Block Store)

Amazon Glacier

备份和归档的存储服务

NAS (Network Attached Storage，网络附加存储）

NFS (Network File System, 解决多个EC2实例之间共享块存储的问题）

Amazon EBS

Amazon EFS

Elastic File System, 基于NFSv4协议

GlusterFS

AWS Transfer Family

数据库

RDS

DB Engine: PostgreSQL/MySQL/Oracle/Microsoft Server SQL/MariaDB.

Amazon Aurora

DB Engine: PostgreSQL/MySQL.

Features:

• Serverless; Multi-Master; Global Database;
• Parallel query; DB cloning; DB backtrack; DB Activity Monitoring.

Comparison for RDS, Aurora

Key differences: architecture design, database engine support, storage, scalability, replication, performance, availability and durability, resiliency, failover, backup, pricing, etc.

DynamoDB (Key-value and document data model)

Amazon DocumentDB (With MongoDB compatibility, JSON data model)

Amazon OpenSearch Service

Amazon Redshift (Data warehouse)

AWS Neptune (Graph database)

缓存 Caching

CloudFront （CDN)

ElasticCache and MemoryDB

Analytics Services / Data Engineering

Amazon Kinesis

• Data Streams

• Data Firehose

• Data Analytics

Data Analytics for SQL Application or Apache Flink

Comparison

Amazon Athena (vs Google Cloud Platform BigQuery)

EMR (Elastic MapReduce)

AWS Glue

Data Pipeline

Amazon MSK (Kafka)

Amazon MQ (Managed ActiveMQ, RabbitMQ)

Amazon Timestream

Amazon QuickSight

机器学习/人工智能（ML/AI）

SageMaker

Amazon Bedrock

Amazon Rekognition

Amazon Translate

Detect and translate text

Amazon Transcribe 

Amazon Polly

Perform speech-to-text and vice versa

Amazon Comprehend

Extract information from text

Amazon Lex

build voice and text chatbox

Amazon Forecast

AWS DeepRacer

TensorFlow on AWS

PyTorch on AWS

Monitoring and Automation Services

Foundations of Monitoring

What's Monitoring & 360 Degree View:

1. Monitoring End-to-End

2. External Monitoring (End User Experience)

External monitoring is anything that happens out HERE, from the USER perspective. Not from AWS centric perspective, like EC2, ALB, CloudFront, etc.

Passive Response

• Alert with Emails, eg: SNS to Email (Individuals and group/distribution list)
• Create help desk tickets, eg: SNS to HTTPS
• Highlight a metric on a dashboard，eg: CloudWatch Dashboard (Turn yellow or red)

Active Response

• Reboot the instance, or stop/start, eg: EC2 Rescue
• Scale horizontally, eg: Auto Scaling
• Custom actions requiring code, eg: Lambda Function

Concept Overview

• Metric & Log collection, aggregation, persistence
• Dashboards
• Alarms
• Actions
• Rules and filters
• Cross-service permissions

Differenct type of Monitoring

• Performance Monitoring
• Availability Monitoring
• Log Monitoring
• Compliance Monitoring

AWS Services

• CloudWatch (Performance metrics, Dashboards, Alarms)
• AWS Trusted Advisor (Canned reports, Recommendations, Limits)
• CloudTrail
• Amazon Macie
• Amazon GuardDuty
• AWS Config
• Amazon Systems Manager

Monitoring Permissions

• Identity-based permissions

        IAM Role allowing CloudTrail to write to CloudWatch

        Logs (when services access other service APIs)

• Resource-based permissions

        Bucket Policy to allow ELB to write access logs to S3

        Bucket

• Access Control Lists

        Primarily used for S3 bucket access

        Primitive - Pre-dates IAM

Permission Combination with Multiple Policy

Logging

Logging in AWS for security and compliance

• To help compliance requirements, AWS provides many service-specific security and audit logs
• Service Logs include:
  • CloudTrail trails - trace all API calls
  • Config Rules - for config & compliance over time
  • CloudWatch Logs - for full data retention
  • VPC Flow Logs - IP traffic within your VPC
  • ELB Access Logs - metadata of requests made to your load balancers
  • CloudFront Logs - web distribution access logs
  • WAF Logs - full logging of all requests analyzed by the service 
• Logs can be analyzed using AWS Athena if they’re stored in S3
• You should encrypt logs in S3, control access using IAM & Bucket Policies, MFA
• Move Logs to Glacier for cost savings

Amazon CloudWatch 监控

CloudWatch Agent

Events

Alarms

Logs

Metrics

Dashboards

Performance Monitoring

Logging Monitoring

• Access logs
• Execution logs
• Event
• Flow logs

        VPC Flow Logs

Record is great for monitoring, troubleshooting and root cause analysis. 

重要字段有：srcaddr, dstport, bytes, action等。

        VPC Traffic Mirroring

Amazon CloudTrail 日志

Amazon EventBridge

AWS Distro for OpenTelemetry

Amazon Managed Service for Prometheus (AMP)

Amazon Managed Grafana (AMG)

ELK with Amazon OpenSearch

CloudFormation

• Create AWS Resources      

• Stack & Template

Create stack from 蓝图 (blueprint)，配置管理服务 AWS CloudFormation 使用的蓝图被称为模版 (template)。

Nested stack: Main stack can call the nested stack, we can pass the parameters to the nested stack.    

• Parameters & Outputs

• YAML & JSON      

        A sample CloudFormation that's creating a SNS topic

        We can automatically convert JSON to YAML or YAML to JSON via CloudFormation Console or Visual Studio Code Editor (Extension)

        Dynamic References: Access Secret from CloudFormation

How to create and retrieve secrets managed in AWS Secrets Manager using AWS CloudFormation templates | AWS Security Blog

Retrieve secret value, then it would be success

AWS Config

Management Tools

Organization and SSO

AWS Systems Manager (SSM)

Parameter Store

Run Command

Session Manager

Change Manager

Inventory

Billing and Cost Management

Cost Explorer

Cost Allocation Tags

Billing Alarms

AWS Budgets

Migration

AWS Database Migration Service (DMS) 

• Replication instances (homogeneous / serverless / instance-based)
• Endpoints
• Database migration tasks

Monitoring       

Schema Conversion Tool (SCT)

其他服务

AWS Marketplace

Amazon SNS

Amazon Simple Notification Service

Amazon SQS

Amazon Simple Queue Service 

Amazon MQ

Amazon SES

Simple Email Service

AWS Cloud Map

AWS X-Ray

distribute tracing

AWS OpsWorks

Automate operations with Chef and Puppet

Amazon DevOps Guru

AWS DataSync

AWS App Mesh

Amazon FSx

• for NetApp ONTAP
• for Windows File Server

AWS FIS

Fault Injection Simulator

部分术语

JSON基础架构标记语言（JSON Infrastructure Markup Language，JIML）。

JMESPath (JSON Matching Expression paths)

ICMP ( Internet Control Message Protocol, 因特网控制报文协议）

RTO (Recovery Time Objective 恢复时间目标)

RPO (Recovery Point Objective 恢复点目标)

TTL （Time to Live 生存时间）

DSL (Domain-Specific Language)

 CICD

(可用Jenkins)

AWS自带的服务有：

CodeCommit / Github / Bitbucket

CodeArtifact

CodeBuild / Jenkins

CodePipeline

CodeDeploy / Jenkins

Elastic Beanstalk

 实验

AWS Credentials 一个小时自动过期！

Cost Saving

Using S3 Lifecycle Policies to Reduce Storage Costs

• Create an S3 Lifecycle rule to transition objects to the S3 IA storage class after a time period of 30 days
• Apply the lifecycle policy to your S3 bucket

$ ls
book_cover.png  lifecycle-rule.json  README.md

$ RANDOM_STRING=$(aws secretsmanager get-random-password \
> --exclude-punctuation --exclude-uppercase \
> --password-length 6 --require-each-included-type \
> --output text \
> --query RandomPassword)
$ aws s3api create-bucket --bucket awscookbook301-$RANDOM_STRING
{
    "Location": "/awscookbook301-4e5oom"
}
$ cat lifecycle-rule.json
{
    "Rules": [
        {
            "ID": "Move all objects to Standard Infrequently Access",
            "Prefix": "",
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,                   
                    "StorageClass": "STANDARD_IA"
                }
            ]
        }
    ]
}
$ aws s3api put-bucket-lifecycle-configuration \
> --bucket awscookbook301-$RANDOM_STRING \
> --lifecycle-configuration file://lifecycle-rule.json
$ aws s3api get-bucket-lifecycle-configuration \
> --bucket awscookbook301-$RANDOM_STRING
{
    "Rules": [
        {
            "ID": "Move all objects to Standard Infrequently Access",
            "Prefix": "",
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "STANDARD_IA"
                }
            ]
        }
    ]
}
$ aws s3 cp book_cover.png s3://awscookbook301-$RANDOM_STRING
upload: ./book_cover.png to s3://awscookbook301-4e5oom/book_cover.png
$ aws s3api list-objects-v2 --bucket awscookbook301-$RANDOM_STRING
{
    "Contents": [
        {
            "Key": "book_cover.png",
            "LastModified": "2023-11-18T07:37:43+00:00",
            "ETag": "\"d38461283ddc63b80044e2af6a7afd0d\"",
            "Size": 255549,
            "StorageClass": "STANDARD"
        }
    ],
    "RequestCharged": null
}