How to pass AWS Developer Associate Exam – Part 3

Last modified date

Comments: 0

Contents

DynamoDB

SQL databases are good for structured relational data, but it can get computationally heavy when there are complex SQL queries applied. Scale vertically.

  • NoSQL
    • NoSQL databases are non-relational databases and are distributed.
    • NoSQL does not support joins
    • Does not support join
    • All the data that is needed for a query is present in one row
    • Don’t perform aggregations such as SUM
    • They scale horizontally
  • DynamoDB Features
    • Fully managed
    • Highly available
    • Replication across 3 AZ’s
    • No SQL
    • Scales, distributed database.
    • Scales to millions of requests per seconds.
    • Fast and consistent performance
    • Integrated with IAM for security, authorization and administration.
    • Enables event driven programming with DynamoDB streams
    • Low cost and auto scaling
  • DynamoDB Basics
    • Made of tables
    • Each table has a primary key
    • Each table can have an infinite number of items (rows)
    • Each item has attributes (can be null and added over time)
    • Max size of a row is 400KB
    • Datatypes supported
      • String
      • Number
      • Binary
      • Boolean
      • Null
      • Document Types – list and map
      • Set Types – set, number set, binary set

Primary Keys

  • Option 1 – Partition key only (HASH)
    • Partition key must be unique for each item
    • Partition key must be diverse so that the data is distributed
    • e.g. user id (unique)
  • Option 2 – Partition key + Sort key
    • The combination must be unique
    • Data will be grouped by the partition key
    • Then sorted by Sort key == range ket
    • e.g. User id for partition key and game id for the sort key

Provisioned Throughput

Tables must be provisioned with read and write capacity units.

Read Capacity Units (RCU) – throughput for reads

Write Capacity Units (WCU) – throughput for writes

They can be independently scaled.

There is an option to setup auto-scaling of throughput to meet demand

Throughput can be exceeded temporarily using “burst credit”.

If burst credit are empty you’ll get a ProvisionedThroughputException. It is then advised to perform exponential backoff retry.

Write Capacity Units

  • One write capacity unit represents one write per second for an item up to 1KB in size.
  • If the items are larger than 1KB, more WCU are consumed.
  • KB’s are rounded to the upper KB

Read Capacity Units

  • Strongly consistent reads
    • If you read straight after a write, you will get the correct data.
    • The request might take slightly longer but guaranteed to get the correct data.
  • Eventually consistent reads
    • If you read straight after a write, it is possible to get an unexpected response because of replication. (old data or no data)
    • If read happens before replication you’ll get eventually consistent reads.
    • If you keep asking then eventually you’ll get the right result
    • This is the default

GetItem, Query and Scan provide a ConsistentRead parameter you can set to true.

  • One read capacity unit represents one strongly consistent read per second.
  • OR one read capacity unit represents two eventually consistent reads per second
  • For an item up to 4KB in size.
  • If items are larger than 4KB, more RCU are consumed.
  • Round up in increments of 4KB if larger than 4KB. e.g. a 6KB read will be classed as 8KB.

Partitions Internal

Data is divided in partitions.

Partition keys go through a hashing algorithm to know to which partition they go to.

WCU and RCU are spread evenly between partitions.

Throttling

If the RCU or WCU is exceeded you get a ProvisionedThroughputExceededException.

  • Reasons for throttling
    • Hot Keys – one partition key is being read too many times
    • Hot partitions
    • Very large items (RCU and WCU depends on size of items)
  • Solutions for throttling
    • Exponential backoff
    • Distribute partitions keys as much as possible
    • If RCU issue, use DynamoDB Accelerator (DAX).

Writing Data

  • PutItem – write data to dynamoDB (create data or full replace)
    • Consumers WCU
  • UpdateItem – update data in dynamoDB (partial update of attributes)
    • Can use Atomic counters and increase them
  • Conditional writes
    • Accept a write or update only if conditions are respected, otherwise reject
    • e.g. if this attribute is this value then perform the write.
    • Helps with concurrent access to items
    • No performance impact
  • BatchWriteItem
    • Up to 25 PutItem or DeleteItem in one call
    • Up to 16 MB of data written
    • Up to 400KB of data per item
    • Batching helps to reduce latency as it allows you to reduce the number of API calls done against DynamoDB.
    • Operations are performed in parallel for better efficiency.
    • Batch writes can fail, so you will have to retry the failed items using exponential backoff.

Deleting Data

  • DeleteItem
    • Delete individual row
    • Ability to perform a conditional delete
  • DeleteTable
    • Delete a whole table and all its items
    • Quicker than calling DeleteItem on all the items.

Reading Data

  • GetItem
    • Read based on primary key
    • Primary key is either a HASH (partition key) or HASH-RANGE (partition and sort key)
    • Eventually consistent read by default.
    • Option to use strongly consistent reads (consumes more RCU and might take longer).
    • ProjectionExpression – to only return certain attributes.
  • BatchGetItem
    • Up to 100 items
    • Up to 16 MB of data
    • Items retrieved in parallel to minimize latency
    • Can be combined with ProjectionExpression

Query

  • Query returns items based on
    • PartitionKey value (must be = operator)
    • SortKey value (=, <, <=, Between etc,) optional
    • FilterExpression for further filtering (client side filtering)
  • Returns
    • Up to 1 MB of data.
    • Or number of items specified in the limit
  • Able to pagination on the results
  • Can query table, a local secondary index or global secondary index

Scan

  • Scans the entire table and then filters out the data (inefficient)
  • Returns up to 1MB of data – use pagination to keep on reading
  • Consumes a lot of RCU
  • Limit impact using limit or reduce the size of the result and pause.
  • For faster scan performance use parallel scans.
    • Scans multiple instances at the same time
    • Increases the throughput and RCU consumed
    • Limit the impact of parallel scans using Limit
  • Can use ProjectExpression + FilterExpression
    • But this doesn’t change the RCU usage.

LSI (Local Secondary Index)

  • Allows for querying a non-key attribute whilst keeping the same partition key.
  • An alternate range key for the table (local to the hash key) Local to partition key.
  • Up to 5 local secondary indexes per table
  • The sort key consists of exactly one scalar attribute
    • Scalar attribute must be either a String, Number or Binary
  • LSI must be defined at table creation time
  • Allows for sorting on a local partition key, rather than having to perform a scan and filter.

Global Secondary Index

To speed up queries on non-key attributes, use Global Secondary index.

GSI allows you to specify a new partition key and an optional sort key.

Allows you to define a whole new index. (whole new table)

This index is a new table and we can project attributes on it.

  • The partition key and sort key of the original table are always projected (KEYS_ONLY)
  • Can specify extra attributes to project (INCLUDE)
  • Can use all attributes from the main table (ALL)

Must define RCU / WCU for the index – acts like a whole new table.

Possibility to add and modify GSI anytime, unlike LSI.

A Global secondary index can be set to the game Id so that it’s easier to query

Indexes and Throttling

Indexes can cause throttling.

  • GSI
    • If writes are throttled on the GSI then the main table will be throttled.
    • Even if on the main table the WCU is fine.
    • Choose GSI partition carefully
    • Assign WCU capacity carefully for the GSI.
  • LSI
    • Uses the WCU and RCU of the main table
    • So if LSI is throttled, then the main table is also throttled.

Concurrency

DynamoDB has a feature called Conditional Update / Delete. Ensures an item hasn’t changed before altering it.

DynamoDB is optimistic locking / concurrency database since multiple clients can be accessing it at once.

DynamoDB DAX

DAX = DynamoDB Accelerator.

  • Caching Technology for DynamoDB.
  • Seamless – once enabled you don’t need to rewrite any parts of your application.
  • Writes go through DAX to dynamo.
  • Micro second latency for cached reads and queries.
  • Solves the HOT KEY problem (too many reads).
  • DAX will cache the hot key
  • 5 minutes TTL default for cache.
  • Up to 10 nodes in the cluster.
  • Multi AZ (3 nodes min recommended for production)
  • Secure (encryption at rest with KMS, VPC, IAM, CloudTrail)

DAX vs ElastiCache

Not much difference in using DAX or Elasticache

But ElastiCache could be used to store the result of a computation on the data so that next time you won’t even have to retrieve the data.

DynamoDB Streams

Changes in DynamoDB (create, update, delete) can end up in a stream.

This stream can be read by lambda, EC2 instances etc.

This can then allow you to react to changes in realtime (e.g. send welcome email, perform analytics)

Could use streams to implement cross region replication.

Streams have 24 hours data retention.

  • How to work with the streams
    • Choose the type of data to be written to the stream.
    • KEYS_ONLY – only the key attribute of the modified item
    • NEW_IMAGE – the entire item, as it appears after it was modified
    • OLD_IMAGE – the entire item, as it appeared before it was modified
    • NEW_AND_OLD_IMAGES – both the new and old images of the item

The streams are made of shards. But you don’t provision them. AWS manages the shards.

Records are not retroactively populated in a stream after enabling it. Only the future changes end up in the stream.

DynamoDB Streams and Lambda

You need to define an event source mapping to read from DynamoDB Streams.

The event source mapping will POLL the stream.

You need to ensure the lambda has the appropriate permissions to POLL.

The Lambda function will be invoked synchronously when there is an event source mapping

TTL (Time to Live)

Automatically delete an item after an expiry date / time.

Have a dedicated TTL column.

TTL is provided at no extra cost and doesn’t use any WCU or RCU.

TTL is a background task operated by DynamoDB itself.

  • Use case
    • Helps reduce storage and manage table size over time.
    • Helps adhere to regulations
    • TTL is enabled per row (define a TTL column and add date there)

Typically it deletes the expired items within 48 hours of expiration

Deleted items due to TTL are also delete in the GSI and LSI.

DynamoDB Streams can help recover expired items.

DynamoDB CLI

  • –projection-expression
    • List of attributes that you want to retrieve from the table
  • –filter-expression
    • Filter results
  • General CLI commands (applicable to other services as well)
    • Optimization
      • –page-size: full dataset is still received but each API will request less data (helps to avoid timeouts)
    • Pagination
      • –max-items: max number of items to be returned by the CLI. CLI returns NextToken
      • –starting-token: specify the las received NextToken to keep on reading.

Transactions

The ability to create, update, delete multiple rows in different tables at the same time.

Transactions are all or nothing type of operation. (all successful or none successful)

There are two write modes in DynamoDB:

  1. Standard
    • If it fails half way through updating rows, half the rows will have been updated, half not updated.
  2. Transactional
    • Update all at once or not

There are three write modes in DynamoDB:

  1. Eventual Consistency
  2. Strong Consistency
  3. Transactional

Transactions use twice 2X the amount of WCU and RCU.

  • API Names
    • TransactWriteItems
    • TransactGetItems

Transactional Capacity Computations

Session State Cache

DynamoDB can be used as a cache to store session state. Serverless.

This can also be done in ElastiCache. This is done in memory.

  • DynamoDB vs ElastiCache
    • DynamoDB is Serverless. (automatic scaling).
    • ElastiCache is in memory.
    • Both are key/value stores
  • EFS could also be used to store and share session state across EC2 instances.
    • EFS would be attached to the EC2 instances as a network drive.
    • So can’t be used with Lambda
  • EBS and Instance Store
    • EBS and Instance Store can only be used for local caching, not shared caching.
  • S3
    • Has a higher latency
    • Designed for large files, not small objects

Partitioning Strategies

Write Sharding.

In the instance where it is not possible to have a highly distributed partition key, a suffix (random/calculated suffix) can be added as part of the partition key.

e.g. people are voting for two candidates. Candidate_id is going to run into partition issues quickly. So add a suffix.

Allows for write sharding.

Write Types

  • Concurrent write
    • Two writes occur at the same time, the first write will get overwritten.
    • This is not always ideal
  • Conditional writes
    • Write a value only if a certain attribute is a certain value
    • This helps deal with the concurrent write problem
    • The second write will throw an error instead of overwriting
  • Atomic Writes
    • Increase or decrease by a certain value
    • If two atomic writes come in at the same time, they will both be processed
  • Batch Writes
    • Write / update many items at a time

DynamoDB and S3

  • Large objects pattern
    • Max object size is 400KB in a Dynamo record
    • Instead upload the file to S3.
    • Then write the metadata to Dynamo which contains a reference to the file being stored in S3.
  • Indexing objects in S3
    • S3 is not designed to be searched.
    • This is a problem if we need to find a file.
    • A solution is to upload to S3, the S3 event will trigger a lambda which will add a record of that file into DynamoDb with all the metadata for that file
    • Then if you need to search for a file, it’s very easy to search Dynamo since that is what it was designed to do.

DynamoDB Operations

  • Table Cleanup
    • Option 1
      • Scan, then delete every row. (this is expensive and consumes a lot of RCU and WCU)
    • Option 2
      • Drop the table, then recreate the table. Fast, Cheap and efficient
  • Copying a table
    • Option 1
      • Use AWS DataPipeline (EMR). Takes the table, places it into S3 and then copy’s it from S3 back to Dynamo
    • Option 2
      • Create a backup and then restore the backup into a new table name (can take some time)
    • Option 3
      • Scan the table and then write it
      • But you have to write your own code.

DynamoDB Security

  • Security
    • VPC Endpoints available to access DynamoDB without the internet
    • Access controlled by IAM
    • Encrypt data at rest using KMS
    • Encrypt in transit using SSL / TLS
  • Backup and Restore feature available
    • Point in time restore like RDS
    • No performance impact
  • Global Tables
    • Multi region, fully replicated, high performance
  • DMS (Database Migration Service) can be used to migrate DynamoDB from other DB’s to Dynamo.
  • Can launch a local DynamoDB for development purposes

Users directly interacting with DynamoDB

Using a federated login service such as Google, Facebook, Cognito, users can get temporary AWS credentials to access DynamoDB.

But we want to ensure that the temporary credentials only have access to the rows they should have, and not the full table.

Fine-Grained Access Control – these temporary credentials have an IAM role for these users but with a condition that it limits their access to the only the DynamoDB API.

This role policies can have LeadingKeys. Limit row-level access for users on their primary key.

Attributes. To limit only certain attributes that the user has access to.

API Gateway

A lambda function can be invoked in several ways. A user can invoke it directly if they have the correct IAM permissions, it can be behind an ALB and be exposed as a HTTP endpoint to the ALB. Or use API Gateway.

  • API Gateway integrations
    • Lambda functions
      • Easy to expose a REST API backed by Lambda
    • HTTP
      • Expose HTTP endpoints in the backend
      • Internal HTTP API on premise, ALB
      • Allows you to have rate limiting, caching, user authentication etc.
    • AWS Services
      • Can expose almost any service
  • Endpoint Types
    • Edge-Optimized (default)
      • For Global clients
      • Requests routed through CloudFront Edge Locations (improves latency)
      • The API Gateway only exists in one region but is accessible globally.
    • Regional
      • For clients within the same region
      • Could manually combine with CloudFront
    • Private
      • Can only be accessed from your VPC using an interface endpoint ENI
      • Use a resource policy to define access.

Deployment Stages

When you make changes to API Gateway, they are not effective immediately.

A deployment has to be made for them to take affect.

Changes are deployed to stages. (e.g. dev, test, prod)

Stages can be rolled back as a history of all deployments is kept.

  • Stage variables
    • Environment variables for API Gateway
    • Use them to change frequently changing configuration values
    • They can be used in:
      • Lambda Function ARN
      • HTTP Endpoint
      • Parameter mapping templates
    • Use Case
      • Configure HTTP endpoints your stages talk to (dev, test, prod)
      • Or pass configuration parameters to AWS lambda through mapping templates
    • Stage variables are passed to the “context” object in AWS lambda

API Gateway Stage Variables and Lambda Aliases

A stage variable can be created to indicate the corresponding Lambda alias.

API Gateway will automatically invoke the right lambda function

Canary Deployment

Canary allows you to test a small amount of traffic on a change in the API Gateway (usually Prod)

Choose the % of traffic the canary channel receives.

This is a blue/green deployment with Lambda and API Gateway.

Metrics and Logs are separate (for better monitoring)

Possibility to override stage variables for Canary.

Integration Types and Mappings

  • MOCK
    • API gateway returns a response without send the request to the backend
    • Good for configuring and testing
  • HTTP / AWS (Lambda and AWS Services)
    • API Gateway will forward a request, but it can be modified.
    • Must configure both an integration request and integration response.
    • Setup data mappings using mapping templates for the request and response
    • This allows for changing the request sent to the backend and the changing the response before it reaches the client.
  • AWS_PROXY (Lambda Proxy)
    • The request from the client will be the input to the lambda
    • Can’t modify the request – it’s a proxy
    • The lambda function is solely responsible for the logic of the request and the response.
    • No mapping templates.
    • Headers, query string parameters are passed as arguments
    • All the work is performed on the backend
  • HTTP_PROXY
    • No mapping template
    • The HTTP request is passed to the backend
    • The HTTP response from the backend is forwarded by API gateway

Mapping templates (AWS and HTTP Integrations)

Mapping templates can only be used when not using the proxy methods.

  • Can be used to modify requests and responses.
    • Rename and modify query string parameters.
    • Modify body content
    • Add headers
    • Uses Velocity Template Language (VTL). A scripting language with for loops etc.
    • Filter output from the response (remove unnecessary data)

Mapping template example

How to integrate a client with API Gateway to a SOAP API.

SOAP API’s are XML based.

REST API’s are JSON based.

So you can create an API Gateway with a mapping template which will transform the payload from JSON to XML before passing it to the server.

  • So API Gateway will
    1. Extract data from the request. Either path, payload or header.
    2. Build SOAP message based on the request data (mapping template)
    3. Call SOAP service and receive XML response
    4. Transform XML response to desired format (JSON) and respond to the user.

Another example is renaming query string parameters.

API Gateway Swagger / Open API spec

Common way to define REST API’s using definition as code.

Open API spec can be imported into API Gateway.

Including the Methods, Method Requests, Integration Requests, Method Response, AWS Extensions.

Can export current API as Swagger / Open API

Swagger can be written as either YAML or JSON.

Using Swagger you can generate an SDK for your applications.

Caching

Reduce the number of calls made into the backend.

Default TTL 300 seconds. (min 0 seconds, max 3600 seconds)

Caches are defined at the stage level.

Possible to override cache settings per method.

Cache encryption option.

Cache size is between 0.5GB and 237GB.

Cache is expensive so make sense to only use in production.

  • Invalidating the cache
    • Able to flush the entire cache immediately (invalidate it)
    • Clients can invalidate the cache with the header Cache-Control:max-age=0. If they have the proper IAM authorization.
    • If you don’t have an InvalidateCache Policy, then any client can invalidate the API cache which is not ideal

API Gateway Usage Plans and API keys

Once you’ve developed your API, you can make it available for people to use and you can charge money for it.

  • Usage Plan
    • Who can access one or more deployed API stages and methods.
    • How much and how fast they can access them.
    • Uses API keys to identify clients and meter access.
    • Configure throttling and quota limits that are enforces on individual clients.
  • API Keys
    • String values to distribute to customers
    • Can be used with the usage plans to control access
    • Throttling limits are applied to the API Keys
    • Quota limits are the overall number of maximum requests.
  • Correct order for creating an API and usage plan
    1. Configure the usage plan
    2. Create one ore more API’s, methods etc
    3. Deploy the API’s to the stages
    4. Generate or import API keys to distribute to application developers who will be using the API.
    5. Create a usage plan with the desired throttle and quota limits
    6. Then associate the API stages and API keys with the usage plan.
    7. Callers of the API must supply the assigned API ket in the x-api-key header in the requests to the API.

Logging and Tracing

  • CloudWatch logs
    • Enable CloudWatch logs at the staging level (with log level).
    • Can overrider settings on a per API basis (ERROR, DEBUG, INFO).
    • Log contains information about request/response body.
  • X-Ray
    • Enable tracking to get extra information about the requests in API gateway
    • X-Ray API Gateway + Lambda gives the full picture. e.g. latency
  • CloudWatch Metrics
    • Metrics are by stage.
    • Possibility to enable detailed metrics
    • CacheHitCount and CacheMissCount. Information about the efficiency of the cache.
    • Count – number of API requests in a given period.
    • IntegrationLatency – the time between API Gateway relaying a request to the backend and when it receives a response from the backend.
    • Latency – time between API gateway receiving a request from the client and when it returns a response to the client. This includes the IntegrationLatency and other API Gateway overhead.
      • So latency will always be higher than integrationlatency
      • Maximum amount of time it can perform a request is 29 seconds. If it is longer than 29 seconds a timeout error will occur.
    • 4XXError (client side error)
    • 5XXError (server side error)

Api Gateway Throttling

  • Account Limit
    • API Gateway throttles requests at 10,000 requests per second (rps) across all API’s in the account.
    • This is a soft limit that can be increased upon request.
    • 429 Too Many Requests error (retriable error but should use exponential backoff.)
    • Client is making too many requests.
    • Can set stage limit and method limits to improve performance
    • Or define a usage plan to throttle per customer.

Just like lambda concurrency, if one API is overloaded it can cause the other API’s to be throttled.

API Gateway Errors

  • 4XX client errors
    • 403 Access denied, WAF filtered
    • 429 Quota exceeded, throttled
  • 5XX means server errors
    • 502 Bad gateway exception. Usually for incompatible output returned from a lambda proxy integration.
    • 504 Integration failure. Endpoint request timed out exception.

CORS

CORS must be enabled if you want to be able to receive API calls from another domain

  • The OPTIONS pre-flight request must contain the following headers
    • Access-Control-Allow-Methods
    • Access-Control-Allow-Headers
    • Access-Control-Allow-Origin
  • CORS can be enabled through the console in API Gateway.

API Gateway Security

The following are different ways to securely access API Gateway

  1. IAM Permissions
    • Create an IAM policy authorization and attach to a User or Role so that they can invoke API Gateway
    • Authentication is done through IAM
    • Authorization is done through an IAM Policy
    • Good way to protect the gateway if being accessed by resources within your AWS account. (EC2 etc.)
    • Leverages Sig v4 capability where IAM credentials are places in the headers. The credentials are signed and then placed into headers.
    • It can be combined with Resource Policies
  1. Resource Policies
    • Similar to lambda resource policies
    • Allows you to set a JSON policy on the API Gateway to define who / what can access it
    • The main use case for it is to allow cross account access (combined with IAM security)
    • Allow for a specific IP address
    • Or Allow for a specific VPC endpoint
  1. Cognito User Pools
    • Database of users
    • Manages full user lifecycle
    • API Gateway can verify the identity of people connecting to gateway with cognito.
    • No custom implementation required
    • Authentication = cognito user pools
    • Authorization = API gateway methods
  1. Lambda Authorizers
    • Most flexible but the most involved
    • Token based authorizer (bearer token)
    • e.g. JWT or Oauth
    • A request parameter-based lambda authorizer (headers, query string, stage var)
    • The lambda can then authorize and will return an IAM policy for the user, resultant policy will be cached
    • Authentication = External (e.g. Auth0)
    • Authorization = Lambda function
  • IAM
    • Great for users/roles already within AWS account
    • Handles authentication and authorization
    • Leverages Signature v4
  • Resource Policy
    • Cross account access on top of IAM
  • Custom Authorizer (lambda authorizer)
    • When using 3rd party authentication systems
    • Flexible in terms of what IAM policy is returned.
    • We have to handle the authentication and authorization in the lambda function.
    • Pay per lambda invocation
    • Results are cached
  • Cognito User Pools
    • You manage your own user pool
    • No need to write any custom code
    • Must implement authorization in the backed lambda function

HTTP API vs REST API

  • HTTP API
    • Low latency
    • Cost effective
    • HTTP proxy
    • Proxy API’s
    • Private integration
    • No data mapping
    • Only support OIDC and OAuth 2.0
    • Has CORS
    • No usage plan or API keys
  • REST API
    • All features as API Gateway (except Native OpenID Connect OAuth 2.0)

WebSocket API

WebSockets is a two way interactive communication between a user’s browser and a server.

Server can push information to the client without the client having to make a request to the server.

This enables stateful application use cases.

WebSocket API’s are often used in real-time applications such as chat applications, collaboration platforms, multiplayer games, financial trading applications.

The client establishes a persistent connection with the server.

Works with AWS Services (Lambda, DynamoDB) or HTTP endpoints

WebSocket API connection Flow

  1. Client connect to the server

There is a WebSocket URL.

The Client connects to the WebSocket and establishes a persistent connection. A unique connectionID is generated for this client

  1. Client wants to send messages to the server

It reuses the connection and sends a message to the server.

The lambda function knows this is the same user so reuses the connection ID

These message are called frames and are sent over the same connection.

  1. Server sends messages to the client

There is a connection URL callback.

It is the same as the websocket url but with /@connections/connectionID at the end.

The server can then send a POST request to this connection URL and send messages back to the client.

  • Connection URL Options
    • POST
    • GET
    • DELETE

WebSocket API Routing

  • Incoming JSON messages are routed to a different backend
  • If no routes are specified then it will be routed to the default route.
  • You request a route selection express, to select the field on JSON to route from.

API Gateway Architecture

  • API Gateway allows you to have a single interface for all your microservices.
  • Allows you to have a single public facing URL to clients, hides complexity.

AWS SAM – Serverless Application Model

Framework for developing and deploying serverless applications.

All the configuration is in YAML code. This simple SAM YAML generates complex CloudFormation code.

Since it transcompiles back to CloudFormation, it supports everything in CloudFormation (e.g. Outputs Mappings, Parameters, Resources).

There are only two commands to deploy to AWS.

SAM can use CodeDeploy to deploy Lambda functions and can help you to run Lambda, API Gateway and DynamoDB locally. Allows for local testing.

  • How to write a SAM Template
    • Transform Header indicates it’s a SAM Template.
    • Transform: 'AWS::Serverkess-2016-10-31'
    • Write Code
      • AWS::Serverless::Function
        • Lambda
    • AWS::Serverless::API
      • API Gateway
    • AWS::Serverless::SimpleTable
      • DynamoDB
    • Package and Deploy
      • aws cloudformation package
      • sam package
      • aws cloudformation deploy
      • sam deploy

Cloud Development Kit (CDK)

Allows you to define cloud infrastructure using a familiar language e.g. JavaScript, Python, Java and .Net

Contains high level components called Constructs using the CDK.

If the code can be compiled, it will be compiled into a CloudFormation Template (JSON/YAML).

This allows you to deploy infrastructure and application runtime code together.

Good for Lambda function and Docker contains.

This is beneficial as it is type safe and you are less likely to get errors (spot errors early)

CDK vs SAM

  • SAM
    • Serverless focussed
    • Write templates declaratively in JSON or YAML
    • Get started quickly with Lambda
    • Leverages CloudFormation
  • CDK
    • Supports every AWS service
    • Written in a familiar programming language
    • Leverages CloudFormation

AWS Cognito

Cognito allows you to give users outside of your AWS account identity so that they can interact with your application.

  • Cognito User Pools
    • Sign in functionality for app users
    • Integrates with API Gateway and Application Load Balancer
  • Cognito Identity Pools (Federated Identity)
    • Provide AWS Credentials to users so that they can access AWS resources directly.
    • Integrate with Cognito User Pools as an identity provider
  • Cognito Sync
    • Synchronize data from device to Cognito
    • Deprecated and replaced by AppSync
  • Cognito vs IAM? Cognito is for:
    • Hundreds of Users
    • Mobile Users
    • Authenticate with SAML

Cognito User Pools (CUP)

Create a serverless database of users for web and mobile apps.

  • Users can use a simple login (Username/email and password)
  • Password Reset
  • Email and Phone number verification
  • MFA Multi Factor Authentication
  • Federated Identities (Facebook, Google, SAML…)
  • Block users if their credentials are compromised elsewhere
  • Login sends back a JWT
  • Integrations
    • API Gateway and Application Load Balancer natively.

Cognito User Pools – Lambda Triggers

Cognito User Pools can invoke a Lambda function synchronously on several triggers:

Cognito User Pools – Hosted Authentication UI

Cognito has a hosted authentication UI that you can add to your app to handle sign up and sign in workflows

Using the hosted UI you have a foundation for integration will social logins, OIDS, SAML etc.

Can customize the logo and CSS.

Cognito Identity Pools (Federated Identities)

Users are outside the AWS environment and want to access services within AWS e.g. DynamoDB

Cognito Identity Pools allows users to obtain temporary AWS credentials.

  • Identity Pools can include
    • Public Providers – Amazon, Facebook, Google, Apple etc.
    • Users in Amazon Cognito User Pool
    • OpenID Connect Providers and SAML Identity providers
    • Developer Authenticated identities (custom login server)
    • Cognito Identity Pools allow for authenticated (guest) access

Users can then access AWS Services directly e.g. SDK, or through API Gateway

The IAM policies applied to the credentials are defined in Cognito. They can be customised based on the user_id for fine grained control.

Cognito Identity Pools
Cognito Identity Pools with Cognito User Pools

Cognito Identity Pools – IAM Roles

There is a default IAM Role assigned to authenticated and guest users.

You can define rules to choose the role for each user based on the users ID.

You can partition your user’s access using policy variables.

These IAM Credentials are obtained by Cognito Identity Pools through STS.

The roles must have a trust policy of Cognito Identity Pools.

Guest user example – Getting a file from S3:

Authenticated users example – POLICY VARIABLE in S3:

User can only access files with that contain their User Id prefix

Authenticated User example – DynamoDB

Row based security

Cognito User Pools vs Identity Pools

  • Cognito User Pools
    • Database of users for your web and mobile application
    • Allows to federate the logins through Public Social, OIDC etc..
    • Can customize the hosted UI for authentication
    • Has triggers with AWS Lambda during the authentication flow
  • Cognito Identity Pools
    • Obtain AWS credentials for users
    • Users can login through Public Social, OIDC, SAML, Cognito User Pools
    • Users can be unauthenticated (guests)
    • Users are mapped to IAM roles and policies, can leverage policy variables

Cognito User Pools + Cognito Identity Pools = manage users / passwords + access AWS services

Cognito Sync

Deprecated – should now use AppSync.

Allows user to store preferences, configurations and state of the app.

Cross device synchronisation (iOS, Android etc.)

Offline capability (synchronized when back online)

Store data in datasets (up to 1 MB), up to 20 datasets to synchronise

  • Push Sync
    • Silently notify across all devices when identity changes
  • Cognito Stream
    • Stream data from Cognito into Kinesis
  • Cognito Events
    • Execute Lambda functions in response to events

Step Functions

Allows you to model workflows as state machines (one per workflow)

e.g. Order fulfillment, Data processing etc.

Define a workflow in JSON then you can visualize the workflow, executions and history.

Workflows can be started with an SDK call, API Gateway, EventBridge (CloudWatch event)

Step functions are made of tasks (Task states).

Tasks States do some work in the state machine.

Tasks can invoke one AWS Service (e.g. Lambda function) or run one activity (e.g. EC2 polling the step function for work to do)

  • Step Function States
    • Choice State – Test for a condition to send to a branch
    • Fail or Success State – Stop execution with failure or success
    • Pass State – Simple pass its input to its output or inject some fixed data, without performing work
    • Wait State – Provide a delay for a certain amount of time or until a specified time/data
    • Map State – Dynamically iterate steps
    • Parallel State – begin parallel branches of execution

Step Functions Error Handling

Step Functions execute many small tasks.

  • Any state can encounter a runtime error
    • State machine definition issues (e.g. no matching rules in Choice State)
    • Task failures (e.g. an exception in a lambda function).
    • Transient issues (network partition events)
  • There are two ways of handling errors in step functions
    1. Use Retry (to retry failed state)
    2. Use Catch (transition to failure path)
  • This should be done in the state machine to handle errors instead of inside the application code
  • Makes the application code much simpler.
  • Predefined Error Codes
    • States.ALL:matches any error name
    • States.Timeout:Task ran longer than TimeoutSeconds or no heartbeat received
    • States.TaskFailed:execution failure
    • States.Permissions:insufficient privileges to execute code
  • The state may also return it’s own errors

This is useful as it allows you to change the error handling flow without handling to rewrite/redeploy code

Step Functions – Retry (Task or Parallel State)

  • Evaluated from top to bottom
  • ErrorEquals match a specific kind of error.
  • IntervalSeconds initial delay before retrying
  • BackOffRate multiply the delay after each retry
  • MaxAttempts
  • When MaxAttempts are reached, the Catch is called.

Step Functions – Catch (Task or Parallel State)

  • Evaluated from top to bottom
  • ErrorEquals matches a specific kind or error
  • Next state to send to
  • ResultPath is a path that determines what input is sent to the state specified in the Next field.

Result Path includes the error in the input.

Step Functions – Standard vs Express

  • Standard
    • Longer Slower Throughput Workflow
    • Maximum duration 1 year
    • Supported execution rate over 2000 per second
  • Express
    • Fast High Throughput Workflow
    • Maximum duration 5 minutes
    • Supported execution start rate 100,000 per second

AppSync

  1. A managed service that uses GraphQL.
    • GraphQL makes it easier for applications to get the data they need.
    • GraphQL allows you to combine data from one or more sources
      • Could combine NoSQL, Relational Databases, HTTP API’s etc
      • Integrates with DynamoDB, Aurora, ElasticSearch
      • Custom sources with AWS Lambda
  2. Retrieve data in real time with WebSocket or MQTT on WebSocket
    • Could use ALB, API Gateway or AppSync for real time websocket
  3. For mobile apps: local data access and data synchronization

To get started with App Sync all you need is to upload one GraphQL schema.

AppSync – Security

There are 4 ways to authorize applications to interact with AppSync GraphQL API

  1. Generate Keys (API_Key)
  2. AWS_IAM: IAM users/ roles / cross-account access
  3. OPENID_CONNECT: OpenID Connect Provider / JWT
  4. AMAZON_COGNITO_USER_POOLS
  5. For custom domains and HTTPS, use CloudFront in front of AppSync

STS – Security Token Service

Allows to grant limited and temporary access to AWS resources for up to 1 hour.

  • AssumeRole – assume roles within your account or cross account.
  • AssumeRoleWithSAML – return credentials for users logged in with SAML
  • AssumeRoleWithWebIdentity
    • Return credentials for users logged in with Identity providers such as (Facebook, Google etc.)
    • But this isn’t used much, instead Cognito Identity Pools instead
  • GetSessionToken – for MFA from a user or AWS account root user.
  • GetFederationToken – obtain temporary credentials for a federated user
  • GetCallerIdentity – return details about the IAM user or role used in the API call
  • DecodeAuthorizationMessage – decode error message when an AWS API call is denied.

STS to Assume A Role

  • Define an IAM role within the account or cross account
  • Define which principals can access this IAM role
  • Then use AWS STS API t retrieve credentials and impersonate the IAM role you have access to (AssumeRole API)
  • Temporary credentials are valid for between 15 minutes and 1 hour.

STS with MFA

  • Use the GetSessionToken from STS. Returns an
    • Access ID
    • Secret Key
    • Session Token
    • Expiration Date
  • Appropriate IAM policy using IAM conditions
  • In the IAM policy add – aws:MultiFactorAuthPresent:true

Advanced IAM – Authorization Model and Evaluation of Policies

  1. If there is an explicit DENY, end decision and DENY
  2. If there is an ALLOW, end decision with ALLOW
  3. Else DENY

IAM Policies and S3 Bucket Policies

IAM Policies are attached to users, roles and groups

S3 Bucket Policies are attached to buckets

Both the policies determine what actions the user can perform on the bucket.

So when evaluating if an IAM Principal can perform an operation X on a bucket, the UNION of it’s assigned IAM policies and S3 bucket policies will be evaluated.

Dynamic Policies with IAM

How would you assign each user a /home/<<user>> folder in S3 bucket?

  1. Create an IAM policy for each user to access their directory
    • e.g. /home/george
    • One policy per user
    • But this doesn’t scale
  2. Create one dynamic policy with IAM
    • Leverage the special policy variable ${aws:username}

Inline vs Managed Policy

There are 3 types of policy within AWS

  • AWS Managed Policy
    • Maintained by AS
    • Good for Power Users and Administrators
    • Updated in case of new services / new API’s
  • Custom Managed Policy
    • Created by you
    • Re-usable, can be applied to many principals
    • Version controlled and rollback
    • Central change management
  • Inline
    • Strict one-to-one relationship between policy and principal
    • Policy is deleted if you delete the IAM principal

Granting a User Permissions to Pass a Role to an AWS Service

When creating a service in AWS, you must pass an IAM Role to this service (usually once during setup)

The service then assumes this role and can perform operations.

To pass a role to another service, you will need the IAM Permission iam:PassRole (and often it comes with iam:GetRole)

In this example role, the user of this role can only pass the role S3 access onto the EC2 instance.

Can a role be passed to any service? NO

Roles can only be passed to what their trust allows

A trust policy for the role that allows the service to assume the role.

Microsoft Active Directory (AD)

Found on any Windows Server with AD Domain Services

Database of objects e.g. users, accounts, computers printers, security groups etc.

Centralized security management e.g. create account, assign permissions.

Objects are organised in trees, a group of trees is a forest.

AWS Directory Services

AWS version of AD

  1. AWS Managed Microsoft AD
    • Create your own AD in AWS, manage users locally, support MFA.
    • Can establish a trust directory with your on-premise AD.
    • Users shared between on-prem AD and AWS Managed AD
  1. AD Connector
    • Directory Gateway (proxy) to redirect to to on-premise AD
    • Users are managed on the on-premise AD
  1. Simple AD
    • AD-compatible managed directory on AWS
    • Cannot be joined with an on-premise AD

The use case is that you can run an EC2 Instance running Windows and it can access the Domain Controller and share the users / logins etc.

Security and Encryption

  • Encryption in flight – SSL
    • Data is encrypted before sending it
    • Server decrypts it after receiving it
    • SSL certificates help with the encryption (HTTPS)
    • This protects with the MITM attack (man in the middle)
  • Server side encryption at rest
    • Data is encrypted after being received by the server
    • Data is decrypted before being sent back to the client
    • It is stored in an encrypted form using a key
    • The encryption / decryption keys must be managed somewhere and the server must have access to it
  • Client side encryption
    • Data is encrypted by the client and never decrypted by the server
    • Data is decrypted by a receiving client
    • The server should not be able to decrypt the data
    • Could leverage envelope encryption

KMS (Key Management Service)

  • Easy way to control access to your data, AWS manages the keys for you.
  • Fully integrated with IAM for authorization.
  • Customer Master Keys (CMK) Types
    • Symmetric (AES-256 keys)
      • Single key for encryption and decryption
      • Necessary for envelope encryption
      • You never get access to the key unencrypted (must call KMS API to use the key)
    • Asymmetric (RSA and ECC key pairs)
      • Public key for encryption
      • Private key for decryption
      • Can be used for Sign/Verify operations
      • The public key is downloadable, you can’t access the private key
      • User case: encryption outside of AWS by users who can’t call the KMS API.
    • Three types:
      • AWS Managed Service Default CMK – free
      • User Keys created in KMS – $1 / month
      • User Keys imported (must be 256 bit symmetric) – $1 / month
    • Pay for every call to KMS – $0.03 / 10000 calls

The keys are fully manageable – Create, Rotate, Disable, Enable etc.

The keys can be audited using CloudTrail

KMS is powerful because you never see the keys, they are fully managed by AWS and can be automatically rotated.

KMS has a limit of encrypting up to 4KB of data per call.

If data is > 4KB, use envelope encryption.

  • To give someone access to KMS there are two things required
    • Make sure the Key Policy allows the user
    • Make sure the IAM allows the API calls

KMS and Regions

KMS Keys are REGION BOUND

KMS Key Policies

You can control access to KMS keys – similar to S3 bucket policies

The difference is that you cannot control access without them.

  • Default KMS Key Policy
    • Created if you don’t provide a specific KMS Key Policy
    • Gives complete access to the key to the root user.
    • Gives access to the IAM policies to the KMS key.
  • Custom Key Policy
    • Define users, roles that can access the KMS Key
    • Define who can administer the key
    • This is useful for cross-account access of your key
  • Copying a Snapshot across accounts
    1. Create a Snapshot, encrypted with your own CMK (Customer Master Key)
    2. Attach a KMS Key Policy to authorize cross-account access
    3. Share the encrypted snapshot
    4. In the target account create a copy of the Snapshot, encrypt it with a KMS key in the target account
    5. Create a volume from the snapshot

Envelope Encryption

KMS Encryption for < 4KB data.

If you want to encrypt more than 4KB you must use Envelope Encryption.

To do this use the GenerateDataKey API – Envelope Encryption. The encryption is done client side.

  • Encrypt API – Up to 4 KB
  • Decrypt API – Up to 4 KB
  • GenerateDataKey – generates a unique symmetric data key (DEK)
    • Returns a plaintext copy of the data key
    • And a copy that is encrypted under the CMK that you specify
  • GenerateDataKeyWithoutPlainText
    • Generate a DEK to use at some point (not immediately)
    • DEK is encrypted using the CMK you specify (but to use it you must you Decrypt first)
  • GenerateRandom returns a random byte string

Encryption SDK

  • The AWS Encryption SDK implements Envelope Encryption for us.
  • The encryption SDK also exists as a CLI tool that can be installed.
  • Implementations in Java, Python, C, JavaScript
  • Data Key Caching
    • Reuse data keys instead of creating new ones for each encryption
    • This helps to reduce the number of KMS API calls therefore reducing cost
    • But has a security trade off
    • Use LocalCryptoMaterialsCache (max age, max bytes, max number of messages etc.)

KMS Limits and Request Quotas

ThrottlingException – exceed a request quota.

Use exponential backoff to retry.

For all cryptographic operations throughout AWS, they all share a quota. This includes requests made on our behalf by AWS.

  • To avoid these quotas and limits
    • Use GenerateDataKey – consider using DEK caching from the Encryption SDK.
    • You can request a quota increase through the API or through AWS Support.

S3 Encryption for Objects

There are 4 methods of encrypting objects in S3:

  1. SSE-S3
    • Encrypts objects using keys handled and managed by AWS
  2. SSE-KMS
    • Use KMS to manage encryption
  3. SSE-C
    • Manage your own encryption keys
  4. Client Side Encryption

SSE-KMS

S3 Encryption using keys handled and managed by KMS.

This gives user control and auditable using CloudTrail

Object is encrypted server side

Must set header: “x-amz-server-side-encryption”: “aws:kms”

SSE-KMS uses GenerateDataKey API as the S3 objects are likely going to be larger than 4KB.

Then uses the Decrypt API Calls.

  • To perform SSE-KMS you need:
    • A KMS Key Policy that authorizes the user/role
    • An IAM policy that authorizes access to KMS
  • S3 calls KMS for SSE-KMS and this uses the quota limit.
    • This could case you to hit the quota limit.
    • So if you are getting throttling exception, it could be KMS and not S3.

S3 Bucket Policies

  • Force SSL
    • Create an S3 bucket policy with a DENY on the condition aws:SecureTransport = false
  • Force Encryption of SSE-KMS
    • Deny incorrect encryption header – make sure is includes aws:kms==SSE-KMS
    • Deny no encryption header to ensure objects are not uploaded un-encrypted.

S3 Bucket Key for SSE-KMS Encryption

Setting to decrease the number of API calls made to KMS from S3 when using SSE-KMS by 99%.

Reduce cost of overall KMS encryption with S3 by 99%.

  • It leverages data keys
    1. A S3 Bucket Key is created
    2. The key is used to encrypt KMS objects with new data keys
  • But you will see less KMS CloudTrail events

SSM Parameter Store

Securely store configuration and secrets – optional encryption with KMS.

  • Features:
    • Serverless, scalable, durable.
    • Versioning for tracking configuration and secrets.
    • Configuration management using path and IAM.
    • Notifications with CloudWatch Events
    • Integration with CloudFormation
  • SSM Parameter Store
    • Can store secrets in a directory hierarchy
    • Can reference secrets in secrets manager
    • Can reference resource id’s.
  • Parameter Tiers
    • Standard
      • 10,000
      • 4KB
      • NO parameter policies available.
    • Advanced
      • 100,000
      • 8KB
      • Parameter policies are available

Parameter Policies (Advanced Parameters)

  • Only for advanced parameters
  • Allows you to assign TTL to a parameter (expiration date) to force updating or deleting sensitive data such as passwords.
  • Can assign multiple policies at a time

Secrets Manager

New service for storing secrets.

  • Secrets manager is more designed for secrets compared to parameter store.
  • It has the capability to force rotation of secrets every X Days
  • Automate generation of secrets on rotation (uses lambda)
  • Integrate with RDS to sync secrets
  • Encrypted using KMS.
  • Mostly meant for RDS.

Difference between Parameter Store and Secrets Manager

  • Secrets Manager
    • Expensive
    • Automatic rotation of secrets with lambda
    • Integration with RDS, Redshift and DocumentDB.
    • KMS encryption is mandatory
    • Integration with CloudFormation
  • SSM Parameter Store
    • Cheaper
    • Simple API
    • No secret rotation
    • KMS encryptions is optional
    • Can integrate with CloudFormation
    • Can pull Secrets Manager secrets using the SSM Parameter Store API.

CloudWatch Logs Encryption

  • Can encrypt CloudWatch logs with KMS keys
  • Encryption is enabled at the log group level by associating a CMK with a log group, either when you create the log group or after it exists.
  • You cannot associate a CMK with a log group using the CloudWatch console.
    • You have to use the CloudWatch API
    • associate-kms-key : associate a key with a log group that already exists
    • create-log-group : if the log group doesn’t exist yet

CodeBuild Security

  • To access resources in your VPC, make sure you specify a VPC configuration for your CodeBuild.
  • Secrets in CodeBuild
    • Environment variables can reference parameter store parameters.
    • Environment variables can reference secrets manager secrets.

AWS SES

Send emails using SMPT or SDK. Needs the correct IAM permissions to send the emails.

Receive emails and integrate using S3, SNS, Lambda.

Other AWS Databases

  • RDS (Relational Database)
    • PostgreSQL, MySQL
    • Aurora and Aurora Serverless
    • Provisioned Database
  • DynamoDB (NoSQL)
    • Managed Key Value Database
    • Stores documents
    • Serverless
  • ElastiCache (In memory database)
    • Redis Memcached
    • Cache capability
  • Redshift (OLAP)
    • Online analytical processing
    • Data warehousing, data lake.
    • Analytics queries
  • Neptune
    • Graph Database
  • DMS
    • Database migration service
  • DocumentDB
    • Managed MongoDB for AWS.

AWS Certificate Manager (ACM)

Used to host public SSL certificates in AWS.

  • This can be done two ways
    1. Buy your own certificates and upload them using the CLI.
    2. Have ACM provision and renew public SSL certificates for you (free of cost)
  • ACM loads SSL certificates on the following integrations
    • Load Balancers (including those created by EB)
    • CloudFront Distributions
    • API’s on API Gateway

JakTech

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.