Select Page
FacebooktwitterlinkedinyoutubeFacebooktwitterlinkedinyoutube

Deploying a global application to build custom pages for end-users has availability and performance challenges. This blog discusses how you can implement the Heimdall Proxy to provide low latency performance without application changes.

 

Amazon Aurora Global Database allows a single database cluster to span multiple AWS regions by asynchronously replicating your data within sub-second timing.  This enables fast, low-latency local reads in each region, and provides disaster recovery from region-wide outages by providing multi-region writer failover.  These capabilities minimize the RTO (Recovery Time Objective) of cluster failure, and thus reduce data loss during failure, allowing the RPO (Recovery Point Objective) to be achieved. 

 

There are considerations that those implementing an AWS Global Database need to be aware of, however.  The most important is that most applications are designed so that you connect to a single hostname, and the database “just works” with ACID consistency across connections and application nodes.  In the case of a global database, you have reader hostname endpoints in each region, and in the primary region, you have two endpoints, one for writes, and one for reads.  It is typically the job of the application developer to properly access endpoints optimally, and on a failover, to change the writer endpoint.  The application must be intelligent to understand 1) When it is safe to use a local reader (or locate the closest reader), or 2) Choose the remote writer for reads due to consistency requirements.  Unfortunately, additional development is typically required to implement the full capabilities of the Amazon Aurora Global Database.

 

In order to resolve the challenges that face those implementing the Amazon Aurora Global Database, the Heimdall Proxy in conjunction with Amazon Route53, allows global database nodes to be optimally utilized while providing application edge database caching functionality.  Global and edge-based applications can now access the global database seamlessly and optimally, without having to rewrite the application.  

 

Solution Overview

 

First, to implement the solution, we will create an Aurora Global Database, with a writer in one region in US-West-2, and additional reader regions in AP-South-2 and AP-Southeast-2.  Then, with this cluster created, we will create a Heimdall proxy cluster in the same regions.  

An Aurora Global Database is created with a primary Aurora cluster in one Region and secondary Aurora clusters in one or more additional regions. Aurora Global Database uses the dedicated infrastructure in the Aurora purpose-built storage layer to handle replication across Regions. Using Heimdall Global Load Balancing Feature Leveraging Aurora Postgres Global Databases allows applications residing anywhere in the world to connect to the Reader Instance closest to the application using latency based routing.

 

In traditional database environments, applications connect to specific Read/Write database instances. The Heimdall Proxy is a transparent data access layer that intelligently routes queries to the most optimal data source, resulting in SQL offload and improved response times. The Heimdall proxy leverages Amazon ElastiCache for Redis to cache SQL results and track SQL queries so they are routed to the appropriate database node for fresh data.

 

In this architecture users can deploy Aurora Postgres Global Databases in Multiple regions (up to 5 regions are supported) at least 1 instance in each region, to ensure resiliency in case of region failure Heimdall Proxy Instances are deployed in each of the regions as well for high availability (Heimdall Proxy can also be deployed behind a Load Balancer within a region as well for multi availability zone resiliency.)

 

The IAM role is added to the Heimdall Proxy host instance and this also contains the Aurora Global cluster ARN which allows Heimdall to track the Readers Available and incase of failover which instance/ region is the writer. 

 

The following diagram shows the Heimdall Proxy with Aurora global databases setup in multiple regions:

  • An application connected to Heimdall Proxy in the primary region routes Write connections to the Aurora Writer Instance and Read queries to the reader instance that responds with the least latency. (This may be in-region or to a secondary region reader instance as well;)
  • Applications connected to Heimdall proxy in the secondary Region(s), routes Write connections to the Aurora Writer Instance in the Primary region and Read queries to the reader instance that responds with the least latency. (This may be in-region or to a secondary region reader instance as well;)
    Incase   Reader instances or the Aurora cluster in the secondary region becomes unavailable the read connections will routed to the Aurora reader which response with the least latency.
  • Built-in Heimdall Proxy features like auto-caching and route queries (read/write splits) are still available to applications in secondary regions as well and the proxy will automatically route connections:
    • See this blog for detailed steps on enabling Read Write splits.
    • See this blog for Advanced Connection pooling.
  • With the architecture up to 5 Aurora Global Database Regions and up to to 15 Reader Instances in the Primary Region and 16, Reader Instances in each of the secondary regions can be set up, the Heimdall smart proxy will automatically be able to automatically detect configuration changes and incase Auto Autoscaling Readers is configured in the primary region, the proxy will dynamically be able to detect additional/removal of Reader Instances and route connections automatically. Note: Autoscaling Replicas is not supported for Secondary DB Clusters in Aurora Global Databases

With an Aurora global database, there are 2 approaches to failover:

  • Managed planned failover – To relocate your primary DB cluster to one of the secondary Regions in your Aurora global database, see Managed planned failovers with Amazon Aurora Global Database. With this feature, RPO is 0 (no data loss) and it synchronizes secondary DB clusters with the primary before making any other changes. RTO for this automated process is typically less than that of the manual failover.
  • Manual unplanned failover – To recover from an unplanned outage, you can manually perform a cross-Region failover to one of the secondaries in your Aurora global database. The RTO for this manual process depends on how quickly you can manually recover an Aurora global database from an unplanned outage. The RPO is typically measured in seconds, but this depends on the Aurora storage replication lag across the network at the time of the failure.

Since Heimdall Proxy automatically detects RDS/ Aurora configuration changes based on the ARN of the Aurora Global Cluster both managed planned and manual unplanned failovers are supported in the above architecture

 

Steps to create the above setup:

Create an Aurora global database or add Aurora Global database for a pre-existing Aurora cluster.

 

The Primary Cluster will be created in ap-southeast-2 region, and secondary clusters in ap-south-1 and us-west-2 

  1. On the Amazon RDS console, choose Databases.
  2. Create Database:-

Choose Amazon Aurora: MySQL or PostgreSQL

Enable the radio button for “Show versions that support the global database feature”. This will ensure you are selecting an Aurora Mysql or Aurora Postgres version that is supported for Global Databases.

Follow the steps to create the Aurora Cluster:- https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html

 

Once the Primary Region Cluster is created: 

On the source Cluster, click Actions -> Choose Add region

On the Add a region page, Enter a name for your Global database identifier:- 

If you need you add multiple regions, select Add region step above to add all the regions as needed.

 

Once the Cluster is created:

Create Heimdall Proxy Instances as needed. You do NOT need to create Heimdall Proxy instances in all the regions where you have created Global Clusters. If needed, you can create Heimdall {roxy instances in other regions where you do not have Aurora Global Instances as well; this will simply route the connections from the application to the neared Aurora Global Database instance.

 

Configuration of Proxy

Once the Aurora Global Databases and the Heimdall Proxy Instances are running:-

Go to the Primary region Heimdall Proxy Instance and Enter the IP address to access the console:

Go to the Configure Section and select Wizard: 

Select AWS Detect:

And select the Global Cluster Name:

Select Next and Ensure the database host name and other details are correct;

Select Next and ensure the Track Cluster changes option is selected:

Select next, for Caching select the Redis Elastic Cache option if you plan to use caching or select Local Only if you do not have Elastic cache setup, this will use Local Cache on the Heimdall Proxy (EC2) Instance:

Select through the options and on the summary page confirm all the details are correct. Click Next and then Submit. 

Once the Configuration is complete, navigate to the Status Page; you will be able to see all the Aurora Instances shown here:

Navigate to the Dashboard page to view the Metrics related to Queries being executed against the Heimdall Proxy:

 

FacebooktwitterlinkedinyoutubeFacebooktwitterlinkedinyoutube