Ryfujk

Following rposcibed pattern here: Scenario 3: VPC with Public and Private Subnets and Hardware VPN Access

Summary Description:
I have configured a VPC with 1 public Subnet and 2 private Subnets
Each subnet is in a different AZ
There are two Route Tables: 1 for public Subnet and 1 for 2 private subnets
There are two NACLs: 1 for public Subnet and 1 for 2 private subnets
There is an internet gateway (routed to the public subnet)
There is a Security Group for the EC2 instances
There is a Security Group for the RDS instances
I have an EC2 instance Apache/Tomcat with web application in the Public Subnet
I have an RDS PostgreSQL database deployed in the 2 private subnets (multi-AZ=YES)

This is what I can do:

1. I can SSH to the EC2 Instance




2. I can access the Tomcat Manager app on port 80




3. I can deploy a war file (my application)




4. If I make the private Subnets public and set the Security Groups and NACLs accordingly, I can access the postgresql database with PGAdmin and I can run the application from my desktop using the RDS instance endpoint as the host.
   So the database is in good order and can be reached.

Here is what is failing:

1. When I deploy my application to Tomcat and attempt to access the database it fails: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connecions.




2. When I ping the RDS endpoint (DNS name) it resolves correctly to the private IP address but cannot reach the node (ALL Traffic is enabled in the NACLs and Security Groups).

So I can get to both EC2 and RDS instances from my desktop from the public internet, but I cannot enable them to connect to each other from within the VPC.

I set all of the NACLs and Security groups to maximum permissiveness just to force a successful connection. But even that is not working.

I will include the pertinent details a little later. If anyone has any suggestions on what to check for would be greatly appreciated!!!

Thanks

I finally figured it out. I'll finish describing the problem and then I will provide the answer. The problem was rooted in the NALCS and Security Groups so I'll share just those details. The routes and internet gateway were straight forward.

VPC NACL

In Bound

80  0.0.0.0/0 Allow

Out Bound

All 0.0.0.0/0 Allow

Public Subnet NACL

InBound

80  0.0.0.0/0 Allow

Out Bound

All 0.0.0.0/0 Allow

5432    172.30.4.0/24 Allow (PostgreSQL)

Private Subnet NACL

InBound

5432    172.30.1.0/24 Allow

Out Bound

5432    172.30.1.0/24 Allow 

VPC Security Group

InBound

80  VPC-Security-Group-ID Allow

Out Bound

All 0.0.0.0/0 Allow

Public Subnet Security Group

InBound

80  0.0.0.0/0 Allow

Out Bound

5432    172.30.4.0/0 Allow

Private Subnet Security Group

InBound

5432    172.30.1.0/24 Allow

Out Bound

5432    172.30.1.0/24 Allow 

f you know any thing about this (maybe you are a network engineer) you probably see right away what the problem is.

The problem is rooted in these concepts:

• Ephemeral ports


• Stateless and Stateful


• Connection Tracking


• NACLs are Stateless


• Security Groups are Stateful

Read about it here:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html

https://en.wikipedia.org/wiki/Ephemeral_port

The corrected confiuration is below. The change that really corrected the connectivity between the Public and Private subnets was enabling response traffic on the Ephemeral ports for both In Bound Public subnet NACL and Out Bound Private subnet NACL rules.
I also cleaned up some redundant and insecure Out Bound rules in the NACLS and Security Groups.

VPC NACL

In Bound

80  0.0.0.0/0 Allow

Out Bound

32768-65535 0.0.0.0/0 Allow

Public Subnet NACL

InBound

80  0.0.0.0/0 Allow

32768-65535 172.30.4.0/0 Allow

Out Bound

32768-65535 0.0.0.0/0 Allow

5432    172.30.4.0/24 Allow (PostgreSQL)

Private Subnet NACL

InBound

5432    172.30.1.0/24 Allow

Out Bound

32768-65535 172.30.1.0/24 Allow 

VPC Security Group

InBound

80  VPC-Security-Group-ID Allow

Out Bound

Public Subnet Security Group

InBound

80  0.0.0.0/0 Allow

Out Bound

5432    172.30.4.0/0 Allow

Private Subnet Security Group

InBound

5432    172.30.1.0/24 Allow

Out Bound

This is a working configuration. Hopefully this helps someone. If you see anyhting that can be improved let me know and let me know if you have questions.