At times you may encounter a need to have U-Turn NAT in place on your firewall to allow internal devices to access resources you host (Such as a web-server) in the same Datacenter, by using their public address. Palo Alto has a great KB article here on the subject. Setting this up is a significant amount of work when used in a environment that is designed for multi-tenancy through the use of Virtual Routers on the PA Firewall. Below are some of my challenges and solutions.
Consider the following network design-
In this setup we have a Customer VM (10.5.1.18) that needs to access a resource we host in the same data-center on it’s public address (200.231.83.250). This resource is in a different VR, and also in a different security zone. The following NAT policy is installed to implement this routing,
set rulebase nat rules SP-N-ABLE-U-TURN-NAT to UNTRUST set rulebase nat rules SP-N-ABLE-U-TURN-NAT from Customer set rulebase nat rules SP-N-ABLE-U-TURN-NAT source any set rulebase nat rules SP-N-ABLE-U-TURN-NAT destination "200.231.83.250" set rulebase nat rules SP-N-ABLE-U-TURN-NAT service any set rulebase nat rules SP-N-ABLE-U-TURN-NAT to-interface any set rulebase nat rules SP-N-ABLE-U-TURN-NAT destination-translation translated-address 10.1.100.15 set rulebase nat rules SP-N-ABLE-U-TURN-NAT disabled no set rulebase nat rules SP-N-ABLE-U-TURN-NAT source-translation dynamic-ip-and-port interface-address interface ethernet1/1.1
Normally, this would work fine. However, in this setup we end up with the traffic going the wrong direction. First, let’s look at the packet dump.
== 2017-06-13 15:53:00.709 -0700 ==
Packet received at slowpath stage
Packet info: len 66 port 26 interface 267 vsys 1
wqe index 228305 packet 0x0x8000000037a3a8d2
Packet decoded dump:
L2: 00:1c:73:b4:3b:e3->00:1b:17:00:01:30, VLAN 5 (0x8100 0x0005), type 0x0800
IP: 10.5.1.18->200.231.83.250, protocol 6
version 4, ihl 5, tos 0x00, len 48,
id 3815, frag_off 0x4000, ttl 127, checksum 48616
TCP: sport 64240, dport 443, seq 3449997169, ack 0,
reserved 0, offset 7, window 8192, checksum 41574,
flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 01 04 02 ........
Session setup: vsys 1
PBF lookup (vsys 1) with application none
Session setup: ingress interface ae1.5 egress interface ethernet1/1.1 (zone 4)
NAT policy lookup, matched rule index 0
Destination NAT, translated IP 10.1.100.15
PBF lookup (vsys 1) with application none
Session setup: egress zone 12 for natted IP
Translated IP in zone 12, egress id 267
Policy lookup, matched rule index 34,
Allocated new session 92273.
Rule: index=0 name=SP-N-ABLE-U-TURN-NAT, cfg_pool_idx=16 cfg_fallback_pool_idx=0
2017-06-13 15:53:00.710 -0700 debug: pan_flow_nat_setup_session(src/pan_flow_nat.c:2147): NAT Rule: name=SP-N-ABLE-U-TURN-NAT, cfg_pool_idx=16; Session: index=92273, nat_pool_idx=16
Packet matched vsys 1 NAT rule 'SP-N-ABLE-U-TURN-NAT' (index 1),
source translation 10.5.1.18/64240 => 69.4.146.232/30661
destination translation 207.231.83.250/443 => 10.1.100.15/443
Created session, enqueue to install
The flow ends up looking like this –
We can see that NAT translation is working, But the packet ends up at 10.5.99.5, another router in the same VR instance that our source machine is in. This is because of our routing table for VR 1 contains the network that we are looking for (Overlapping subnets are in place between customers) –
VIRTUAL ROUTER: Customer(id 1) ========== destination nexthop metric flags age interface next-AS 0.0.0.0/0 69.4.146.225 10 A S ethernet1/1.1 10.0.0.0/8 10.5.99.1 10 A S ae1.5
The problem here is the fact that our translated NAT address (10.1.100.15) lies in a different virtual router (VR2). The resolution for this is to insert a PBR rule to force the egress traffic out the proper interface. In this instance, our PBF rule looks like this –
set rulebase pbf rules R1 action forward nexthop ip-address 10.255.254.254 set rulebase pbf rules R1 action forward egress-interface ae1.2000 set rulebase pbf rules R1 from zone Customer set rulebase pbf rules R1 enforce-symmetric-return enabled no set rulebase pbf rules R1 source 10.5.1.18 set rulebase pbf rules R1 destination 10.1.100.15 set rulebase pbf rules R1 source-user any set rulebase pbf rules R1 application any set rulebase pbf rules R1 service any
After doing this, we still need to insert routes into both Virtual Routers in the firewall to make the networks aware of each other. This can present a security risk, but we are able to lock it down with rules. The routes will look like this —
#Set route on intenal router to allow access to customer subnet) set network virtual-router Default routing-table ip static-route Customer-10.5.1.0 destination 10.5.1.0/24 set network virtual-router Default routing-table ip static-route Customer-10.4.1.0 nexthop next-vr Customer #set route on customer VR to allow access to internal resources set network virtual-routerCustomer routing-table ip static-route 10.5.1.18 nexthop next-vr Internal set network virtual-router Customer routing-table ip static-route 10.5.1.18destination 10.1.100.15/32
Even after doing this, you might still not have traffic passing. That’s because we need a security policy in place to allow this traffic. Looking at the following packet capture –
== 2017-06-13 16:13:07.009 -0700 == Packet received at slowpath stage Packet info: len 66 port 26 interface 267 vsys 1 wqe index 229027 packet 0x0x8000000037ad28d2 Packet decoded dump: L2: 00:1c:73:b4:3b:e3->00:1b:17:00:01:30, VLAN 5 (0x8100 0x0005), type 0x0800 IP: 10.5.1.18->200.231.83.250, protocol 6 version 4, ihl 5, tos 0x00, len 48, id 1006, frag_off 0x4000, ttl 127, checksum 51425 TCP: sport 65272, dport 443, seq 3049991729, ack 0, reserved 0, offset 7, window 8192, checksum 20342, flags 0x0002 ( SYN), urgent data 0 TCP option: 00000000: 02 04 05 b4 01 01 04 02 ........ Session setup: vsys 1 PBF lookup (vsys 1) with application none Session setup: ingress interface ae1.5 egress interface ethernet1/1.1 (zone 4) NAT policy lookup, matched rule index 0 Destination NAT, translated IP 10.1.100.15 PBF lookup (vsys 1) with application none Packet matches PBF policy (vsys 1) rule idx 0 (id 3) for c2s. PBF policy rule 0. Session setup: egress zone 5 for natted IP Translated IP in zone 5, egress id 256 Policy lookup, matched rule index 36, Packet dropped, Session setup failed
A great way to troubleshoot this is to use of debug packet-filters. First, set your match criteria –
“debug dataplane packet-diag set filter index 1 match source 10.5.1.18 destination 200.231.83.250”
Then, run some traffic and watch your counters.
“show counter global filter severity drop-packet-filter yes delta yes”
nsadmin@DSA-FW-1(active)> show counter global filter severity drop packet-filter yes delta yes Global counters: Elapsed time since last sampling: 9.165 seconds name value rate severity category aspect description -------------------------------------------------------------------------------- flow_policy_deny 2 0 drop flow session Session setup: denied by policy -------------------------------------------------------------------------------- Total counters shown: 1 --------------------------------------------------------------------------------
Using the above, we can see the reason for the drops now
Session setup: denied by policy
We are going to need a security policy to allow this traffic to pass. To determine the exact policy needed, I like to setup a “Allow-Any” rule, determine the traffic we need to allow, then implement the rule. The final rule to allow this traffic looks like this –
set rulebase security rules CUSTO-INT to Internal set rulebase security rules CUSTO-INT from Customer set rulebase security rules CUSTO-INT source any set rulebase security rules CUSTO-INT destination "200.231.83.250" set rulebase security rules CUSTO-INT source-user any set rulebase security rules CUSTO-INT category any set rulebase security rules CUSTO-INT application any set rulebase security rules CUSTO-INT service application-default set rulebase security rules CUSTO-INT hip-profiles any set rulebase security rules CUSTO-INT tag sparetime set rulebase security rules CUSTO-INT action allow set rulebase security rules CUSTO-INT log-start yes set rulebase security rules CUSTO-INT disabled no
To review, the following is needed when you setup U-Turn NAT on a Palo Alto Firewall, with the two host devices between in different virtual routers.
- Nat Policy in place to translate traffic destined for the external IP of the internal device, to the internal IP.
- Routes on each VR Instance to expose the devices to each other.
- PBF Rule to send the traffic out the correct interfaces on the return.
- Security Policies to allow this traffic to pass.