Setting up and troubleshooting Palo Alto U-Turn NAT with multiple Virtual Router Instances

At times you may encounter a need to have U-Turn NAT in place on your firewall to allow internal devices to access resources you host (Such as a web-server) in the same Datacenter, by using their public address. Palo Alto has a great KB article here on the subject.  Setting this up is a significant amount of work when used in a environment that is designed for multi-tenancy through the use of Virtual Routers on the PA Firewall. Below are some of my challenges and solutions.

Consider the following network design-

 

In this setup we have a Customer VM (10.5.1.18) that needs to access a resource we host in the same data-center on it’s public address (200.231.83.250). This resource is in a different VR, and also in a different security zone. The following NAT policy is installed to implement this routing,

set rulebase nat rules SP-N-ABLE-U-TURN-NAT to UNTRUST
set rulebase nat rules SP-N-ABLE-U-TURN-NAT from Customer
set rulebase nat rules SP-N-ABLE-U-TURN-NAT source any
set rulebase nat rules SP-N-ABLE-U-TURN-NAT destination "200.231.83.250"
set rulebase nat rules SP-N-ABLE-U-TURN-NAT service any
set rulebase nat rules SP-N-ABLE-U-TURN-NAT to-interface any
set rulebase nat rules SP-N-ABLE-U-TURN-NAT destination-translation translated-address 10.1.100.15
set rulebase nat rules SP-N-ABLE-U-TURN-NAT disabled no
set rulebase nat rules SP-N-ABLE-U-TURN-NAT source-translation dynamic-ip-and-port interface-address interface ethernet1/1.1

Normally, this would work fine. However, in this setup we end up with the traffic going the wrong direction. First, let’s look at the packet dump.

== 2017-06-13 15:53:00.709 -0700 ==
Packet received at slowpath stage
Packet info: len 66 port 26 interface 267 vsys 1
  wqe index 228305 packet 0x0x8000000037a3a8d2
Packet decoded dump:
L2:     00:1c:73:b4:3b:e3->00:1b:17:00:01:30, VLAN 5 (0x8100 0x0005), type 0x0800
IP:     10.5.1.18->200.231.83.250, protocol 6
        version 4, ihl 5, tos 0x00, len 48,
        id 3815, frag_off 0x4000, ttl 127, checksum 48616
TCP:    sport 64240, dport 443, seq 3449997169, ack 0,
        reserved 0, offset 7, window 8192, checksum 41574,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 01 04 02                             ........
Session setup: vsys 1
PBF lookup (vsys 1) with application none
Session setup: ingress interface ae1.5 egress interface ethernet1/1.1 (zone 4)
NAT policy lookup, matched rule index 0
Destination NAT, translated IP 10.1.100.15
PBF lookup (vsys 1) with application none
Session setup: egress zone 12 for natted IP
Translated IP in zone 12, egress id 267
Policy lookup, matched rule index 34,
Allocated new session 92273.
Rule: index=0 name=SP-N-ABLE-U-TURN-NAT, cfg_pool_idx=16 cfg_fallback_pool_idx=0
2017-06-13 15:53:00.710 -0700 debug: pan_flow_nat_setup_session(src/pan_flow_nat.c:2147): NAT Rule: name=SP-N-ABLE-U-TURN-NAT, cfg_pool_idx=16; Session: index=92273, nat_pool_idx=16
Packet matched vsys 1 NAT rule 'SP-N-ABLE-U-TURN-NAT' (index 1),
source translation 10.5.1.18/64240 => 69.4.146.232/30661
destination translation 207.231.83.250/443 => 10.1.100.15/443
Created session, enqueue to install

The flow ends up looking like this –

 

We can see that NAT translation is working, But the packet ends up at 10.5.99.5, another router in the same VR instance that our source machine is in. This is because of our routing table for VR 1 contains the network that we are looking for (Overlapping subnets are in place between customers) –

VIRTUAL ROUTER: Customer(id 1)
==========
destination                                 nexthop                                 metric flags      age   interface          next-AS
0.0.0.0/0                                   69.4.146.225                            10     A S              ethernet1/1.1
10.0.0.0/8                                  10.5.99.1                               10     A S              ae1.5

The problem here is the fact that our translated NAT address (10.1.100.15) lies in a different virtual router (VR2). The resolution for this is to insert a PBR rule to force the egress traffic out the proper interface. In this instance, our PBF rule looks like this –

set rulebase pbf rules R1 action forward nexthop ip-address 10.255.254.254
set rulebase pbf rules R1 action forward egress-interface ae1.2000
set rulebase pbf rules R1 from zone Customer
set rulebase pbf rules R1 enforce-symmetric-return enabled no
set rulebase pbf rules R1 source 10.5.1.18
set rulebase pbf rules R1 destination 10.1.100.15
set rulebase pbf rules R1 source-user any
set rulebase pbf rules R1 application any
set rulebase pbf rules R1 service any

After doing this, we still need to insert routes into both Virtual Routers in the firewall to make the networks aware of each other. This can present a security risk, but we are able to lock it down with rules. The routes will look like this —

#Set route on intenal router to allow access to customer subnet)
 
set network virtual-router Default routing-table ip static-route Customer-10.5.1.0 destination 10.5.1.0/24
set network virtual-router Default routing-table ip static-route Customer-10.4.1.0 nexthop next-vr Customer
 
#set route on customer VR to allow access to internal resources
 
set network virtual-routerCustomer routing-table ip static-route 10.5.1.18 nexthop next-vr Internal
set network virtual-router Customer routing-table ip static-route 10.5.1.18destination 10.1.100.15/32

Even after doing this, you might still not have traffic passing. That’s because we need a security policy in place to allow this traffic. Looking at the following packet capture –

== 2017-06-13 16:13:07.009 -0700 ==
Packet received at slowpath stage
Packet info: len 66 port 26 interface 267 vsys 1
wqe index 229027 packet 0x0x8000000037ad28d2
Packet decoded dump:
L2:     00:1c:73:b4:3b:e3->00:1b:17:00:01:30, VLAN 5 (0x8100 0x0005), type 0x0800
IP:     10.5.1.18->200.231.83.250, protocol 6
version 4, ihl 5, tos 0x00, len 48,
id 1006, frag_off 0x4000, ttl 127, checksum 51425
TCP:    sport 65272, dport 443, seq 3049991729, ack 0,
reserved 0, offset 7, window 8192, checksum 20342,
flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 01 04 02                             ........
Session setup: vsys 1
PBF lookup (vsys 1) with application none
Session setup: ingress interface ae1.5 egress interface ethernet1/1.1 (zone 4)
NAT policy lookup, matched rule index 0
Destination NAT, translated IP 10.1.100.15
PBF lookup (vsys 1) with application none
Packet matches PBF policy (vsys 1) rule idx 0 (id 3) for c2s.
PBF policy rule 0.
Session setup: egress zone 5 for natted IP
Translated IP in zone 5, egress id 256
Policy lookup, matched rule index 36,
Packet dropped, Session setup failed

A great way to troubleshoot this is to use of debug packet-filters. First, set your match criteria –

“debug dataplane packet-diag set filter index 1 match source 10.5.1.18 destination 200.231.83.250”

Then, run some traffic and watch your counters.

“show counter global filter severity drop-packet-filter yes delta yes”

nsadmin@DSA-FW-1(active)> show counter global filter severity drop packet-filter yes delta yes
 
Global counters:
Elapsed time since last sampling: 9.165 seconds
 
name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------
flow_policy_deny                           2        0 drop      flow      session   Session setup: denied by policy
--------------------------------------------------------------------------------
Total counters shown: 1
--------------------------------------------------------------------------------

Using the above, we can see the reason for the drops now

 	
Session setup: denied by policy

We are going to need a security policy to allow this traffic to pass. To determine the exact policy needed, I like to setup a “Allow-Any” rule, determine the traffic we need to allow, then implement the rule. The final rule to allow this traffic looks like this –

set rulebase security rules CUSTO-INT to Internal
set rulebase security rules CUSTO-INT from Customer
set rulebase security rules CUSTO-INT source any
set rulebase security rules CUSTO-INT destination "200.231.83.250"
set rulebase security rules CUSTO-INT source-user any
set rulebase security rules CUSTO-INT category any
set rulebase security rules CUSTO-INT application any
set rulebase security rules CUSTO-INT service application-default
set rulebase security rules CUSTO-INT hip-profiles any
set rulebase security rules CUSTO-INT tag sparetime
set rulebase security rules CUSTO-INT action allow
set rulebase security rules CUSTO-INT log-start yes
set rulebase security rules CUSTO-INT disabled no

To review, the following is needed when you setup U-Turn NAT on a Palo Alto Firewall, with the two host devices between in different virtual routers.

  • Nat Policy in place to translate traffic destined for the external IP of the internal device, to the internal IP.
  • Routes on each VR Instance to expose the devices to each other.
  • PBF Rule to send the traffic out the correct interfaces on the return.
  • Security Policies to allow this traffic to pass.

 

Share this content:

Leave a Comment

Your email address will not be published. Required fields are marked *