A couple of years ago I wrote a post about a dual ISP config with a Juniper SRX firewall. At the time I ran into some challenges regarding the DHCP client functionality of the SRX. For some reason it couldn't get a lease from the Ziggo ISP DHCP servers. Any other DHCP server on my local network worked just fine. Since I created a work-around at the time (by using an additional NAT router and static IP addresses) I didn't give it much thought.... Until last week.
Last week I ran into a networking challenge that kinda freaked me out. For some reason my Apple TV wouldn't connect to my NAS, but it could connect to the Internet. For some reason my Apple TV got a public IP address while it was located on my internal network. The public IP address was completely unknown to me. So, WTF was giving my Apple TV a public IP address?
It wasn't until the next day that I discovered what was causing this; A while back I reconfigured a switch. I reassigned some vlan's in the ports, and I must have assigned the VLAN connected to the second ISP to the VLAN where the Apple TV was in. This resulted in a VLAN which was connected to both the (bridged) network of ISP 2 and my internal VLAN. Basically a network where two DHCP services were present. One public, and one private.
Internal network - red lines are trunks serving all applicable VLAN's
Re-configuring the switch-port back to its original state solved this problem, and got me thinking about the original problem a couple of years back. I did have an upgrade of the firewall and its software, so perhaps the problem was solved, and I could get the public IP address of ISP 2 on my firewall....
Unfortunately, it still didn't work, so I started troubleshooting again. As before, every device connected to cable modem got a public IP address, but the DHCPDISCOVER messages of the SRX (sub)interface came up with nothing. So for some reason the DHCPDISCOVER messages of the SRX were ignored by the ISP DHCP server, or they didn't arrive.
First plan was to clone the MAC address of the SRX to a laptop and see if that would work. And that did worl, so there was no blacklisting of MAC addresses going on at the ISP. That meant that the DHCPDISCOVER messages didn't reach the DHCP server.
Some quick Google searches pointed me in the direction of the SRX config. Not the config itself, but the underlying operating system (a lean-and-mean Linux OS). It turns out that the SRX attaches a Time-To-Live (TTL) of 1 to the DHCPDISCOVER packets. And a packet with a TTL of 1 is not gonna make it to the ISP 2 DHCP server. Some devices in its path decrement the TTL counter, and to make sure the DHCPDISCOVER packet reaches the DHCP server the TTL needs to increase at the source. Meaning increase the TTL on the SRX. Fortunately, because of the underlying Linux OS, this is relatively simple. All you need is Shell access on the SRX and execute the following command:
user@srx300> start shell user root Password: % sysctl -w net.inet.ip.mcast_ttl=64
The command sets the TTL of the DHCPDISCOVER packet to 64. This is basically overkill. After some testing on my end, I got a public IP address on the sub-interface with a TTL setting of 4. So it turns out that the components in my network (the Juniper and HP switch) decrease the TTL. Something I wasn't aware of (FYI: my expertise start at layer 3).
To make sure the command sticks (because it goes back to default after a reboot), you need to create a special file that holds the command. This is also done on in the Shell (note that in the following example the TTL is set to 4).
user@srx300> start shell user root Password: % vi /etc/rc.custom #!/sbin/sh sysctl -w net.inet.ip.mcast_ttl=4 % chmod 777 /cf/etc/rc.custom
From this moment on, I can start experimenting with my new Dual ISP configuration. This time with the public IP address on the SRX instead of adding a transit network with NAT.