This page describes how to configure dynamic routing between an end user and two data centres. The scenario involves going through a pair of external firewalls and then a pair of internal firewalls at each location. In practice, I would suggest that merging the external firewalls into the internal firewalls and then making the two firewalls in each data centre an active/passive HA pair will drastically simplify what can be seen as an over complicated design. Still, it is a good learning opportunity for dynamic routing.

We have a scenario where there are two data centres; DC1 and DC2. Data centre 1 has network 10.10.0.0/16 and Data centre 2 has network 10.20.0.0/16. The two data centres have a direct connection to each other and the core switches are members of OSPF area 0.0.0.0 and share all routes.

The end user has a MPLS network that terminates on BGP routers on the edge of the data centres. Each data centre has two BGP routers. Each Data centres has two external firewalls and two internal firewalls. Traffic from the end user is from 10.30.0.0/16.

Traffic from the end user 10.30.0.0/16 to data centre 1 10.10.0.0/16 goes though the BGP routers, external firewall and internal firewall of data centre 1. If this path is broken, the end user can access 10.10.0.0/16 through data centre 2.

Traffic from the end user 10.30.0.0/16 to data centre 1 10.20.0.0/16 goes though the BGP routers, external firewall and internal firewall of data centre 1. If this path is broken, the end user can access 10.20.0.0/16 through data centre 1.

The two BGP routers in data centre 1 have a full mesh relationship with the two external firewalls in data centre 1. The two external firewalls in data centre 1 redistribute the BGP routes into OSPF area 0.0.0.1 which the internal firewalls of data centre 1 are also connected. The two internal firewalls in data centre 1 redistribute these OSPF routes from area 0.0.0.1 to area 0.0.0.0. That same thing happens in data centre 2. Every pair of firewalls exists as an active-primary and an active-secondary. That is to say, we do not use ECMP. There is always a single path for routing.

Further more, the data centre 1 network 10.10.0.0/16 network is redistributed by the data centre1 internal firewalls from area 0.0.0.0 to area 0.0.0.1. This in turn is picked up by the data centre 1 external firewalls and redistributed into BGP. The same happens for data centre 2 and redistributing 10.20.0.0/16 via the data centre 2 internal firewalls to area 0.0.0.2. It is important to note that data centre 1 routes are also shared out through data centre 2 but at a lower priority and vice-versa with data centre 2 through data centre 1.

How do we do this?

* BGP on external firewalls * We establish a eBGP relationship from the primary and secondary external firewalls to the primary and secondary MPLS routers. The MPLS routers share an AS number which is different from the AS number that the external firewalls share. The AS number of the MPLS routers in DC1 is different from the MPLS routers in DC2. The AS number of the external firewalls in DC1 is different from the AS number in DC2. The primary firewall peers to the primary MPLS router with aggregate confed AS Path:enabled, soft reset with stored info: disabled, type:ebgp, import next hop:use peer, export next hop:use self, Remove Private AS:enabled. We also have an import rule for each MPLS router peer with nothing special. YOu can specify in import match filter for address prefix if you want. We rely on the MPLS router adding path prepending from the secondary router. More specifically, both MPLS routers in data centre 1 and both MPLS routers in data centre 2 advertise 10.30.0.0/16 to the external firewalls. However, The primary MPLS to primary firewall is advertised by the MPLS router with AS path prepended once, from primary MPLS to secondary firewall is prepended twice, from the secondary MPLS router to the primary firewall is prepended three times and the secondary MPLS router to the secondary external firewall is prepended four times. This specifies that the internal firewalls will always prefer the primary external firewall unless the primary external firewall looses access to both MPLS routers (of unless the primary external goes offline). Do the same for the external firewalls in data centre 2.

Now we have to configure eBGP export rules for the external firewalls in data centre 1. 1 We have an export rule for the primary external firewall to the primary MPLS router that exports 10.10.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:1”. 2 We have (the same) export rule for the primary external firewall to the secondary MPLS router that exports 10.10.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:1”. 3 We have an export rule for the primary external firewall to the primary MPLS router that exports 10.20.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:2”. 4 We have (the same) export rule for the primary external firewall to the secondary MPLS router that exports 10.20.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:2”. → Notice how the firewalls in DC1 are exporting the routes of DC2 but with “:2” on the append field. This means that the end user will prefer the DC2 firewalls when accessing DC2 unless access through BGP to the external firewalls is lost. We rely on the BGP routers using the :1 or :2 append fields to control how it then relays routing preferences to the end user. → We do the same in DC2 except that they export 10.20.0.0/16 with a :1 and 10.10.0.0/16 with a :2.

* OSPF on external firewalls * We also have to create an OSPF redistribution profile on each external firewall to allow OSPF routes to be redistributed into BGP. Just create the redistribution profile, tick OSPF as the source type. Priority doesn't really matter here as that is for another scenario. Endure you click the “redist” radio button. Under the BGP configuration, add OSPF to the list of redistribution rules and enable it. Set origin can be incomplete and nothing else needs to be specified.

For the external firewalls, we now configure OSPF. The DC1 external firewalls need to be configured as area 0.0.0.1, DC2 firewalls need to be configured as area 0.0.0.2. This is because the internal firewalls summarise routes and that can only happen between different areas. If it were not for the route summaries, we could put both sets of external firewalls into area 0.0.0.0 and then let the MPLS routers summarise routes. While this example doesn't really cover it, imagine that the core switches are exchanging a load of smaller subnets that can be summarised by 10.10.0.0/16 and 10.20.0.0/16.

You will need to create a redistribution profile for BGP routes and one for static routes like we did for OSPF. We then create two export rules on each external firewall: one for BGP and one for Connected (connected is needed if the internal firewalls and dc networks need to know about the network links between the MPLS routers and the external firewalls - may not be the case but you never know - e.g. for ping/traceroute from dc to MPLS router). Set the path type as ext-2 and the metric to 41 (in this example, I picked 41 out of thin air, make your own up). It is important that the metric used is the same on both the primary and the secondary firewalls. What is important here is the area configuration. Under area 0.0.0.1, set the interace to be the external firewall's “internal” interface. In OSPF, set the interface metric to 10 and the priority to 1. This last bit is very important. For the primary external firewall, the priority should be 1. The secondary firewall should have the priority set to 2. It is this priority that will tell the internal firewalls to prefer the external primary and use the external secondary as a failover option. The metric (10) should be the same on both the external primary and the external secondary.

* OSPF on internal firewalls * FOr the OSPF on the internal firewalls, because we are doing OSPF to OSPF, we do not need any redistribution profiles or any export rules. We create two areas, 0.0.0.0 which uses the internal firewalls “internal” interface which touches the dc core network and then we create 0.0.0.1 (or 0.0.0.2) on the internal firewalls “external” interface which touches the external firewalls. For 0.0.0.0, we set the interface metric to 10 and priority to 1 on the primary firewall and metric to 11 and priority to 2 on the secondary firewall. The metric tells the external firewall to prefer the primary firewall and the priority tells the internal core switches to prefer the primary. For 0.0.0.0 we also need to configure two “ranges” under the “Range” tab next to “Interface” tab. We configure the 10.10.0.0/16 range with action set to “advertise” and we set the 10.20.0.0/20 range with the action set to “advertise”. This is how we distribute a summary of the internal networks to the external firewalls in addition to redistributing all the smaller internal networks. This allows, for any given data centre, the external firewalls to export the two /16 networks into BGP as long as the internal firewalls keep advertising the two summary routes. All the other smaller routes that the internal firewall sends to the external firewalls never actually make it into BGP. This is because advertising the two /16 networks is cleaner for the MPLS routers. We could just advertise all the smaller routes (i.e. not summarise them) but this might impact the MPLS routing table (i.e. fill it up). For 0.0.0.1 in DC1 (0.0.0.2 in DC2), we configured the internal firewall's external interface that is in the same subnet at the external firewalls' internal interface and set the metric to 10 and priority to 1. In retrospect, It may be good for the secondary firewalls to have priority set to 2. on the 0.0.0.1 (0.0.0.2) areas. I think we get away with it because the BGP does something to it?“