This page describes how to configure dynamic routing between an end user and two data centres. The scenario involves going through a pair of external firewalls and then a pair of internal firewalls at each location. In practice, I would suggest that merging the external firewalls into the internal firewalls and then making the two firewalls in each data centre an active/passive HA pair will drastically simplify what can be seen as an over complicated design. Still, it is a good learning opportunity for dynamic routing.

In this scenario, we have no visibility or control over the MPLS routers and we rely on the MPLS router provider to correctly configure the MPLS routers to accurately import and export routes with respect to various metric, priorities, AS paths, AS path pre-pending, etc.

We have a scenario where there are two data centres; DC1 and DC2. DC1 has a number of /24 networks that can be summarised as 10.10.0.0/16. DC2 has a number of networks that can be summarised at as 10.20.0.0/16. The two data centres have a direct connection to each other and the core switches of each data centre are members of OSPF area 0.0.0.0 and share all routes with each other. Access from DC1 to DC2 and vice-versa is unrestricted.

The end user has offices that access the data centers through a MPLS network that terminates on routers on the edge of the data centres. Each data centre has two routers at the edge. Each data centre has two external firewalls and two internal firewalls. Traffic from the end user is from 10.30.0.0/16.

Traffic from the end user 10.30.0.0/16 to DC1 10.10.0.0/16 goes though the routers to the external firewalls, then to the internal firewalls and then to the DC1 core switches. If this path is broken, the end user can access 10.10.0.0/16 through DC2.

Traffic from the end user 10.30.0.0/16 to DC2 10.20.0.0/16 goes though the routers to the external firewalls, then to the internal firewalls and then to the DC2 core switches. If this path is broken, the end user can access 10.20.0.0/16 through DC1.

The edge routers of DC1 have eBGP peering relationships to the external firewalls of DC1. The external firewalls of DC1 have OSPF peering on area 0.0.0.1 with the internal firewalls. The internal firewalls of DC1 have OSPF peering on area 0.0.0.0 with the internal switches of DC1.

The edge routers of DC1 have eBGP peering relationships to the external firewalls of DC1. The external firewalls of DC1 have OSPF peering on area 0.0.0.1 with the internal firewalls. The internal firewalls of DC1 have OSPF peering on area 0.0.0.0 with the internal switches of DC1.

The reason that areas 0.0.0.1 and 0.0.0.2 exist is purely to allow the internal firewalls to summarise routes to the external firewalls in OSPF. If route summary (covered later) is not needed, then we can use area 0.0.0.0 everywhere.

The two routers in DC1 have a full mesh relationship with the two external firewalls in DC1. The two external firewalls in DC1 redistribute the BGP routes into OSPF area 0.0.0.1 to which the internal firewalls of DC1 are also connected. The two internal firewalls in DC1 redistribute these OSPF routes from area 0.0.0.1 to area 0.0.0.0.

The two routers in DC2 have a full mesh relationship with the two external firewalls in DC2. The two external firewalls in DC2 redistribute the BGP routes into OSPF area 0.0.0.2 to which the internal firewalls of DC2 are also connected. The two internal firewalls in DC2 redistribute these OSPF routes from area 0.0.0.1 to area 0.0.0.0.

Every pair of firewalls exists as an active-primary and an active-secondary. That is to say, we do not use ECMP. There is always a single path for routing.

Further more, the DC1 network 10.10.0.0/16 network is redistributed by the DC1 internal firewalls from area 0.0.0.0 to area 0.0.0.1. This in turn is picked up by the DC1 external firewalls and redistributed into BGP. The same happens for DC2 and redistributing 10.20.0.0/16 via the DC2 internal firewalls to area 0.0.0.2 and out to the DC1 external firewalls. It is important to note that DC1 routes are also shared out through DC2 and vice-versa. When the DC1 external firewalls export DC2 networks into BGP they do so in such a way that the MPLS network will prefer the DC2 networks via DC2. When the DC2 external firewalls export DC1 networks into BGP they do so in such a way that the MPLS network will prefer the DC1 networks via DC1.

We establish a eBGP relationship from the primary and secondary external firewalls to the primary and secondary MPLS routers. The MPLS routers share an AS number which is different from the AS number that the external firewalls share. The AS number of the MPLS routers in DC1 is different from the MPLS routers in DC2. The AS number of the external firewalls in DC1 is different from the AS number in DC2.

Each external firewall peers to both of the MPLS routers with the following settings in the Peer Group (one connection per peer group - i.g each external firewall has two peer groups with one peering in each.).

• Aggregated Confed AS Path enabled
• Soft Reset With Stored Info disabled
• Type EBGP
• Import Next Hop Use Peer
• Export Next Hop Use Self
• Remove Private AS enabled

Each external firewall needs to have two import rules - one for each MPLS router. It may be possible to merge them into a single import rules that is used by both peering groups. I specified an import filter of 0.0.0.0/0 with “explicit” NOT enabled so we import everything from the MPLS routers. Note that this requires us to put a lot of trust in the administrators of the MPLS routers. You may want to add filtering to prevent the MPLS administrators from being allowed to advertise any old route into the data centre networks.

We rely on the MPLS router adding path pre-pending from the secondary router in each data centre. More specifically, both MPLS routers in DC1 and both MPLS routers in DC2 advertise 10.30.0.0/16 to the external firewalls. However, The primary MPLS (in both data centres) to primary firewall (in both data centres) is advertised by the MPLS router with AS path pre-pended once, from primary MPLS to secondary firewall is pre-pended twice, from the secondary MPLS router to the primary firewall is pre-pended three times and the secondary MPLS router to the secondary external firewall is pre-pended four times. This specifies that the internal firewalls will always prefer the primary external firewall unless the primary external firewall looses access to both MPLS routers (of unless the primary external goes offline). Do the same for the external firewalls in DC2.

This also means that the internal network in DC1 will route to the internal firewalls in DC1 for 10.30.0.0/16 unless the route through DC1 is broken. Then they will route over to the internal network in DC2 and access 10.30.0.0/16 through the DC2 firewalls.

This also means that the internal network in DC2 will route to the internal firewalls in DC2 for 10.30.0.0/16 unless the route through DC2 is broken. Then they will route over to the internal network in DC1 and access 10.30.0.0/16 through the DC1 firewalls.

Now we have to configure eBGP export rules for the external firewalls in DC1. 1 We have an export rule for the primary external firewall to the primary MPLS router that exports 10.10.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:1”. 2 We have (the same) export rule for the primary external firewall to the secondary MPLS router that exports 10.10.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:1”. 3 We have an export rule for the primary external firewall to the primary MPLS router that exports 10.20.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:2”. 4 We have (the same) export rule for the primary external firewall to the secondary MPLS router that exports 10.20.0.0/16 exactly (match criteria). Under the action tab, we set the community type to “Append” and Append field to “<dc1_external_fw_as>:2”. → Notice how the firewalls in DC1 are exporting the routes of DC2 but with “:2” on the append field. This means that the end user will prefer the DC2 firewalls when accessing DC2 unless access through BGP to the external firewalls is lost. We rely on the BGP routers using the :1 or :2 append fields to control how it then relays routing preferences to the end user. → We do the same in DC2 except that they export 10.20.0.0/16 with a :1 and 10.10.0.0/16 with a :2.

We also have to create an OSPF redistribution profile on each external firewall to allow OSPF routes to be redistributed into BGP. Just create the redistribution profile, tick OSPF as the source type. Priority doesn't really matter here as that is for another scenario. Endure you click the “redist” radio button. Under the BGP configuration, add OSPF to the list of redistribution rules and enable it. Set origin can be incomplete and nothing else needs to be specified.

For the external firewalls, we now configure OSPF. The DC1 external firewalls need to be configured as area 0.0.0.1, DC2 firewalls need to be configured as area 0.0.0.2. This is because the internal firewalls summarise routes and that can only happen between different areas. If it were not for the route summaries, we could put both sets of external firewalls into area 0.0.0.0 and then let the MPLS routers summarise routes. While this example doesn't really cover it, imagine that the core switches are exchanging a load of smaller subnets that can be summarised by 10.10.0.0/16 and 10.20.0.0/16.

You will need to create a redistribution profile for BGP routes and one for static routes like we did for OSPF. We then create two export rules on each external firewall: one for BGP and one for Connected (connected is needed if the internal firewalls and dc networks need to know about the network links between the MPLS routers and the external firewalls - may not be the case but you never know - e.g. for ping/traceroute from dc to MPLS router). Set the path type as ext-2 and the metric to 41 (in this example, I picked 41 out of thin air, make your own up). It is important that the metric used is the same on both the primary and the secondary firewalls. What is important here is the area configuration. Under area 0.0.0.1, set the interace to be the external firewall's “internal” interface. In OSPF, set the interface metric to 10 and the priority to 1. This last bit is very important. For the primary external firewall, the priority should be 1. The secondary firewall should have the priority set to 2. It is this priority that will tell the internal firewalls to prefer the external primary and use the external secondary as a failover option. The metric (10) should be the same on both the external primary and the external secondary.

FOr the OSPF on the internal firewalls, because we are doing OSPF to OSPF, we do not need any redistribution profiles or any export rules. We create two areas, 0.0.0.0 which uses the internal firewalls “internal” interface which touches the dc core network and then we create 0.0.0.1 (or 0.0.0.2) on the internal firewalls “external” interface which touches the external firewalls. For 0.0.0.0, we set the interface metric to 10 and priority to 1 on the primary firewall and metric to 11 and priority to 2 on the secondary firewall. The metric tells the external firewall to prefer the primary firewall and the priority tells the internal core switches to prefer the primary. For 0.0.0.0 we also need to configure two “ranges” under the “Range” tab next to “Interface” tab. We configure the 10.10.0.0/16 range with action set to “advertise” and we set the 10.20.0.0/20 range with the action set to “advertise”. This is how we distribute a summary of the internal networks to the external firewalls in addition to redistributing all the smaller internal networks. This allows, for any given data centre, the external firewalls to export the two /16 networks into BGP as long as the internal firewalls keep advertising the two summary routes. All the other smaller routes that the internal firewall sends to the external firewalls never actually make it into BGP. This is because advertising the two /16 networks is cleaner for the MPLS routers. We could just advertise all the smaller routes (i.e. not summarise them) but this might impact the MPLS routing table (i.e. fill it up). For 0.0.0.1 in DC1 (0.0.0.2 in DC2), we configured the internal firewall's external interface that is in the same subnet at the external firewalls' internal interface and set the metric to 10 and priority to 1. In retrospect, It may be good for the secondary firewalls to have priority set to 2. on the 0.0.0.1 (0.0.0.2) areas. I think we get away with it because the BGP does something to it?“