3. NSX-T Multi-Tier Routing Architecture
And let’s start by talking about the use cases. So, in multi-tier architecture, you’re going to add some complexity here. And so we have to have a good reason to do that, right? We want to keep things as simple as we possibly can, but we can make them more complex if we need to. But ideally, we’re going to keep things as simple as we possibly can because Nsxt is complex enough. So the primary purpose of multi-tier routing is to create multi-tenant support. We can have different routing domains for different tenants within our environment. We want to provide logical separation between a provider router and a tenant router. So the tier-zero router is essentially the top tier; it’s kind of like the provider router. And it can be difficult to keep these straight at times because the names aren’t all that descriptive; however, with edge or distributed routers, it’s very easy to keep track of where they reside.
Tier Zero is the provider router. Tier Zero requires an edge cluster. So anytime you have to enable services that run centrally, like, for example, north-south routing or the edge firewall or NAT or DHCP, those services are going to run on a services router, and the services router is going to run in your edge cluster. Now just bear in mind that the edge cluster is not a cluster of ESXi hosts. And if you’re familiar with Nsxv, that’s what the edge cluster was. It was a cluster of ESXi hosts. It is a cluster of either virtual machines running on top of ESXi or bare metal nodes in Nsxt. Tier Zero is the provider router; that’s the top tier, and the bottom tier is the tier one gateway. Those are our tenants’ routers. And we can allow different tenants to have complete control of their Tier 1 gateway routing, and then we can maintain control as the provider at Tier Zero. Tier-one gateways can be used to allow tenants to configure their own routing, but we have control over how those tenants communicate with one another, the Internet, or external resources in my data center.
We control all of that at the provider level in the Tier 0 gateway. And multi-tier routing is definitely not mandatory, but in many cases it’s recommended. So if you’re going to have multiple tenants, multitier routing is a good solution to utilize. So let’s switch over to the NsXT reference design guide, and we’re going to look at a diagram in that guide called Two Tier Routing and Scope of Provisioning. Here, you can see we have multiple tenants. And so in this scenario, we have one tenant with two different segments. East-west routing is happening between those segments through the Tier 1 distributed router. So I’ve got one tenant with web segments and app segments, and they are able to set up all of their own routing between those segments. Here’s another tenant with their own Tier 1 distributed router. These are separate routers. They’re not distributed router instances on different hypervisors. These are completely different router instances. So this is a logical diagram. There may be 50 different transport nodes in the physical diagram. But from a logical perspective, we’ve got two different tier-one routers. Those are the tenant routers, and the Tier Zero router is the provider router. So the only way that we can get from one tenant’s network to another is through that provider-controlled Tier Zero gateway. And at the Tier Zero gateway, we do things like set up firewall rules. We can control our route tables. Southbound, the Tier Zero gateway can connect to one or more Tier 1 gateways.
It can even connect directly to logical segments. And then, from tier zero and northbound, we’re connecting up to a physical router. So that’s the physical router in our datacenter, and the key characteristic of the TierZero gateway is essentially that it’s able to support northbound connectivity to our external networks. Tier Zero also supports static routing. Furthermore, BGP, Tier One gateways do not support this. They don’t support static routing. In BGP, the Tier 1 gateway can only connect to a Tier 0 gateway northbound. So remember, in the single-tier example that we went through in the last video, we were using a Tier-Zero gateway for everything. In this case, we’ve introduced a tier of routers beneath that Tier Zero gateway. And these Tier 1 routers cannot connect to a physical router northbound. They can only connect to a Tier Zero gateway going north. And then we may have a Tier 1 service router as well. There are certain services, like NAT or load balancing, that are configured on a tier-one service router. So let’s take a quick look at another diagram in the Nsxt reference design guide. And in this case, we see we’ve got our northbound interface from the Tierzero gateway to the external physical router. So the Tier Zero gateway is our northbound connection. So here we see a router link interface that is our inter-tier transit network.
and that connection is automatically created. So this is just a simple interface connecting Tier Zero and Tier One gateways. And there’s an automatically created subnet. It starts with 1064, which is automatically created when you establish the Tier Zero and Tier One gateways. Now, here we see two service interfaces. These are interfaces connected to VLAN segments. You’ll notice that we have VLAN segment one here and VLAN segment two. That’s what the service interfaces are connecting to. And so these are used to connect physical or virtual VLAN workloads. And in certain use cases, they can also be used to connect to a load balancer. And the service interfaces are supported on both Tier 0 and Tier 1 gateways. And in order to configure a service interface, you have to have the service router component deployed. So it’s a centralised interface that actually exists on the service router. And then finally, as you can see here, the Tier Zero gateway and the Tier One gateway can both be connected to overlay segments. And if you watched the last video, in our single-tier architecture, the Tier One gateway was the only router in our environment, and it was connected to all of the overlay segments. So what I’m looking at in this scenario is maybe the Tier Zero gateway. The service provider has an overlay segment on which virtual machines are deployed, and we could potentially have multiple tenants, each with their own Tier 1 gateways and overlay segments. Let’s take a look at another diagram from the Nsxt reference design guide.
This one is on page 55, and this document is going to walk you through how the routes are actually propagated between Tier One and Tier Zero gateways and between the physical routers in our data center. So we start by kind of focusing down here at the Tier One level. And Tier One has routes for its own locally connected segments, right? So here you can see we’ve created an overlay segment, a layer-two segment for 170, 216, and 100. We’ve also got a VLAN segment and a service interface that we’ve created on that Tier One gateway. So those two local, directly connected networks are going to be advertised to Tier Zero so that my provider’s router is also aware of where those routes exist. And there is a default route that’s automatically created on the Tier One router so that it has this inter-tier connection. And if it needs to send traffic northbound, the traffic is flowing using that default route up to that Tier Zero gateway.
So because of that, there’s no need for things like BGP or any kind of routing protocol in the Tier1 gateway; it routes between its locally connected segments. And if it’s not a locally connected segment, it just sends it up to Tier Zero. Now, at Tier Zero, it’s going to be 170, 216, ten, and 192, 160, and 810. They are seen as Tier I-connected routes. So if it’s got traffic for those networks, it’s going to send it to this 1064, 2240 address, which is the address of the Tier One gateway. We’ve also got some routes that are connected to the Tier Zero gateway itself. It’s got some local interfaces as well. It’s got one local interface connected to an overlay segment, which is 170, 216, and 200. And it’s got an interface connected to this physical network. 192, 168, 240. Those are directly connected networks that are automatically present in the routing table. And then we’ve got our physical router. So we could be doing equal-cost multipathing here, or we could have an active standby configuration. But regardless, the Tier Zero gateway will redistribute Tier Zero and Tier One routes to BGP. So what that essentially means, if you’re not familiar with route redistribution, is that the Tier Zero gateway is essentially going to do this: learn routes from a source that is not BGP. If we look at this diagram here and we’ve got some of the logical networks, we’ve got 192, 168, 100, 170, 216, 100, 170, 216, 20, all of these logical networks that the tier-zero gateway is learning about, it’s not learning about them from BGP. And that’s what route redistribution is. It’s essentially saying, “Hey, we’re feeding these routes into the Tier 0 gateway.” It’s learning about them through some protocol that is not BGP. It’s going to take those routes and redistribute them into the BGP, right?
That’s what we mean by “route redistribution,” which is taking routes that are learned one way and injecting them into a dynamic routing protocol such as BGP. So essentially, what the Tier Zero Gateway is going to do is send an advertisement to the physical router saying, “Hey, by the way, physical router, if you ever need to get to this network, this network, or this network, you can reach them through me.” And that way, the physical router learns about these networks that are behind the Tier 0 and Tier 1 gateways and can send traffic to them. So again, that’s on page 55 of the Nsxt reference design guide if you want to check that out. And again, those page numbers may change, and new versions of the reference design guide may come out, but you should be able to find a similar diagram in any version of that bers may changSo now let’s start to work through a couple of diagrams that I’ve created here. And we’re going to take some packet walks through the two chair routes within the same host. So here you can see we’re focused on a single ESXi host, ESXi zero one. And within ESXi 0.1, I’ve got two segments. VNI 5001 is a segment that is owned by tenant 1, and tenant 1 has their own tier 1 distributed router. Tenant 2 has a segment that they’ve created.
We’re going to call it VM Three on that segment, and they have their own distributed router as well, their own tier one. And then down here we can see the physical underlay network, even though that’s really not going to come into play in this particular scenario. So VM One sends a packet to VM Three. Here comes the packet out of VM One. It’s going to hit the Tier One Distributed Router because that’s the default gateway for VM One, and the Tier One Distributed Router is going to determine that. Hey, I’ve got this packet, and it just hit me. As the default gateway, the Tier One Distributed Router automatically has a default route that sends traffic to the Tier Zero Distributed Router. So here comes this traffic out of the Tier 1 DR, and it’s going to flow into the Tier 0 distributed router. Both of these are now distributed router components.
They are running locally on ESXi 1. So I haven’t had to hit any kind of service router at this point. So next, the tier-zero distributed router learned about routes from the tier-one distributed router. As I create these logical networks, these layer-2 segments on my tier one routers, they’re letting the tier zero router know. So the Tier Zero router knows about this 192.168.0.1 network. It knows because the tier-one router told it. So now the tier-zero router has an entry in the routing table saying any traffic destined for 192.168.dot, zero.dot, anything, is going to get sent to this tier-one distributed router for tenant two. And the tier one router for tenant two has a logical interface on VNI 5002. And so outgoes the packet andit hits the destination virtual machine. So again, real quick, let’s talk about those steps. VM One sends out a packet that’s destined for VM Three. It hits the default gateway, which is the tier one Dr. Tier one DR has a default route pointing to tier zero DR, which is automatically created. The tier-zero doctor knows about that destination network. It learned about it from tenant two’s tier one doctor. And so all of this routing process occurs right within the same ESXi host. Let’s change up our diagram a little bit here. So now we’ve got our Tier 0 distributed router, which is running on multiple transport nodes and multiple ESXi hosts. We’ve got tenant one with their own tier-one distributed router and their own layer-two segment. We’ve got tenant two with their distributed router and their layer 2 segment. So nothing has really changed from the previous diagram. The only thing that’s changed is that now these virtual machines are distributed across multiple physical hosts. As a result, VM-1 will send another packet to VM-3. The packet will be sent to the default gateway because that’s on a different network. And so VM One is always going to send anything to a different network than its default gateway. And the Tier 1 distributed router is going to use its default route. It’s going to say, “
Okay, this isn’t on any of the segments connected to this one.” That’s what the Tier 1 distributed router is going to do. It may have 20 other layer-two segments connected to it. This traffic isn’t bound for anything connected to this one specific tier-one distributed router. So it’s going to just use its default route and send this packet up to the Tier Zero distributed router. And again, the tier-zero distributed router has learned about that 192-1680 network from the tenant to distributed router. And so it’s going to route that packet to that tenant’s distributed router. And the distributed router’s tenant will dump it onto the locally connected segment. So at this point, the routing process is complete. All of the routing occurred on the source host, but the destination VM is on a different host. The destination VM is on the same host too.So that begins the next phase of our process. Here. The routing is done. At this point, the traffic is on the correct layer to segment, but it has to get encapsulated. So depending on the Mac address of VM3, the Mac table will be used. The tap will determine, okay, to get to that Macaddress, I need to send it to this tap. So the ESXi one tap will encapsulate it, forward it over the physical network towards the destination tap, and the destination tap will decapitate it. It’ll pull off that outer header, and once it’s decapsulated, it will be sent to the appropriate VNI. And at that point, it can be delivered to the destination virtual machine.
So two-tier routing It is still doing all of the east-west routing on the source host. All of the east-west routing is still happening using the kernel modules on the source host. All right, let’s look at one more example here. But now we’re getting out of the east-west thought process. We’re actually going to do north-south. So we’ve got some traffic that is bound for a machine that is on an external network, right? So we’re not doing east-west anymore. Now we’re doing north-south. So let’s say that VM 1 here wants to send some traffic outbound to this external machine that you see here on the right. Let’s think of it as an amorphous physical server in our data center. So when VM One sends out that packet, it’s going to say, “Okay, this is something on a different network.” Let me send it to my default gateway. The default gateway is the Tier One Distributed Router for this particular tenant. And the Tier 1 distributed router is going to say, “Okay, this is not a network that I am connected to.” I don’t have a connection to this network. So there’s no connection in that Tier1 distributed router for that particular network. So now the Tier One Distributed Router sends that to its default route, and the default route is the Tier Zero Distributed Router on that same host.
Every host has a Tier-0 doctor. And the Tier Zero Distributed Router is going to check its route table and determine that this is not a network that I’m directly connected to. This is not a network that I’ve learned about from any of the Tier One DRS, right? The Tier Zero Distributed Router may be connected to a bunch of different tenants, but it’s never learned that specific route. So it says that this is a route that I don’t know about. Let me just send it to my default route. And so the default route for the tier-zero distributed router is the tier-zero service router. And if we back up here a little bit, you’ll notice there’s a transit segment. This is called an intra-tier transit segment. So within Tier 0, there’s a segment that connects the Tier 0 distributed routers to the Tier 0 services router. So off it goes to the Tier Zero Services router and the Tier Zero Services router. Maybe it knows about that network. Perhaps it discovered it through BGP. Or, who knows? Maybe the Tier Zero Services router is actually going to use its default route. One way or another, that Tier Zero Services router is going to route that traffic out to the physical router in our data centre so that it can reach the external machine that is the destination.
4. Demo – Configure East West Routing
In this video, I’ll show you how to configure East-West routing in an Nsxt-30 environment, and we’ll use the free labs available at Hol vmware.com to do so. Now, before I get started, I just want to mention that we’re going to be using a multi-tier routing architecture in this lesson. And there are certain parts of that architecture that are already built for you in the hands-on lab environment. And one of the parts that has already been created is a Tier 0 gateway. And the tier-zero gateway is connected to some upstream router that represents the northbound interface to the external network. And we also have a number of segments that are currently connected to this Tier 0 gateway. So what we’re going to do is actually reconnect those segments to a tier-one gateway because we’re setting up a multi-tier routing topology. So let’s just take a moment to explore what we already have configured here. So I’m logged into the NSX user interface, and I’m just going to click on Tier Zero Gateway. And you can see here that we’ve got a Tier 0 gateway that’s already been created. Now remember, the tier-zero gateway is required for many services, and it’s also required for north-south routing.
So if I want to be able to reach any networks outside of my NSS domain, I have to have a Tier 0 gateway. And at the moment, if I click on linked segments, I can see that the tier-zero gateway has four segments that are already connected to it. So what we have right now is a single-tier routing environment. My segments are not connected to a Tier 1 distributed router; they’re connected to Tier 0. And if we’re not doing multitenancy, a tier-zero routing topology can work just fine for us, and it can be simpler and easier to maintain. But if we do want to have multiple tenants, then having a tier-one gateway will become really important. And we’ll kind of start to think of our tier-zero gateway as the provider gateway. And all of our tenants will each have their own Tier 1 gateway. Those tenants could be different parts of our organization. They could be test, development, and production environments. So regardless of the use case, if we’re going to have multiple tenants, a multi-tier routing architecture starts to make sense. So let’s go ahead and set this up. I’m going to click on Tier 1 gateways, and I’m going to add a new Tier 1 gateway. And so I’m just going to name my new tier-one gateway “Tier One Rick,” and I have to associate a tier-one gateway with a tier-zero gateway. So if I’m doing single-tier routing, I don’t have a tier-one gateway. Tier One is only used with multi-tier, and it always has to be associated with an upstream tier-zero gateway. If we want the ability to do north-south traffic, Now here, I can choose my edge cluster.
And the edge cluster is going to be a requirement if we intend on running services at Tier 1. So for services like, for example, network address translation, we’re going to need a services router. That service router is going to run on the edge cluster, and then I can choose whether or not I want that service router failover to be preemptive or non-preemptive. So on the edge nodes, one of those service routers is going to be active. The other one is going to be on standby. In a non-preemptive model, if the active fails, the standby takes over, and the standby becomes the active. If the original active edge node comes back, it doesn’t matter. It still sticks with whichever is currently active, whereas the preemptive one of them is always kind of in charge. And even if it fails, if it comes back, it takes over again. Now, in this scenario, I don’t need to enable any services, so I’m not going to bother with the edge cluster here. I don’t need this for north-south routing. I’ve got my Tier Zero gateway for that. So I’m just going to skip the edge cluster and click on Save. So now it asks me if I want to continue configuring this Tier 1 gateway. I’m just going to go ahead and click “yes” there. And as I move on through this demo, I’m going to be taking some of the segments that currently exist and moving them over to this Tier 1 gateway. So as I move on with this demonstration, I’m going to be taking some of the segments that exist in this hands-on lab environment and moving them over to this Tier 1 gateway. Currently, they’re connected to the Tier Zero gateway. I’m going to move them to Tier One.
So what I need to do is make sure that any segments that are connected to this Tier 1 gateway are being advertised. I want the Tier 1 gateway to tell the Tier 0 gateway, “Hey, this network is now available through me.” It’s directly connected to the Tier 1 router. And bear in mind that the Tier 1 gateway does not run BGP. The Tier 0 gateway runs BGP. The tier one gateway basically has a proprietary routing mechanism where it communicates directly with tier zero. So if I go to Route Advertisement and tell the tier one gateway to advertise all connected segments and service ports, whenever I connect a segment to this tier one router, it will inform the tier zero router that this segment is now connected to me. I can also inform the tier-zero router of any static routes that are configured on this tier-one gateway. or if I configure NAT. I’ll have to enable these NAT-IPS. We’ll talk more about that later. And once the Tier Zero gateway starts learning about these segments from Tier One, The Tier Zero gateway can then redistribute those routes and advertise them into BGP so that the routers in the physical environment can learn about these segments that are connected to my Tier One gateway. So I’m going to go ahead and allow the advertisement of all connected segments and service ports. That’s all I need to enable this. in this situation. I’m not going to set up any static routes or anything like that, and I’ll go ahead and click Save. And so my configuration is now saved. I’m going to close editing, and let’s take a little look at the Tier One gateway that I’ve created now.
And, yeah, here it is. And it currently has zero linked segments. So let’s move over to our segments here and under segments. And let’s start with LS Web. I’m going to click on the ellipsis here. Next to LS Web, I’m going to choose edit, then connectivity. It’s currently connected to my Tier Zero gateway. I’m just going to go ahead and change that to the new Tier One gateway that I’ve created. And, remember, associated with each of these segments is a subnet. So when I change the connectivity for this segment now, my Tier One gateway is going to have a directly connected interface on this network, on the 170 216 10 network. This interface is going to be created on my Tier 1 distributed router. And so I’ll go ahead and close editing for the Web segment, and I’m just going to go ahead and complete the same process for the LSDB segment. I’m going to edit the segment. I’m going to move it over to the Tier 1 gateway. I’ll save that change, and that’s the only thing I’m changing on any of these segments: I’m moving from Tier Zero to Tier One, and then I’ll do the same thing on my LS App segment. So now I have successfully migrated all three of these segments—the Web, App, and DB segments—over to the new Tier One gateway that I just created. So basically, what I’ve now done is create a Tier One router and connect it directly to these three segments. Now, I might have other segments for other tenants. I could create another Tier One router, and I could connect other segments to them, but they all can connect upstream to the same Tier Zero gateway, which will provide routing between all of those tenants. So let’s experiment a little bit. Let’s run a couple of tests and see how this has impacted our environment. And I’m just going to launch a command prompt at the home screen of this console here. You can assume that this console basically represents a computer that is outside of my NSF domain. And what we just did was modify the Web, App, and DB segments. So what I’m going to do is see if, from this external workstation, I can still ping the interfaces of the default gateways for all of those networks and assume that, if I can successfully ping that, that traffic is flowing from the outside world through the tier zero gateway and over to the tier one gateway.
So if I can hit this, that means that the tier zero gateway has successfully received route updates from the tier one and that the tier zero gateway has redistributed those routes into BGP so that these external machines could learn about them. And look at that. There we go. It works for numbers ten and one. Let’s try 21. That’s working as well. Let’s try 31. That’s working as well. So for the app, web, and DB segments, I can hit the default gateway for any of those segments from an external machine. So now that we’ve validated that the default gateways are up for each of these segments, let’s go to Putty and connect to one of the virtual machines. So using Putty, I’m going to connect to the Web machine, a virtual machine. And this virtual machine is connected to the LS Web segment that I just moved over to my Tier 1 router. So let’s start by running the ifconfig command to see the interfaces of this machine. and here you can see the IP address. It’s 170, 216, and 1011. And so let’s try to ping the default gateway from this virtual machine. 170, 216, and one hundred and one. And look at that. Those pings are coming back successfully. So that’s now hitting an interface on the Tier 1 distributed router. So can this virtual machine ping other segments that are also connected to that Tier 1 distributed router? Let’s try to hit one of the other addresses on that router: 170, 216, or 21. and we can hit that segment as well. And that just automatically works because my Tier 1 distributed router knows about all of the directly connected networks, so it knows about the 170, 216, 1020, and 30 networks. Let’s try to ping a virtual machine. Now I’m just going to clear my screen.
I’ll just type “ping app 1,” a corp local. And there we go. My pings are successful to one of the servers that is connected to the application logical switch. So it looks like my east-west routing between these segments is working successfully. And while we’re in here, we may as well also just verify that we’re able toping systems on the DB segment as well. So let’s give that a shot. That is also working. So, in summary, here’s what we’ve done. We’ve got a Tier 1 router that’s doing all of the east-west routing. It connects the three segments to which I’ve connected. It is within my NST domain. If I need to send traffic somewhere else, like, for example, if I’m trying to reach some destination on the Internet, that’s when north-south routing is going to start to come into the equation, and we’re going to have to use that tier-0 gateway. But we’re not quite there yet. So this is a relatively simple configuration at the moment. I’ve got my Tier One gateway, and that Tier One gateway has multiple segments that are connected to it: the app, web, and DB segments. And I am now able to do east-west routing between those segments. And really, most of the configuration is actually done within the segment. So, for example, let’s say I want to create a new segment. So I have the LS app, the LSDB, and the LS web.
I’m going to create a new segment. I’m going to call it LS Rick, and I’m going to connect it to my Tier One gateway. And then I’m going to choose my transport zone. I’m going to choose my overlay transport zone, and I’m going to establish a subnet. So let’s do 170, 216, 41, and 24. And so when I configure this subnet on the segment, I’m setting up an IP address that’s going to be used on the logical interface of that Tier One gateway. So, if I go ahead and click on Savehere, a new interface with the IP addresses 170, 216, and 41 will be created at the TierOne gateway. And just to prove that that’s actually working the way that I said it was, let’s go back into this virtual machine one more time. You saw me run some commands toping 170, 216, 10, 121, and 31. Let’s try 41, the address that I just assigned to that new segment. And look at that. Those pings are coming back successfully. So in reality, for East-West routing, the bulk of the configuration is actually done on the segments themselves. What I’m doing at the Tier One router is essentially deciding: do I want to redistribute certain networks up to the Tier Zero gateway? And that’s what we’ll look at in the next video. How to set up the Tier Zero gateway
