Blade Server Networking Basics pt. 2

Part 1 of this entry discussed some architectural basics for setting up a blade environment. Part 2 will discuss some of the configuration details required to make the designs in Figure 1 and Figure 2 work.
As with any network build you will first need to figure out how you plan to segment you environment. This will help determine your VLAN and subnet needs. For this discussion we will keep things fairly simple: 3 VLANs.

1. VLAN20 - server access on 10.0.20.0/24 (NAT at upstream firewall)
2. VLAN172 - private network for backups on 172.16.0.0/24
3. VLAN100 - private network for device management on 10.0.100.0/24

To get things rolling you will need to IP and configure the management modules of your chassis. Depending on the vendor, this may require an IP per each module (primary/secondary config) or just a single IP (active/standby config) that will move if the active module fails. Refer the to specific configuration instructions for the vendor's chassis. Once the management access is setup, you can setup the blade switch modules for IP access. Once again, how this is achieved will depend on the device but you will want to use an IP from the same range (10.0.100.0/24 in this case) for the internal connection. Setting things up this way provides network access (ssh or telnet) to configure the switch through the chassis' management port in the event connectivity is lost to the bladeswitch's external ports. Continuing with our hardware from Part 1 we can now begin to configure the switch via the Cisco CLI.

First thing that should be done is to configure an interface for the switch on another VLAN. This setup will provide another point of access to the switch when needed.

interface Vlan20
ip address 10.20.2.20 255.255.255.0
no ip route-cache

The IBM BladeCenter H in this example has 14 slot for blades with servers in the first 4 slots. To start I recommend shutting down all the ports for the unused blade slots. This will prevent you from potentially hosing your network when plugging in new servers.

ciscoswitch# conf term
ciscoswitch(config)# int range gi0/4-14
ciscoswitch(config-if)# shut

Next configure the port channels that will connect to the external 3750 switches. Lets assume we are working with the model illustrated in Figure 1. The main difference for setting up is that 2 x 2 port portchannels would be created rather than a 1 x 4 port portchannel as in this example. The external ports for the Cisco blade switch are numbered 17-20. (Notice ports 15 and 16 are skipped? These ports are for the internal connectivity between the management modules and the blade switches.) The following show a sample config for a port-channel using ports 17-20.

interface Port-channel6
switchport trunk allowed vlan 20,30,172
switchport mode trunk
switchport nonegotiate
link state group 1 upstream
!
...
!
interface GigabitEthernet0/17
description To-3750-Top-25
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,172
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active
spanning-tree link-type point-to-point (Recommended for rapid-PVST+ mode only)
!
interface GigabitEthernet0/18
description To-3750-Top-26
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,172
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active
spanning-tree link-type point-to-point
!
interface GigabitEthernet0/19
description To-3750-Top-27
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,172
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active
spanning-tree link-type point-to-point
!
interface GigabitEthernet0/20
description To-3750-Top-28
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,172
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active
spanning-tree link-type point-to-point
...

A matching config will need to be created on the 3750 (top location, ports 25-28) to complete the port-channel. The 3750 config should not include the link state group 1 upstream statement in the port-channel config though. The reason will be explained shortly.

Now we are ready to configure the ports connecting the servers. For this example we are using a standar IBM HS21 blade that has two NICs, one to each blade switch. I want high availability to these servers but also need to connect to two different subnets. To accomplish this, configure the ports are trunks that have access to both VLANs. On the servers using the NIC vendor's teaming software (Windows) or NIC Bonding and 802.1q Tagging (Linux) to create virtual interfaces built from NIC pairs that pass VLAN tagged traffic.

The following is an example of the blade switch config for ports 1-4.

interface GigabitEthernet0/1
description blade1
switchport trunk allowed vlan 20,172
switchport mode trunk
link state group 1 downstream
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet0/2
description blade2
switchport trunk allowed vlan 20,172
switchport mode trunk
link state group 1 downstream
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet0/3
description blade3
switchport trunk allowed vlan 20,172
switchport mode trunk
link state group 1 downstream
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet0/4
description blade4
switchport trunk allowed vlan 20,172
switchport mode trunk
link state group 1 downstream
spanning-tree bpdufilter enable
!

The most important part of this config is the statement link state group 1 downstream. This statement is paired with the upstream statement in the port-channel config. What this does is monitor the status of the port-channel. If the entire port-channel goes down (i.e. all 4 links are lost, perhaps to a 3750 failure), all of the ports in group 1 down stream will go into shutdown mode. Why would I want to do this? At any one time, the teamed NICs of the servers may be passing traffic across either switch. Which switches and how much traffic depends on if teaming mode is load balancing or failover. Teaming/bonding monitors the NIC health at layer 2. Without the link state changes, the teaming software will not see any change in the upstream connecitivity and continue to pass traffic through this port. This will result in intermittent connectivity (load balancing method) or possibly complete loss of connectivity (failover method) to the server. With the link state groups set up correctly, the server may only experience a few dropped pings at worst.

Blade Server Networking Basics pt. 1