[c-nsp] best way to network servers with management (iLO/IPMI)

[Nate Carlson]

On Thu, 9 Apr 2009, Drew Weaver wrote:
> Ideally, I would like to be able to assign the management device a RFC 
> 1918 IP, have the actual server be on a different subnet altogether but 
> use a shared port. I know that BMCs generally can use VLAN tagging, but 
> I'm really not sure how I can do all of this with just one port.

Yeah, you can.. enable VLAN trunking on the port, set your native VLAN to 
whatever you want the OS's main interface to be on, allow the VLAN you 
want management on as tagged, and set up your IPMI card to tag on that 
VLAN. Works fine. The OS can still get at the management VLAN, though, if 
you enable VLAN tagging on the OS-level and set up an interface on that 
VLAN.

> The 'easy' way is to simply assign a dedicated NIC to the management 
> device and run another cable, but I'm not sure that is the best way to 
> do it.

..however, it is always my recommendation to use a dedicated NIC if you 
can. I've seen lots of issues with IPMI cards and shared interfaces.. for 
example, with many of the Supermicro motherboard and ipmi combos, when you 
assign an IP to the shared interface under Linux, all the sudden the IPMI 
stops working. Ooops. I think this is fixed in 2.6.27+, but still..

With the dedicated NIC, it's usually a completely isolated PHY, etc, so no 
matter what happens to the system itself, you can still get at IPMI, 
unless you lose power to that interface card.