Out-of-Band Management for Public Utilities Applications

Challenge:

In any large, enterprise network that includes remote equipment sites, a carefully designed out-of-band management solution can help to minimize downtime and ensure that vital services and capabilities are available when they’re needed. The reason for this is simple.

An effective out-of-band management solution allows support personnel to deal with many issues at remote equipment sites without the need to send a tech team to the site to deal with the problem in person.

A well-planned out-of-band management solution can eliminate the time delays, expenses and hassles of long truck rides or helicopter flights to deal with problems at distant sites. Instead of sending a small fleet of service vehicles up a treacherous mountain road and facing washed out roadways and inclement weather, support personnel can deal with equipment outages safely, quickly and remotely, without even leaving the NOC.

The task of implementing an out-of-band management solution in a large enterprise network often involves a multitude of different considerations that might surprise even the most experienced network technician. The larger the size and scope of the enterprise network involved, the more likely it is that the most effective out-of-band management solution possible will need to address dozens of different variables, challenges and unique problems. This is especially true in networks found in public utilities, oil and gas and communications applications.

Large enterprise networks usually include an assortment of diverse types of remote equipment sites with each type of site presenting its own unique set of challenges. For example, the network might include both AC and DC powered devices, sites with varying communication needs, sites that are easily accessed, sites that are almost impossible to access and sites of all different configurations and sizes. With this in mind, it quickly becomes apparent that most importantly, an effective enterprise-wide out-of-band management solution must offer scalability, adaptability and the option to customize out-of-band solutions to meet unique challenges.

Although the specific problems faced for each deployment will differ from case to case, the most common requirements in large, enterprise level, out-of-band management installations are as follows:

• An Enterprise Management Solution that enables support personnel to quickly locate and control specific console servers and switched PDUs within network infrastructure
• A scalable solution that offers a range of products to fit the needs of different sized equipment sites
• Support for both AC and DC powered applications
• A domestic/local vendor who can respond quickly to support issues
• A strong warranty that provides at least five years of coverage

The most common, (and perhaps the most important,) capability required in enterprise level out-of-band management implementations, is the ability to control and configure a large number of power control and console access devices from a centralized command interface. When the out-of-band support solution includes dozens or hundreds of switched PDUs and console servers spread across a wide geographic area, an effective management solution is needed in order to simplify the task of quickly locating and controlling specific switched outlets and console ports during service outages. While many popular enterprise management programs do a great job of cataloging devices and collecting data and alerts, they typically do not provide a means to directly access remote console ports or control power reboot/switching functions.

In order to simplify the process of accessing power reboot and console port functions quickly, it’s best if both power control and console access products can be controlled by the same enterprise management solution. This means that ideally, both the switched PDU products and console server products in a large out-of-band solution should be supplied by the same manufacturer in order to assure that a single enterprise management solution will be able to manage both power control and console access functions.

Another factor to consider when evaluating an enterprise management solution, is the ability to communicate with compatible PDUs and console servers on LANs located behind a cellular router. In order to allow efficient management of remote devices via cellular, it’s imperative that the management solution must be able to recognize power reboot units and console servers via a cellular connection. Many enterprise management solutions lack the ability to communicate with devices at port numbers on a network segment served by a cellular router. In this case, the enterprise management solution cannot be used in applications that require communication via cellular broadband.

Another important factor to consider when planning an enterprise level out-of-band management solution is scalability. In large networks, an effective out-of-band management must cover the needs of a wide range of different types of remote network equipment sites of varying sizes and functions. Larger equipment sites might require power switching and reboot capabilities for dozens of managed devices, while smaller equipment sites might only need console access and power reboot switching for five or six.

This means that in order to serve the needs of both larger equipment sites and smaller equipment sites, an effective out-of-band management solution should include switched PDUs and console servers of various sizes. Larger sites might need power reboot switching and console access for 20 or more devices, while smaller sites might only need power switching and console access for seven or eight devices.

As mentioned previously, out-of-band management in enterprise network applications is much easier when both power reboot and console access devices originate from the same manufacturer. With this in mind, it follows that ideally, the various sizes of switched PDUs and console servers present in the out-of-band management solution should also originate from the same manufacturer in order to ensure a uniform command interface for all managed sites and to eliminate the need for operators to be familiar with the control and configuration options for several different product lines.

In addition to network access, an effective out-of-band management solution should also support several different alternate communication options. Depending on the nature of your application, the various remote sites in your network infrastructure can present an assortment of unique challenges due to the methods of communication available at each site. While accessible sites such as branch offices and data centers generally have ready access to a network cable, remote sites such as monitoring stations and microwave antenna sites might not provide easy network access. This forces you to rely on other avenues for out-of-band communication such as cellular broadband, satellite or dial-up. Console servers and switched PDUs in an enterprise level out-of-band management solution should support communication via network, plus one or two other options such as cellular broadband, satellite or dial-up.

Another variable that should be considered when implementing an out-of-band management solution is the type of power available at each remote equipment site. Some remote sites may rely on AC power, others might use DC power and other sites may also require high amp power due to the nature of the devices present at the site. This is another case where it’s wise to choose a manufacturer that offers both AC and DC powered console and power control products as well as switched PDUs designed for high amp power. If you can find one manufacturer with a product range that can fill the power requirements of all of your remote equipment sites, this drastically simplifies out-of-band management by providing a uniform control interface for your entire out-of-band management infrastructure.

A common challenge faced when creating an effective out-of-band management solution, is the need to deal with completely unique situations at remote equipment sites that might not be solvable using only off-the-shelf switched PDUs, console servers and out-of-band management solutions. Each application can present unusual obstacles that a given manufacturer might not have anticipated or planned for. In order to deal with these unforeseen issues, it’s important to choose a manufacturer who can provide customized solutions to deal with problems that are unique to your particular industry or application.

Given the often mission critical nature of the capabilities provided by remote equipment sites in large enterprise networks, it’s important to choose power reboot and console access products that are covered by an extensive, long-term warranty. In addition to providing evidence that the manufacturer is confident in the quality of their product, a strong warranty ensures that in the event of product failure, your switched PDU or console server can be quickly repaired and returned to service. In order to minimize downtime when one of these devices malfunctions, it’s also helpful if the manufacturer’s warranty includes an arrangement to provide replacement/loaner units.

The physical location of the manufacturer is another important factor to consider in minimizing downtime. If you choose a manufacturer located overseas or far away from your facility, this can increase the time and expense required to ship units for repair. In cases where the manufacturer’s service center is located overseas, communication difficulties can also negatively impact the time required to diagnose and repair units when service is required.

Another factor to consider when choosing a vendor for your out-of-band management solution is that electronic devices that are manufactured in the USA are more likely to be UL and CE certified. This provides additional evidence of quality and reliability and helps to ensure that the devices used in your out-of-band management strategy are properly documented and meet all applicable safety regulations.

Solution:

After assessing out-of-band communication options and power availability, WTI’s switched PDU and console server products are installed at each remote site and WTI’s WMU Enterprise Management Software is used to coordinate, configure and control the various remote sites. This allows support personnel at a central NOC to easily access reboot and console port functions at remote equipment sites without the need for a physical service call or truck roll.

When an equipment outage disrupts communication with a remote site, WTI’s WMU Enterprise Management Solution allows support personnel at your NOC to quickly locate and control a switched PDU or console server at the remote site and initiate a power reboot or access console port command functions in order to restore normal operation. This drastically simplifies the task of managing remote devices at multiple equipment sites by providing support personnel with a centralized command interface that can be employed to rapidly determine exactly which console port or switched outlet is required to address the problem.

In addition to providing fast access to remote power switching and console access, the WMU also streamlines the process of updating firmware and managing user accounts on multiple WTI devices, without the need to individually address each individual WTI unit. When firmware updates for WTI products are available, the WMU allows administrators to easily select all applicable units and initiate the upgrade procedure for multiple units via a single command. Likewise, the WMU also cuts the time required to change user passwords or update access privileges in the same manner.

Unlike many other enterprise management solutions, the WMU also supports the ability to find and communicate with WTI devices located at ports behind a cellular router. While the enterprise management solutions offered by other vendors have no problem finding the cellular modem or router itself, many are unable to find and communicate with their own devices at ports on a LAN behind a cellular modem. Without this ability, network administrators are sometimes forced to eliminate cellular broadband as an option for out-of-band or in-band communication in out-of-band management applications.

Since each type of remote equipment site offers slightly different communications and power options, a variety of different WTI power reboot control products and remote console access products have been deployed in order to provide each type of remote site with an appropriate out-of-band management solution.

Given the variety of different types of sites involved, it quickly becomes clear that in this case, the best possible out-of-band management solution will differ from remote site to remote site, depending on the power needs and communication options present at each site. Sites such as data centers, where network cable connections are readily accessible will most likely rely on redundant network connection to provide in-band and out-of-band access. In contrast, microwave antennae sites where a network cable might be impractical, will rely on communication options such as cellular broadband and dial-up for in-band and out-of-band access.

Obviously then, each type of remote equipment site will have a different set of requirements that the out-of-band management solution will need to meet. In addition to fitting the communication and power needs of each equipment site, an effective out-band-management solution will also need to address the fact that each type of equipment site served will also have differing requirements for out-of-band management.

This is where the need for a scalable out-of-band management solution becomes apparent. Larger equipment sites such as data centers and microwave antenna installations will likely include more managed devices and also offer more rack space for out-of-band management units. Accordingly, out-of-band management solutions for these types of sites will require a larger number of remotely switched outlets and more capacity for console port access. On the other end of the scale, smaller sites will naturally include fewer managed devices and may also be somewhat cramped, rack-space-wise. In addition, smaller equipment sites will generally require fewer switched outlets, less capacity for console port access and will also offer less available rack space for out-of-band management devices.

As a result, an out-of-band management solution for an enterprise network can essentially be broken down into two levels: the management level, which provides a single, centralized enterprise management interface for controlling reboot and port access functions at multiple remote locations, and the site level, which includes the actual switched PDUs and console servers installed at each equipment site. We have already discussed the WMU and the management level, so now we will move on to examples of various types of site level solutions for managing remote networked devices. Four typical types of remote equipment sites will be discussed:

1. Data Centers or NOCs
2. Microwave Antenna Sites
3. Branch Offices
4. Remote Pipeline Monitoring Stations

1. An Out-of-Band Management Solution for Data Centers and NOCs

Aside from the sheer number of managed devices present, data centers and NOCs generally present an extremely straightforward example of out-of-band management. This is primarily due to two factors; communication via Ethernet cable is readily accessible and numerous power options are also available. For the most part, the most daunting challenge faced when implementing an out-of-band management solution in a data center or NOC is the sheer number of managed devices that must be supported.

For the purpose of this example, let’s assume that the data center is AC powered and that both primary and secondary 10/100/1000Base-T Ethernet connections are available for in-band and out-of-band communication with the site. Since there are a large number of devices present in the data center that require remote management, the out-of-band management solution will need to provide a sufficient number of switched outlets and serial ports to maintain these devices. In order to adequately support the various routers, switches and firewalls present, remote console access for up to 40 devices is provided by WTI’s DSM-40-E Dual GigE Console Server while power reboot switching capabilities for up to 20 devices are provided by WTI’s NBB Series Network Boot Bar.

Due to the mission critical nature of the data center, a power redundancy solution is also required. Fortunately, the DSM-40-E console server includes dual power inlets with built in ATS capabilities, allowing the DSM to be easily connected to a primary power source such as utility power plus a secondary power source such as a UPS coupled with a generator.

In order to provide in-band and out-of-band communication with the DSM-40-E Dual GigE Console Server, the primary Ethernet cable is connected to one of the DSM’s two available 10/100/1000Base-T Ethernet ports and the secondary Ethernet cable is connected to the DSM’s other 10/100/1000Base-T Ethernet port. Remote power reboot and switching capabilities for up to 20 devices are provided by the NBB Network Boot Bar, which is in turn connected to a free serial port on the DSM-E. If console access for more than 40 devices or power reboot capabilities for more than 20 devices are required, then additional DSM and/or NBB units can be deployed.

In this example, support personnel can remotely manage devices at the data center or NOC using WTI’s WMU Enterprise Management Software. The WMU can also collect alarms and status messages generated by WTI devices, allowing support personnel to keep an eye on conditions and events at the site. When firmware or user accounts for WTI devices at the remote site need to be updated, new firmware can be installed and user account information can be edited for multiple WTI devices via the WMU, eliminating the need to individually access and update each WTI device.

The out-of-band management solution shown in this example features the following elements:

• Primary AC Power from Utility Line
• Secondary AC Power from UPS
• In-Band Communication via Primary Ethernet Cable
• Out-of-Band Communication via Secondary Ethernet Cable
• Remote Console Access for up to 40 Devices via the DSM-40-E Dual GigE Console Server
• Remote Power Switching and Reboot for up to 20 Devices via the NBB Network Boot Bar
• Power Redundancy for the DSM via Connection to a Utility Line plus Secondary Power Source such as a UPS
• Power Fallback via the DSM’s built in ATS capabilities

2. An Out-of-Band Management Solution for Microwave Antenna Sites

Due to their nature, microwave antenna sites are often located far off the beaten path, where utility power lines and network cables are not readily available. Fortunately, since DC power is typically used at microwave antenna sites, the lack of a public utility line generally doesn’t pose a problem. However, the lack of easy access to an Ethernet cable does dictate that in-band and out-of-band communication with the microwave antenna site will probably rely on alternate avenues for communication, such as dial-up, cellular broadband or satellite. Although a microwave antenna site generally doesn’t include as many managed devices as a data center, there are still a fair number of devices at the site that will need remote reboot and console access capabilities as part of an effective out-of-band management strategy.

Given the DC powered environment, the number of managed devices present and the lack of access to an Ethernet cable at a microwave antenna site, WTI’s DSM-24DC DC Powered Console Server is deployed to provide access to console port functions on devices at the remote site, and a pair of WTI’s RPC-40L8A4-48 DC Power Switches are used for remote reboot and power switching capabilities. This provides out-of-band access to console ports on up to 24 managed devices at the microwave site, plus power switching and reboot capabilities for up to sixteen devices.

In order to provide in-band and out-of-band access in the absence of a network cable, a cellular router is connected to the DSM-24DC’s 10/100/1000Base-T Ethernet port to allow in-band access and a PSTN line is connected to the DSM-24DC’s optional internal modem port for out-of-band access. In the event that cellular access is unavailable, support personnel at the NOC can still use the PSTN line to manage reboot functions and console access for switches, routers, alarm collectors and other devices at the microwave antenna site.

Since the communication services provided by the microwave antenna site are absolutely essential, the DSM-24DC Console Server and RPC DC Power Switches are connected to both a primary DC power source and a secondary DC power source to provide redundant power. In the event that the primary DC power source is unavailable, the TCM and RPC units will draw power from the secondary DC power source, ensuring that remote reboot capabilities and access to console port command functions are always on hand, even during power emergencies.

In this example, support personnel at the NOC can remotely manage devices at the microwave antenna site using WTI’s WMU Enterprise Management Software. In addition to providing a centralized interface for initiating power switching functions and console port connections, the WMU also simplifies the task of updating firmware and managing user accounts on multiple WTI units at the remote site.

The out-of-band management solution shown in this example features the following elements:

• Primary and Secondary DC Power from Batteries (or DC Converter if AC power is available)
• In-Band and Out-of-Band Communication via Cellular Broadband and Dial-Up
• Remote Console Port Access for up to 24 Devices via the DSM-24DC DC Powered Console Server
• Remote DC Power Switching and Reboot Control for up to Sixteen Devices via two RPC-40L8A4-48 DC Power Switches
• Power Redundancy for the DSM and RPC via Battery Back Up or DC Converter

3. An Out-of-Band Management Solution for Branch Office Applications

In branch office applications, an effective out-of-band management solution allows support personnel to deal with network problems without the need to travel to the remote site in person. This cuts the time required to deal with network issues, reduces downtime and helps to ensure that support personnel are also available to deal with network problems at other equipment sites in a large enterprise network.

Like data centers and NOCs, branch offices usually have ready access to both an Ethernet cable and AC power. The major difference between an out-of-band management solution for a branch office and an out-of-band management solution for a data center is that branch offices generally include fewer managed devices.

In this case, WTI’s DSM-24-E Dual GigE Console Server is deployed to provide remote console access for up to 24 devices, while WTI’s NPS-16HD20-1 Network Power Switch provides remote reboot capabilities for up to 16 managed devices. Since both the DSM and NPS include dual power inlets and a built-in ATS, these devices can be connected to both a primary power supply such as a utility line and a secondary power supply such as a UPS in order to provide a power redundancy/fallback solution to ensure that power outages will not interfere with out-of-band management capabilities.

The Ethernet cable is connected to one of the DSM’s two available 10/100/1000Base-T Ethernet ports to provide in-band communication and a PSTN line is connected to the DSM’s optional internal modem to supply out-of-band access to console port command functions. In order to provide remote access to power switching and reboot functions, the NPS is connected to a free serial port on the DSM console server.

This type of configuration enables support technicians at the NOC to use the WMU Enterprise Management Software to remotely reboot devices in the branch office or access console port command functions on those devices in order to run troubleshooting routines, change configuration parameters or diagnose and correct equipment malfunctions.

The out-of-band management solution described in this branch office application example features the following elements:

• Primary AC Power from Utility Line
• Secondary AC Power from UPS
• In-Band Communication via Ethernet Cable
• Out-of-Band Communication via Dial-Up
• Remote Console Port Access for up to 24 Devices via the DSM-24-E Dual GigE Console Server
• Remote Power Switching and Reboot Control for up to Sixteen Devices via the NPS-16HD20-1 Switched PDU
• Power Redundancy for the DSM and NPS via a Primary Power Source such as a Utility Line and a Secondary Power Source such as a UPS and/or Generator

4. An Out-of-Band Management Solution for Pipeline Monitoring Stations

Pipeline monitoring stations are usually located in extremely isolated locations and often the only way to reach them is via a long truck ride down a bumpy, poorly maintained, unpaved road. In spite of their isolation though, pipeline monitoring stations often have access to a network cable that follows the path of the pipeline that these monitoring stations keep watch over. Due to the small size and limited functionality of these monitoring stations, they generally contain fewer networked elements than would normally be found in a data center or branch office.

The small size of most pipeline monitoring stations presents a different sort of challenge when implementing an out-of-band management solution. Due to the limited amount of rack space available in a pipeline monitoring station, often there’s just not enough room for both a console server and switched PDU in addition to the leak detection equipment, alarm collection devices, routers, switches and other devices that already crowd the packed equipment cabinet.

In order to allow both power switching and console access capabilities without using up valuable rack space, WTI’s CPM-800-1-ECAM Console Server + Switched PDU Hybrid is deployed. The CPM-800-1-ECAM requires only 1U rack space, yet still provides both power reboot and console access functions. As an added bonus, the CPM-800-1-ECAM also includes dual power inlets and a built in power transfer switch, allowing the unit to also function as a power fallback solution.

An Ethernet cable is connected to one of the CPM-800-1-ECAM’s two available 10/100/1000Base-T Ethernet ports while the other Ethernet port is connected to a satellite modem. This provides in-band communication via network, plus out-of-band communication via satellite.

In the event that a managed device at the pipeline monitoring station crashes and takes down network communication, administrators at your NOC can still contact the CPM-800-1-ECAM via satellite and then either remotely reboot the unresponsive device or access console port command functions to restore normal operation and determine the cause of the problem.

The out-of-band management solution in this pipeline monitoring station application includes the following elements:

• Primary AC Power from Utility Line
• Secondary AC Power from UPS
• In-Band Communication via Ethernet Cable
• Out-of-Band Communication via Satellite Modem
• Remote Console Port Access plus Remote Power Reboot Switching for up to 8 Devices via the CPM-800-1-ECAM Dual Ethernet Console + Power Hybrid is deployed
• Power Redundancy for the CPM-800-1-ECAM via a Primary Power Source such as a Utility Line and a Secondary Power Source such as a UPS and/or Generator

Results:

The combination of WTI’s console management products, switched PDU products and WMU Enterprise Management Software allows personnel at your NOC to deal with network problems at remote equipment sites without the need to send a support team to the site to deal with the problem in person. This cuts the time required to respond to problems, minimizes downtime and helps to ensure that vital, network-based services and capabilities are available when needed.

When routine problems with network elements at remote sites disrupt network communication, the effective out-of-band management solution provided by WTI eliminates the need for expensive truck rolls, dangerous trips up poorly maintained mountain roads, the hassles of dealing with interdepartmental red tape and misunderstandings and scheduling conflicts with third party support vendors. In addition to the basic benefits of remote console access and remote reboot control, WTI’s out-of-band management solution also provides the following:

• Scalability: WTI products are available in a number of varying models to fit the needs of many different sizes and types of remote equipment sites
• Adaptability: WTI products offer an extremely versatile design that can easily adapt to satisfy your specific out-of-band management requirements
• Customization: WTI is glad to work with customers to solve unforeseen problems and develop solutions and capabilities to fit customer needs
• Comprehensive Product Line: WTI offers both console access and switched PDU products to provide a simple comprehensive out-of-band management solution
• Enterprise Management Solution: WTI’s WMU Enterprise Management Software provides a single, centralized interface to control your entire out-of-band management deployment
• Confidence: WTI Offers a five year warranty on all of our products and provides US based support in the event of problems

WTI console servers and switched PDUs are available in a wide variety of sizes and configurations, providing the scalability required to meet the console access and remote reboot needs of any size or type of remote equipment site. Regardless of whether the site runs on AC or DC power, regardless of the number of devices at the site and regardless of the communication avenues present at the site, WTI has an out-of-band management solution that will provide the perfect fit.

In addition to offering a wide range of different console server and switched PDU models, WTI products also feature an extremely flexible, adaptable set of configuration and command options which enable our products to serve the needs of a multitude of different applications. The reason for this is simple: for many years, we’ve listen to our customers’ suggestions and problems and have gladly incorporated these ideas into our products in order to provide a practical and efficient out-of-band management solution that anticipates the many unexpected challenges that arise when designing an enterprise level remote management solution. WTI welcomes opportunity to work with customers in order to provide customized solutions to customers’ challenges while simultaneously improving our own products.

When combined with our WMU Enterprise Management Software, WTI console servers and switched PDUs allow a single vendor solution for out-of-band management, drastically simplifying the task of dealing with network outages by providing a single, uniform, centralized control interface. This eliminates the need for support personnel to familiarize themselves with multiple, separate command interfaces for power and console access functions and supplies a much more cohesive solution than can be attained by cobbling together a patchwork of remote reboot and console access solutions from a mixture of several different vendors.

In addition to providing a centralized command interface for all WTI console server and switched PDU products, the WMU also serves as a collection point for alarms generated by various WTI devices in your enterprise infrastructure. When WTI devices detect high rack temperatures, unresponsive devices or other potential signs of trouble, alarm messages can be sent to the WMU, and also sent in SNMP format to another enterprise management systems or emailed or texted directly to the appropriate support personnel.

When firmware for WTI devices needs to be upgraded or user account information on your WTI devices needs to be edited, the WMU also allows you to manage firmware updates and revise user accounts for multiple WTI devices or all of your WTI devices in a single operation. This saves an incredible amount of time and eliminates the bothersome task of accessing each individual console server and switched PDU and updating them individually.

One of the most important results of the out-of-band management solution described in this white paper is that WTI stands behind its products and has sufficient confidence in product quality, that all WTI products are covered by a five year warranty. WTI’s warranty covers free parts and labor, free US based support, free firmware updates and loaner devices to keep you up and running in the rare event that one of our devices needs to be repaired. Since WTI is based in the US, you can count on a quick response to your questions and service issues, without the confusion and delays often experienced when dealing with overseas support centers. WTI products are UL/CE certified and are proudly designed and manufactured at our headquarters in Irvine, California.