What’s surprising about enterprise network discovery and assessment with virtual infrastructure
Recently, I had the pleasure of sitting down with John Cavanaugh, CTO at NetCraftsmen, to discuss the convergence of security and network assessments that is occurring in enterprise environments, and the need to avoid potential security exposures that can creep into complex networks with what we might term NFV sprawl. Here are some of the highlights from our Q&A session...
As CTO at NetCraftsmen, John leads a team of consultants focused on designing and delivering scalable, highly available and secure infrastructure solutions to customers across multiple industry verticals and technologies. Before joining NetCraftsmen, John held several positions including Executive Director and Chief Architect for Global Network Services at JPMorgan Chase. Prior to his role at JPMorgan Chase, he was a Distinguished Engineer at Cisco.
Kelly: John, thank you for making the time to discuss network assessments and planning for automation with your enterprise clients. Can you explain how these engagements usually get started?
John: Happy to do it. In terms of network assessments, what’s interesting is that when we conduct network security assessments, we also need to concurrently examine network topology, pathing, and configuration management. In reality, you cannot assess network security without addressing the network itself and the lifecycle control that you use to operate it. So, these disciplines are converging. In fact, leading analyst firms are advising organizations to knock down the organizational walls that have traditionally existed between networking and security teams, and to get out of the siloed approach. SASE and Zero Trust Security are the new market segments being driven from these views.
Kelly: What is your typical approach to network assessment when you work with a client organization?
John: We use automation tools to help us to discover the devices in a network, virtual and physical. We look at their current status in terms of the device configuration and software versions; it’s really helpful for us to have the EoX (end-of-life, support, and development) dates from the various network vendors right in the assessment tools and reports so we can plan forward pretty quickly.
Our approach uses an OVA agent that we drop into a VM remotely along with a seed file of the first 60 or so device IPs from the customer network. We use that seed list to navigate and discover the connected devices. We poll over XML and SNMP versions 2 and 3 (and SSH when these are not available). We take an iterative approach for a few reasons, not the least of which is that SNMP polling if used all at once can put too much pressure on the network. In fact, customers most often avoid auto-discovery because of this.
There was a case in the past of which I’m aware, where a large financial institution was performing auto-discovery on their network. They brought the network down. It turned out that they had configured to perform a full network inventory every second, which put a ton of load on the network – and crashed it. It was incredibly disruptive, obviously, and the loads were way above what is expected and healthy for network management – which really should never get above a maximum of about 5% of available bandwidth.
Kelly: How do you assess the impact of the network for apps and users?
John: We spend quite a bit of time speaking with the end-users to get their input. It’s interesting how passionate people can be about their experience on networks. Information Technology (IT) Departments can make or break their reputation based on how users perceive network performance, and we often find IT teams are insulated from what the users really think. So, this is a key set of inputs to consider in a network assessment.
Kelly: What are some of the notable results of assessments that you’ve found?
John: It’s interesting to realize that organizations frequently lose track of their legacy devices. As a somewhat dramatic example, we worked with an IT team at a large enterprise that was expecting our assessment to discover about 2500 network devices. What we actually found was closer to 12,000 devices, including a large collection of about 3,500 AP’s that nobody knew were there (Shadow IT).
The problem for organizations with unmanaged - and untracked - legacy devices is manifold. First, of course, are the inherent security exposures from having these potential entry points in the network which are unmonitored. Second, it’s also true that too many devices can degrade LAN and wireless network performance in weird ways. Also, just the extra complexity of having these devices can impact how a network operations center (NOC) team operates and erode their efficiency: the devices fall out of version control, make problem isolation and troubleshooting more difficult, increase staff burdens, and so on.
Kelly: When are organizations more likely to need a network assessment?
John: Well, in the last example blind spots developed due to multiple turnovers in leadership, making it easy to lose track of equipment, so I recommend an assessment when significant changes in leadership or organizations occurs – such as in cases of mergers and acquisitions (M&A).
These blind spots are often referred to as technical debt which is a symptom of the complexity that accrues when technology teams jury rig systems together – never taking the time to redesign or streamline processes. This is especially true of M&A situations where diverse technological systems are often just bolted together.
Also, as I previously said, network security assessments often trigger network assessments, with the need to change and evolve network management approaches.
In a similar vein, regulatory compliance turns out to be a case in which network assessments are critical (think about network segmentation to support PCI or HIPAA/HiTrust) – and where orchestration capabilities are the key to enabling teams to manage their networks better over the longer term.
Kelly: Please say more about that: we hear a lot about how orchestration is essential for Cloud Service Providers (CSPs) but we don’t often hear about how enterprise teams use orchestration.
John: Yes, that’s because enterprise teams do not often use orchestration – but they need it. In particular for Virtual Network Functions (VNFs) like virtual switches and firewalls, and for software-defined capabilities. A lot of enterprise IT teams are familiar with VMware orchestration, and we’ve seen some teams try to apply VMware to network device orchestration (in VMware NSX – a virtual firewall – DFW – is supported). But VMware just handles virtual devices; what teams need is to manage and orchestrate their virtual and physical workloads and the virtual and physical appliances in a single service chain, using the same approach. By the way, service chaining is a CSP term and not one that many enterprise infrastructure teams know. But it’s a term that I think enterprise teams will need to know as their virtual device counts continue to grow.
Let me give you a real-world example. I worked with a healthcare team that had some older infrastructure that failed a compliance audit for improperly handling data that was governed by HIPAA privacy rules. Over the course of a few years the team had spun up a lot of new VNFs – virtual firewalls in this case - to support often transient workloads. Unfortunately, once the workload use was complete, the virtual firewalls remained; the team was somewhat large – and individual admins could not recall why the virtual firewalls had been created or which workloads they were intended to support. So, they didn’t want to take them down and risk a security incident. Over a few years of time, this created a lot of resource overhead and added to network complexity. But Firewalls were only one part of the complexity, they also had load balancing rules in place that were out of date. Again, it was a case of the NOC team being too busy and too compartmentalized to track this information across the devices and workloads and maintain and share the data with the entire team, on an on-going basis.
The assessment results identified that the team needed to define and enforce policies for how application workloads would be supported across the network. In that environment, the team put in place a process for attesting to the intended use of network devices and for associating workloads with VNFs. They started with a manual process, but to ensure that the VNFs were taken down and decommissioned along with the workload – they needed automation.
Kelly: This reminds me of VM sprawl that can happen when teams lose track of why VM’s are instantiated, and I can see that enterprise teams might be newer to thinking through their processes to managing the whole lifecycle associated with NFV. Is this where you think orchestration can help?
John: Yes. Enterprise teams need the service chaining provided by Blue Planet orchestration solutions, as well as service chain lifecycle management. The ability to dynamically create and provision a service chain consisting of multiple VNF’s and physical devices, and the ability to visualize the service chain and to associate the workload and its traffic to the correct service path, – and take them down when they are no longer needed - will relieve IT personnel of significant manual burdens, potential errors, and related costs. It will also help IT organizations maintain and implement consistent policies and ensure compliance in a more automated approach, which I know you are offering in the Blue Planet Enterprise suite, as well.
Some solutions exist for single vendor, single technology stacks, but enterprise reality is multi-vendor and multi-technology. Enterprises need to abstract workloads that are VMware, OpenStack and Bare Metal with Routers, Firewalls, Load Balancers, and other network-based appliances from a variety of vendors.
Blue Planet leads the market on orchestration, and that makes them very interesting for use in network discovery, assessment, along with compliance and policy management, as well.
Blue Planet Enterprise has full functionality for defining policies across networks, as well as ensuring compliance with these policies in network device configurations and traffic flow. Look for details about these valuable capabilities in future blogs. You can also request your live demo or free trial to see these capabilities in action for yourself. Blue Planet continues to aggressively move forward to provide CSP-proven capabilities like orchestration, discovery, configuration and compliance management, and layer 3 optimization, all packaged and priced for the enterprise with SaaS pay-as-you-go delivery.