John Kristoff's blog @ Team Cymru

Security: NETCONF in the Wild

[Editor's note: This work was conducted in collaboration with James Schaefer at DePaul University. Maciej Leja, also at Depaul University, participated in this work as well.]

NETCONF, an XML-based RPC mechanism aims to help network operators programmatically manage their network devices. You will find NETCONF capabilities in network gear from a handful of common backbone network equipment providers (e.g. Juniper Networks NETCONF XML Management Protocol Developer Guide, Cisco Network Configuration Protocol). While generally not enabled by default, when used it is commonly running over SSH on IANA assigned TCP port 830 (see IETF RFC 6242 - Using NETCONF Protocol over Secure Shell (SSH)). A newer (don't let the RFC number fool you) and perhaps less well known NETCONF transport is defined in IETF RFC 5539 - NETCONF over Transport Layer Security (TLS), using a default TCP port of 6513. There is also IETF RFC 4744 - Using NETCONF Protocol over the Blocks Extensible Exchange Protocol (BEEP) specification assigned to TCP default port 831, but this is even less common and not widely implemented.

NETCONF has its roots in something called JUNOScript, as implemented by Juniper Networks. JUNOScript the protocol was essentially brought to the IETF and evolved into what is now NETCONF. Juniper still maintains backwards compatability in their gear for JUNOScript, but there is little difference between the two and those differences don't much matter for our purposes here. You may be wondering why NETCONF arose when SNMP, the so-called Simple Network Management Protocol has been around since the 1980's. Isn't SNMP the means by which "network operators programmatically manage their network devices"? While SNMP has been widely used, particularly for network status and statistics gathering, it never quite caught on for device configuration management to any significant degree. SNMP has often been seen as an obtuse and syntactically complicated protocol, rarely intuitive for operators who have a tendency to prefer text-based interfaces and messages. Furthermore, much of the relevant device detail an operator is often interested in can only be found in the vendor specific branches of a MIB tree, making interoperability and management across platforms cumbersome. For this reason you often find network operators avoiding SNMP-based solutions, instead preferring to get syslog messages over SNMP traps and leveraging scripting tools like Expect and RANCID to mimic the common interactive terminal access methods. In fact, even while NETCONF has seen some adoption, some network operators are still not satisfied with the automation options.

So while NETCONF has seen some deployment and appears to have made some inroads over SNMP in network equipment management it is not ubiquitous. Even with XML, a text-based self-described message format, there is still a natural resistance, reluctance and general lack of motivation for operators to all begin deploying and using NETCONF en masse. Nonetheless, we have recently realized that NETCONF accessible network gear can be found on the Internet and this fact piqued our interest. We are not aware of any prior concerted effort to evaluate Internet-accessible NETCONF service interfaces, over SSH or otherwise so we spent some time reviewing NETCONF history, the protocols, implementations and of course what we can see in the wild. We recently presented an initial summary of our findings of an equally named lightning talk at NANOG 63, NETCONF in the Wild. (Note: you can also find a copy of the slides on the NANOG 63 page along with an associated video of the presentation).

Our overriding concern is that NETCONF accessible systems may pose a significant threat to network operators. If a savvy intruder were able to obtain management access to network gear, it could very well be "game over" as they say. Thankfully, on either Juniper or Cisco gear, the network gear we tend to be most familiar with, NETCONF capability is not enabled by default. Furthermore, NETCONF capability from both of these vendors is currently accessible only over SSH as a specific SSHv2 subsystem component. This means SSHv1 is not supported for NETCONF and other than the server public key, no NETCONF communications will traverse the network in clear text. That is more good news. However, we remained concerned for two reasons. One, we speculated that some operators have NETCONF over SSH services exposed publicly and do not realize it. We reasonably came to this conclusion, because we were surprised to discover network gear with NETCONF over SSH on TCP port 830 services remotely accessible without restriction on our own networks. Two, we know from prior experience that just because a system runs SSH does not mean it is impenetrable. Dictionary password attacks against well known or guessed user names is an often cited example, undermining access controls of network gear (see for example the paper A Quantitative Analysis of the Insecurity of Embedded Network Devices: Results of a Wide-Area Scan). We set out to see who else may have NETCONF over SSH running and accessible on the public Internet.

We first generated a random list of about 10 million IPv4 addresses and sent one TCP SYN probe to port 830 (NETCONF over SSH) to each address in the list. If we obtained a SYN,ACK response we would then conduct a ssh-keyscan follow up probe to TCP port 830 to a responder. ssh-keyscan was able to quickly and easily collect the SSH implementation version information as well as the SSH server public keys if there was indeed an SSH listener on TCP port 830. As one might expect, some systems that return SYN,ACK responses will not have an associated SSH daemon listening there. In fact, for most TCP port 830 responders in our random probe set, this was typically the case. We then ran the same sequence of Internet probes using a list of about 6700 IPv4 Internet addresses we had previously identified as Juniper JUNOS network devices on the Internet. We wanted to evaluate known Juniper systems, because based on our background research, these are the among the most likely types of devices where you might see NETCONF over SSH running. Since Juniper helps power some of the biggest networks with the some of the most powerful gear, focusing on these systems was of particular interest. In the JUNOS-specific probe set, ssh-keyscan ultimately identified a few dozen SSH over TCP port 830 listeners and their associated SSH public keys on a variety of networks. While the total number of responders was relatively small we found that we could only successfully obtain ssh-keyscan results from half of those same systems on TCP port 22. It seems clear that some operators are intentionally shielding the interactive SSH service port, but have forgotten to do the same with the NETCONF over SSH service port.

Next we wondered what we would find if we randomly probed a different set of 10 million random IPv4 addresses for NETCONF over TLS listeners. Here we were surprised to find, while a still small number, a handful of responders, more than what we found from random NETCONF over SSH probing. We should note, it is not entirely clear if these were indeed NETCONF systems, but they certainly were providing an X.509 certificate on the NETCONF over TLS port, so it seems likely that at least some of them were running NETCONF over TLS services. In fact, we did try to point a web browser at some of those systems and found we could not negotiate HTTP over any of those we tested. Since NETCONF over TLS has it own unique XML-based negotiation involving server and client certificates, this would support the idea that these were probably NETCONF service listeners.

With neither NETCONF over SSH nor NETCONF over TLS did we attempt to verify beyond the initial SSH or TLS handshaking that any of the responders were truly were running NETCONF services. This would have required authentication attempts, which we did not feel justified in undertaking. We think it is safe to assume most, if not all of the systems previously identified as JUNOS systems, that performed an SSH handshake on TCP port 830 were indeed running NETCONF services. The 6513 responders we are less sure of, but the X.509 certificate properties can reveal some clues about these systems. Based on the certificate attributes, we typically found what appear to have been a handful of systems likely affiliated with shopping cart web sites, financial systems and some network appliances. While some patterns of system type seen in the 6513 responders emerge, we could not quickly identify any particular NETCONF over TLS implementation. We would have liked to audit some network gear that we knew ran NETCONF over TLS services, but we could not obtain any in the short amount of time we dedicated to this project. We also briefly attempted to to build our own using libnetconf and netopeer, but we were not successful in uncovering any additionally useful insight. So while we abandoned any further investigation, our fear is that some NETCONF over TLS implementations or configurations may lack strong client certificate authentication. Are there any NETCONF over TLS systems in the wild that may be susceptible to access control bypass simply by presenting them a seemingly valid or properly formatted handshake and certificate? We hope not, but we just do not know for sure. We briefly considered writing a simple NETCONF over TLS specific penetration testing tool, but we will leave further research in this area to others if they care to take it on.

We will close by suggesting that while the threat of an accessible NETCONF system in the wild may pose a significant risk for the associated network, we did not find evidence that this is any sort of grave risk for the Internet at large, at least when considered alongside all the other potential risks that are already well known. Nevertheless, any NETCONF service exposure to anonymous Internet clients is probably at best, unnecessary. Even if a device is resistant to break-in or compromise at the application layer, that the NETCONF application layer is accessible poses a potential risk of its own. Network devices generally have enough to do and you don't want to worry about the NETCONF service being interrupted with any unnecessary connection processing. If you're deploying gear with NETCONF capabilities or you use NETCONF to manage your network devices, do remember to shield those services as you would other management and control plane interfaces.

posted at 4:52 pm | permanent link