John Kristoff's blog @ Team Cymru

Security: NETCONF in the Wild

[Editor's note: This work was conducted in collaboration with James Schaefer at DePaul University. Maciej Leja, also at Depaul University, participated in this work as well.]

NETCONF, an XML-based RPC mechanism aims to help network operators programmatically manage their network devices. You will find NETCONF capabilities in network gear from a handful of common backbone network equipment providers (e.g. Juniper Networks NETCONF XML Management Protocol Developer Guide, Cisco Network Configuration Protocol). While generally not enabled by default, when used it is commonly running over SSH on IANA assigned TCP port 830 (see IETF RFC 6242 - Using NETCONF Protocol over Secure Shell (SSH)). A newer (don't let the RFC number fool you) and perhaps less well known NETCONF transport is defined in IETF RFC 5539 - NETCONF over Transport Layer Security (TLS), using a default TCP port of 6513. There is also IETF RFC 4744 - Using NETCONF Protocol over the Blocks Extensible Exchange Protocol (BEEP) specification assigned to TCP default port 831, but this is even less common and not widely implemented.

NETCONF has its roots in something called JUNOScript, as implemented by Juniper Networks. JUNOScript the protocol was essentially brought to the IETF and evolved into what is now NETCONF. Juniper still maintains backwards compatability in their gear for JUNOScript, but there is little difference between the two and those differences don't much matter for our purposes here. You may be wondering why NETCONF arose when SNMP, the so-called Simple Network Management Protocol has been around since the 1980's. Isn't SNMP the means by which "network operators programmatically manage their network devices"? While SNMP has been widely used, particularly for network status and statistics gathering, it never quite caught on for device configuration management to any significant degree. SNMP has often been seen as an obtuse and syntactically complicated protocol, rarely intuitive for operators who have a tendency to prefer text-based interfaces and messages. Furthermore, much of the relevant device detail an operator is often interested in can only be found in the vendor specific branches of a MIB tree, making interoperability and management across platforms cumbersome. For this reason you often find network operators avoiding SNMP-based solutions, instead preferring to get syslog messages over SNMP traps and leveraging scripting tools like Expect and RANCID to mimic the common interactive terminal access methods. In fact, even while NETCONF has seen some adoption, some network operators are still not satisfied with the automation options.

So while NETCONF has seen some deployment and appears to have made some inroads over SNMP in network equipment management it is not ubiquitous. Even with XML, a text-based self-described message format, there is still a natural resistance, reluctance and general lack of motivation for operators to all begin deploying and using NETCONF en masse. Nonetheless, we have recently realized that NETCONF accessible network gear can be found on the Internet and this fact piqued our interest. We are not aware of any prior concerted effort to evaluate Internet-accessible NETCONF service interfaces, over SSH or otherwise so we spent some time reviewing NETCONF history, the protocols, implementations and of course what we can see in the wild. We recently presented an initial summary of our findings of an equally named lightning talk at NANOG 63, NETCONF in the Wild. (Note: you can also find a copy of the slides on the NANOG 63 page along with an associated video of the presentation).

Our overriding concern is that NETCONF accessible systems may pose a significant threat to network operators. If a savvy intruder were able to obtain management access to network gear, it could very well be "game over" as they say. Thankfully, on either Juniper or Cisco gear, the network gear we tend to be most familiar with, NETCONF capability is not enabled by default. Furthermore, NETCONF capability from both of these vendors is currently accessible only over SSH as a specific SSHv2 subsystem component. This means SSHv1 is not supported for NETCONF and other than the server public key, no NETCONF communications will traverse the network in clear text. That is more good news. However, we remained concerned for two reasons. One, we speculated that some operators have NETCONF over SSH services exposed publicly and do not realize it. We reasonably came to this conclusion, because we were surprised to discover network gear with NETCONF over SSH on TCP port 830 services remotely accessible without restriction on our own networks. Two, we know from prior experience that just because a system runs SSH does not mean it is impenetrable. Dictionary password attacks against well known or guessed user names is an often cited example, undermining access controls of network gear (see for example the paper A Quantitative Analysis of the Insecurity of Embedded Network Devices: Results of a Wide-Area Scan). We set out to see who else may have NETCONF over SSH running and accessible on the public Internet.

We first generated a random list of about 10 million IPv4 addresses and sent one TCP SYN probe to port 830 (NETCONF over SSH) to each address in the list. If we obtained a SYN,ACK response we would then conduct a ssh-keyscan follow up probe to TCP port 830 to a responder. ssh-keyscan was able to quickly and easily collect the SSH implementation version information as well as the SSH server public keys if there was indeed an SSH listener on TCP port 830. As one might expect, some systems that return SYN,ACK responses will not have an associated SSH daemon listening there. In fact, for most TCP port 830 responders in our random probe set, this was typically the case. We then ran the same sequence of Internet probes using a list of about 6700 IPv4 Internet addresses we had previously identified as Juniper JUNOS network devices on the Internet. We wanted to evaluate known Juniper systems, because based on our background research, these are the among the most likely types of devices where you might see NETCONF over SSH running. Since Juniper helps power some of the biggest networks with the some of the most powerful gear, focusing on these systems was of particular interest. In the JUNOS-specific probe set, ssh-keyscan ultimately identified a few dozen SSH over TCP port 830 listeners and their associated SSH public keys on a variety of networks. While the total number of responders was relatively small we found that we could only successfully obtain ssh-keyscan results from half of those same systems on TCP port 22. It seems clear that some operators are intentionally shielding the interactive SSH service port, but have forgotten to do the same with the NETCONF over SSH service port.

Next we wondered what we would find if we randomly probed a different set of 10 million random IPv4 addresses for NETCONF over TLS listeners. Here we were surprised to find, while a still small number, a handful of responders, more than what we found from random NETCONF over SSH probing. We should note, it is not entirely clear if these were indeed NETCONF systems, but they certainly were providing an X.509 certificate on the NETCONF over TLS port, so it seems likely that at least some of them were running NETCONF over TLS services. In fact, we did try to point a web browser at some of those systems and found we could not negotiate HTTP over any of those we tested. Since NETCONF over TLS has it own unique XML-based negotiation involving server and client certificates, this would support the idea that these were probably NETCONF service listeners.

With neither NETCONF over SSH nor NETCONF over TLS did we attempt to verify beyond the initial SSH or TLS handshaking that any of the responders were truly were running NETCONF services. This would have required authentication attempts, which we did not feel justified in undertaking. We think it is safe to assume most, if not all of the systems previously identified as JUNOS systems, that performed an SSH handshake on TCP port 830 were indeed running NETCONF services. The 6513 responders we are less sure of, but the X.509 certificate properties can reveal some clues about these systems. Based on the certificate attributes, we typically found what appear to have been a handful of systems likely affiliated with shopping cart web sites, financial systems and some network appliances. While some patterns of system type seen in the 6513 responders emerge, we could not quickly identify any particular NETCONF over TLS implementation. We would have liked to audit some network gear that we knew ran NETCONF over TLS services, but we could not obtain any in the short amount of time we dedicated to this project. We also briefly attempted to to build our own using libnetconf and netopeer, but we were not successful in uncovering any additionally useful insight. So while we abandoned any further investigation, our fear is that some NETCONF over TLS implementations or configurations may lack strong client certificate authentication. Are there any NETCONF over TLS systems in the wild that may be susceptible to access control bypass simply by presenting them a seemingly valid or properly formatted handshake and certificate? We hope not, but we just do not know for sure. We briefly considered writing a simple NETCONF over TLS specific penetration testing tool, but we will leave further research in this area to others if they care to take it on.

We will close by suggesting that while the threat of an accessible NETCONF system in the wild may pose a significant risk for the associated network, we did not find evidence that this is any sort of grave risk for the Internet at large, at least when considered alongside all the other potential risks that are already well known. Nevertheless, any NETCONF service exposure to anonymous Internet clients is probably at best, unnecessary. Even if a device is resistant to break-in or compromise at the application layer, that the NETCONF application layer is accessible poses a potential risk of its own. Network devices generally have enough to do and you don't want to worry about the NETCONF service being interrupted with any unnecessary connection processing. If you're deploying gear with NETCONF capabilities or you use NETCONF to manage your network devices, do remember to shield those services as you would other management and control plane interfaces.

posted at 4:52 pm | permanent link

Security: DVMRP Ask Neighbors2: an IGMP-based DDoS/leak threat

At Team Cymru, we have got into the habit of using BLUF, bottom line up front. Allow me to do so here as well. There exists a little known IP multicast tracing and troubleshooting capability referred to as DVMRP Ask Neighbors2 (the request) and DVMRP Neighbors2 (the response) that can leak router configuration detail and be abused in amplification and reflection attacks. Now, for a fuller accounting of the story.

As I am wont to do, when I take myself out to lunch I will often bring with me some reading material, usually some recent research paper. Approximately four years ago this month, on one such dining and study occasion I happened to bring with me a paper entitled Extracting Intra-Domain Topology from mrinfo. IP multicast is a subject that has long held my attention since many of my formative years operating networks involved inter-domain IP multicast deployment. mrinfo, from the mrouted package. is a command line tool I was familiar with so this paper along with the overall theme of network discovery and measurement immediately appealed to me. After having read the paper and then after a long hiatus from IP multicast troubleshooting tools like mrinfo, I rediscovered how useful this was for mapping Internet topology not necessarily specific to IP multicast by obtaining otherwise unobtainable information from some Internet routers. Fast forward three years. In October 2013, my mind was often preoccupied with amplification and reflection attacks, but I had an opportunity to experiment with the mrinfo tool again when I almost immediately realized the Internet community had another potentially worrisome DDoS threat vector on our hands. From then on up to the present day we have set out to better understand and study all the ramifications of what many would deem to be a case of "protocol abuse" in two forms, an amplifcation and reflection threat and a network topology disclosure threat. We felt both of these threats needed to be brought to the attention of certain vendors and the Internet community so mitigation could begin before miscreants discovered and took advantage of the threats. This blog post provide some background technical material about the threats and the small part we have been playing in mitigation response.

The Internet Group Multicast Protocol (IGMP), IP protocol 2, is essentially a signaling mechanism used between hosts and local IPv4 routers. In essence, end hosts use IGMP to join and leave IP multicast groups and routers use IGMP to maintain group membership on the local network. In practice, there is no operational requirement that these group membership maintenance messages ever leave the local network. Now, I'm not one to advocate for wholesale prohibition of otherwise harmless IP datagrams across internetworks, but here is where the trouble initially begins, but first we must say something about an IP multicast routing protocol that utilizes IGMP.

A now little used and largely obsolete IP multicast routing protocol, the Distance Vector Multicast Routing Protocol (DVMRP), the first widely deployed approach to help move IP multicast data packets around an internetwork, specifies the use of IP protocol 2, making DVMRP a subset type of IGMP messages. The initial specification of DVMRP, version 1, was published as experimental IETF RFC 1075 in 1988. Some years later, work began on updating DVMRP, including bundling in support for additional features that perform "tracing and troubleshooting" capabilities. This latter effort never left Internet Draft status and was ultimately abandoned in final publication draft-ietf-idmr-dvmrp-v3-11. It is the implementation of the DVMRP Ask Neighbors2 and DVMRP Neighbors2 request and response messages in the draft specification of DVMRP that worried us. The former message is specified to be a unicast request, directed to a DVMRP router. If supported, the target router sends a DVMRP Neighbors2 response that includes, primarily, a list of logical interfaces and each interface's properties such as a local IP address and neighbor IP addresses. See Appendix B - Tracing and Troubleshooting support in the draft for further details. Not only was this feature and capability widely implemented in two of the most prominent router vendors' products, but soliciting an Ask Neighbors2 response does not require DVMRP to be running, only some other basic IP multicast component. As you might guess, not only do some routers that respond to these messages disclose information about a router an operator might otherwise wish to keep private, but because a single, small request can potentially solicit a sizable response, this tracing and troubleshooting capability makes for a novel new amplification and reflection attack vector.

We might express some outrage that this little known capability, never formally ratified in an IETF RFC has not only been widely implemented, but has lay lurking as a potential problem for years. We may also be inclined to fret about this threat existing on some of the largest, most well connected and vital pieces of equipment on the Internet. These reactions may be justified, and we do not want to belittle the issue, but we think the potential for widespread abuse may not be as bad as we have seen with other recently disclosed threats through the NTP or the DNS. Two reasons lead us to this belief. First, we believe the changes, fixes or other approaches to limiting abuse are more easily and likely deployed when compared to other protocol abuse threats. By in large, IGMP does not need to transit routers so as a last resort, network-wide filtering of IGMP is not only feasible, but with the exception of being able to use a tool like mrinfo, come with few adverse side effects. Second, it turns out that some networks and some widely deployed network gear already do not forward IGMP messages so widespread scanning and ultimately attacks, will presumably never be able to travel along certain paths of the Internet.

According to our initial research, we believe at least tens of thousands of Internet routers as of this writing will respond to Ask Neighbors2 request messages. While many routers provide minimal responses, many will amplify the request with large or multiple responses, in some cases we have seen an amplification factor of over 100 to 1. While we have discovered numerous responders and have cataloged a number of interesting properties about many routers and responses we have seen, we have partnered with the good folks at The ICSI Networking and Security Group to perform more rigorous and detailed analysis of our findings. We hope to have a proper research paper related to this work appear soon. In the meantime, I'll close with a few final technical details, how to go about mitigating the issue and a warm thank you to a couple of vendor security teams.

The two vendors that are most associated with support for IP multicast routing are Cisco Systems, Inc. and Juniper Networks. Both of these vendors have long supported the DVMRP tracing and troubleshooting features in many of their products. While this capability is not enabled by default, it does not require DVMRP specifically to be activated. For instance, on many Cisco products, simply enabling PIM (a more modern and common suite of IP multicast routing protocols) on any interface will suffice. Likewise on Juniper, PIM on an interface or enabling the igmp protocols group globally will do. We are unfamiliar with other implementations, but based on our experience with the Internet-enabled IP multicast infrastructure, we suspect others will be relatively uncommon.

Where Cisco and Juniper routers support and respond to DVMRP Ask Neighbors2 requests, there are some distinct behaviors they each exhibit. Cisco equipment will limit each IP datagram response to 576 bytes, but send as many messages as necessary to provide a complete list of interface detail. Cisco also encodes the major and minor IOS version numbers in the associated named fields in the DVMRP Neighbors2 response header. Juniper will fragment it's responses. It is worth pointing out, that as a result of correspondence with both Cisco and Juniper, both vendors are in the process of deprecating and removing support for the DVMRP troubleshooting and tracing feature in future versions of their software.

Mitigation of this issue is relatively straightforward. Short of completely disabling any and all IP multicast capabilities, both Cisco and Juniper have provided simple configuration guidelines that should be easily applicable in practically all environments. For Cisco equipment, as documented in The Multicast Security Tool Kit, the following statements can be applied in global configuration mode (be sure to use an access-list number that works for you).

    ip multicast mrinfo-filter 52
    access-list 52 deny any

For Juniper networks, the following firewall filter to the loopback interface can be applied:

    filter igmp {
        term igmp_accept {
            from {
                destination-address {
                 protocol igmp;
            then accept;
        term igmp_drop {
            from {
                protocol igmp;
            then {

The configuration guidelines outlined above are those that are recommended by Cisco and Juniper respectively. Both behave and operate slightly differently and this difference should be noted by network operators. A Juniper device will still respond to Ask Neighbors2 requests directed to a IP multicast group address, but will not forward unicast-directed Ask Neighbors2 probes. The Cisco mrinfo-filter directive prevents only the local router from responding to Ask Neighbors2 requests, but may forward unicast-directed probes to other networks and hosts. To limit unicast forwarding on a Cisco device, an access-list on an interface could be used to mimc the behavior of the Juniper filter:

interface FastEthernet 0/0
 ip address
 ip access-group 101 in
access-list 101 permit igmp any
access-list 101 deny igmp any any

Today, in the security track at NANOG 62 we present a brief talk entitled DVMRP Ask Neighbors2: an IGMP-based DDoS leak/threat, in coordination with bulletins released by both Cisco and Juniper. Let me end by expressing our gratitude and appreciation for a pleasant experience working with both Cisco PSIRT and Juniper SIRT. The representatives we worked with throughout the last year, as we made progress and plans to publicly discuss this threat, handled our concerns with the utmost professionalism and seriousness. We thank them for their cooperation and are happy to report both exhibited an ability to address the issue completely, discretely and in coordinated fashion.

posted at 2:49 pm | permanent link

Paper: Hold-On: Protecting Against On-Path DNS Poisoning

Securing and Trusting Internet Names (SATIN 2012), a conference with the goal of promoting industry and academic attention on the operational security of the Internet's DNS, recently took place in Teddington, UK at the National Physical Laboratory (NPL). This is the second annual meeting and while DNS is an area I've spent a fair amount of time on over the years, I've attended neither one since travel to such a remote destination for a relatively modest agenda and limited outreach opportunity is hard to justify. Thankfully, like most conferences the slide decks and accepted papers are freely available so we still get the benefit of at least some of what was discussed, but we miss out on the always valuable hallway discussions. I read and will provide a less than rigorous review and critique of one of the submitted papers that, if for no other reason, has the distinction of having the largest number of co-authors.

Haixin Duan, Nicholas Weaver, Zongxu Zhao, Meng Hu, Jinjin Liang, Jian Jiang, Kang Li and Vern Paxson Hold-On: Protecting Against On-Path DNS Poisoning, SATIN 2012 [pdf] (Editor's note: the conference web page shows a slightly different title for the paper, I use the one from the paper itself.)

The authors point out that DNS resolvers are designed to accept and act on the first valid response to a DNS query. Unfortunately it has been demonstrated most recently and with no lack of fanfare by the clever Dan Kaminsky how dangerous this can be in practice. The authors categorize a poisoning attack into three areas: off-path, on-path and in-path, but each amounts to what is essentially known in the Internet security community more generically as an impersonation or spoofing attack.

In this context it is worth considering how other systems and protocols deal with spoofed or even benign duplicate messages. Some, such as source-route bridging, which many of you have probably never seen much less heard of, reacts in a manner similar to a DNS resolver. In source-route bridging, it is normal and expected for multiple route explorer messages to arrive at a destination, each enumerating a unique path between the end hosts. The receiver of these explorer messages simply has to choose one and in practice it is the first one received under the assumption that is usually the correct and simplest choice. We could also look for lessons in TCP, such as when a TCP receiver returns duplicate ACKs or in ICMP, when a group of receivers may respond to an ICMP echo request to a limited local, directed broadcast or multicast address. The authors however do not draw any parallels even if it might have been useful to do so.

The paper proposes a fundamental change to the DNS resolver design and while the authors acknowledge the scope of the change, readers unfamiliar with the architecture are unlikely to realize how truly radical such a proposal is. IETF RFC 1034 - Domain Concepts and Facilities discusses the DNS resolver algorithm in section 5.3.3 by stating four recommended priorities for any DNS resolver. The fourth recommendation says simply "Get the answer as quickly as possible". The authors argue for the introduction of a "Hold-On" timer. This timer is intended to help the resolver validate an answer by waiting for a valid response if an earlier one evaluates as suspicious. Their proposed hold-on timer may remind of you a hold-down timer in routing protocols and there are similarities, but where a routing protocol uses a hold-down timer to achieve stability in the face of routing changes, the hold-on timer proposed here aims to minimize surreptitious censorship if not outright injection of malicious data.

A common challenge wherever timers are used in protocols is in determining the initial timer value. While a static value is easy to implement, rarely does such a rigid approach come without drawbacks. Timers often adapt to network conditions and the authors propose just such a thing here. They propose using an expected round trip time (RTT) to help filter out responses that arrive too soon. To determine the expected RTT they first conduct a series of queries that would not normally result in spoofed responses. It is worth noting that their proposal only considers the case of protecting a stub resolver or a resolver that simply forwards to another limited set of resolvers, hence a single expected RTT value is used for all returning answers. Otherwise, the expected RTT could vary widely for a large array of Internet authoritative DNS servers. The authors implement the algorithm using what they call a DNS proxy that sits between stub resolvers or forwarders on the one hand and a set of full resolvers on the other. This does make their experiment much simpler to conduct, but seems to suggest little consideration has been given to how this proposal might scale if deployed between large resolvers and the rest of the Internet DNS.

As an additional measure used to validate returned answers, the authors also examine the returned IP TTL of responses and compare this value with an expected value, obtained in a manner similar to the expected RTT. Presuming DNSSEC is not enabled, a reply that evaluates to a valid RTT and IP TTL will be accepted by the resolver. Their initial evaluation of the RTT and IP TTL mechanisms were done using a Bro policy script. Implementation of the DNS proxy demonstrated that the hold-on timer resulted in no noticeable delay.

Spoofing attack mitigation is a serious and interesting problem and this paper contributes favorably to our current thinking about how to address it, especially in the context of censoring systems as the authors are most interested in here. However, I did not find authors' proposal a convincing practical solution. There are a number of issues that don't seem to be fully addressed. For instance, for an off-path poisoning attack, such as the Kaminsky attack, the RTT validation is easily defeated. Eventually spoofed packets will fall within the validation window. The IP TTL may be harder to overcome, but unless the legitimate resolver initializes it's IP TTL to 255 and is closer than the attacker to the destination, even this will be relatively easy to defeat. Granted, the authors do note that they limit their proposal to an on-path threat and the paper's title says as much, but even here their proposal only works based on how censoring networks are operating today. It would be easy for an on-path adversary to send multiple responses, one of which could satisfy the expected IP TTL and RTT values. Furthermore, this proposal appears ill designed for anything but stub or forwarder to resolver validation. Putting it on a large resolver and maintaining one or more RTT timers for what might be hundreds if not thousands of queries per second associated with literally thousands of the Internet's authoritative DNS servers does not seem likely to gain much acceptance from the DNS operational community. Even if scaling issues are overcome, the system can be easily defeated by the censoring networks. They could limit access to resolvers of their choosing where they might be able to apply censorship directly into the DNS. Ultimately, on-path adversaries could be motivated to migrate to an in-path mode if necessary, rendering these defenses completely ineffective.

Some may say DNSSEC is the answer and that may be partially true, but if overcoming censorship is the goal, which for these authors it appears to be a primary one, encryption, which DNSSEC doesn't provide, is likely needed. Others will also harangue on at length about IETF BCP 38 - Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing and IETF BCP 84 - Ingress Filtering for Multihomed Networks as if we can cajole all networks to limit IP address spoofing. This would ignore that censors are going to claim operational sovereignty to spoof within their own networks as well as the practical limits to deploying anti-spoofing measures throughout the entire Internet.

posted at 4:16 pm | permanent link

Code: Watch out for someone else's srand()

You may have heard me say, only half jokingly at best, "I don't know Perl, I know Combat Perl". When it comes to C I am even less confident. In that arena I'm prone to say, "I don't know C, I know Windmill C Technique". Picture the grade school kid whose only move against the playground bully is to closes his eyes, start his arms spinning wildly perpendicular to the ground like a human mowing machine while he nervously approaches his target. That's me writing C code. It was with some trepidation that I even post this blog entry, making all Windmill C coders look good.

In a recent battle involving a pcap parser I needed to implement a routine that "sampled" packets to help ease the processing load for the user when encountering large pcap files. A packet should be selected at random using a configurable predefined rate. The random process needn't be cryptographically secure, and I wasn't going to worry about bias already present in the order of packets in the pcap file, but my code ought to be statistically random enough to obtain a good cross section of packets in a typical pcap file. For example, instead of processing every single packet in a pcap file, the user should be able to process approximately one in every 100 packets, a reduction by two orders of magnitude.

My job seemed simple enough. First, setup a command line option to accept a sampling rate. Then when reading packets serially, perform a test using a random number with the given sampling rate to determine if the current packet should be processed or not. At the start of a packet processing routine, I implemented something that looks like this:

if ( sample_flag > 0 && rand() % sample_rate > 0 ) {

sample_flag is essentially a boolean. sample_rate is the user-defined sampling rate, which might be some sane integer value n where 1 < n < 231 - 1 (RAND_MAX in my stdlib.h is 231 - 1). Upon reaching the if conditional test, the rand() function returns a value modulo the sample rate if sampling is enabled. This value would be between zero and the sample rate minus one, inclusive. If the result of the modulo operation is not equal to zero, the packet is skipped, otherwise, only when the modulo result is zero will the packet be processed. Statistically speaking, the value zero will appear, or a packet will be processed, approximately one in every sample_rate times. This random sampling should be sufficient for my task. Again, it may not be cryptographically secure and it doesn't account for any bias in the packet capture itself, but it should be reasonably random across distinct runs of any pcap file.

Not so fast. When I tested this sampling code using any pcap file, the same exact sequence of random numbers was generated each time through. In other words, the so-called random sequence of numbers was the same sequence for each incarnation of the application. That certainly isn't what I expected. In order to use the rand() function effectively you must properly seed or initialize the random number generator using the srand() function. To my chagrin, I did this and was still getting what should have been a practically impossible duplication of random number sequences. I initialized the random number generator in the same fashion as one recently suggested in a Stack Overflow answer to "How to 'randomize()' random numbers in C(Linux)?":

FILE *urandom_fp = fopen( "/dev/urandom", "r" );
if ( urandom_fp == NULL ) {
    perror( "/dev/urandom" );

unsigned int seed;
int result = fread( &seed, sizeof(int), 1, urandom_fp );
if ( result != 1 ) {
    fprintf( stderr, "/dev/urandom read error\n" );

if ( fclose(urandom_fp) == EOF ) {
    perror( "/dev/urandom" );


Now again, this may not be the most cryptographically secure way to initialize the random number generator, but it ought to be more than adequate for my purposes. I was reading an integer's worth of data for the seed from /dev/urandom, a special non-blocking file that provides pseudo-random data and is found on the Unix system I'm working on. The seed essentially determines the sequence of the random numbers that rand() generates. If you don't seed the random number generator, the default seed value is 1. Unless you want perfectly predictable random numbers, which is usually only done in testing and measurement situations, you typically want to seed the random number generator with as random a number as you can get. Hence my call to read from urandom. Yet, while it looked like what I was doing should produce reasonably decent random number sequences across application invocations, I was getting entirely predictable sequences each time.

Recall that this coder codes using the Windmill Technique so it wasn't immediately obvious how I had seemingly implemented this sampling process incorrectly. After a few minutes of head scratching and rewriting the sampling routine in different ways to debug what was going on, it finally dawned on me to to see if srand() was being called somewhere else, after my own call to use urandom for the seed. I happened to be borrowing some code from the Bloom filter C library in the Bloom::Faster Perl module on CPAN. It didn't take me long to discover that bloom.c contained the following:


and in bloom.h this:

#define CONS 1234567

D'oh! Since I was calling the code in this Bloom filter library after I called srand() myself, the random number generator was being reinitialized using the static seed of 1234567 instead of the better, pseudo-random seed from urandom for each invocation of the application. I immediately removed the call to srand() from the Bloom library code I was using and got the randomness I originally expected.

For a fleeting moment I considered that perhaps I'm not as bad a C programmer as I thought. That idea quickly faded as I proceeded to Windmill my way through the rest of the application I was building. To be fair, the Bloom filter library I was adapting didn't require as good randomization as I needed and maybe the original implementation wanted to achieve some measure of reproducibility. Regardless, in the future I'm likely to be more cognizant of how my applications' random numbers are being generated even when at first glance they appear to have been seeded effectively.

For a good introduction to generating random data with pointers to other sources of information, including the challenges in obtaining random data for cryptographic applications, see Ferguson and Schneier's Practical Cryptography chapter entitled Generating Randomness. Note, I realize there is an updated version of the book with a new name, I just don't have it. The new version, Cryptography Engineering, appears to have retained the chapter with presumably equal or better content.

On a related note, sadly Google Code Search is reportedly shutting down on January 15, 2012. If you're reading this before then, you might want to hurry up and search for code to see how others have called srand(). If you do, you will notice that not all seeds are created equal. Is there code that calls rand(), but never srand()? Can you find predictable sequences that lead to unintended consequences?

posted at 5:33 pm | permanent link

Ops: TCP port 1024 and 3072 traffic

Have you ever wondered what all the unsolicited TCP SYN/ACK or RST packets to destination ports 1024 and 3072 are? Go watch those ports for awhile if you've just not noticed you're getting them (it is not constant, but I predict you will see some if you keep an eye out). For nearly 10 years many of my colleagues and I puzzled over them too. It was in August of 2000 when I first noticed the activity and sent off a report about it to the source network. They were nice enough to respond and told me their service had been attacked, indicating they were not purposely originating any malicious traffic towards the network I had been monitoring. In other words, I was reporting traffic that was nothing more than backscatter.

If you keep a careful watch over your network you may have undoubtedly seen what may at first look like TCP scans to those ports, often with the source port of a well known service such as IRC and a source address that maps back to a well known IRC server. For years, I occassionally watched as others would occasionally mention a situation that fit this very pattern. It was 2009 when I realized someone would ask about these signature packets once or twice a year. Yet, I never saw a satisfactory answer as to their ultimate origin. Then it came up internally at Team Cymru when our NOC noticed many packets with this pattern associated with some address space I was using for another project. I was asked about it and noted that this traffic pattern has been seen periodically for many years, but I didn't have a more specific answer than "backscatter". Knowing that there must be something specific that caused this particular pattern of packets, but not knowing what that was really bugged me. I asked on the Freenode IRC #dragon channel if anyone knew what tool was responsible. As a guess, jlc3 suggested perhaps a variant of an old DDoS tool named juno was responsible, but then he quickly recanted that suggestion after not seeing any reference to 3072 in its C source. juno is a relatively simple TCP SYN flooder. It spoofs the source address so that any SYN/ACK or RST response from the target will be reflected to random destinations. I looked at the source code anyway and noticed the following line:

syn->sport           = htons(1024 + (random() & 2048));

Eureka! This line was apparently originally intended to randomize the source port value, but in fact, rather than randomize the source port value to any 16-bit value, or perhaps even some more reasonably-looking set, it will only set the source port to a value of either 1024 or 3072 (in decimal). It will choose one randomly, but out of all possible source ports, this is a very small set to choose from and certainly not very random in the grand scheme of things.

Working from inside out beginning on the right side of the statement, this algorithm says to take a random number using random(), then AND it, using the & operator, with the static decimal value 2048. The random() function will generate a random long integer value. The AND operation will compare the corresponding bits of the random long integer value with the static integer value of 2048. 2048 written in binary, ones and zeros, is 100000000000. A bitwise AND operation will set the result to 1 only if both bits being compared are 1, otherwise the result is 0. When you AND any other binary number to an integer value of 2048, the result, as an integer, will only ever be 0 or 2048 since only the 12th bit from the right of the integer value 2048 is a 1. The algorithm takes that ANDed result then adds it to the static integer value of 1024. The resultant sum will only ever be an integer value of 1024 or 3072. Quite simply, the simple addition operation will be either 1024 + 0 or 1024 + 2048, giving a sum or source port of either 1024 or 3072.

The original and still surprisingly widely used juno code therefore generates packets where the source port is only ever those two values, matching the fingerprint we have been seeing for years. Much to my chagrin, only after the relationship between the source code of the juno attack tool and years of bewilderment did I happen to stumble upon a post at Web Hosting Talk from 2001 where the the relationship was already known. D'oh! Oh well. With any luck this blog post is easily and quickly found in a net search for those looking for an answer to a similar question many of us had for years.

Finally, I should note, not long after I first reported this finding to others in a different forum, and still unaware of the Web Hosting Talk post, Conor McGrath, who I first saw inquire publicly about this pattern, was happy and relieved to have a definitive answer, but he did leave me with one humbling thought, "One annoying mystery down, millions to go!". Indeed. :/

posted at 10:12 pm | permanent link

Ops: Deep Darknet Inspection - Part 3 of 3

In Deep Darknet Inspection - Part 1 and Deep Darknet Inspection - Part 2 we were able to determine not only the origin and nature of packets we received, but also postulate with a high degree of confidence, characteristics about the systems that sent them. Before completing the series, I need to correct something I said in part one. I had suggested that we would end the series with a mystery, but it turns out I solved it, or at least sufficiently so. The delay in posting this last segment is in part due to following a trail of clues I had thought were going to lead to a dead end. I still don't believe deep darknet inspection will always aid in uncovering the desired insight, but in this case I proved myself wrong. We begin by examining a set of TCP SYN scan packets that share some interesting characteristics:

TCP SYN packets towards a single destination IPv4 address in a darknet session spanning 36 hours:

Top 5 IP identification field values
% of total     id value

    12          0x0100
    <1          0xcdb4
    <1          0xccc2
    <1          0xaa03
    <1          0x4bff
Top 5 TCP source ports
% of total     source port

    7.5         12200
    5.6          6000
     <1          4369
     <1          3348
     <1          1558

Clearly, you can see an anomaly in the IP identification and TCP source port fields. In fact, based on the percentages, you might guess that the 6000 and 12200 source port SYN segments have their IP identification value set to 0x0100. In fact, this is true about 88% of the time. It is possible that some other random TCP SYN segments arriving at the darknet just happened to use one of those two source ports or perhaps the IP identification field was overridden by a middle box. What is true 100% of the time is that the advertised TCP window size is always 16384 for source port 6000 segments and 8192 for source port 12200 segments. If we examine just the TCP segments that have one of those anomalous properties (i.e. IP id = 0x0100, TCP source port = 6000 with an advertised window of 16384 or TCP source = 12200 with an advertised windows of 8192), do they have anything else in common? You know the answer is going to be yes, or I wouldn't be asking.

All of those selected TCP SYN segments are exactly 20 bytes in length. We know that a TCP header without options is 20 bytes. You might think there isn't anything so special about that, but in fact there is. What modern operating system's TCP/IP stack doesn't include any TCP options in the initial SYN segment by default? None that I know of. In my experience, there are relatively few modern systems that generate legitimate traffic matching that signature. Older systems (e.g. SunOS 4.1) and some VPN software are amongst the few that do. However, many tools that assemble TCP packets (e.g. port scanners) usually omit TCP options in the SYN segments they create. Normally these packets are created through the raw sockets interface. Raw sockets support is severely limited on modern Microsoft Windows systems, but is readily available on other systems.

As an aside, this 20-byte TCP SYN (and SYN flag only) with no options signature we stumbled upon makes a pretty good anomalous network traffic detector. Here is a tcpdump filter expression you can tailor to suite your needs that captures TCP segments with no options and only the SYN flag set:

'tcp[12] == 0x50 and tcp[13] == 0x02'

If you only have NetFlow data all is not lost. Watch for 40 byte TCP flows that have only the SYN flag set. What do you see? Can you easily filter out the false positives?

Lets get back to analyzing our anomalous SYN packets. The TTL field in the IP header might normally give us a clue as to the operating system, since many operating systems start with different default values. In fact, many of the TTL values we see here are around the 100 to 110 range, and judging from the path back, this would suggest they all started out at or around 120. That would be an uncommon default value for any OS. We might speculate that they all began at 128 since many Microsoft systems default to that, but is it likely we're consistently always 8 hops off in the reverse direction? It is possible, but unlikely with so many varied sources.

Determining the OS is actually going to be tricky with what we know thus far because we can assume not only is the TCP portion of the packet being assembled using raw sockets, but the IP header appears to be assembled with raw sockets as well. Why do we think this? Even discounting the discrepancy of the TTL values, the common IP id value of 0x0100 would be highly improbable if raw sockets weren't being used. All is not lost however.

From 2005, Jose Nazario's Dasher.C Now In the Wild blog post details an amazingly similar set of packet characteristics. Notice the constant source port of 6000, the constant IP id value of 256 (0x0100 in hex) and the initial TTL of 120. Aha! This seems like our culprit or at least a relative. A comment in a thread entitled "win_dasher" at Offensive Computing shows some additional disassembled output of Dasher's TCP Port Scanner module, the piece of the malware originally and apparently written by someone that goes by the moniker of WinEggDrop. We can easily find what looks like the original version of WinEggDrop's TCP SYN scanner source code on a Chinese web forum. Examining that source code we see a hard coded IP TTL of 120 and a hard coded TCP window size of 16384, just like we've seen in our source port 6000 scans:

ip_header.ttl = 120; 
tcp_header.th_win = htons(16384);

Have you noticed we haven't even bothered to characterize destination ports or source IP addresses yet? This might be what casual investigators look at first. It is often what generic statistical analysis and monitoring systems care most about. TCP packets with a source port of 6000 has been mentioned in a Internet Storm Center Diary entry. Likewise, TCP packets with a source port of 12200 has come up on a thread at Yet, both discussions end without a satisfactory explanation of what is going on. Maybe by doing "deep" darknet inspection we can get a bit closer to the ground truth. However, before we can pass judgment on what the data means we first want to better understand what the data actually is and what it is not.

There is something else peculiar about many of the source port 6000 packets. About 20 different sources are using either 622723072 or 1344798720 as the TCP initial sequence number (ISN). Looking back at WinEggDrop's original code we see:

dwSeq = 0x19831018 + nPort;   // Set A Sequence 
tcp_header.th_seq = htonl(dwSeq); // Syn Sequence 

428019736 (0x19831018) is nowhere near either of the ISNs we are seeing, but it helps support the hypothesis that we are seeing a derivative of the WinEggDrop's code. The source port 12200 scans appear to be another step removed, because they don't use common sequence numbers, but there is much less randomness than there ought to be. Perhaps those scans are from a later generation of the original code or a minor branch of an ancestor?

What you may want to know is what are all these Dasher / WinEggDrop TCP scanner derivatives doing? Judging by the frequency of source addresses (i.e. overwhelmingly Chinese) and destination ports (i.e. largely ports such as 1080, 7212, 8000 and 8080), they are most likely looking for open proxies. We probably didn't need deep darknet inspection to figure that out, but I for one have a much clearer picture of the nature and origin of the packets than we did when we started. Particularly in this case, when I thought I was going to end with more questions than answers, there is some personal gratification in knowing more now than when I started.

posted at 7:46 pm | permanent link

Ops: Deep Darknet Inspection - Part 2 of 3

In Deep Darknet Inspection - Part 1 we tried to unravel as much as we could out of a pair of TCP SYN packets. The result was a picture of what appeared to be a Conficker-infected host. By looking at IP and TCP header detail as well as packet timing intervals we were able to put forth a pretty reasonable profile as to the origin of those packets. Now, in part two, we consider a new pattern of packets from that darknet session - UDP packets arriving on destination port 5060, a well-known port assigned to the session initiation protocol (SIP). We'll hone in on 17 such packets received at a single destination IPv4 address over the course of 36 hours. All of these were SIP OPTIONS request messages with very similar content. The SIP payload of two such messages, with the original destination IP address in the "OPTIONS sip" field obfuscated to, are shown below:

SIP payload 1:
OPTIONS sip:100@ SIP/2.0
Via: SIP/2.0/UDP;branch=z9hG4bK-1047900479;rport
Content-Length: 0
From: "sipvicious"<sip:100@> tag=34353131313634613133633401353835363538393133
Accept: application/sdp
User-Agent: friendly-scanner
To: "sipvicious"<sip:100@>
Contact: sip:100@
Call-ID: 965192181434457134540520
Max-Forwards: 70
SIP payload 2:
OPTIONS sip:100@ SIP/2.0
Via: SIP/2.0/UDP;branch=z9hG4bK-17038962;rport
Content-Length: 0
From: "sipsscuser"<sip:100@> tag=77194458456158489721772408023096992174807017852
Accept: application/sdp
User-Agent: sundayddr
To: "sipscc"<sip:100@>
Contact: sip:100@
Call-ID: 124296019799750175018545983
Max-Forwards: 70

The SANS ISC 5060 port report and Team Cymru's Top 10 UDP ports graph both show greater than average activity for UDP port 5060. A veritable source of public information on emerging scanning trends can be found by watching the Internet Storm Center Diary hosted by SANS. In July, the Handler on Duty posted an entry entitled Targeting VoIP: Increase in SIP Connection on UDP port 5060 and you could subsequently find more information about the nature of these scans, which would undoubtedly lead you to the SIPVicious toolkit.

SIPVicious is widely used for SIP scanning today. Earlier this year the South West Anarchy Team (SWATeam) published swat0013.txt, a text describing how to use SIPVicious to locate open VoIP PBX systems that can be later tested for open extensions as a means to make free phone calls. Old school phreaks meet the new school phreaks.

The SIP content shown above might not seem too interesting, especially when an automated tool like SIPVicious is being used, but it might suit the pedantic to know more. In this case it helps to have some familiarity with the SIPVicious code and in particular the script. Some of the SIPVicious generated fields are based on local host system properties. These values include a local source address and source port number in the Via and Contact fields. SIPVicious, as of this writing, attempts to discover a local address, using a gethostbyname(gethostname()) call, for the Via and Contact fields. It also uses a hard coded IP address in other places.

If the system calls used to determine the local IP address fail, is used instead. This is noteworthy because we've seen other SIP scans from multiple sources that always use an IP address of in all fields except the OPTIONS field. In addition, unique identifiers in the From, User-Agent and To fields have been altered from their original SIPVicious counterparts. See the SIP payload 2 in the listing above for an example. It is unlikely those scanning hosts all happen to be behind a NAT using the same address so we can infer that there is some modified version of in the wild. This is a reasonable assumption since other parts of the SIP payload in those scans correspond perfectly to those fields generated by the SIPVicious code. It would be highly improbable for someone to come up with the same patterns independently.

The source ports shown above are also 5060, which, after examining the SIPVicious code, suggests that there are no locally running SIP listeners on the local sending system's UDP port 5060. However, we sometimes see other source ports used in these scans, usually some marginal value greater than 5060. This suggests a couple of scenarios. One, the source host already has a SIP listener in use on port 5060, maybe it is the same end host where both the scanning and subsequent follow up call attempts are being run from. Two, by considering the distance the source port value is from 5060, we can infer the approximate number of concurrent SIPVicious processes running on the end host at the time we received our scan packet. SIPVicious will increment, by one, the source port value until it finds an available local socket.

Believe it or not, there is more to the SIP scans if we consider the IP identification fields, not shown above. We talked about the IP id field in deep darknet inspection part one and implied that it often acts as a simple 16-bit packet counter. Of course, many systems do different things with different protocol fields, including the IP id field, as we are about to discover.

Out of the 17 SIP scans from this darknet session, all had a unique source IP address, but seven had an IP id field value of zero. Assuming all 17 packets were generated by independent origin hosts and any IP id value is roughly equally probable, the chance of that occurring naturally are astronomical. It turns out that Linux always sets the IP id field value to zero for UDP messages when the don't fragment bit in the IP header is also set, which is typically the case here. CVE-2002-0510 has been assigned to this behavior, but it was not considered an issue worth changing the IP stack implementation for. What about those 10 scans that did not have an IP id field value of zero? It is likely many of those systems are not Linux. Since SIPVicious is written in Python, a likely platform in those cases are a BSD-based system. It is also possible that source hosts responsible for those 10 scans were behind some sort of middle box, such as a NAT, that rewrote the IP id field to a seemingly random value. DNS PTR queries, passive DNS data, OS fingerprinting techniques, examining the IP TTL field, checking if the DF bit is set and other tricks might help characterize each of those further, but we'll leave those ideas as an exercise for the reader to ponder.

Here again, with a deeper look into darknet packets we were able to uncover not only the nature of the packets seen at a darknet, but we were also able to better profile systems sourcing the traffic. In this case we were lucky to have access to the source code used to generate these packets. Not surprisingly, we won't always be so fortunate as we will soon see in part three.

posted at 4:19 pm | permanent link

Ops: Deep Darknet Inspection - Part 1 of 3

Darknets can be used to help provide valuable network and security insight with little to no risk for the darknet operator. Examining packets sent to unused IP addresses may highlight new threat vectors, a misconfiguration, information leaks and various types of Internet backscatter such as active denial-of-service activity. Team Cymru uses darknets for statistical data gathering and reporting, but from time to time I like to take a deeper look into the raw content that an average, publicly accessible Internet host is exposed to. Doing so can help you get closer to the ground truth of what is ultimately responsible for those packets and in a geeky sort of way, that can be a fun and educational thing to blog about. In the first of a three-part series running over the course of the next few days, here is the relevant summary detail of two packets seen at a darknet recently in slightly modified Wireshark display notation:

delta     saddr           IP_id  sport  dport  info

0.000000  32569  21269  445    TCP SYN Win=65535 MSS=1410 WS=1 TSV=0 TSER=0  
2.968866  32638  21269  445    TCP SYN Win=65535 MSS=1410 WS=1 TSV=0 TSER=0

The source IP address is purportedly originating from Indonesia and appears to be allocated specifically for mobile or cellular data usage. The destination port is 445, widely used by some Microsoft Windows systems for SMB over TCP and a common target for a number of worms and malware. In fact, according to the SANS Internet Storm Center Top 10 Reports as of this writing, port 445 leads all others by a comfortable margin. Even when considering Team Cymru's own Top 10 TCP ports graph, which is all sampled Internet traffic we see, port 445 is near the top. However, in this exercise we are doing deep darknet inspection. What more can we say, with some level of confidence, about these two packets? Perhaps more than what appears on the surface.

The IP identification field's primary and original purpose is to aid IP datagram reassembly in the face of fragmentation. For many systems however, the originating host simply treats this value as a per packet counter, incrementing the IP id field value by one for each datagram transmitted. Its possible that a middle box would re-write this field, however the distance between the first and second value is relatively small, a difference of only 69. This suggests that the identification field is being used as a counter and that there have been a handful of packets sent to other hosts between the two we received. Its possible that many of those intervening packets were SYN scans, about 20 per second, to other hosts. With today's modern connectivity options that is not a high rate of traffic, but it should be easy to identify as anomalous at the source network. Time to look higher up the stack.

We can assume the second packet is a retransmission since it has the same TCP characteristics as the first and it arrived approximately 3 seconds later, which is the default retransmission time for many operating systems. However, no more packets were received from that host for this session, which is odd because most systems default to sending at least two TCP connection retransmissions attempts. Conficker/Conflicker/Downadup as seen from the UCSD Network Telescope identified this seemingly unique worm behavior last year. So there was probably some version of Microsoft Windows running at that address infected with Conficker. Lets dig deeper.

The maximum segment size (MSS) TCP option from this host was set to a value of 1410 bytes. Normally this value might be set to 1460, if at all, which allows for 1460 bytes of payload in a TCP/IP packet without options on top of a typical Ethernet frame. While this value can vary, 1410 is a sometimes used by PPPoE-connected hosts. Lets be thorough data packet archaeologists and see what else we can discover.

The window scale and time stamp options are set in the SYN segments. These are not typically seen from Windows clients except for more recent versions such as Vista or Microsoft Windows Server 2008. Based on the route origin block and detail thus far, I'd say chances are good that the original host was a consumer-oriented Vista box, but we're not done yet.

Consider the source port value of 21269. At first glance this is just a random looking ephemeral source port. However, earlier versions of Microsoft Windows machines used a default ephemeral port range of 1024 to 4999, while newer systems such as Vista and Windows 2008 server use 49152 to 65535 by default. Since 21269 is outside of both ranges, it seems likely the original host has gone through some sort of middle box, maybe a NAT-PT gateway.

In summary, we've identified what appears to be a transient Microsoft Vista host infected with Conficker connected to the Internet with PPPoE and traversing some sort of middle box from Indonesia. That is a fair bit of insight from just a couple of TCP packets.

Given even just a small number of packets we can often infer a lot. We might not be able to guarantee a precise analysis, but with continuous evidence gathering and some practice, just like any good scientist, we evaluate our hypotheses and hone our results. I tend to believe there is some value in better understanding the packets our darknets receive rather than just lumping them together as protocol or port noise. It certainly doesn't hurt to be able to explain with more clarity to a colleague, reporter, student or supervisor what all the noise is about. In part two on our deep darknet inspection expedition, we look for answers not just in packets, but in source code. In part three, we'll wrap up our journey with more insight, but end with a mystery, highlighting the limitations of using darknets to fully comprehend the origin and nature of some packets that arrive at a darknet. Til then bit mechanics, may all your evil bits be 0x0.

posted at 4:08 pm | permanent link

Code: Parsing the DNS-based IP address to route mapping service

For me, writing code usually means writing combat Perl code. Its my standard joke, but also my standard disclaimer. When I recently used C for another project I included an addendum to my standard disclaimer. In that case, it went like this, "I wouldn't advise running this as root". Yet, being able to construct tools with your own code is a tremendously useful skill to have and I frequently urge my networking undergrad students to attain some competency in tool building. Being able to parse and summarize logs is a great first tool to attempt to build. In Perl mine was an ISC BIND named log parsing and summarization tool called Its an abomination, but it served its purpose and still seems to work well enough for when I need it that I haven't bothered to redo it. I've since gone on to build a number of tools with Perl, many of the parsing and summarization variety, but hopefully each successive incarnation a little better than the last. The one Perl book that has helped me take a little of the combat out of my more recent code for which I love to recommend to other Perl coders has been Damien Conway's Perl Best Practices. Perl is what I tend to reach for first, mainly out of habit. Regardless of your preferred weaponry for combat coding, lets just agree that being able to build useful tools is a great thing and instead discuss what it takes to build a tool around a service a number of us have used for many years.

If you're like me, you make regular use of the Team Cymru IP address to BGP route mapping service. I tend to use the whois-based service interface for queries that I do by hand and the DNS-based service interface in code. The Route Views Project offers a similar DNS-based service, but it is not as widely known nor does it have the registry-associated data that can be handy when trying to uncover some quick insight about an address. However, parsing the Team Cymru DNS-based service can be a bit tricky, something I hope this post provides some insight into if not a few good laughs at my code along the way.

Our task here is a seemingly simple one: pass an IP address to a subroutine and get back an autonomous system number (ASN). Here is how the start of such a Perl subroutine might look:

1. sub get_asn {
2.     my $address = shift || return;
3.     my $res     = Net::DNS::Resolver->new;
4.     my $qname   = get_ptr_name($address);
5.     my $query   = $res->send( $qname, 'TXT', 'IN' );
6.     my $asn;
8.     return if !$query;
9.     return if $query->header->ancount < 1;

The routine above expects a scalar value parameter, an IPv4 or IPv6 address, and assigns it to the $address variable in line 2. We set up a DNS query by using the Net::DNS module in line 3 to create a new resolver object. In line 4 we need the reverse or PTR name that will be used in the query so we pass the address to a utility function called get_ptr_name() and expect the appropriate query name back and assign it to the $qname variable. We are then ready to send the query and attempt to do so at line 5. If the query fails or no answer data is returned, we abruptly leave the subroutine at line 7 or 8 respectively. At any time we leave the subroutine early we will return with an undefined value, so it will be up to the caller to handle such a condition gracefully.

As an aside, let us take a quick look at the get_ptr_name() utility routine and see what it might do:

a.    sub get_ptr_name {
b.        my $addr = shift || return;
d.        if ( $addr =~ /:/ ) {
e.            $addr  = substr new Net::IP ($addr)->reverse_ip, 0, -10;
f.            $addr .= '';
g.        }
h.        else {
i.            $addr  = join( '.', reverse split( /\./, $addr ) );
j.            $addr .=  '';
k.        }
m.        return $addr;
n.    }

This subroutine uses a simple regular expression to test for an IPv6 address. If a colon (':') character is found in the address string, the address is presumed to be an IPv6 address, otherwise it must be an IPv4 address. In line e. we use the power of CPAN and the Net::IP module to get the reverse nibbles for an IPv6 address, because doing so by hand is a pita. However, in that case we also must strip off the trailing zone the module includes by default and in its place append the Team Cymru IPv6 route origin zone. We perform a similar, but simpler transformation on an IPv4 address and return the final result.

Presuming everything has gone well up to this point, we want to process a DNS answer we get back. What does an answer look like? This is where things can get a little hairy. We might get multiple answers and there may be multiple ASNs listed in each answer. In DNS-speak, here is what the general format of the RDATA in an RRset will look like:

    "49152 [...] | | AA | registry | 1970-01-01"

There are five fields per answer, separated by a pipe ('|') symbol. The first field is an ASN list. Often it will be a single ASN, but due to multiple origin autonomous system (MOAS) routes there may be more separated by whitespace. The second field is the covering route prefix. The third field is a two-letter country-code based on IP address registry allocation information. The fourth field is the registry responsible for the address allocation. The fifth and final field is the date the registry allocated the covering prefix. If there is a route, you should get an answer and at least one ASN and prefix. Beyond that, you should code defensively. Most of the time you get a single answer and a single ASN, but don't count on it. In our case, we won't care about more specific prefixes nor multiple ASNs. Continuing on then...

11.     for my $answer ( $query->answer ) {
12.         next ANSWER if $answer->type ne 'TXT';
13.         ($asn) = $answer->rdatastr =~ m{ \A ["] (\d+) }xms;
14.         $asn ? last ANSWER : next ANSWER;
15.     }
17.     return $asn;
18. }

We conclude our simple ASN mapping routine by finding the first TXT RR in the set and capturing the first ASN in that RR before returning it to the caller. Keep in mind that this routine is very simple and likely not suitable for any truly robust project where you care about multi-homing, MOAS, different covering prefix announcements, upstream routes or a descriptive name for an ASN. Constructing code that deals with those situations is probably more appropriate for a library than a blog post (not a bad idea eh?). The Net::Abuse::Utils module contains a subroutine called get_asn_info() which uses our mapping service and goes a little further than I show here. I wrapped the routines above into a small script called which you may freely use and expand on for your projects. It will take a list of IPv4 or IPv6 addresses, one per line via STDIN, and give back a pipe-delimited list of the first associated ASN it finds or 'NA' if none. Go forth and do battle.

posted at 4:56 pm | permanent link

Ops: Port filtering

For many years I've resisted and lobbied against implementing most every middlebox in a network I've had administrative influence over. I've been especially adamant when the proposed middlebox would reside at large aggregation points, such as the so-called "border" of large organizations. I've found these middleboxes to be short term hacks for more fundamental problems that could be better addressed another way. While arguably middleboxes fix otherwise difficult to solve problems and many people like them, they often come at great expense. That expense is not only felt financially, but architecturally in the erosion of hard to quantify, but intuitively useful network properties such as flexibility, simplicity and transparency.

One consequence of a common middlebox function is when a perfectly legitimate packet is prohibited as a result of an unlucky combination of bits in said packet. You may often see this in the form of a rudimentary router packet filter mechanism or a more sophisticated firewall filter rule system. By way of example, here is a situation where an innocuous attempt to resolve to an IPv4 address will fail because of an unlucky set of bits that are not permitted due to a UDP port filter between Cisco's network and the rest of the Internet:

dig -b0.0.0.0#1434

Altering the command slightly, the following query will succeed:

dig -b0.0.0.0#1435

In the first query attempt a source port value of 1434 is specified. In the second, 1435 is used. Why does 1434 not work? Many years ago there was a nasty computer worm commonly referred to as Slammer that took advantage of a remotely exploitable Microsoft SQL Server vulnerability over UDP port 1434. Many operators applied port 1434 filters to stem its spread. The worm outbreak began in early 2003 and it was widely reported that virtually all the Internet-accessible vulnerable systems, approximately 75,000 of them, were infected within the first 10 minutes. The artifacts of the 1434 filter persist to this day.

While there are still remnants of Slammer infected hosts, our insight indicates that on average there are only a few hundred unique source addresses per day showing signs of an infection. I recently sampled the extent of the port 1434 filter by issuing UDP DNS queries for all the nameservers for each domain in the .edu zone using a source port of 1434. Approximately 11% of the unique name servers used by .edu's failed to respond. Moreover, of those that filter UDP port 1434 messages, approximately 37% also filter TCP port 1434 segments.

Do note, while I tested port filters using DNS queries to DNS servers, the likelihood of the filters being applied only to DNS queries and DNS servers is highly improbable. Its reasonable to assume that these filters apply to all applications between the tested networks and the greater Internet indiscriminate of intent. DNS service was simply a convenient application for which I could test. Ephemeral source port selection is a bit of a gamble. Its difficult for endpoints to know when middleboxes are present or to know what unlucky combination of bits will result in a packet being filtered. Certainly there are other applications and ports that share the same fate as 1434. Examining them in detail may be the subject for a future post.

A typical security practitioner might ask what the big deal is. Even if source port selection was uniform, which it isn't, the chance that an unlucky source port will be selected is low and even then, some applications gracefully recover without user intervention. You can probably guess my feeling toward port filtering as generalized security tool is that it often lacks rigor and imagination, but I don't expect to change any minds or port filters with this post. Having gotten anyone to read this much about it is enough.

posted at 9:38 pm | permanent link

Paper: End-to-End Arguments in System Design

Those of you who hang out with me on a restricted mailing list may have seen me post short summaries of research-oriented papers that cover some aspect of internetworking or information security. I was recently asked about this, because its been awhile since I've done it. Since its not the first time I've received positive feedback about these summaries, I'm going to open my new blog with a review of one of the most enduring and important papers in all of computer communications.

J.H. Saltzer, D.P. Reed and D.D. Clark, End-to-End Arguments in System Design, ACM Transactions on Computer Systems (TOCS), Volume 2, Issue 4, November 1984. [pdf]

The e2e paper wants designers to consider the proper placement of functionality in a communications system. Its argues for placing functionality upwards, in a layered protocol model and outwards, towards the application that uses the functionality. However, it does not say that functionality must not exist in lower layers. In some cases functionality at lower layers may be appropriate and worthwhile. All too often the e2e argument is trotted out as careless dogma in debates amongst protocol, system, network or security designers to support a stance on how to best implement a system. I sometimes wonder if these people have even read the paper all the way through. I'm sure some have never read it and equate what is in it with a flawed set of associated ideas and statements they see on mailing lists and in other forums.

I teach a variety of computer networking and security courses at DePaul University. Inevitably I will briefly discuss the end-to-end argument. I attempt to convey as best I am able the main ideas of the paper and encourage my students to carefully consider its implications. I always feel a bit squeamish in doing so. On occasion I read something that David Reed, one of the paper's original authors, has written on the e2e argument. Sometimes I have found myself coming to a conclusion that is confirmed by what he's written. Other times my notion of what the e2e argument says or how it is applied will be challenged, making me feel less confident about my ability to have fully grasped the paper's message. I may not be the most enlightened thinker in the world of communication systems, but if this happens to someone like me who's been doing this stuff for many years, surely others might fail to fully grasp its message also.

I've recently been referring back to a sentence in the paper which I think highlights a key point of the argument, "A great deal of information about system implementation is needed to make this choice intelligently." For me, this says a lot about how one must correctly interpret the paper. It is the only place in the paper where a form of the word "intelligent" is used and it implicates the person doing the implementing rather than a system component. Often, and I've done this myself, people interpret the e2e argument as saying something about the stupid network, but that isn't quite right.

The e2e argument does suggest a set of guiding principles that may lead to specific designs, but rather than consider it as a set of commandments, remember it is an argument. Its not only a must-read paper, but its a must-re-read paper. If you haven't read it recently, spend the time to do so. It says a lot, but maybe not what you thought it did.

posted at 7:47 pm | permanent link