Duct-tape and bubblegum networking

In August 2008, Utah State University flipped the switch on new infrastructure that completely and radically changed the way that USU’s network had been run for decades.

The original was a classic legacy system—archaic, unmaintainable, and unintuitive systems that had been hacked together over the years, constantly frustrating students, staff, faculty. A system that the expanding needs of the University was quickly outgrowing.

I was the technical and project lead over the open source, in-house re-creation of USU’s network registration stack. This included a bottom-up replacement of Utah State University’s DNS, DHCP, IP address management, and computer registration infrastructure. Accomplishing this was my first professional opportunity to be over such a challenging and multifaceted problem that affected tens of thousands of people.

See the open source project at openipam.googlecode.com.

The resulting products were: openIPAM, an open source IP address management solution built in-house; and Bluezone, an internal USU site where everyone at the university now registers their computers to use the campus network. I will explain here the glaring legacy problems our team was faced with, how we overcame them, and some of the major lessons I learned from this project.

Problems

We all look back on the old system and laugh now—it’s incredible to see the amount of frustration we just accepted as “normal” for so long. Below are some of the problems that plagued the old system, which I’m happy to say were all fixed. This is a critical examination of the drastic positive change we were able to accomplish by starting from the ground up.

Disparate systems

This was truly duct-tape and bubblegum networking—amongst the havoc, we could say that there were four systems:

• Guest Wireless Access
• Wired Access for Housing Residents
• Wireless Access for Faculty/Students
• Wired Access for Faculty/Students.

Yes, that’s right—the registration for being on the wireless was a completely different system than LAN registrations (and the wireless had two campus-spanning network SSIDs: one for guests, and one for everyone else). There were also two separate captive portals in place: one for guests and one for unregistered computers. On top of that, the guest ticket system only worked on the wireless and there was no easy way for visitors to use the LAN.

It was a mess.

Since we created the new system from the ground up we simply left out the cruft of multiple networks, captive portals, registrations portals, etc. Now, when an unknown device is connected to USU’s network, the user is automatically redirected to a single site, the USU Bluezone. With a login and two clicks any device can be registered instantly. Sounds simple—because it is, and always should be. The magic part is that there really is only one system in the end—openIPAM itself, which provides a webservice API that Bluezone calls behind the scenes to keep a single authoritative source of information.

Unintuitive interfaces abound

The old interfaces looked something like this:

And yes—of course—every student, staff and faculty member made sure to methodically read all of the text every time.

I spent innumerable hours ensuring that the new interfaces were kept intuitive and straightforward, and fighting off those who wanted to clutter it up with unnecessary links and explanatory text (I follow Steve Krug’s philosophy: “Happy Talk Must Die”). They all went through an iterative process of change—for example, here is how the Bluezone homepage progressed, starting from its initial form:

Asking 30,000 people to understand MAC addresses and DHCP

Probably the worst part about the old system was that we required everyone who registered a computer to understand MAC addresses. Everyone—including all students—couldn’t just register a computer; they had to run command line programs or dig through preference panes. We had pages and pages of instructions scattered throughout University websites that gave steps for finding out MAC addresses on all major operating systems. Given the diversity and sheer number of the student body it’s no wonder that the IT Service Desk was constantly inundated with people calling who wanted to register their computers, but couldn’t figure it out.

In the new system, we eliminated 90% of the need for Service Desk customer support of campus network registrations—down from nearly a thousand calls and walk-ins during peak registration weeks to around a hundred. A drastic and welcome change for those fielding the calls, especially when the beginning of each semester comes around.

Flat text files and The Incredible Machine

Just imagine this: in the backend, USU’s entire IPv4 class B address space was managed through flat text files (that’s managing 65,536 routable IP addresses, not to mention more than that amount in non-routable addresses). Flat text files. One more time: flat text files. Every DNS record, every computer registration, all stored with crazy wrapper scripts you’d have to run to edit them (for important things like locking against concurrent edits). Complete with embedded special meanings and syntax for the comments—which were parsed after save to provide information to the DHCP servers.

Now comes my favorite part: after the files were saved, the new data would be sent over to the bind servers and then a trigger file would be created on the server. Every so often the server would see if this file existed and, if it did, would delete the file and then reboot itself to apply the changes. If you ever played the old classic game called The Incredible Machine, you’ll see why I think that game is a perfect metaphor for USU’s networking at the time.

We created the openIPAM infrastructure to be extremely modular, including central, redundant PostgreSQL databases, which serve data to root nameservers, DHCP servers, and the openIPAM backends.

openIPAM’s modular infrastructure:

Extraordinarily long wait times

After spending hours muddling through registering their computers, students would then have to wait until the DHCP servers would restart themselves at 10 minutes before each hour for their registration to become active. Static registrations for servers and any other wired LAN device took “one business day” to be manually processed by a member of the Networking Team and placed into the bind flat files.

You can imagine the headache that this caused network managers across campus who were constantly bringing up new server infrastructure and managing client computers, not to mention the pain that students had to go through. Now, anyone can have a computer running within seconds of connecting to USU’s network, with granular permissions distributed out to network managers so that servers can be brought up just as easily.

Security

Who’s in charge? Since we were storing all of USU’s computer registration data in text files, no contact information could truly be trusted because so much of it was out of date. When the USU Security Team found a vulnerable or compromised computer that was hurting the network, often the only way to contact the network admin in charge was to shut off that computer’s network access and wait until someone complained.

Compromised guest computers: Because of the convoluted way that guest registrations were implemented, it was almost impossible for the Security Team to monitor malicious activity from guest computers. This was obviously a Very Bad Thing considering how many compromised guest computers came to campus every day.

Proxy servers: This was one of the greatest social challenges we faced because the proxy servers gave people a false sense of security. It had become ingrained into the USU community that registering computers on private, non-routable addresses and then connecting to the Internet through a proxy would somehow reduce their security footprint.

We solved the problem of contact information by requiring that every host has at least one up-to-date contact associated to it, and by creating a host expiration system. No more nebulously owned hosts that never expire—every computer was directly tied to a person and would expire if they didn’t take care of it.

For guest computers, we decided to treat guest registrations in openIPAM like every other computer registration (rather than wrapping them behind any kind of proxy or captive portal once they had authenticated). In terms of security, this put guest computers on the same playing field as every other computer, and made them have to follow the same security rules and deal with the same consequences.

The proxy server problem took a bit of prodding and generated fiery discussions, but the USU IT community quickly understood and adapted to the change in thinking.

Long-term Scalability

One of the major reasons for creating a new system was the concern over address space usage. The old system had no procedure and no facility to expire registrations. This caused a huge waste of space because there were many registrations that hadn’t touched USU’s network in years—some hadn’t been used in over a decade. We were quite literally running out of space; there were only 8,499 routable IPs left to allocate, out of the 65,536 routable IPv4 addresses in USU’s class B subnet. There were nearly 30,000 expired IP address allocations and hundreds of thousands of expired DNS records. That’s a lot of cruft.

In the new system, we imported everything and then only kept what people claimed. This was quite a challenge, both socially and technically, but saved the campus down the road when more IPv4 addresses were needed. We also required expiration dates on all network registrations, ensuring that contact information and registrations were kept up-to-date.

Big lessons learned

Solve the 90% case

A constant and pervasive theme throughout this year of my life was asking the question, “What is going to happen here 90% of the time?” As simple as it is, to constantly ask this question was probably the most important thing that I learned in this entire project. It helped us avoid feature creep, evade the trap of spending exorbitant amounts of time on edge-cases, instantly provided the best defaults for any situation, and gave the proper excuse for moving past a problem. Make the 90% case 100% intuitive, not surprising, and simple—your users will thank you.

And, in the same vein, constantly re-evaluate, reflect, and prioritize. Yes, you might be solving a 90% case now, but you might be solving the tiny one, not the important one.

It’s amazing that the Internet even works at all

Seriously.

When you really dive into networking, you gain a new understanding and acceptance of unplanned outages, and a new mind for security. There are so many pieces that have to be working perfectly, all over the world, for you to be able to read this website right now—and so many of them are built on implicit trust. For example, let see how ARP works: imagine shouting into a crowd of people saying, “Hey! Does anybody know who Bob is?” Alice then responds, “That’s me!” And you believe her. For a very long time. Much of networking is build on this kind of trust. This project gave me a great appreciation for network security and how incredibly easy it can be to compromise or overload a network. The most amazing thing is how little these attacks happen, relative to how easy they really are.

So why does it keep working most of the time? Simple: incredibly talented people. System admins and network managers who know what they’re doing, never stop learning, and who spend inordinate amounts of overtime fixing things when they break.

Program with an open source mindset

I found that when I stopped thinking about programming in a USU-specific way—creating a product that would only ever be able to work internally—my code became much more modular, maintainable, and ultimately usable by others. This is a powerful way to think, especially when you realize that the organization you’re programming for today will be a different organization tomorrow. Keeping an open source mindset helps your creation scale to fit the needs of the future.

Test, test, test.

We learned this lesson the hard way. Let’s just say, there were some very scary hours and days during our implementation that I prefer never to revisit. The important lesson: thoroughly test everything, especially when dealing with something as crucial as networking connectivity.

Partially due to lacking of testing, and partially due to the inherent nature of dynamic DNS—USU’s network began to experience a huge amount of network churn in the days following implementation.

This chart shows information about ARP entries in USU’s core switches. The X-axis of the chart represents time, and you can clearly see the number of ARP entries more than double the day we went live. This churn has calmed down significantly over time as we have refined the code and found more efficient methods of handling DHCP requests. The jump in ARP entries is not indicative of a resounding problem on its own, but the graph is actually a pretty good visual indicator of our stress levels during that time. Network connectivity began to break for a small subsection of campus (following Murphy’s Law, it was some of the most critical and vocal departments on campus) due to some hastily tested, but critical, code. It was the kind of stress I won’t let myself experience again, and the kind that could have been avoided by more extensive testing.

People first

Communication is key. Establish trust long before coding standards.

This project solidified my views on programming: in the end, it’s all about people. And, people are far more complex than programming. It is imperative, an absolute must, to spend time learning to communicate with and trust each other. Programming is a means to connect with people and help them connect with each other; it’s not about the code. If you get wrapped up in only thinking about the low-level details, you’ll completely lose track of the high-level perspective.

And, when I say “people first,” I also learned that you have to take care of yourself as well. Work can be all-consuming, if you let it. It can become your whole world, but balance is where true inspiration lies.

Users honestly appreciate intuitive UI design

One final story: a group of students walked into the Service Desk one day, preparing themselves for the same pain they experienced every semester trying to register their computers. When they saw how easy the new system was and how quickly they could be online, they all started clapping and cheering right there in the Service Desk. That made me smile.

We knew that we were not alone…

Many universities and businesses were in the same boat we had just jumped out of—holding on to an aging, difficult to use IPAM system they were looking to replace. So, we released openIPAM as an open source project for anyone to use freely and contribute to.

Leading this transition for Utah State University was a challenging and irreplaceable experience which helped me grow immensely as a programmer and as a person. It was a radical change for its time and place, which hopefully will grow with USU and others as the years go on, helping us become more and more easily connected with each other.