Slow Software: The Case for High-latency Systems Development
Systems research prizes efficiency and performance. For the last decade, I’ve worked on ultra-low-latency systems; the goal of Demikernel and its related nanosecond-scale systems (e.g., Cornflakes, Capybara) is to do more with fewer cycles. AI coding has brought the same level of efficiency to building systems, letting fewer programmers write more code. However, it has also decoupled the speed of development from the importance of the software, leading to more frequent failures with a “wide blast radius”. These problems will only get worse unless we find a way for slow software to enforce thoughtful design and development for our most critical systems.
Background: Naturally Slow Systems
Until now, systems building has always been time-consuming. First, low-level systems development is tied to hardware, which must be physically built and deployed. Next, deployment of systems software can be slow; it often required software to be physically delivered and installed or for the entire system to be rebooted or replaced. Finally, building large systems using basic tools – usually assembly or C – meant that implementation complexity limited development speed.
These barriers imposed a rough correlation between a system’s development pace and importance. The more software relied on a system, the more effort was required to implement and update it. Likewise, the larger the blast radius of a system failure, the more time naturally was spent building the system and making sure that it did not fail.
In the last 20 years, these barriers have been systematically removed. Hyperscalars have been building datacenters at a faster rate than ever and continuously deploy new hardware. Most software runs on a centralized cloud which enables weekly or even more frequent updates. AI, while not perfect, has made it vastly faster to write systems code without using basic tools. Over time, the importance of a piece of software has been completely decoupled from the time to develop it.
This decoupling lets people treat systems software like any other software. The same push to use AI is being applied equally across the board, to our testing infrastructure, network routing, datacenter management and operating systems. Recent work has highlighted how to use AI to improve system performance without understanding the entire system or implementing the optimizations. Programmers are rapidly making changes without understanding the system or the guarantees that we use to reason about underlying infrastructure.
While this approach works well for a lot of software, it has implications for systems building. As a concrete example, distributed systems depend on consistency guarantees provided by an underlying storage layer. However, storage systems today commonly have complex weak consistency models that are not well documented and may not be correctly understood by AI models without adding the storage system’s entire code base to the context (which might not be available or is constantly changing itself). As a result, the AI model may change some code that violates the consistency assumptions and the programmer may not find it unless she knows where to look.
This attitude has led to predictable side effects. Inevitably, something will break (because neither humans nor AI are perfect). Only then does the difference between systems software and other software become apparent, as the wide blast radius takes down many other systems. As an analogy, we all use real world products that are made quickly and cheaply (e.g., processed food, IKEA). However, we also want some things that are not (e.g., fruits and vegetables, roads and bridges). Software is exactly the same; we do not have a good way to distinguish important infrastructure systems from other software until something goes wrong (e.g., health problems arise, the bridge collapses).
The Slow Software Movement
We need to re-establish some of the coupling between the importance of a software system and the time to develop it. People need time to consider their design decisions and forcing them to move more slowly imposes this time. Thus I propose the slow software movement, akin to the slow food movement. The goal is not to slow down all software development but to identify the important infrastructure software, where the slower speed of development is a feature, not bug.
More concretely, the more dependencies on a piece of software, the more effort should be put into developing it. And the more impactful the failure, the more effort should be put into avoiding failures. Experienced programmers can make this distinction. Mark Russ recently talked about how he knows what code he needs to read and doesn’t, and argued for a mentorship program to impart the same experience on others. However, I believe this doesn’t necessarily need to be entirely learned from experience, but we could build something into the development environment to help.
Not only do new programmers need to spend time thinking about these things and learning, experienced programmers should as well. And it should be clear which software requires more thought and what does not. In lieu of natural impedence, we could perhaps introduce artificial barriers to push programmers to have more skin in the game, so to speak, when they decide to implement something important. They will eventually need to invest time in maintaining the software and making it faster to write up front means more pain later when it breaks, especially for younger programmers.
Note that I am not saying that we should not use AI to write code. It is clearly useful for quickly prototyping ideas or writing tedious or repetitive code. If I am building a new system, I should not need an entire team of engineers to build compilation scripts, testing infrastructure or evaluation frameworks. If I want to try different caching algorithms, I shouldn’t have to implement all of them by hand. However, this code has a very small blast radius when it fails, so it should be developed faster. On the other hand, faster systems building has costs and those costs should be apparent when writing the code, not only when fixing it later.
Systems Research Opportunities
What does this mean for systems research? It has always been important to think about the motivation behind any system we build. In the past, it was expensive to prototype an idea, so almost any idea that was worth building was worth publishing (e.g., the first SOSPs). Now, that barrier has been removed because it has become extremely cheap to implement a prototype, and, as a result, our conferences are being flooded with poorly considered submissions. Instead of students and researchers thinking hard about whether an idea is good before implementing it, they simply implement it and submit it, leaving it up to the reviewers to consider whether the idea will be a useful contribution.
Unfortunately, this is like trying to guess whether an architectural model of a building will stand up or last decades under less than ideal weather conditions. Experienced reviewers that have been building and maintaining systems for decades can often make a judgment but it moves the hard part of the research from students to reviewers, and will rapidly burn out experienced reviewers. We need some kind of throttling function – previously imposed by the difficulty of prototyping – to reinforce the need for systems research to be done at a slower more thoughtful pace.
We should not blame or ignore the factors that have accelerated the pace of systems research in the last decade, but we should not lean into it either. Instead, we need to focus on research to re-establish best practices for systems development. We should actively try to figure out how to judge the practicality and maintainability of systems, along with performance and generality, and force paper writers to justify their decisions on those axes. While I don’t have all the answers, the systems community has adapted in the past and I trust them to do it again.