Modern software development is sometimes colourfully described as being similar to firing tracer bullets at a target. Rather than spending time doing a lot of research, design and specification up front, the smallest, simplest version of the software is built and the feedback gathered from its use is used to improve the next version. Being able to continuously integrate, test and deploy software means that we can make this feedback loop incredibly tight but because the same engineers are developing the software and the tests we constantly have to think carefully about the testing investments we make and their opportunity costs.
Software engineers allow themselves to use knowledge of how the software works to pick good white box tests. They rely on code reviewers to catch important test cases that they’ve missed. They rely on unit tests to exercise pathological cases which are hard to exercise in end-to-end or integration tests. They use bugs discovered in previous releases to inform tests cases developed for the next. The goal is to ensure that the next version of the software provides feedback on how close it is to the target rather than just exploding on impact.
The decisions around which tests to write are hard because tests have costs. They take time to write and maintain and need to be changed and updated along with the software they test. What would happen if some or all of the tests could be free?
This question has tantalised computer scientists since Turing’s work on the halting problem in the 1930s and while academic progress has been made, tools based on that work have tended to either only work on small codebases; perform trivial analysis; generate a high percentage of false positives; produce difficult to understand reports or impose other overheads on the development process that make the resulting testing pretty far from free. As a result while we theoretically know how to do sophisticated analysis, in practice we tend to rely mostly on linters doing trivial checks alongside automated tests which check a handful of paths through the software written by engineers.
So it was wonderful when my mixture of hope, anticipation and scepticism gave way to delight when the Infer static analyser delivered on the promise of useful, non-trivial static analysis at Facebook. While Infer currently only reports on a small subset of the errors that it can detect, for those classes of errors Infer’s output is generally as useful as seeing a failing test case: Infer’s 80% fix rate on the non-trivial bugs it finds and reports on is extraordinary in an environment like Facebook’s.
Infer and tools like it won’t completely replace test cases written by engineers. To paraphrase Cristiano, without application specific information, no formula will be able to determine whether a piece of software is working as intended. As a companion to test suites though, static analysis will be transformative. It’s likely that sophisticated static analysers will soon be used by everyone from the smallest software engineering teams to the biggest tech companies. When high quality tests are free, why wouldn’t you run them?
Working with Infer in production at Facebook over the last year has been like living in the future. It has changed the way I think about testing and about the testing investments that should be made. I’m incredibly excited to see its public release as an open source tool that can be used by everyone and very grateful to have been able to help distribute this future more widely.
Investing In Testing
Last year I was talking to an engineer at Droidcon London who was working on an Android app with 100% test coverage. I immediately asked whether he thought 100% test coverage was worthwhile: many software engineering teams strive to achieve 100% test coverage, but few succeed because it’s an ...read more
There have been a number of blog posts recently about exciting new organisational structures. As Cory points out “Every early stage company thinks it has reinvented management”: a very dangerous belief when betting on a new organisational structure can be much riskier than betting on the wrong product.
It starts ...read more
Beyond Time Dilation?
EVE online is a remarkable game. On Monday over 2000 people spent over 20 hours destroying virtual spaceships worth 200,000 USD in real money in what was the likely the largest battle in a video game ever. That EVE is capaple of supporting such large engagements is an amazing ...read more
In mid-2004 I first started exploring Second Life. Version 1.4 had just been released and Philip Rosedale had said in the press release “My fantasy is to be Uma Thurman in Kill Bill, and now I can. I’d pay $10 for her yellow jumpsuit and sword moves and ...read more
Parse By The Sea
Brighton Digital Festival
The Brighton Digital Festival starts this week and I’m very happy to be helping out with Facebook London‘s contributions: Parse By The Sea, a mobile app Hackathon featuring Parse on the 26th of September, and helping to Connect The Brighton Digital Festival by sponsoring Metranet to provide high ...read more
One Universe, Many Scales
Following In My Father’s Footsteps
From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large ...read more
Caching Shared, Private Data With Ningx
As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry ...read more
Adding Vary Header Support To Nginx
Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified ...read more
Load Balancing Stateful Services With Nginx
The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are ...read more
Brighton Mini Maker Faire: The Movie
A few years ago nearly all the code I wrote was in C++, but increasingly I’m finding myself writing in a variety of mostly C-style languages and having to perform crunching mental gear changes as I switch between them.
In the interests of making these language switches less painful ...read more
100 robots Vs The Audience
100 robots Attack!
Lots of exciting 100 robots news! Our debut album, Attack!, has been professionally mastered by Chris at Melograf Mastering who has done an amazing job and made the album sound incredible. The new version is already available at bandcamp and will be available on itunes, amazon and many other download ...read more