Modern software development is sometimes colourfully described as being similar to firing tracer bullets at a target. Rather than spending time doing a lot of research, design and specification up front, the smallest, simplest version of the software is built and the feedback gathered from its use is used to improve the next version. Being able to continuously integrate, test and deploy software means that we can make this feedback loop incredibly tight but because the same engineers are developing the software and the tests we constantly have to think carefully about the testing investments we make and their opportunity costs.

Software engineers allow themselves to use knowledge of how the software works to pick good white box tests. They rely on code reviewers to catch important test cases that they’ve missed. They rely on unit tests to exercise pathological cases which are hard to exercise in end-to-end or integration tests. They use bugs discovered in previous releases to inform tests cases developed for the next. The goal is to ensure that the next version of the software provides feedback on how close it is to the target rather than just exploding on impact.

The decisions around which tests to write are hard because tests have costs. They take time to write and maintain and need to be changed and updated along with the software they test. What would happen if some or all of the tests could be free?

This question has tantalised computer scientists since Turing’s work on the halting problem in the 1930s and while academic progress has been made, tools based on that work have tended to either only work on small codebases; perform trivial analysis; generate a high percentage of false positives; produce difficult to understand reports or impose other overheads on the development process that make the resulting testing pretty far from free. As a result while we theoretically know how to do sophisticated analysis, in practice we tend to rely mostly on linters doing trivial checks alongside automated tests which check a handful of paths through the software written by engineers.

So it was wonderful when my mixture of hope, anticipation and scepticism gave way to delight when the Infer static analyser delivered on the promise of useful, non-trivial static analysis at Facebook. While Infer currently only reports on a small subset of the errors that it can detect, for those classes of errors Infer’s output is generally as useful as seeing a failing test case: Infer’s 80% fix rate on the non-trivial bugs it finds and reports on is extraordinary in an environment like Facebook’s.

Infer and tools like it won’t completely replace test cases written by engineers. To paraphrase Cristiano, without application specific information, no formula will be able to determine whether a piece of software is working as intended. As a companion to test suites though, static analysis will be transformative. It’s likely that sophisticated static analysers will soon be used by everyone from the smallest software engineering teams to the biggest tech companies. When high quality tests are free, why wouldn’t you run them?

Working with Infer in production at Facebook over the last year has been like living in the future. It has changed the way I think about testing and about the testing investments that should be made. I’m incredibly excited to see its public release as an open source tool that can be used by everyone and very grateful to have been able to help distribute this future more widely.


Investing In Testing

Wed 10 June 2015 by Jim Purbrick

Droidcon London

Last year I was talking to an engineer at Droidcon London who was working on an Android app with 100% test coverage. I immediately asked whether he thought 100% test coverage was worthwhile: many software engineering teams strive to achieve 100% test coverage, but few succeed because it’s an ...

read more

buckd

Mon 18 August 2014 by Jim Purbrick

BuckGraffiti

One of the things I’ve been working on since joining Facebook is Buck, an open source Android & Java build tool which is significantly faster than many other Java build tools for a number of reasons.

As well as being fast, Buck gains a lot of power and flexibility by ...

read more

Following In My Father’s Footsteps

Mon 12 November 2012 by Jim Purbrick

Tintin Hair

From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large ...

read more

21st Century JavaScript

Sat 12 March 2011 by Jim Purbrick

The slides and video of my talk at AsyncJS on Thursday are now online. The video is pretty murky, but the sound has come out fine and you can see enough of the slides to be able to follow along at home. The talk focuses on ways to bring useful ...

read more

Spawning Django Blogs

Mon 18 October 2010 by Jim Purbrick

Since leaving Linden Lab I have been talking to a number of people about doing freelance consulting and development work while I get my start-up off the ground and last week got round to setting up a UK limited company so that people will actually be able to pay me ...

read more

FOSDEM X: The Movie

Sun 14 March 2010 by Jim Purbrick

A video of my FOSDEM talk about Mono in Second Life and our plans for the future of scripting is now online (the slides are also available here ):

Watching back, I was surprised to hear myself say “Hooray!”, “Shit” and “Crap” quite so often…

While you’re catching up on ...

read more

FOSDEM X

Wed 10 February 2010 by Jim Purbrick

Last weekend I went to the 10th Free and Open Source Developers European Meeting in Brussels. This year was the first time that FOSDEM had hosted a track on Mono, so I went along to find out what’s going on with Mono, tell the Mono folk what our plans ...

read more

@scalecamp

Mon 07 December 2009 by Jim Purbrick

On Friday I jumped on the train to London to attend the first scalecampuk, an unconference about scalability, at the Guardian offices.

The sessions were all very interesting and mostly very relevant. I learned new things about XSS and CSRF and Django’s defences against them from Simon Willison, new ...

read more

Bouncaline

Tue 03 November 2009 by Jim Purbrick

Last week I took some time off to spend with Luke and Natty during half term and we spent Wednesday having a lovely time finishing off a game we started a couple of months ago: Bouncaline.

Luke has been interested in making games for a while: he made a level ...

read more

New Widgets

Tue 25 November 2008 by Jim Purbrick

It’s that time of year again where people start asking what I’d like for Christmas and I start wondering what they’d like in return. It’s just the sort of problem that should be solved with social software. Over the last few years I’ve had an ...

read more

dConstructing dConstruct

Thu 18 September 2008 by Jim Purbrick

A couple of weeks ago the great and the good of web development descended on Brighton for the wonderful clearleft produced dconstruct conference and once again I’m glad I went along.

Steven Johnson kicked off with a talk about how Dr. John Snow’s innovative data visualization of a ...

read more
Fork me on GitHub