Opening a couple of browser windows and clicking around was more than sufficient for testing the initial version of ReactVR pairs. Implementing a simple middleware to log actions took advantage of the Redux approach of reifying events to allow a glance at the console to reveal precisely which sequence of actions caused a problem.

Adding support for optimistic consistency made testing more challenging. In order to test conflict resolution, conflicting actions needed to be generated on multiple clients almost simultaneously. After a couple of sessions testing broken versions of pairs with friends it was clear that a more efficient process was required. Fortunately, Redux actions are independent of the UI events which generate them. This separation of concerns made it trivial to randomly generate and dispatch actions without driving the UI. Opening clients dispatching several randomly generated actions per second made it easy to generate conflicts to test optimistic consistency policies while watching games play out made it easy to eyeball the results to check that they were correct. This random action generation mechanism can be enabled by adding ?random as the query string when opening the Pairs example in a browser.

One of the problems found by this approach was that clients didn’t always end up eventually consistent. One client would end up with all squares shown and all pairs scored, while another would have some squares hidden. After some digging it turned out that in these cases the master would be reducing a hide action followed by a score action, while the other client would reduce the actions in the reverse order, causing the hide acton to be invalid. Without a way for a non-master client to let the master know about the conflict the master would not send a sync action and the clients would not end up eventually consistent.

This problem identified a limitation with the optimistic clientPredictionConsistency policy: if any sequence of actions causes a conflict then every ordering of those actions must also cause a conflict in order for the clients to end up eventually consistent. The fix for the hide-score case seemed clear: if the score action was only valid if the pair was shown then both orderings of those actions would generate a conflict and so the master would generate a sync action regardless of the order in which it reduced the actions. Some more eyeballing seemed to suggest that the problem had been solved, but a better way to test the property that sync action generation is independent of action order was to write a property based test.

Some Googling revealed that my Facebook colleague Lee Byron had written a JavaScript property based testing framework called test-check which was compatible with the Jest framework used by ReactVR tests, so I started hacking. I soon had a test which could generate an arbitrary sequence of actions, dispatch them and check that if the sequence of actions generated a sync action then dispatching the same sequence of actions in reverse would also generate a sync action.

I could now test that the property held for thousands of action sequences in a few seconds and so I found the next bug almost immediately. While my change to make any ordering of test then hide generate a sync had fixed one problem it had introduced another. The validity of score events was now dependent on the preceding show events, so it was possible for show-show-score to be valid but for every other order of those events to cause the score event to be invalid and so not reduced.

At this point I took a step back. The only situation that should cause a conflict that needs to be resolved is when more than one player tries to score the same pair. In this situation clients don’t have enough information to resolve the conflict and a master client needs to pick an ordering and communicate the result to the other clients. In the case of hide and score actions every client can do the right thing. Hide actions can be made to not hide scored squares and score actions can be made to show pairs. With the reducers changed to work in this way hide actions can always be reduced and score actions are only invalid when they conflict with each other. With these changes in place the validation logic becomes dramatically simpler to reason about and the property based tests were unable to find any more cases which would not be eventually consistent even after thousands of runs.

Testing distributed systems is hard, but combining replicated Redux with property based tests has proved to be a powerful way to gain a high degree of confidence that applications will work correctly despite limitations in the current simplistic clientPredictionConsistency mechanism. The same property based tests will enable new optimistic consistency mechanisms without those limitations to be developed far more quickly in future.

If you’d like to play the ReactVR version of pairs or see the rest of the code, it’s available on github here.

All code in this post is made available under the ReactVR examples license.


ReactVR Redux Revisited

Tue 04 July 2017 by Jim Purbrick

There were a couple of aspects of my previous experiments building networked ReactVR experiences with Redux that were unsatisfactory: there wasn’t a clean separation between the application logic and network code and, while the example exploited idempotency to reduce latency for some actions, actions which could generate conflicts used ...

read more

VR Redux

Wed 04 January 2017 by Jim Purbrick

Mike and I have been talking about how to easily build simple networked social applications with ReactVR for a while, so I spent some time hacking over the Christmas break to see if I could build a ReactVR version of the pairs game in Oculus Rooms. Pairs is simple and ...

read more

Free Tests For Everyone!

Thu 11 June 2015 by Jim Purbrick

Modern software development is sometimes colourfully described as being similar to firing tracer bullets at a target. Rather than spending time doing a lot of research, design and specification up front, the smallest, simplest version of the software is built and the feedback gathered from its use is used to ...

read more

Investing In Testing

Wed 10 June 2015 by Jim Purbrick

Droidcon London

Last year I was talking to an engineer at Droidcon London who was working on an Android app with 100% test coverage. I immediately asked whether he thought 100% test coverage was worthwhile: many software engineering teams strive to achieve 100% test coverage, but few succeed because it’s an ...

read more

buckd

Mon 18 August 2014 by Jim Purbrick

BuckGraffiti

One of the things I’ve been working on since joining Facebook is Buck, an open source Android & Java build tool which is significantly faster than many other Java build tools for a number of reasons.

As well as being fast, Buck gains a lot of power and flexibility by ...

read more

Evolving Develop

Mon 20 July 2009 by Jim Purbrick

As usual I headed down to the Metropole on the sea front last week to attend the annual Develop conference in Brighton. Unusually, this time I was attending the Evolve day which shifts the focus from console development to online, mobile and social games, which I had helped create as ...

read more

dConstructing dConstruct

Thu 18 September 2008 by Jim Purbrick

A couple of weeks ago the great and the good of web development descended on Brighton for the wonderful clearleft produced dconstruct conference and once again I’m glad I went along.

Steven Johnson kicked off with a talk about how Dr. John Snow’s innovative data visualization of a ...

read more
Fork me on GitHub