Caching Shared, Private Data With Ningx

As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry in the region for example.

In order to enforce these rules, the EVE cluster performs a number of access control checks whenever a request is made from an EVE client to the cluster. As a large fraction of calls to the CREST API require these checks to be performed, it would be nice to perform them in Nginx to avoid the overhead of having to make a request to the EVE proxy before returning the cached responses from Nginx. However, duplicating the access control logic within Nginx and trying to keep the two access control implementations in sync is likely to be error prone. As the spying metagame in EVE is arguably bigger than the game itself the consequences of getting the access control logic wrong could be huge. Internet spaceships are serious business.

Fortunately, it’s possible to combine and reuse the load balancing and vary header support techniques previously discussed to avoid both excessive calls from Nginx to the cluster and access control logic duplication.

In addition to annotating responses from the cluster with the address of the proxy containing the character’s session, we also annotate the response with the character’s location, corporation and various other character meta data. The same logic that performs access control checks in the cluster can then add these response headers to the list of vary headers when generating a cache key for a later request on behalf of the same character. Rather than duplicating access control logic, Nginx just needs to make sure that only response headers from the cluster are used for these access control vary headers. If a particular URI is annotated to vary on language and region for example, Nginx will allow the language to be supplied by the client, but the region must be supplied by the cluster in a previous response for the same character.

By reusing the stateful load balancing and vary header support we added to Nginx we’re able to cache data shared between multiple characters without duplicating complex access control logic implemented by the EVE cluster: reducing the CREST load on the EVE cluster without breaking the metagame.

Thanks to @jonastryggvi for working with me on the Caching support and @CCPGames for allowing me to blog about it.