As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry in the region for example.

In order to enforce these rules, the EVE cluster performs a number of access control checks whenever a request is made from an EVE client to the cluster. As a large fraction of calls to the CREST API require these checks to be performed, it would be nice to perform them in Nginx to avoid the overhead of having to make a request to the EVE proxy before returning the cached responses from Nginx. However, duplicating the access control logic within Nginx and trying to keep the two access control implementations in sync is likely to be error prone. As the spying metagame in EVE is arguably bigger than the game itself the consequences of getting the access control logic wrong could be huge. Internet spaceships are serious business.

Fortunately, it’s possible to combine and reuse the load balancing and vary header support techniques previously discussed to avoid both excessive calls from Nginx to the cluster and access control logic duplication.

In addition to annotating responses from the cluster with the address of the proxy containing the character’s session, we also annotate the response with the character’s location, corporation and various other character meta data. The same logic that performs access control checks in the cluster can then add these response headers to the list of vary headers when generating a cache key for a later request on behalf of the same character. Rather than duplicating access control logic, Nginx just needs to make sure that only response headers from the cluster are used for these access control vary headers. If a particular URI is annotated to vary on language and region for example, Nginx will allow the language to be supplied by the client, but the region must be supplied by the cluster in a previous response for the same character.

By reusing the stateful load balancing and vary header support we added to Nginx we’re able to cache data shared between multiple characters without duplicating complex access control logic implemented by the EVE cluster: reducing the CREST load on the EVE cluster without breaking the metagame.

Thanks to @jonastryggvi for working with me on the Caching support and @CCPGames for allowing me to blog about it.

Adding Vary Header Support To Nginx

Sun 14 October 2012 by Jim Purbrick

Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified ...

read more

Load Balancing Stateful Services With Nginx

Mon 30 July 2012 by Jim Purbrick

The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are ...

read more
Fork me on GitHub