As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry in the region for example.

In order to enforce these rules, the EVE cluster performs a number of access control checks whenever a request is made from an EVE client to the cluster. As a large fraction of calls to the CREST API require these checks to be performed, it would be nice to perform them in Nginx to avoid the overhead of having to make a request to the EVE proxy before returning the cached responses from Nginx. However, duplicating the access control logic within Nginx and trying to keep the two access control implementations in sync is likely to be error prone. As the spying metagame in EVE is arguably bigger than the game itself the consequences of getting the access control logic wrong could be huge. Internet spaceships are serious business.

Fortunately, it’s possible to combine and reuse the load balancing and vary header support techniques previously discussed to avoid both excessive calls from Nginx to the cluster and access control logic duplication.

In addition to annotating responses from the cluster with the address of the proxy containing the character’s session, we also annotate the response with the character’s location, corporation and various other character meta data. The same logic that performs access control checks in the cluster can then add these response headers to the list of vary headers when generating a cache key for a later request on behalf of the same character. Rather than duplicating access control logic, Nginx just needs to make sure that only response headers from the cluster are used for these access control vary headers. If a particular URI is annotated to vary on language and region for example, Nginx will allow the language to be supplied by the client, but the region must be supplied by the cluster in a previous response for the same character.

By reusing the stateful load balancing and vary header support we added to Nginx we’re able to cache data shared between multiple characters without duplicating complex access control logic implemented by the EVE cluster: reducing the CREST load on the EVE cluster without breaking the metagame.

Thanks to @jonastryggvi for working with me on the Caching support and @CCPGames for allowing me to blog about it.

Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified via the Accept header.

Fortunately it’s relatively easy to add support for the Vary header using the Nginx Lua module and a small amount of Lua, which is much easier than building and maintaining a 3rd party module and doesn’t greatly impact performance.

First, we define a dictionary in the nginx config which will store a mapping from URIs to Vary headers:

lua_shared_dict uriToVary 10m;

Next we define the default location in the nginx config, which will use the lua ProxyRequest function to make a subrequest to /proxy_request then store the Vary header in the response in the dictionary.

location / {

    # make subrequest to /proxy_request, then store response headers
    content_by_lua '
        local vary = require("vary")
        local response = vary.ProxyRequest()
    ';
}

function ProxyRequest()

    -- make subrequest and capture response
    local response = ngx.location.capture("/proxy_request", {
        method = GetRequestMethod(ngx.var.request_method), body = ngx.req.get_body_data()})

    -- forward HTTP headers from response
    for k,v in pairs(response.header) do
        ngx.header[k] = v
    end

    ngx.shared.uriToVary:set(ngx.var.request_uri, response.header["Vary"])

    -- forward status and body from response
    ngx.status = response.status
    ngx.print(response.body)

    return response

end

Finally, we define the /proxy_request location which will use the lua RewriteCache function to combine the uri with the vary headers to generate the final cache key used by the proxy_cache module.

location /proxy_request {
    internal;

    # set defaults
    set $noCache 1;
    set $cacheBypass 1;
    set $cacheKey nil;

    # rewrite using stored data
    rewrite_by_lua '
        local vary = require("vary")
        vary.RewriteCache()
    ';

    # proxy request
    proxy_cache_bypass $cacheBypass;
    proxy_no_cache $noCache;
    proxy_cache_key $cacheKey;
    proxy_cache API_CACHE;
    proxy_pass $proxy$request_uri;
}

function RewriteCache()

    local varyOn = ngx.shared.uriToVary:get(ngx.var.request_uri)
    local cacheKey = nil

    -- if vary unknown for this uri, bypass cache and do not cache
    if varyOn == nil then
        return
    end

    cacheKey = ngx.var.request_uri .. GenerateCacheKey(varyOn, ngx.req.get_headers())
    ngx.var.noCache = 0
    ngx.var.cacheKey = cacheKey
    ngx.var.cacheBypass = 0

end

function GenerateCacheKey(varyOnStr, requestHeaders)

    local cacheKey = ""
    for part in string.gmatch(varyOnStr, "([^,%s+]+)") do

        if requestHeaders[part] then
            cacheKey = cacheKey .. ":" .. requestHeaders[part]
        end

    end

    return cacheKey

end

The first time a URI is requested the cache will be bypassed, but the Vary header from the response will be stored in the shared dictionary. The second time the URI is requested the cache key will be generated from the URI and the appropriate request headers specified in the vary header and the response will be cached. When the URI is subsequently requested with the same set of headers it will be served from the cache.

Note that when the shared dictionary is full it will evict old entries using an LRU scheme. Nginx will generate “ngx_slab_alloc() failed” errors when this occurs, but these can safely be ignored.

Thanks to @jonastryggvi for working with me on the Vary support and @CCPGames for allowing me to blog about it.

The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are sent on to the client.

In developing the CREST API we extended the EVE proxies to talk HTTP, then added nginx reverse proxies to the service to provide SSL termination and caching while shielding the EVE proxies from potentially malicious requests.

So, how does nginx know which EVE proxy to send a request to? In the first instance, it just guesses. We set up a set of proxies and use proxy_pass to have nginx just pick one.

upstream eveproxies {
    # List all eveproxies
}

location / {
    proxy_pass http://eveproxies;
}

The proxy can then use the CCP cluster’s RPC machinery to find the character’s session. If nginx has been lucky the request is processed and the response sent back to nginx and from there to the player. If no session exists for the character on any proxy a new session is created and then the request processed as above. If the character session is on a different node the proxy returns an X-accel response to a location which extracts the correct proxy URI from the path and resends the request.

location ~* ^/internal_redirect/(.*) {
    internal;
    proxy_pass http://$1$is_args$args;
}

The performance of this approach can be greatly improved by caching the mapping of authorization headers to proxies, which can be done using a dict and a small piece of lua.

lua_shared_dict tokenToProxy 10m;

location / {

   content_by_lua '
        -- make subrequest and capture response
        local response = ngx.location.capture("/proxy_request", {
            method = GetRequestMethod(ngx.var.request_method),
            body = ngx.req.get_body_data()})

        -- forward HTTP headers from response
        for k,v in pairs(response.header) do
                ngx.header[k] = v
        end

        -- forward status and body from response
        ngx.status = response.status
        ngx.print(response.body)

        -- cache backend for next request
        ngx.shared.tokenToProxy:set(ngx.var.http_authorization,
            response.header["X-Backend"])
    ';
}

location /proxy_request {

    internal;
    set $crestProxy "http://eveproxies";

    rewrite_by_lua '
        ngx.var.crestProxy = ngx.shared.tokenToProxy:get(
            ngx.var.http_authorization)
    ';

    proxy_pass $crestProxy$request_uri;
}

In a configuration with multiple loadbalancers we potentially have to pay the price of one proxy redirection per nginx process. This could potentially be improved by using a shared cache for the authorization to proxy mapping or by using ip affinity to map all requests from a client to a single nginx box, but in practice where the number of requests from a client is much larger than the number of loadbalancers, this improvement is likely to be negligible.

This mechanism ensures that most HTTP requests go straight to the correct proxy without the load balancers having to maintain any state. A new load balancer can be added to the cluster just be being told the addresses of the eve proxies and will quickly start routing requests to the correct location.