Varnish: Setting up Varnish as your ESI cache (with a Django backend example)

at 10:52 PM

Varnish is a load balancer, an extra layer of server software which sits in place of your web server (Apache, nginx, etc) listening to port 80 (or whatever you configure) and waits for HTTP requests. It then forwards the request to the server which isn't overloaded.

After the response is returned by your web server, Varnish will detect any ESI fragments which need replacing and cache the fragment. Varnish will not request the fragment from the server again until the content expires.

The data fragments are returned from Varnish's memory cache, which is pretty damn quick. This saves your server a whole heap of computational and database load.

This setup has an additional benefit of making it easy to clear the cache of a specific ESI fragment.

Setting up Varnish

You configure Varnish by editing the file at /etc/varnish/default.vcl. The VCL script file tells it how to behave when certain information comes through.

Since Varnish is running on port 80 as your primary point of contact, you'll have to tell it:

  • where your web servers are
  • clear any varying HTTP headers and cookie information when a URL contains "esi" (so caching works for every request)
  • check response for a custom header called "X-ESI-max-age" or "esi-enabled" and enable ESI parsing
  • "X-ESI-max-age" is removed before the request is returned to the user
# Server/port where your HTTP server runs
backend default {
.host = "127.0.0.1";
.port = "82";
}

sub vcl_recv {
# Assuming that all your cacheable fragments/views contain the pattern /esi/ in the URL.
# You can add other patterns too, but for the sake of this tutorial we'll just use this.
if (req.url ~ "/esi/") {
# Remove variances so it caches for every request
unset req.http.Vary;
unset req.http.cookie;
}
}

sub vcl_fetch {
if (beresp.http.Pragma ~ "nocache") {
return(pass);
}

# This tells Varnish that we want to "varnish-cache" this page
# Check for our custom header
if (beresp.http.X-ESI-max-age == "1") {
# Remove custom header
unset beresp.http.X-ESI-max-age;
unset beresp.http.set-cookie;
esi;
}

# This tells Varnish that we want ESI processing on this page
if (beresp.http.esi-enabled == "1") {
esi;
}
}

To test Varnish, run it and type "start". If there are any errors, check your indenting or syntax. Now that's Varnish all set up.

Coding!

Now for the fun part, tweaking your webpages! Add in some settings to "settings.py" so it's easier to configure.

VARNISH_USE_ESI = True # Quick kill-switch
VARNISH_SERVER = "localhost" # Your HTTP server
VARNISH_PORT = 80 # The port of your HTTP server

I've added a module called "varnish" to store the following code. This helper function goes into "varnish/utils.py"

from django.utils.cache import patch_cache_control

def esi(response, use_esi = False, max_age = 0):
"""
This is a helper function which sets the HTTP response headers
required to enable ESI caching and allow for configurable cache lifetimes.
"""
# Use ESI so template fragments are parsed
if use_esi:
response['esi-enabled'] = "1"

# Specify cache time on the ESI views
if max_age > 0:
response['X-IDG-ESI-max-age'] = "1"
patch_cache_control(response, max_age = max_age)

return response

As you can see, it conditionally sets the custom headers which the VCL script is expecting.

A Python decorator simply wraps around a function and can modify the input/output of the given function. The following code is for a decorator varnish() in "varnish/decorators.py", which basically makes it easier to use esi().

def varnish(use_esi = False, max_age = 0):
"""
This decorator calls the esi() function to modify
the response headers from a view.
"""
def wrap(func):
def wrapper(*args, **kwargs):
response = func(*args, **kwargs)
esi(response, use_esi = use_esi, max_age = max_age)
return response

return wrapper

return wrap

In order for Varnish to know that we want a cacheable code fragment view, we need to add into the HTML:

<esi:include src="/path/to/your/esi/cached/view/" />

It's important that we have "/esi/" in the URL pattern as we've configured that pattern in the VCL script. Varnish will attempt to fill it in automatically from cache, or fetch the include URL from your server if necessary.

The following code is for a Django template tag in "varnish/templatetags/varnish.py" which I use to quickly write "esi:include" tags when ESI is enabled, or output the fragment content directly into the template when ESI is disabled.

You can find ContextNode here.

from django import template
from django.conf import settings
from django.template import TemplateSyntaxError, resolve_variable
from django.core import urlresolvers

from twig.utils import ContextNode

register = template.Library()

@register.tag
def esi_cache(parser, tokens):
"""
Usage: Output a HTML fragment, either cache request
to Varnish ESI or full HTML output.
{% url esi-view-name object.id as esi_url %}
{% esi_cache esi_url %}
"""
bits = tokens.split_contents()

if len(bits) != 2:
raise TemplateSyntaxError('%s expects 2 arguments' % bits[0])

def wrap(context):
url = resolve_variable(bits[1], context)

if settings.VARNISH_USE_ESI:
esi_url = "http://%s:%s%s" % (settings.VARNISH_SERVER, settings.VARNISH_PORT, url)
return '<esi:include src="%s"/>' % esi_url

else:
# If we're not using ESI, we can just plug in the view output directly
esi_url = url

# Otherwise call the view and return the data
# @see http://djangosnippets.org/snippets/1568/
resolver = urlresolvers.RegexURLResolver(r'^/', settings.ROOT_URLCONF)
view, args, kwargs = resolver.resolve(esi_url)

if callable(view):
return view(context['request'], *args, **kwargs).content
else:
# This gives a nicer error email in case it ever happens
raise TemplateSyntaxError('Error retrieving "%s"' % esi_url)

return ContextNode(wrap)

Putting it all together

For example you have a URL pattern named "esi-product-summary" in your urls.py file.

urlpatterns = patterns('',
url(r'^product/esi/summary/(?P<id>\d+)/$', 'products.views.esi_summary', name = 'esi-product-summary'),
)

To use ESI fragments in the template:

<div class="reviews">
{% for product in products %}
{% url esi-product-summary product.id as esi_url %}
{% esi_cache esi_url %}
{% endfor %}
</div>

The built-in tag {% url %} generates the URL and stores it into a variable called "esi_url".

If VARNISH_USE_ESI is enabled, then {% esi_cache %} outputs the <esi:include src="/product/esi/summary/123"/> element into the skeleton template.

This skeleton template is then returned to Varnish, which detects the missing fragments and fills them in (if it's not already cached) by making extra HTTP requests to the server for each fragment.

When all the fragments are collected, all the parts are put together and returned to the user. Sounds lengthy but it all happens very quickly, especially when it's already cached.

Controlling cache expiry

So how long does it take to expire? You can configure that in your view. Just add a simple decorator and it'll "just work".

# This tells Varnish to cache it for 1800 seconds (30mins)
from varnish.decorators import varnish

@varnish(max_age = 1800) def esi_summary(request, id):
# The following code has no impact on the ESI stuff.
c = {
'product': Product.objects.get(id = product_id),
}
return render(request, 'products/esi/summary.html', c)

This code is a bit more verbose than it needs to be, but that's mainly due to the extra option USE_ESI.

Now your site is less likely to crumble when a thundering herd of traffic comes your way!

0uNuQ
Prepare yourselves, reddit/digg/slashdot is only a link away!

Enabling ESI on ALL views

If you're using an ESI fragment on a base template, then it may be in convenient for you to enable ESI site-wide. You can either do this through the VCL config file or using Django middleware.

Here's the middleware if you need it:

class VarnishEnableSitewideEsi():
"""
This enables ESI across ALL views on the site which are text/html.
"""
def process_response(self, request, response):
mimetype = response['Content-Type']
mimetype = mimetype.split(';')[0]

if mimetype == "text/html":
return esi(response, use_esi = True)

return response

Sources

Tags: , ,
 
Copyright © Twig's Tech Tips
Theme by BloggerThemes & TopWPThemes Sponsored by iBlogtoBlog