Varnish: Setting up Varnish as your ESI cache (with a Django backend example)

at 10:52 PM 0 Comments

Varnish is a load balancer, an extra layer of server software which sits in place of your web server (Apache, nginx, etc) listening to port 80 (or whatever you configure) and waits for HTTP requests. It then forwards the request to the server which isn't overloaded.

After the response is returned by your web server, Varnish will detect any ESI fragments which need replacing and cache the fragment. Varnish will not request the fragment from the server again until the content expires.

The data fragments are returned from Varnish's memory cache, which is pretty damn quick. This saves your server a whole heap of computational and database load.

This setup has an additional benefit of making it easy to clear the cache of a specific ESI fragment.

Setting up Varnish

You configure Varnish by editing the file at /etc/varnish/default.vcl. The VCL script file tells it how to behave when certain information comes through.

Since Varnish is running on port 80 as your primary point of contact, you'll have to tell it:

  • where your web servers are
  • clear any varying HTTP headers and cookie information when a URL contains "esi" (so caching works for every request)
  • check response for a custom header called "X-ESI-max-age" or "esi-enabled" and enable ESI parsing
  • "X-ESI-max-age" is removed before the request is returned to the user
view sourceprint?
01.# Server/port where your HTTP server runs
02.backend default {
03.  .host = "127.0.0.1";
04.  .port = "82";
05.}
06. 
07.sub vcl_recv {
08.  # Assuming that all your cacheable fragments/views contain the pattern /esi/ in the URL.
09.  # You can add other patterns too, but for the sake of this tutorial we'll just use this.
10.  if (req.url ~ "/esi/") {
11.    # Remove variances so it caches for every request
12.    unset req.http.Vary;
13.    unset req.http.cookie;
14.  }
15.}
16. 
17.sub vcl_fetch {
18.  if (beresp.http.Pragma ~ "nocache") {
19.    return(pass);
20.  }
21. 
22.  # This tells Varnish that we want to "varnish-cache" this page
23.  # Check for our custom header
24.  if (beresp.http.X-ESI-max-age == "1") {
25.    # Remove custom header
26.    unset beresp.http.X-ESI-max-age;
27.    unset beresp.http.set-cookie;
28.    esi;
29.  }
30. 
31.  # This tells Varnish that we want ESI processing on this page
32.  if (beresp.http.esi-enabled == "1") {
33.    esi;
34.  }
35.}

To test Varnish, run it and type "start". If there are any errors, check your indenting or syntax. Now that's Varnish all set up.

Coding!

Now for the fun part, tweaking your webpages! Add in some settings to "settings.py" so it's easier to configure.

view sourceprint?
1.VARNISH_USE_ESI = True # Quick kill-switch
2.VARNISH_SERVER = "localhost" # Your HTTP server
3.VARNISH_PORT = 80 # The port of your HTTP server

I've added a module called "varnish" to store the following code. This helper function goes into "varnish/utils.py"

view sourceprint?
01.from django.utils.cache import patch_cache_control
02. 
03.def esi(response, use_esi = False, max_age = 0):
04.  """
05.  This is a helper function which sets the HTTP response headers
06.  required to enable ESI caching and allow for configurable cache lifetimes.
07.  """
08.  # Use ESI so template fragments are parsed
09.  if use_esi:
10.    response['esi-enabled'] = "1"
11. 
12.  # Specify cache time on the ESI views
13.  if max_age > 0:
14.    response['X-IDG-ESI-max-age'] = "1"
15.    patch_cache_control(response, max_age = max_age)
16. 
17.  return response

As you can see, it conditionally sets the custom headers which the VCL script is expecting.

A Python decorator simply wraps around a function and can modify the input/output of the given function. The following code is for a decorator varnish() in "varnish/decorators.py", which basically makes it easier to use esi().

view sourceprint?
01.def varnish(use_esi = False, max_age = 0):
02.  """
03.  This decorator calls the esi() function to modify
04.  the response headers from a view.
05.  """
06.  def wrap(func):
07.    def wrapper(*args, **kwargs):
08.      response = func(*args, **kwargs)
09.      esi(response, use_esi = use_esi, max_age = max_age)
10.      return response
11. 
12.    return wrapper
13. 
14.  return wrap

In order for Varnish to know that we want a cacheable code fragment view, we need to add into the HTML:

view sourceprint?
1.<esi:include src="/path/to/your/esi/cached/view/" />

It's important that we have "/esi/" in the URL pattern as we've configured that pattern in the VCL script. Varnish will attempt to fill it in automatically from cache, or fetch the include URL from your server if necessary.

The following code is for a Django template tag in "varnish/templatetags/varnish.py" which I use to quickly write "esi:include" tags when ESI is enabled, or output the fragment content directly into the template when ESI is disabled.

You can find ContextNode here.

view sourceprint?
01.from django import template
02.from django.conf import settings
03.from django.template import TemplateSyntaxError, resolve_variable
04.from django.core import urlresolvers
05. 
06.from twig.utils import ContextNode
07. 
08.register = template.Library()
09. 
10.@register.tag
11.def esi_cache(parser, tokens):
12.  """
13.  Usage: Output a HTML fragment, either cache request
14.  to Varnish ESI or full HTML output.
15.  {% url esi-view-name object.id as esi_url %}
16.  {% esi_cache esi_url %}
17.  """
18.  bits = tokens.split_contents()
19. 
20.  if len(bits) != 2:
21.    raise TemplateSyntaxError('%s expects 2 arguments' % bits[0])
22. 
23.  def wrap(context):
24.    url = resolve_variable(bits[1], context)
25. 
26.    if settings.VARNISH_USE_ESI:
27.      esi_url = "http://%s:%s%s" % (settings.VARNISH_SERVER, settings.VARNISH_PORT, url)
28.      return '<esi:include src="%s"/>' % esi_url
29. 
30.    else:
31.      # If we're not using ESI, we can just plug in the view output directly
32.      esi_url = url
33. 
34.    # Otherwise call the view and return the data
35.    # @see http://djangosnippets.org/snippets/1568/
36.    resolver = urlresolvers.RegexURLResolver(r'^/', settings.ROOT_URLCONF)
37.    view, args, kwargs = resolver.resolve(esi_url)
38. 
39.    if callable(view):
40.      return view(context['request'], *args, **kwargs).content
41.    else:
42.      # This gives a nicer error email in case it ever happens
43.      raise TemplateSyntaxError('Error retrieving "%s"' % esi_url)
44. 
45.  return ContextNode(wrap)

Putting it all together

For example you have a URL pattern named "esi-product-summary" in your urls.py file.

view sourceprint?
1.urlpatterns = patterns('',
2.  url(r'^product/esi/summary/(?P<id>\d+)/$', 'products.views.esi_summary', name = 'esi-product-summary'),
3.)

To use ESI fragments in the template:

view sourceprint?
1.<div class="reviews">
2.  {% for product in products %}
3.    {% url esi-product-summary product.id as esi_url %}
4.    {% esi_cache esi_url %}
5.  {% endfor %}
6.</div>

The built-in tag {% url %} generates the URL and stores it into a variable called "esi_url".

If VARNISH_USE_ESI is enabled, then {% esi_cache %} outputs the <esi:include src="/product/esi/summary/123"/> element into the skeleton template.

This skeleton template is then returned to Varnish, which detects the missing fragments and fills them in (if it's not already cached) by making extra HTTP requests to the server for each fragment.

When all the fragments are collected, all the parts are put together and returned to the user. Sounds lengthy but it all happens very quickly, especially when it's already cached.

Controlling cache expiry

So how long does it take to expire? You can configure that in your view. Just add a simple decorator and it'll "just work".

view sourceprint?
01.# This tells Varnish to cache it for 1800 seconds (30mins)
02.from varnish.decorators import varnish
03. 
04.@varnish(max_age = 1800)  def esi_summary(request, id):
05.  # The following code has no impact on the ESI stuff.
06.  c = {
07.    'product': Product.objects.get(id = product_id),
08.  }
09.  return render(request, 'products/esi/summary.html', c)

This code is a bit more verbose than it needs to be, but that's mainly due to the extra option USE_ESI.

Now your site is less likely to crumble when a thundering herd of traffic comes your way!

0uNuQ
Prepare yourselves, reddit/digg/slashdot is only a link away!

Enabling ESI on ALL views

If you're using an ESI fragment on a base template, then it may be in convenient for you to enable ESI site-wide. You can either do this through the VCL config file or using Django middleware.

Here's the middleware if you need it:

view sourceprint?
01.class VarnishEnableSitewideEsi():
02.  """
03.  This enables ESI across ALL views on the site which are text/html.
04.  """
05.  def process_response(self, request, response):
06.    mimetype = response['Content-Type']
07.    mimetype = mimetype.split(';')[0]
08. 
09.    if mimetype == "text/html":
10.      return esi(response, use_esi = True)
11. 
12.  return response

Sources

Tags: , ,
 
Copyright © Twig's Tech Tips
Theme by BloggerThemes & TopWPThemes Sponsored by iBlogtoBlog