Mapproxy - the customizable tile proxy

We have recently been contacted by a French client about some small functional improvements in mapproxy. Without extensive prior experience with this software we were able to provide the customer with a complete solution in a short period of time.

Tool selection

Well isn’t this a frequent situation ? We want to set up a tile server for a new project and have to choose among those available. In the end, we always choose the one we are most familiar with, and always hope it will never be phased out, right ?

Let’s check some options we often encounter at camptocamp:

Mapcache: great along with mapserver, somehow the reference, well supported, feature-rich, reliable
GeoWebCache (Integrated with GeoServer or available as stand-alone application): yet another “good-old-trusty” tool with limited cache drivers
ArcGIS: complete server solution with every imaginable configuration, but proprietary, of course we prefer the open-source solutions
QGIS server with cache plugin: feature-rich and easy to set up, but with a rather big overhead

Enter mapproxy, a tile server with multiple backends written in python.

	Source data backends	Cache backends
Compatibility	WMS sources (1.0.0–1.3.0), WMTS/TMS sources, Mapserver and Mapnik configurations, any TileCache, Google Maps or Bing compatible source, ArcGIS REST servers and compact cache files	Filesystem, MBTiles/SQLite, ArcGIS Compact Cache, S3, Redis, Riak, CouchDB

If you have ever tried to do some source code modifications in mapserver, the first thing to do is brew a huge thermos of coffee. Digging into highly optimized C code feels like a trip back in time and I have never felt comfortable about potential side effects of a small code change.

On the contrary, making a change in mapproxy has proved to be a rather joyful experience. The software is entirely written in python which makes a readable and maintainable code base. The source code is well structured and has good test coverage.

Tile cache setup

Purpose

A tile cache is placed between a geodata source and potential clients.

It will take load off the geodata generation server for resource-hungry data
Recurrent computation of identical data is avoided
It can be used for advanced operations like watermarking, reprojection or generation of data sets in multiple spatial reference systems.

It has 2 main methods of operation: seeding and serving.

Seeding is the process of loading a defined selection of data from a source and storing it in an optimized way as a cache in storage.
- Seeding consists of generating an initial data set as well as refreshing existing data in cache
- Seeding can be done as a one-off manual operation or as a recurrent automated job.
Serving means listening to clients and serving requests in a given protocol from cached data. Optionally, this service may fetch source data on the fly in case of failed cache lookup.

Configuration

Mapproxy mainly uses two distinct configuration files:

mapproxy.yaml
seed.yaml

Many aspects of cache operations are specified in these configuration files:

Data sources
Grid definition
Cache types and location
Cache ages
Default data (empty tiles)
etc.

Performance tuning

Performance of the mapproxy server can be improved by deploying the app in a robust web server (apache + mod_wsgi, NGINX, gunicorn, etc.)

A pre-generated tile pyramid can also be accessed locally by a front-end application or can be served directly as static files by a high performance file server.

Our code modification in detail

Our wish was to add some more detail to cache handling, with additional options for stale tile handling and re-seeding.

The main design drivers of the parent project were:

Reduction of both load on the source server
Minimisation of storage space for cache
Handling unreliable source servers

Code organization

The code is clearly separated into main functions, each part having usually a virtual parent class with basic shared functionality and child classes implementing a precise function (eg. parent = MapLayer; children = [WMSSource, TiledSource, ArcGISSource, etc.])

Main functions are separated into:

Cache management
Source Handling
Published services
Seeding
Etc.

This modular design means that it is rather easy to propagate an option through the different consumer classes.

New options

Refresh stale tiles while serving
Seeding improvements
1. Refresh only existing tiles (already cached through client queries)
2. Keep stale tiles in case of source error

These were very precise changes, requiring a deep dive into the code structure. However, the design of mapproxy proved robust so that only minimal code changes were needed without any major refactoring.

For point 1, the modifications were well localized:

changes in the configuration system
changes in the cache -> tile internal service
minor accessory modifications for blank tile source on error

https://github.com/mapproxy/mapproxy/pull/518

2 only required cross-wiring of existing options, so that some default values can be overridden from the command line.

https://github.com/mapproxy/mapproxy/pull/515

Without going further into detail, this showed that changes in very different parts of the software were feasible without any side effects on other parts of the code.

Conclusion

Cache handling in mapproxy is already extremely fine-grained. But in case the current configuration options are not sufficient, it has proved to be easy to change the source code in order to take into account additional options.

Mapproxy is a good choice if you have high requirements on the cache configuration and if you enjoy tweaking a tool’s internals. The mapproxy community is also open for new developments, so new functionalities can be made available to a large number of users.

The tool will definitely enter our company’s toolbox, since the experience of working with mapproxy was very positive.

In summary: highly recommended!