Take Control of Your Maps

We live in the era of Google Maps. What started off as an impressive refresh of Mapquest-style maps now fuels web mashups. With APIs official and unofficial, Google Maps is simple enough for front-end designers to embed and for back-end programmers to target. Along the way to becoming nearly ubiquitous, it has played a major role in the “democratization of mapping.” For the practical developer who wants to add geospatial information to a site or application, the Google Maps API has been an easy call.

But, perhaps no longer. As websites mature and the demand for geographic applications grow, the old mashup arrangement is starting to chafe. Mapping components are more and more vital, and so we demand greater control, expressiveness, and functionality from them.

Fortunately, as in many aspects of internet technology, an ecology of open source online mapping tools has emerged alongside the market leader. It is now possible to replicate Google Maps’ functionality with open source software and produce high-quality mapping applications tailored to our design goals. The question becomes, then, how?

Google Maps is the best (right?)#section2

Google Maps is an impressive application. It’s fast, responsive, and nicely rendered, and it exposes a ton of functionality via its well-documented and well-understood API. Why on earth wouldn’t you outsource this bit of functionality to Google?

Ask yourself this question: why would you, as a website developer who controls all aspects of your site, from typography to layout, to color palette to photography, to UI functionality, allow a big, alien blob to be plopped down in the middle of your otherwise meticulously designed application? Think about it. You accept whatever colors, fonts, and map layers Google chooses for their map tiles. Sure, you try to rein it back in with custom markers and overlays, but at the root, the core component—the map itself—is out of your hands.

The result is Google Maps fatigue. We’ve all experienced it. It manifests not only when we yawn at YAGMM (Yet Another Google Maps Mashup), because there are high-quality web apps deploying the Google Maps API seamlessly and with great success. Despite this, and despite the fact that Google itself continues to refine and improve the base application, the fatigue remains. It’s the effect of seeing the same elements over and over again across the web. As web developers, we live with constraints, so to a certain extent, the Google Maps API is similar to Verdana and Georgia—they’re common components we know will work well. But if it were possible and practical to make a substitution, wouldn’t you do so?

Depending on your application, Google’s choices about what display on the map and how to display it might not work for you. On a general-purpose map, for example, it’s great to see the outlines of building footprints, as Google is starting to display in certain cities—but such outlines might only constitute visual clutter for your application. Google decides what information is conveyed to your users, but it’s not necessarily what they need or want.

Further, you’re only allowed to mess with Google’s maps within limits placed on you by the API’s terms and conditions—which you accept when you begin using it. You may not conduct post-processing or other manipulation. You may not obscure the copyright mark. Not that it would be practical to do so: part of the API’s appeal is a speedy response to requests for map tiles, so any on-the-fly mucking around would likely result in a sub-optimal user experience.

You’re also limited to the functionality Google decides to implement. To a degree, you can reverse-engineer the API to insert your own custom functionality (and hackers certainly have). But that can take serious JavaScript-fu, to say nothing of the inherent fragility of such a strategy.

If not Google Maps, then what?#section3

It’s easy to acknowledge your mapping solution doesn’t afford you all the design and development control you want. It’s quite another thing to consider tearing out that factory module (from an admittedly pretty impressive factory) and replacing it with your own finely tuned custom build. Rolling your own maps need not be an intimidating affair, provided you understand the problem and the tools to fix it.

The map stack#section4

The key to this kingdom is understanding the map stack. Modern online mapping applications all adhere to this architecture, which winds up in your user’s browser as the click-and-draggable view of the world we all know and love (the view, not always the world). There are four major layers in the map stack, from the highest to lowest level.

Browser UI#section5

Users interact with your mapping application primarily through a JavaScript or Flash library that listens to user events, requests tiles from the map server, assembles tiles in the viewport, and draws additional elements on the map, such as popups, markers, and vector shapes. This is the layer in the stack where the Google Maps API sits.

OpenLayers is one example of a browser mapping UI library written in JavaScript. It’s free software released under an open source license. Even though it’s still under heavy development, OpenLayers is a remarkably mature library that supports multiple projections and map server backends, geospatial data format reading and writing, event handling, and vector drawing. “View source” in these examples to see what’s possible and how to integrate these features into your application.

Tile cache#section6

A tile cache is a server that sits between the browser and the map server. It checks to see if a requested map tile is already hanging around in a cache somewhere, where it can be served up quickly to short-circuit the call to the map server. If the map tile is not already generated, the tile cache gets it from the map server and saves it to speed up subsequent requests. For the most part, this layer should be all but transparent to the user, and is typically a simple set up and configuration. Instead of pointing the client to the map server, point it at the tile cache, and point the tile cache at the map server.

You can consider the tile cache as one half of a layer in the map stack, because it’s technically not required in order to close the loop from the user request for a tile to the response from the map server. However, in the real world, our sites are under loads that make constant, on-the-fly map tile generation expensive, impractical, and needlessly wasteful. The tile cache is the secret sauce which makes open source web mapping a viable alternative.

The aptly-named TileCache is an open source tile caching server that’s simple to set up. It can store tiles on local disks, in memcached, or on Amazon’s S3 storage service.

Map server#section7

Here’s where you get to flex your design muscle. A map server is an application that takes geospatial data as input and renders graphical output. Specifically, for modern online mapping apps, it spits out a series of map tiles, which are uniformly sized graphic files (256×256 PNGs, for example) that are ultimately served to and assembled in the browser as the displayed map.

You can control the final appearance of your maps by styling the geospatial elements, in a manner not unlike CSS. The exact mechanism varies by implementation, but all map servers provide hooks for defining styles such as stroke width, stroke color, fill color, opacity, and layer stacking order.

Map servers can be exposed directly to the internet, where a well-defined interface takes HTTP requests representing specific geographic places and returns the corresponding tiles. However, map servers are usually hidden from direct access by the tile cache layer in the stack, for good reason.

Mapnik is an open source map server that renders beautifully and has a developer-friendly interface. At EveryBlock, where I work, we use Mapnik to generate our map tiles, and we’ve been pleased with the results.

Geospatial data#section8

The building blocks of any mapping application are the databases that define the points, lines, and polygons that represent real-world places, roads, and areas. Geospatial data connects geometric shapes with a coordinate system (latitude and longitude, most commonly) and with attribute or feature data such as school names, interstate designations, or congressional district numbers and representatives.

The big kahuna in the geospatial data world is a “streets” database. While not every mapping application needs it, most do, and users expect it as a primary way to orient themselves. Adding streets to your app is a key step that I’ll get into in more depth below.

Map yourself#section9

Now that you’re familiar with the building blocks of the web map stack, I’ll address some key considerations for putting your own stack together. Because there are several alternatives for each layer, depending on your platform, existing data and code, programming language, and application requirements, I won’t dive into too much detail, but I will try to provide general hints and tips that apply regardless of the specific choices you make.

Where to store the “where”#section10

Geospatial data is stored and transmitted in many different flavors: the open source OGR library, which is used to open, read, and write vector files such as these, lists over 30 supported formats. Fortunately, there are a few that are far more common than the rest. The Shapefile is the granddaddy of them all, and you’ll find many graphical information systems (GIS) organizations distribute their data in this format. Shapefiles are actually bundles of files: usually three of them, with the extensions .shp, .shx, and .dbf.

An increasingly popular format is KML, which is an XML-based format first used in Google Earth. Google Earth and Maps can both produce and consume KML files, making it a good format for interchange with these applications. Since it’s XML-based and therefore just text instead of the binary format of Shapefiles, the KML format is a bit verbose, but it’s easy to open in a text editor for a good, old-fashioned “view source” session.

If you use a relational database management system (RDBMS) in your application, you should consider using its built-in geometric data type, if it has one, or adding an extension that enables one. For example, PostGIS is a library for PostgreSQL which provides a geometry type and functions that operate on it. You can then simply add an additional column to your tables and associate geospatial data with your existing rows. This gives you the ability to make SQL queries that include “spatial predicates.” Spatial predicates are statements like “within five miles of this latitude and longitude,” or “this line intersects with this line.”

No matter how you store your geospatial data, I recommend that you add the OGR library to your toolbox. It’s the Swiss Army knife of geospatial data, allowing you to translate between formats, change projections, and filter and combine datasets. There’s a command-line utility, ogr2ogr, that provides this functionality, and there are also OGR bindings for your favorite programming language. The great thing about OGR is that it smooths out the differences between the various geospatial data serializations. You don’t have to care about what format, or in what crazy projection, you’ve got your data layer—just translate it into the format that makes sense for the rest of your application and move on.

Just browsing#section11

One of the first things I do when I download a new geospatial dataset is get a rough visualization of it. I want to see, quickly, what kind of geometries—points, lines, polygons, or combinations thereof—it has, and where it is in relation to my other datasets. Basically, what will it look like, and how well does it match my expectations of what it should cover?

I’ve found that a GIS desktop application is a great tool for getting a quick look at a new Shapefile. QGIS is a cross-platform, open source app that fits the bill. Essentially, QGIS is the kind of application that web-based maps are emulating: click-and-drag and zoom, with layering. It gives you an opportunity to mock-up your custom map application and see how the various layers hang together. You can test things such as labeling and color palettes, or even edit the underlying data, which can be useful for one-off fixes and tweaks.

Street beat#section12

As I mentioned above, many mapping applications require a streets layer. Streets are a primary way that people orient themselves—the closest thing we have to a practical coordinate system in the world. As you might imagine, a database of streets is a large, complex, and valuable asset. Companies, such as NAVTEQ and TeleAtlas employ a fleet of vehicles to scour the world’s roads, arterials, and expressways, amassing the exact coordinates of each street they travel. You can purchase a license to use this type of commercial database for your own applications—usually at a per-page-view rate. It can be expensive, but if your organization can afford it, you’ll have the benefit of a constantly-updated, high-quality streets layer, with near-comprehensive coverage.

An alternative to a commercial streets database, at least in the United States, is the TIGER/Line dataset, produced by the U.S. Census Bureau. TIGER/Line is periodically updated to aid in the various programs of the Census and it’s free to download and use for your own applications. It has fairly comprehensive coverage of U.S. streets—it might not have the latest subdivision in your community, but it can be as good as the commercial products, especially in major metropolitan areas.

Many municipalities have GIS departments that publish Shapefiles of their streets. These are great datasets because they are often more up-to-date and more accurate than TIGER/Line, since they are used to provide city services.

An interesting project to keep your eye on is OpenStreetMap, or OSM. It’s a kind of wiki for streets that allows anyone to upload their GPS traces or data from free sources like TIGER/Line. The great thing about OSM is that it aspires to be worldwide. This is a good thing, because unlike the United States, most countries don’t put their geospatial data in the public domain. While there’s still a long way to go, you can already start downloading and using OSM data, which is fairly complete in some areas, especially in Europe.

The scales fall from your eyes#section13

“Scale” is a key concept in maps: it’s the correspondence between a unit of measurement on the map and the thing being mapped, (usually the world, but it could be the moon, the solar system, or a blood cell). On the map, 1:10000 means that an inch on the map represents 10,000 inches in the world. Note that scales are unit-independent: 1:10000 also means 1 cubit on the map represents 10,000 cubits in the world.

For example, let’s say that the largest scale—in other words, the scale at which things in the map are as big as possible—is 1:10, and the smallest scale is 1:160. Typically, there is a doubling of the second number in the scale at each zoom, so in between we will have zoom levels at scales of 1:20, 1:40, and 1:80. Altogether then, we have five total scales, and five zoom levels. By convention, we give the smallest scale—which is the most-zoomed-out—zoom level number one, which in this case is 1:160. 1:80 would be two, 1:40 would be three, 1:20 would be four, and 1:10 would be zoom level five.

In the web map stack, it‘s helpful to determine the scales at which your maps will appear upfront. One way to think of this is the “zoom levels” in Google Maps: there are a fixed number of them, and each level represents a particular scale. In fact, in your application, there will be a one-to-one relationship between a zoom level and a scale.

Here’s why it’s important to define your scales in advance: the browser UI, the map server, and the tile cache work together best when they work with the same set of scales. For the browser UI, you’ll typically want to give your users a fixed number of zooms for consistency, as well as for fitting your application’s domain—if you only cover the United States, for instance, you don‘t need to zoom all the way out to see the rest of the world. For the map server, you’ll want a defined set of scales for your styling, because layers will turn on or off at certain scales, and different line widths and font sizes will apply at those scales. Finally, the tile cache will cache your tiles at each zoom level, which are determined by the allowable scales. Defining your scales is one way to reduce the complexity of your map application and improve coordination across the layers of the map stack.

Layers within layers#section14

In the previous section, I mentioned that defining your scales across your entire application helps when styling the map server. As you think about your web map application as a stack of layers, also think about the visible layers within the maps themselves: streets, parks, rivers and lakes, building footprints, neighborhood boundaries, et cetera. At each scale or zoom level, these layers come together in a certain way, one on top of the other, defined by your styles to render the final output. Each layer is styled for each scale so that all the elements come together harmoniously.

For example, at a city-wide scale, we might display only the major streets and label only the expressways:

We obviously wouldn’t want to label every street at this scale, regardless of size, or the map would be too cluttered to read.

As we zoom in to a neighborhood-wide scale, however, we add labels to the arterials, and display all the rest of the streets:

Notice too that we increase the thickness, in pixels, of all street segments, to convey their greater scale. Varying color intensity is another way to reduce or increase emphasis on elements at different scales. Which method you choose should depend on the nature of the layer.

Embrace and extend#section15

Browser mapping UI libraries such as OpenLayers are ready to use out of the box: just link up the requisite resource in your site’s <head> and start coding. Since you’re going through the trouble to replace the map stack because you wanted more control over its appearance, you’re probably not going to just accept the stock icons and other styles these libraries provide. As fond as I am of OpenLayers, its zoom, pan, and other widgets leave a lot to be desired.

Fortunately, we’re talking about JavaScript and CSS, which means we can override to our heart’s content. At EveryBlock, we wanted a simple zoom-in/zoom-out widget for our maps. We started by defining our controls explicitly, overriding the default controls OpenLayers sets up:

/* JavaScript */
var options = {
  ...
  controls: [],
  ...
};
var map = new OpenLayers.Map("map-div", options);
var customControls = [new OpenLayers.Control.DragPan(),
                      new OpenLayers.Control.PanZoom()];
for (var i=0; i;
  map.addControl(control);
  control.activate();
}

The DragPan control gives you the click-and-drag interface we expect from online maps. PanZoom sets up the widget that allows you to click to zoom in and out, and pan by the cardinal directions. We’re not interested in panning with this widget, so we simply found the corresponding CSS styles and hid the panning elements:

/* CSS */
#OpenLayers_Control_PanZoom_zoomworld,
#OpenLayers_Control_PanZoom_panup,
#OpenLayers_Control_PanZoom_panright,
#OpenLayers_Control_PanZoom_pandown,
#OpenLayers_Control_PanZoom_panleft { 
  display: none !important; 
}

Finally, we replaced the zoom elements with our own, again simply overriding the CSS (Line wraps marked » —Ed.):

/* CSS */
#OpenLayers_Control_PanZoom_zoomin {
  background: url(/img/zoom-plus-mini.png) no-repeat » 
  !important;
  height: 20px !important;
  width: 20px !important;
  top: 13px !important;
}
#OpenLayers_Control_PanZoom_zoomout {
  background: url(/img/zoom-minus-mini.png) no-repeat »
  !important;
  height: 20px !important;
  width: 20px !important;
  top: 37px !important;
}
#OpenLayers_Control_PanZoom_zoomin:hover {
  background-image: url(/img/zoom-plus-mini-over.png) »
  !important;
}
#OpenLayers_Control_PanZoom_zoomout:hover {
  background-image: url(/img/zoom-minus-mini-over.png) »
  !important;
}

Note the use of the !important directive in each case. This is obviously not ideal, but unfortunately it’s necessary in this case, because OpenLayers hard-codes some style information in the JavaScript.

This is just an example, but it demonstrates how easy it is to use the vast functionality of these libraries and still exert control over the display.

Conclusion#section16

One of the great things about online mapping is that it straddles the line between the artistry and communication of cartography, and the precision and programmability of GIS. You can produce great-looking maps that are highly functional and integrate smoothly with your application. It’s my hope that this article demystified the web map stack and will get you thinking about how you can take control of the maps in your site.

Resources/external links#section17

There are many open source projects related to online mapping and GIS. This article touched on these:

• OpenLayers
• TileCache
• Mapnik
• QGIS

In addition, just to name a few: Modest Maps and Mapstraction are browser UI libraries similar to OpenLayers, in Flash and JavaScript, respectively. GeoServer and MapServer are alternatives to Mapnik in the map rendering department. You owe it to yourself to investigate these alternatives, as they each excel in different ways and one may meet your needs better than the others.