After a while a new status update. As usual even while being silent for a while, the project is still healthy and alive and in fact there were a number of interesting changes – not only by me but by the rest of the community.

Changes since the last update

• Speed increase and reduced memory usage: As usual we did a number of efforts to improve performance while using less memory. Major improvement was the implementation of the low zoom optimizations, dramatically reducing the amount of data required for drawing ways in low zoom. Also in general ways with the same attributes are merged if possible. Also a optional renumbering and resorting step was introduced, better organizing the data in the database files, thus reducing IO during data loading. Future optimization will do the same for areas. But we also optimized the style sheet, indexes and rendering code to load and render data faster.
• Support for multipolygon relations: We now handle multipolygon relations, supporting most standard variants. The code is currently refactored again to correctly handle some more complex cases better.
• Modularisation: We have splitted the code into a number of libraries,so there is now a core library, a library for importing of data, a library for the basic rendering engine and one for each backend.
• Improved style sheet: The style sheet syntax has been changed. The new syntax allows better and more compact definitions while on the same time, preparing support for more features. One feature is, that now values are given in mm, allowing better support on different devices (desktop,mobile) without the requirement to modify the style sheet. We also slightly improved the label placing code and the rendering of bridges and tunnels. Also the rendering of the Qt backend was improved in some situations.
• Improved routing speed and quality: The routing speed was increased by around factor 5-6, while the routing quality has been improved (roundabouts, more flexible profiles, support for max speed, better support for turn restrictions). Much work has been put in the resulting route description, now being of comparable quality to other solutions. The route description generator is now also extensible, allowing you to further enhance the quality without changing the core engine.
• Improved import: Some changes were done to improve import speed and better handle buggy data in the import files.
• Better support for non-Linux platforms, especially Windows (mingw but also VisualStudio) and MacOS/IOS, but also Android. We now have project files for VisualStudio 2008 as part of the repository and some JNI mapping code for using libosmscout under Android.
• There is now also better support for 64bit system and for database files >2GB.
• Bug fixing: A tons of bugs were fixed in all code areas.

Some current screenshots

The first screenshot shows the routing output in an IOS application (this is not the libosmscout renderer, though the cairo renderer has successfully been ported to IOS, too. Rendering speed was around 0.1-0.5 seconds, depending of the scene to render). The community member used the router during his holiday on Costa Rica and was happy to use it – and came successfully back to tell The next two screenshots are by another community member. He uses a custom 3D renderer on top of libosmscout to draw OSM data in 3D. He thus gains all the data structures, import features, the fast database, concentrating on the 3D aspect. Additional OSM tags for 3D description of building were injected into the database, to get evaluated in the renderer. The project can be found here: https://github.com/canurabus/libosmscout-render.

Next steps

This is a “no guarantee” list of changes planed for the near future:

• Work on drawing seas and coastlines. There is already some code, I started a while ago. Now is the time to get back to it to add the last obvious “must have” feature in the renderer.
• Some work on the new style sheet language. The alternateColor for ways will be dropped. Instead a style sheet condition will be added to explicitly differentiate between drawing a simple way and a more complex variant.
• Some more work on the router to make it faster. Currently a path in the graph starts and ends at a junction. However the way merging feature allows to have longer paths and thus skip junctions if they are not the target. This should reduce the number of visited nodes.
• Some work on the style sheet to improve look. Move away from the 1:1 copy of the mapnik style.
• Add some more features to the renderer like drawing arrows for oneway streets.
• Add a tool to do some validation on a style sheet, like hints about types that have no style…
• Promote some demos to first class tools by moving them n the repository.

• The style sheet has been tuned regarding what is shown on certain zoom levels. This results in general drawing speed improvements while at the same time the map is more readable. We still miss a style sheet especially for mobile devices, which should bring further speed improvements while rendering is optimized for less details on hires screens.
• Improved clipping leading to faster rendering
• Improved multipolygon parsing. The “lookup by name” index does directly profit from it, since boundaries are now far better recognized but also some objects not rendered before are now rendered correctly. We still miss real multipolygon drawing including layered drawing and clipping, but this is now planed to be realized real soon now.
• Added support for low zoom optimizations. Rendering for low zooms is now faster because of special preprocessing during import. Also memory footprint on low zoom should be reduced.
• Added support for displaying house numbers. Interpolation is still not supported.
• The tiler demo should now work correctly for all magnifications. Now more clipping artefacts or disappearing objects.
• Some optimizations reducing file sizes.
• Initial version of SVG backend (no text yet). The SVG backend create a local stylesheet that allows to adapt the map via an external style sheet.
• Moved import code into its own library, thus reducing code size and dependencies of the core library. Splitting of backends into their own library is also planned.
• Redesigned rendering code. preprocessing of rendering data before drawing should now be simpler,allowing advanced algorithms for label placement and similar. First benefit is that streets are now drawn by style sheet priority.
• Added some more simple example styles.
• Added API and example for POI lookup.
• A number of bug fixes, making thing more stable during import and map drawing. libosmscout now compiles under Visual Studio 2005.

Performance:

On desktop we now have data fetch rates and rendering times for a 800×480 bitmap always below 200ms (each). For high zoom each is far below 100ms.

Next steps:

• Create a separate library for each backend.
• Support full multipolygon drawing.
• Add support for lane drawing.
• Move from pixels measures in style sheet to mm.
• Change style sheet syntax to support different style dependent on magnification.
• Change to a more CSS like style sheet syntax.

As always some screenshots:

motorways in NRW, Germany. Green dots show service stations, red dots show important cities.

Boundaries in NRW, Germany. Red dots mark important cities.

• I reworked the way index. Now way index again uses a tiled index instead of the quadtree index as implemented for areas. However now tile size depends on the individual type of the way – thus improving index scan performance.
• You can now add a minimum magnification for areas in the style sheet. This allows you to force the drawing engine to start drawing areas of a certain type starting from a certain magnification even if the area would be big enough to be visible on a smaller magnification. The style sheet now uses this for buildings to avoid map overloading (by moving display of building to a bigger zoom level than before).
• Added an IGNORE option for the type definition. This allows you to define types but ignore them in turn during import. This reduces the risk that automatic type detection mechanism guess wrong.
• I did a number of type definition and style sheet file improvements.

Over all this should improve the visual quality, in this case especially for low magnifications, and drawing speed. This also should avoid gaps in motorways as rendered before. See the screenshots (which also features the improved plate labeling algorithmus recently implemented).

New screenshot showing the result of recent changes

New screenshot showing the result of recent changes (zoomed in)

Analysis shows that currently the number of vector points dominates drawing speed and memory consumption. In further steps I will try to reduce the number of points passed to the drawing engine.

Next steps are:

• A demo showing how to efficiently draw tiles.
• Rename the “area index” into “area area index”.
• Add an optimization flag to the type definition. Ways with this flag will be scanned and rebuild into longer ways with fewer points (optimized up to the certain magnification). Map drawing (if requested) will then prefer such data. This should further reduce drawing speed for low magnification tiles.

Reworked configuration files

I did switch from XML based configuration files for preprocessing (*.ost) and style definition (*.oss) to DSL style configuration files (using Coco/R for scanner parser generation). As part of this change I got rid of manually defining Ids to auto generated ids, making configuration simpler.

Improved numeric index

The numeric index now generates index pages of a given block size (4096 by default). This should improve OS low-level file block caching/handling. As a result preprocessing got a bit faster. There are still some code optimizations left, that might increase performance even further.

libagg backend improvements

While the libagg backend is not finished (missing support for external images, some smaller missing features that should be rather easy to add), I added path dash and font support so that resulting maps are now usable. Image quality between cairo, Qt and agg backends is only visibly different in font handling (libagg being better IMHO). I did not yet do any in deep performance comparison, but it looks like performance between cairo and libagg backend is comparable, with libagg possibly being faster in some cases. Main benefit of libagg is the possibility to draw into offscreen buffers without any OS support. Here are screenshots of the same map drawn with cairo and libagg.

cairo backend example

libagg backend example

What happened between this and the last blog entry:

• Improved demo applications. Now the Illumination and Qt demos have the same feature set.
• Improved speed of import (germany in around 40 minutes).
• Support for *.pbf files (which improved execution time of the preprocessing step by factor 12!).
• General improvements of the code base (more documentation and comments, improved interfaces).
• Dramatically reduce amount of memory required (now around 20 MB or less for map of germany).
• Finished Qt backend (visually equal to cairo backend).
• Improved interface for map drawing backends, now even more code is implemented in the abstract base class. This should make implementing new backend even simpler).
• Improved standard style.
• Initial version (no dashes, no labels, no icons) of libagg backend (very fast software vector renderer).

It should now be possible to integrate libosmscout into your applications without much problems.

Next goals:

Performance: It looks like now the physical layout of numeric indexes on disk (not block aligned) and the cache implementation doing a deep copy have most impact on performance. The numeric index just needs to calculate page context based on a given block size and make data align to block starts – and much, much testing. To avoid deep copies I started implementing access methods for data (instead of directly accessing class attributes). This is a necessary precondition for switching to a reference counting solution.

Memory consumption: Above reference counting solution and index optimization will likely further reduce memory requirements. However today libosmscout is already in a state where cache settings and map drawing style have may more influence on memory consumption than the engine itself.

Driver: I will put some more effort into the libagg driver to add support for labels and dashes, however I would prefer if somebody with more libagg experience would do the job, since the documentation is not sufficient and requires reverse engineering of the various examples. libagg is a very fast software renderer allowing fast map drawing on non-accelerated platforms.

However a svg driver would far more interesting, especially if it is style aware, thus allowing to change style in SVG to change all object of one type.  I possibly will do some prototype driver for this, too.

Finally drivers should support drawing of layered multipolygon relations, which in turn should improve rendering time (no need for hardcoded layers for areas and the required drawing of areas in layer order).

Definition files and internal type model: Currently definition files are based on XML and thus a little difficult to understand and change. Also they enforce a dependency on libxml2. I’m now working on switching to a Coco-based DSL, allowing a number of syntax improvements. After that I will do some more changes to the internal representation resulting in:

• Drop of magic ids.
• More flexible type definitions (complex conditions).
• More attribute preprocessing (keyword ‘aspects’), which should result in improved visualisation and possibly smaller data encoding.

Location of todays soccer match: Germany - Italy

Here is the required screen shot showing the location of todays soccer game Germany vs. Italy. As one can see, stadium is not 100% correctly drawn since still lacking support for multipolygons (the importer already supports it, it is just the backends  not making use of it yet).

As always I’m still searching for people helping me in all of the above

I also added OSMScout, a simple Qt application that mimics parts of TravelJinni (the libillumination/cairo counterpart). It not yet has all the features TravelJinni has, but already shows how map integration should be done in Qt.

Below is a screenshot of OSMScout

Screenshot of the OSMScout application

…and the same area as display by TravelJinni:

Screenshot of the TravelJinni application

While visually both maps look very similar (small differences in text rendering) the are performance differences. While Qt claims to be fast, cairo in fact is faster for simple image surfaces (under X11). Judging from Google investigation using a OpenGl painter should burst Qt performance – but this has yet to be tested (and new version of cairo support OpenGl, too).

• I finished basic relation support and simplified the code. We do have support for multipolygon parsing, do support the same attributes like ways and areas have. What is missing is fixing label drawing and correct (at least for common cases) drawing of multiploygon relations – which should enable us to drop the ugly layer hack for areas.
• The “give me everything in an area” index, used for getting the data for drawing a map has also been improved and simplified, resulting in reduced loading and faster scanning. However there is still room for a number macro and micro optimizations to make the index even master and especially loading time shorter.
• The drawing code has been refactored to separate the concrete drawing code (the code that uses cairo)from the drawing algorithm. I did this to support other drawing engines beside cairo – especially Qt. It should also enable us to support other export formats directly (cairo also supports pdf, ps and svg).
• I changed the drawing algorithm and the style sheet to draw outlines only if the way is wide enough. Ways will now be drawn in the “real” with.
• I enhanced the way data to pass  way width as define din the *.osm file to the drawing engine.
• I changed the icon drawing code, so that one can define any icons (no internal enumeration). A few more icons have been added.
• Areas can now have external image patterns. A few examples have been added.

My next plans are:

• Fix some final relation attribute stuff
• Correctly draw multipolygon relations
• Add support for Qt

After that I will concentrate on further  improvements like:

• Better labeling
• Better style language(allowing to use further tags to define a type)
• Optimizing the drawing engine, allowing more parameters etc..
• In parallel I will continue to work on routing improvements