Posts filed under 'Tech'
I haven’t updated this blog in a while because I’ve been hard at work on some very important changes to the OSM server stack. Have a look at Steve’s announcement for the high-level vision and continue on here for the low-level techniques we’ve used to speed the server up by at least two orders of magnitude.
The new server backend is MonetDB, a well-known and widely deployed open-source database with a proven track record of stability and scalability. It is definitely not an academic project.
But thats not the best part – the secret of the new blindingly fast performance is our adoption of a fixed ontology. Basically, free-form tags were slowing us down, both on the server and in the community. We’ve adopted one of the best industry standard ontologies fronted by several simple APIs . And, in order to prevent piracy of OSM data, the server now supports DRM.
The result of all this hard work by TomH and myself is that we can now run the OSM server faster on less hardware. And the fixed ontology reduces the size of the daily diffs to less than 100 bytes, so we’re distributing them via Twitter. Its what all the cool kids are doing.
April 1st, 2009
Here is a short script to expire tiles based on osmchange diffs. Michal’s comment got me thinking about if there were an easy way to do it and today’s answer is yes, as long as you don’t mind it not being 100% accurate.
The script basically takes all the node locations in the diff, or referenced by ways in the diff, and expires the (meta)tile which contains that point. There are several problems with this approach:
- If a named node (e.g: a pub) is on the corner of a meta tile and its name is modified then the tile(s) containing the name should be expired as well.
- The non-local effects of mapnik’s text placement mean that changes to a way may affect its neigbours, meaning they should also be expired.
- If a way crosses the corner of a meta tile, but has no nodes within it, and is changed then that meta tile should be expired.
However, it seems to work well enough for me at the moment, although I reckon it is probably prudent to expire all the tiles every week or so, just to make sure.
Or the script can be easily extended to expire all the neighbours of expired meta tiles.
September 12th, 2008
I’ve been playing with the diff import for osm2pgsql, which now works wonderfully, and trying to set up a (nearly) live tile-server. To test it, I set it running over the weekend after the Euston micro mapping party, grabbing some tiles every hour and turning them into a movie. It covers the period from about 5pm Friday to 10am this morning with one frame per hour.
September 8th, 2008
After a bit of a quick and dirty hack, there is now a version of Mapnik capable of rendering SVG icons directly to the Cairo back-end. The result is a lovely print-quality map, such as this. Sadly, when I try to load it KPDF crashes. Xpdf is OK, but I don’t know about other renderers… I thought they were all based on poppler these days anyway.
I’ll spend some time cleaning up the new additions and hopefully this’ll make its way into Mapnik’s trunk real soon now.
August 20th, 2008
The icons seemed popular, so here are the original SVGs licensed under the CC-BY-SA license. Enjoy!
August 15th, 2008
I’ve been spending some free time trying to design an icon set for OpenStreetMap. There are several design goals for this icon set:
- They need to work, or nearly work, in black and white as well as colour. This is so they could be used in printed low-colour or greyscale maps.
- They should be visually identifiable by POI class, for rapid readability.
- They should all be the same size and “weight”, so that no single icon overwhelms the others.
You can grab the icons here, version 0.1, with icons for the following POIs; airports, ATMs, bars, cafés, parking, pubs, recycling points, restaurants, bus stops, places of worship (Buddhist temples, Christian churches, Jewish synagogues, Muslim mosques) and supermarkets.
I’ve tried to keep the icon design consistent by using the same rounded-square outline for all the icons at 16×16 pixels. Each different class uses a different flat background colour, chosen to be distinct from all the others (although I should probably check they’re OK for colour-blind people). The icon classes are totally arbitrary, but I’ve been working with the following:
- Blue: transport.
- Orange: Food, drink and socialising.
- Green: Services and shops.
- Grey: Religious points-of-interest.
- Red: Emergency services.
All of the icon strokes are drawn in white on top and are supposed to be internationally recognisable, though some don’t quite live up to that. The pub icon, for example, shows the “nonic” style of pint glass which is common in the UK, but probably not elsewhere. On the other hand, the concept of a “pub” isn’t itself international, so maybe it doesn’t matter. There is an icon for a bar, when this is more appropriate, but I don’t think we have many of them tagged at the moment.
This screenshot shows how different the map looks when POIs like restaurants and cafés are rendered. Hopefully if these are rendered on a layer of the main map more people will start to tag these kinds of features.
Compare the above with the original, which looks rather barren by comparison:
Feedback, requests and suggestions are welcome. The set is still at a very early stage and is missing many icons, but I hope to make it complete at some point in the not-too-far future.
August 13th, 2008
I just got myself a Toshiba U400-108, which works really well with (K)ubuntu 8.04. I haven’t found anything yet which doesn’t work out-of-the-box.
The following stuff all works:
- CPU frequency scaling.
- Suspend-to-RAM.
- The LCD screen (haven’t yet tested the external monitor).
- Wireless networking
The only problems that I’ve found are:
- The laptop gets quite hot under the trackpad (i.e: in the crotch area).
The battery reports zero charge The battery just takes a really long time to charge on the first use.
I don’t think any of these are big issues though, and I’d recommend the laptop to anyone who is looking for a new one. The laptop also has an HDMI socket which appears to be supported by the intel xorg server, but I haven’t tested it yet.
June 28th, 2008
So the new Macbook Air looks very, very cool. Its just the sort of laptop I like and would buy, if I could find a good enough excuse. However, a simple look (as of 17th Jan 2008) at the Apple store reveals that the lower end model retails for $1,799 or £1,020.43 (ex VAT), or an exchange rate of 1.763. Given that the current exchange rate is 1.971, this means Apple is charging £107.76 ($212.42, 11.8%) above what the raw exchange rate is.
Of course, Apple doesn’t have to use the current exchange rate and is probably protecting itself from future fluctuations, but it is really annoying to always be on the wrong side of the exchange rate.
January 17th, 2008
As a necessary first step to getting my wiimote talking to my computer I bought a little USB bluetooth dongle (Linksys USBBT100 which works fine with Linux) and installed the necessary bluetooth stack. When I turned it on and scanned I found three devices within range, all phones, none of which belong to me. Its pretty common-knowledge that bluetooth has a longer range than advertised, but I was surprised to learn that it quite easily connects to other devices through walls or floors.
The people who designed the bluetooth protocol aren’t stupid and they clearly put some thought into making it difficult to eavesdrop or man-in-the-middle the connection with the pairing/encryption thing, but given the number of bluetooth stack exploits, it’ll be a while before I trust a wireless keyboard with my passwords…
July 29th, 2007
A (long) while back I blogged about needing something to backup photos stored on my camera’s memory card while I’m on holiday in case my camera gets stolen again. In the end, I bought the Camera Buddy (40Gb) which, despite a confusing manual, takes xD cards and works absolutely fine under Linux as a usb-storage removable hard disk. Apparently it should also work as an 8-in-1 card reader, but that didn’t work out of the box and I can’t be bothered to get it to work. If I need to read the card directly I can do it via my camera anyway.
July 29th, 2007
Previous Posts