Posts filed under 'Tech'
I gave a talk at State of the Map, OSM’s annual conference, about the minutely diffs; why they should be used, a little introduction on how they could be used and some examples of things that already use them. I also used the talk to advertise the OWL system I’ve been writing.
Here are the slides from the talk – click on any of them to see the original SVG. I tried writing the slides in OpenOffice, but couldn’t figure out how to make it do what I wanted. So I ended up writing them in Inkscape instead, which I think gives them a much nicer look. Finally, the font I used is an awesome open display face called Cantarell.
The slides are interspersed with my notes. Unfortunately, these were totally wiped from my mind the moment I went up on stage, so they probably bear little resemblance to what I actually said.
Hi. [Insert introduction here].
[The usual "tell the audience what you're about to tell them" stuff]
Why not just use the planet? OSM releases the full dataset on a weekly basis, which is still better than Google, TA/NT or OS. Compared to them, our feedback cycle is almost instant. But it could be better; I’m here to convince you there’s good reasons to move from weekly updates…
…to minutely.
There’s a fascinating study by these guys on the effects of feedback frequency on motivation. Basically, they asked a bunch of school children to give presentations, a bit like this one, and they were told they’d be given their grades after one day, one week or three weeks. Which group performed better? The ones who were given their feedback faster. Faster feedback can motivate people; it can motivate the users of your tools.
The other part of it is that it’s confusing – I’m sure many of us remember when people would ask, “I did an edit yesterday – why hasn’t it appeared on the map yet?” After you’ve explained that the rendering updates each Wednesday or Thursday for the Nth time, you start to like this minutely updated thing.
There’s another good reason, which anyone who processes planet files each week will recognise – speed.
Waiting for planet files to download is boring. Waiting for them to be processed is boring. And keeping a few around can fill up a disk pretty quickly. Your hardware could be doing 35x less work each week and, what’s more, it can be doing it spread out over the week instead of in one intense period, which means you need less hardware.
So there’s three really good reasons to go minutely: Your users will be more enthusiastic and motivated to use your tools; they’ll not be confused because there’s very little lag between their edits and stuff appearing; and it’s faster with less data to process.
That’s the “why”, but processing diffs is harder than just processing the planet. So here’s the “how”.
Working with the diffs pretty much falls into one of these three categories.
You can just work very, very quickly. Using the diffs you could just update a planet file or database and just re-process it. If you can do a planet’s worth of data in less than a minute I’d like to hear from you!
A more typical approach is to apply the diffs to a manageable subset of the data. This works great, but sometimes a subset of the data just isn’t enough.
By the end of this talk I hope to convince you that this third option is the best. Basically it’s trying to figure out what the changes in output will be for any diff input, so you don’t have to recompute huge amounts of the output or running expensive queries over large amounts of data. It’s all very well to say this, but let’s see how it can actually be done.
Every technical talk has to have a slide with way too much text, code or XML on it. So here’s mine.
It’s an excerpt of an “osmChange” file – the format that’s used for the minutely, hourly and daily diffs. It’s pretty much the same as the normal OSM format, which I assume you’re all intimately familiar with, but with the addition of…
…these top-level elements; create, modify and delete. These tell you what’s happening to the elements they contain.
There’s not always going to be three of them, but you get the general idea.
Let’s look at a simple example.
[And this is where I ran out of time to make notes!]
July 12th, 2010
I haven’t updated this blog in a while because I’ve been hard at work on some very important changes to the OSM server stack. Have a look at Steve’s announcement for the high-level vision and continue on here for the low-level techniques we’ve used to speed the server up by at least two orders of magnitude.
The new server backend is MonetDB, a well-known and widely deployed open-source database with a proven track record of stability and scalability. It is definitely not an academic project.
But thats not the best part – the secret of the new blindingly fast performance is our adoption of a fixed ontology. Basically, free-form tags were slowing us down, both on the server and in the community. We’ve adopted one of the best industry standard ontologies fronted by several simple APIs . And, in order to prevent piracy of OSM data, the server now supports DRM.
The result of all this hard work by TomH and myself is that we can now run the OSM server faster on less hardware. And the fixed ontology reduces the size of the daily diffs to less than 100 bytes, so we’re distributing them via Twitter. Its what all the cool kids are doing.
April 1st, 2009
Here is a short script to expire tiles based on osmchange diffs. Michal’s comment got me thinking about if there were an easy way to do it and today’s answer is yes, as long as you don’t mind it not being 100% accurate.
The script basically takes all the node locations in the diff, or referenced by ways in the diff, and expires the (meta)tile which contains that point. There are several problems with this approach:
- If a named node (e.g: a pub) is on the corner of a meta tile and its name is modified then the tile(s) containing the name should be expired as well.
- The non-local effects of mapnik’s text placement mean that changes to a way may affect its neigbours, meaning they should also be expired.
- If a way crosses the corner of a meta tile, but has no nodes within it, and is changed then that meta tile should be expired.
However, it seems to work well enough for me at the moment, although I reckon it is probably prudent to expire all the tiles every week or so, just to make sure.
Or the script can be easily extended to expire all the neighbours of expired meta tiles.
September 12th, 2008
I’ve been playing with the diff import for osm2pgsql, which now works wonderfully, and trying to set up a (nearly) live tile-server. To test it, I set it running over the weekend after the Euston micro mapping party, grabbing some tiles every hour and turning them into a movie. It covers the period from about 5pm Friday to 10am this morning with one frame per hour.
September 8th, 2008
After a bit of a quick and dirty hack, there is now a version of Mapnik capable of rendering SVG icons directly to the Cairo back-end. The result is a lovely print-quality map, such as this. Sadly, when I try to load it KPDF crashes. Xpdf is OK, but I don’t know about other renderers… I thought they were all based on poppler these days anyway.
I’ll spend some time cleaning up the new additions and hopefully this’ll make its way into Mapnik’s trunk real soon now.
August 20th, 2008
The icons seemed popular, so here are the original SVGs licensed under the CC-BY-SA license. Enjoy!
August 15th, 2008
I’ve been spending some free time trying to design an icon set for OpenStreetMap. There are several design goals for this icon set:
- They need to work, or nearly work, in black and white as well as colour. This is so they could be used in printed low-colour or greyscale maps.
- They should be visually identifiable by POI class, for rapid readability.
- They should all be the same size and “weight”, so that no single icon overwhelms the others.
You can grab the icons here, version 0.1, with icons for the following POIs; airports, ATMs, bars, cafés, parking, pubs, recycling points, restaurants, bus stops, places of worship (Buddhist temples, Christian churches, Jewish synagogues, Muslim mosques) and supermarkets.
I’ve tried to keep the icon design consistent by using the same rounded-square outline for all the icons at 16×16 pixels. Each different class uses a different flat background colour, chosen to be distinct from all the others (although I should probably check they’re OK for colour-blind people). The icon classes are totally arbitrary, but I’ve been working with the following:
- Blue: transport.
- Orange: Food, drink and socialising.
- Green: Services and shops.
- Grey: Religious points-of-interest.
- Red: Emergency services.
All of the icon strokes are drawn in white on top and are supposed to be internationally recognisable, though some don’t quite live up to that. The pub icon, for example, shows the “nonic” style of pint glass which is common in the UK, but probably not elsewhere. On the other hand, the concept of a “pub” isn’t itself international, so maybe it doesn’t matter. There is an icon for a bar, when this is more appropriate, but I don’t think we have many of them tagged at the moment.
This screenshot shows how different the map looks when POIs like restaurants and cafés are rendered. Hopefully if these are rendered on a layer of the main map more people will start to tag these kinds of features.
Compare the above with the original, which looks rather barren by comparison:
Feedback, requests and suggestions are welcome. The set is still at a very early stage and is missing many icons, but I hope to make it complete at some point in the not-too-far future.
August 13th, 2008
I just got myself a Toshiba U400-108, which works really well with (K)ubuntu 8.04. I haven’t found anything yet which doesn’t work out-of-the-box.
The following stuff all works:
- CPU frequency scaling.
- Suspend-to-RAM.
- The LCD screen (haven’t yet tested the external monitor).
- Wireless networking
The only problems that I’ve found are:
- The laptop gets quite hot under the trackpad (i.e: in the crotch area).
The battery reports zero charge The battery just takes a really long time to charge on the first use.
I don’t think any of these are big issues though, and I’d recommend the laptop to anyone who is looking for a new one. The laptop also has an HDMI socket which appears to be supported by the intel xorg server, but I haven’t tested it yet.
June 28th, 2008
So the new Macbook Air looks very, very cool. Its just the sort of laptop I like and would buy, if I could find a good enough excuse. However, a simple look (as of 17th Jan 2008) at the Apple store reveals that the lower end model retails for $1,799 or £1,020.43 (ex VAT), or an exchange rate of 1.763. Given that the current exchange rate is 1.971, this means Apple is charging £107.76 ($212.42, 11.8%) above what the raw exchange rate is.
Of course, Apple doesn’t have to use the current exchange rate and is probably protecting itself from future fluctuations, but it is really annoying to always be on the wrong side of the exchange rate.
January 17th, 2008
As a necessary first step to getting my wiimote talking to my computer I bought a little USB bluetooth dongle (Linksys USBBT100 which works fine with Linux) and installed the necessary bluetooth stack. When I turned it on and scanned I found three devices within range, all phones, none of which belong to me. Its pretty common-knowledge that bluetooth has a longer range than advertised, but I was surprised to learn that it quite easily connects to other devices through walls or floors.
The people who designed the bluetooth protocol aren’t stupid and they clearly put some thought into making it difficult to eavesdrop or man-in-the-middle the connection with the pairing/encryption thing, but given the number of bluetooth stack exploits, it’ll be a while before I trust a wireless keyboard with my passwords…
July 29th, 2007
Previous Posts
