Thanks for the feedback on the previous post on imports; it got me thinking about whether my model had been over-elaborate. Specifically, the limit on the model agents which capped the maximum “completeness” levels at which they would contribute. I thought this was a necessary part of the model, but after investigation it turns out it isn’t.
The following graph shows what happens when this restriction is removed. Note that I’ve had to run the simulation at higher levels of import for the effects to be really visible — there’s no doubt that, if editors really do continue to contribute after the map has passed their personal threshold of “good enough” (or they don’t have such a threshold), then the imports have less of an effect than they do assuming such a threshold. But they still have a detrimental effect.
Let’s examine the evidence, taken from the changeset history of OpenStreetMap. If people don’t have a threshold then we would expect to see, in areas which are very well-mapped, a similar editor distribution in later edits as we do in earlier edits. The editor distribution shows the cumulative number of edits (in this case changesets) against the cumulative editors themselves. For comparison, it looks a lot like the income inequality charts (Lorenz curves) used to calculate the GINI coefficient. Here’s the chart for London, UK.
The red curve represents those edits which are in the top 100 most recent for any particular bounding box, the green line represents those which aren’t. The reason for using recency count, rather than the time of the changeset, is so that there’s no bias towards recently-mapped areas; the 100th changeset in an area is always the 100th changeset regardless of history.
It’s clear that the more recent edits are less equally distributed amongst the users, but the gap between them isn’t huge. Possibly this is an artefact of choosing London as the test-area, as it was the first place to be edited in OpenStreetMap, so maybe has a weird editing history. Let’s look at Den Haag, NL instead.
The same effect is visible, but not as pronounced. There is no firm conclusion to this, but maybe a slight suggestion that there is an inequality between recent editor activity and older editor activity in the same area. There are some problems with this approach, however, primary amongst them; all changesets are not equal and counting them in this fashion makes assumptions about the statistical distribution of work (FSVO “work”) within changesets.
Let’s look at some primary data then; the unique number of users per month in particular areas. First up, the USA.
The grey areas behind the curve are the periods in which TIGER data was being imported. Interestingly the first import doesn’t seem to have had much effect. About three months after the second, though, the editor growth rate seems to have dropped off. Maybe it’s an artefact, due to the low population density or something. Let’s look at somewhere with a higher density, the Netherlands.
The grey area, again, is the import. And, again, there’s a drop-off about three months after the import is finished. Maybe everywhere has a drop-off around the beginning of 2008? Let’s look at some more places.
Both Germany and Denmark had drop-offs in editor growth, albeit about 6 months after the Netherlands. But the UK seems to have had it’s drop-off much earlier — or maybe it just hasn’t had it yet. The kink in the graph, then, seems to be something natural (well, i fitted two curves, so I’d be surprised if it didn’t kink somewhere), but USA and NL kink earlier and lower than any of the others except the UK. The UK, though, has the highest current fit growth rate, so maybe it wasn’t a good candidate for a two-line fit.
In conclusion; even without the assumption that agent-editors have a threshold for contributions, the theoretical model still predicts that imports damage the growth of the editor community. There’s no conclusive evidence for this in practice, although there is some circumstantial evidence. At this point it’s difficult to say for sure whether the effects shown here are due to imports, or due to seasonal effects or anything else. For answers to those questions we’ll need more sophisticated analyses or more data.
September 10th, 2009
I’ve been thinking recently on the effects of imports on the OpenStreetMap community. But there’s no real hard data — and repeatable experiments would be difficult if not impossible. So I’ve turned to that old “what if” machine; the Monte-Carlo simulation.
A caveat before reading the whole post: I’ve no way of backing any of this up, so treat it as an “informed guess”. There are assumptions about the dynamics of the community which I’ve used in the model which may, or may not, be true in reality.
The first case is looking at the effect of an initial import (i.e: before I’ve started the community simulation) on the completion of the map. What I’m simulating is thousands of contributing agents, generated based on levels of local activity on the assumption that many mappers are recruited either through friends or by local events. There’s also a background, random level of recruitment from news articles, long-range friends, etc… All contributor sign-up is linked to a population density field so that the distribution of mappers is greater in urban areas, as it is in real life. Mappers also have a “comfort radius” within which they will map, but not go beyond, which is probably the case for most casual mappers. Finally, there is a “completeness” threshold for each mapper which controls the extent to which they’ll “complete” the map. This simulates an effect I’ve seen in real life — that different people care about, and will map, different things and few people will map absolutely everything and anything.
Anyway, on to the results. The first graph is instantly informative:
Basically; the more we start with, the less you end up with and the longer it takes to get there. The worrying thing is that the asymptotic level of “completeness” (for whatever value you consider as an indicator of quality) is inversely related to the level of initial import. The asymptotic level of completion isn’t 100% because the model accounts for very rural areas, where recruitment is poor, and doesn’t account for the mobility of mappers. A simplifying assumption is that they’ll stay within their “comfort radius” of home.
But we’ve got communities everywhere. So what happens when the import is done after the community has started growing, and started mapping its area?
Hmmm… Pretty much the same thing, but less so. A word here, though, about the “import” that I’m doing into the model. It’s a best-case import which completes the map to a certain degree everywhere and doesn’t stomp on, duplicate or otherwise maul anyone’s existing data, so I don’t model the corresponding drop in the community which might result from people feeling their efforts were steam-rollered. Also, the “import” is considered to be good data, something which is rarely the case in the real world.
But what happens when the import covers only a part of the total area?
This is better still, if only because the impact of the import is localised to a part of the community. But what happens if the import isn’t contiguous, but spread randomly over a fraction of the area?
Ah… Finally, some benefit to the imports — there doesn’t seem to be any long-term damage to the community and the “completeness” levels out at roughly the same place. The “import” here was to a small fraction of the area and, again, I didn’t simulate any direct effect on the community, or any time lost in fixing-up and integrating such data into the existing map.
So what happens if we break an import up into lots of mini-imports and spread them out so that the rate of importing is roughly constant?
Interestingly, here we can see even less of a long-term effect on the rate of completion. In fact, all the curves seem to converge pretty rapidly over time. However, if you’re looking for 90% completion, it seems the approach with incremental imports makes no measurable difference to the time taken, or to the final levels of completion.
What, if any, conclusions can we draw from this? It seems that small, non-local imports aren’t detrimental, and may even be beneficial, to the growth of the community and the map. Imports of specialised features, such as bus stops or water areas, can be very useful in filling in the gaps where other contributors may not want to contribute.
However, it seems that large-scale imports, particularly of “fundamental” features such as road networks, can cause problems with the growth of a community, particularly if the import is done at a very early stage in the growth of that community.
In conclusion, there’s no evidence here that imports are all bad, but it’s my personal opinion that imports, particularly if they’re badly thought-through, too wide-ranging or don’t do enough to engage the existing community, can be bad. And often are.
September 6th, 2009