Where Does My Money Go

Open Spending Meeting Minutes 2011-02-10

Rufus Pollock — Tue, 15 Feb 2011 19:15:04 +0000

Participants

Rufus Pollock
Friedrich Lindenberg
Stefan Wehrmeyer
Nadia El imam
Stefan marsiske

Agenda

Previous meetings: http://wdmmg.okfnpad.org/meetings
Updates
Becoming OpenSpending
- http://okfnpad.org/openspending
- http://wiki.openspending.org/Status_2011-02-10
Currently interested countries, cities, places
Immediate tasks
Why is it hard to get data …
- Because gov have it internally and don’t want to / not interested in giving it out

Updates

Friedrich

converted the whole system to mongodb
new aggregator system
- performance
- reified aggregations – i.e. aggregations (the bubbles) are now real things in the system
load eu, fts
new view system – bespoke views
- treemaps

Stef: got uk 25k spending data working, looked at needlebase

Stefan:

refactored js explorer http://rufuspollock.org/wdmmg/explorer.html
UX -> looking at front page (Merge http://data.wheredoesmymoneygo.org/ and http://wheredoesmymonegyo.org/ and http://offenerhaushalt.org/) http://wiki.okfn.org/p/Where_Does_My_Money_Go
flagging system

NEXT STEPS:

Identify theme questions, questions we are interested in having answered

Use: http://ask.wheredoesmymoneygo.org/
Examples questions
- Little money to many small actors, versus big actors
- Energy policy (lisa evans)
Pursue them using the site, ask others interested in answering the same question and engage them into hunting down answers, finding datasets, research
Blog about the experience of trying to do this.
Find out what’s hard & engage others in discussing why that is the case
Announce on the discuss list and document process from start to end
- Announce the challenge – make sure you establish value system ( this is not a witchhunt)
- Story
- Openspending, wheredoesmymoneygo, mailinglist,
- Format for blogs & findings ( encourage people to look for info themselves)

Find out how users navigate the site

Generality versus Specificity

Generality and specificify

Generality is important for the model
Specificity is important for presentation

Where do we spend time

Visualization
Analytics
Community
UX
Acquisition
Usability/Theming/Layout +1 (FL)
- Front page
- Call to action
- Treemap
- Navigation
- Search: entry & dimensions
Content prodution, analysis
Community
- Meetings +1 (stef)
- Data from different countries
- Blogging / Tweeting
Load more data
- ourselves
- enable other people +1 (stef)
Small way to contribute site (interesting)
Technical foundation
- Proper ORM, maybe use general API controller library
- Controller code to model
- de-Wordpress-ify?

Moving to Mongo: Where Does My Money Go’s new back-end

Friedrich Lindenberg — Mon, 14 Feb 2011 19:23:48 +0000

If there was a coder’s lesson in last November’s release of UK departmental spending above 25k, it was this: while there is already a lot of financial information available at the moment, the number of sources is set to grow rapidly – with many local and state governments not only in the UK but all over the world opening their books to an interested public.

For us this poses an interesting problem: how can we create a platform for exposing those records in a way that is easy to use for citizens, easy to extend by anyone to hold new, ever-larger datasets and that supports new modes of exploration and visualization?

Dropping schema

For us, this question had to first be answered in the back-end data store of Where Does My Money Go. Originally, the site was based on a relational emulation of a star schema database. That is, we had a central list of facts (financial records) to which we attached several dimensions (such as the spender and recipient, the affected region or their classification under various schemes) stored in key-value tables. While this allowed us to keep data of different schemata with their associated metadata in a single relational DB, it was both hard to load data into that model and to query the resulting structure.

To replace the existing database, two approaches initially seemed viable: using an RDF triple store or a schema-less NoSQL database. While the RDF route would allow storing a nice graph structure of transactions, spenders and recipients, I was left unsatisfied both by the complexity of the modelling and query methods involved in the RDF stack and the maturity of the available tool chain for Python. Using MongoDB, on the other hand, a running prototype was up and running within two days.

In the resulting MongoDB solution, we’re now using an extremely simple structure to store the data: a largely denormalized, nested collection of payment entries. Each entry contains an abridged representation of the spender and recipient involved, as well as any applicable classifications, such as the COFOG code for UK state spending. This way, we can query the collection using a wide range of criteria. Retrieving, for example, European Commission commitments to companies in the UK that are part of EURATOM can be done with a trivial query:

{ “dataset.name”: “fts”, “to.country”: “United Kingdom”, “article.title”: /euratom/ }

We also decided to create addition collections to explicitly represent the entities and classifiers involved in the entries: While entities are representations of the spenders and recipients of spending, classifiers correspond to the function and programme codes used to distinguish different types of spending. Each of those items was made adressable in the application to allow for detailed research, annotation and visualization of each involved department, company, or any government function.

Aggregating

While individual spending items can contain interesting information or be relevant to the research of journalists, most of the interesting questions are only answered when we start adding up data: How much is spent on Education in my country? How much does the EU spend on biomedical research?

These questions are only answered by aggregating spending. As calculating such aggregates on the fly is a costly thing to do, we’ve now started to pre-aggregate revenue and spending for certain programmes. For this, we’re generating synthetic spending records that express the sum of various underlying data. Aggregation allows us to include a large variety of aggregate figures into pages, allowing for nice visualizations in almost arbitrary places, for example entity and classifier pages.

Data, lots of data

Having upgraded the core of WDMMG, we’re now in a position to easily load a wide variety of different data sources. To explore and test this, we’ve started importing whatever financial we can get our hands on. Some of the exciting new data sources include the European, German, Israeli state and Bergen local budgets. Comparing the EU and German budget, one can discover clear signs of a fierce rivalry to create the most incomprehensible and bureaucratic document on the planet – both are therefore nice tests for our new system.

The European Commission’s Financial Transparency System, a large database of anyone who has been promised money by the EC.
The German Förderkatalog, a national database of federal research grants from the 1960s to today.

With all those data sources, we’re now able to find the same entities as spenders and recipients in various places. As we learn to associate the money flows, our database is going to turn into a Facebook of government funds recipients, connecting the dots between local, state and trans-national spending.

Including more state budgets and spending records from all around the world, for example the UK local government spending that is being released in many places, will provide for a better picture of where to money goes, while improvements to out UI over the next weeks will help to invite others to explore, annotate and collaboratively understand ever more about how government works.

Finally Some Answers on Where My Money Goes

lisa — Fri, 04 Feb 2011 18:42:59 +0000

It’s an innocent enough question:

what does the government spend money on?

Now, I’m not an accountant and I’m not a statistician and personally I don’t have a political axe to grind, I just want some answers to this question that make sense.

This attitude gives me a lot of freedom: I don’t have to get bogged down in any system of understanding spending unless it really helps. But this attitude also gives me a lot of opportunity to look a right idiot, because I’m effectively feeling my way around this unfamiliar financial and political world, pushing for information that may or may not be useful in the end, asking questions that must seem really stupid to experts.

I think it’s only fair that I document my fools progress so that other people who might be on a similar mission can be helped along their way.

So, from the beginning to right now, this is what I’ve found out about government money.

I should add that public spending is making more and more sense to me these days and while I don’t have all the answers right now, I do have a vision for the future and some useful stuff to share.

In the beginning

When I first thought about government spending I had some vague idea that there would be lists of spending, one for each department or maybe one big list for all of government, showing money in and money out for each department. It would just be a matter of finding these sacred and quite possibly hidden lists. I mean someone in the department must be keeping track of the money in and out, right?

The people at Where Does My Money Go? had something that matched my idea of a list of spending. The list they found on the Treasury website showed different government departments and how much they had spent on certain types of public services. I helped to interpret this and the team went on to make this colourful interactive display.

The beauty of this display is allows drilling down into the details of the spending. This is a relatively new technology and lends itself to spending data nicely, it makes those lists accessible and it doesn’t feel like work to look through them they way it does to look into a spreadsheet.

But we quickly found that the list we had used raised more questions than it answered. We only had spending for certain departments, nothing in detail on local councils, and the only details we had were in these odd sounding categories of spending that were designed to allow comparisons between UN countries.

I had a feeling that there must be more detail that we weren’t getting, but where was it?

Therein’ I pestered the heck out of the poor folks at the Treasury about where these spending lists are stored and what else is stored there.

The short answer is that it was all stored in something aptly called COINS, which is a database for frequently revised budgets and something else that is really, really interesting and that I’ll come back to shortly.

A moment of clarity

While I was looking at this spending it slowly dawned on me that just looking at money in and out isn’t the full story of the way public money is managed. For one thing the lists of money in and out, on their own, soon become unwieldy and make it hard for people to answer questions about public spending.

For another thing, Governments don’t just receive and spend money, their wealth is also made up of the things they own like buildings, roads and also determined by the money they owe — the debts as we all know too well.

This is when I found out that accountants have elegant ways to deal with all of this financial jumble.

Let me explain accountancy in the terms I’ve come to understand it.

There are three main figures to consider.

Let’s split the finances of an organisation into either 1) something it is owed or owns and 2) something it owes to someone else. That is two of the 3 main figures explained already.

Just these two figures can give us a simple but important measure of how healthy the finances are. Hypothetically if the organisation received everything it was owed and then sold off everything it has and used that to pay off all its debts, how much would be left? What is left is our third figure. This is what all the people who invested in the organisation would be left with for their return on their investment.

For the mathematically minded this can be expressed at:

1 – 2 = 3 or owed and owned – debts = return to investors

We should find that if we sum the amount of money that people invest and the profit the organisation makes, the result is equal to the third figure.

Just to put this into context for the case where the organisation is a council or government body, the investors are the taxpayers and the bond buyers.

Another way of looking at these three figures is that organisation is ultimately balancing the money it has for its activities and the money it receives and borrows to make its activities happen. So it balances what it owns and is owed (figure 1) with its debts and investments and profits (figures 2 and 3).

For the mathematically minded that is:

2 + 3 = 1 or investment and profits + debts = owned and owed

just a rearrangement of first expression.

If you ever heard of double entry book-keeping then keeping this equation equal is basically it: accounting for what is produced with what is needed to produce it.

There’s lots more to accounting than this, obviously, but I hope I’ve conveyed that this way of categorising financial information is neat and yields useful information.

Here’s the killer

You might ask: wouldn’t it be ideal if each publicly funded body published their accounts under these categories so we can understand their financial health?

It turns out that the government have been producing resource accounts for every department and every local authority for years, and this is the killer: in 2001, yes, 10 years ago, they even produced a guide to interpreting these accounts so that we, the public, can extract all kinds of useful information.

You might ask: wouldn’t it be ideal if someone collected together all the public accounts? Even better if they summed the accounts into one set by eliminating all double counting where public bodies exchange money between themselves.

It seems that at the same time as the resource accounts started to emerge in the early 2000s, the Treasury started to run something called ‘Whole of Government Accounts’. It brings together, or consolidates, all the resource accounts of pretty much every public body into one set of resource accounts. What a perfect way to categorise all public spending at a high level.

Guess where Whole of Government Accounts is stored? In COINS. Disappointingly when COINS was published all of the Whole of Government Accounts was removed. The Treasury plan to publish the first ever version of Whole of Government Accounts this spring.

The vision for the future

The government is currently publishing more spending information than any of their predecessors. We have all central government spending over £25,000 and soon we’ll have all local government spending over £500.

How do we make some sense out of all this spending data?

Lots of information can be made manageable if it can be categorised well.

I’m interested in using the categorisation system in the resource accounts. According to the introduction to resource accounts, it is possible to calculate departmental costs ranging from staff wages to money spent on programmes with their own objectives, as well as lots of useful information about assets and liabilities (what I referred to as figures 1 and 2 above).

I believe using this accounting system and allowing a user to ‘drill down’, as they currently do in Where Does My Money Go?, from this high level organised system into the details of the spending will give ‘accountability’ to the accounts.

There are many difficulties in data compatibility between the resource accounts and the lists of transactions that are currently being published, but I think with some iterations of the data being published, these difficulties could be overcome.

What do you think?

How to find the money in COINS linked data

psychemedia — Wed, 08 Dec 2010 16:17:33 +0000

When I went In at the deep end: getting started with COINS Linked Data, I didn’t actually manage to find any of the money; that is, I couldn’t work out a SPARQL query that would actually return any spending amounts. But after a little bit more digging, I struck gold…!

Before we get to the money, though, it’s worth just doing a little simplification on the way we write the queries. Looking at an example query such as:

PREFIX rdf:
PREFIX rdfs:

SELECT distinct ?deptCode ?deptName WHERE {
?dept ?deptCode.
?deptCode rdfs:comment ?deptName.
} LIMIT 20

The query comes in two parts: one or more PREFIX statements, and then a SELECT statement which actually constructs the query.

Rather than cluttering the body of the query (the SELECT part) with long URL identifiers in angle brackets, we can define a PREFIX to make things a little easier to read. So for example, in the above query, we can replace with something like coins-dimension:departmentCode if we first define PREFIX coins-dimension:

Now, when the query engine sees coins-dimension: in the SELECT part of the query, it knows to replace the whole phrae with a string constructed by appending the bit after the colon on to the end of whatever is inside the angle brackets in the PREFIX definition. Got that?!;-) (At least, I think that’s how it works?!)

Here’s the re-written query:

PREFIX rdf:
PREFIX rdfs:
PREFIX coins-dimension:

SELECT distinct ?deptCode ?deptName WHERE {
?dept coins-dimension:departmentCode ?deptCode.
?deptCode rdfs:comment ?deptName.
} LIMIT 20

(You should be able to just copy and paste it into the SPARQL query box at http://openuplabs.tso.co.uk/sparql/gov-coins)

So, now we can go looking for the money

(Just a quick note before we go any further though… crafting a query to get exactly what you want seems to be non-trivial… the following notes should be seen as work in progress, fumbling attempts on the way to writing a proper query on the COINS Linked Data store. I think it’ll take another post from someone who knows what they’re doing with COINS to write a properly meaningful query…)

A key relation for finding the cash appears to be:

If we define: PREFIX coins-measure: then we can more easily refer to coins-measure:amount. Here’s an example query that looks for a handful of programmes that BIS appears to have spent cash on. (Amount terms are thousands of pounds, I think?)

PREFIX rdf:
PREFIX rdfs:
PREFIX coins-dimension:
PREFIX coins-measure:
PREFIX coins-attr:
PREFIX coins-def:

SELECT ?deptCode ?objectGroupCode ?objectGroupDesc ?amount WHERE {
?transaction coins-dimension:departmentCode ?deptCode.
?transaction coins-dimension:dataType coins-def:outturn.
?deptCode rdfs:comment 'Department for Business Innovation and Skills'.
?transaction coins-attr:programmeObjectGroupCode ?objectGroupCode.
?objectGroupCode rdfs:comment ?objectGroupDesc.
?transaction coins-measure:amount ?amount .
} LIMIT 5

So coins-measure:amount is what you need

But what does the amount actually refer to? Reading through the COINS guidance suggests that we actually need a few more settings.

budget boundary: of the form coins-attr:budgetBoundary ?boundary. ?boundary rdfs:label ?boundaryLabel.
resource capital: of the form coins-attr:resourceCapital ?resCapital. ?resCapital rdfs:label ?resCapitalLabel
data type (i.e. whether the amount is an outturn): using coins-dimension:dataType . Note that there may also be a data subtype, such as whether an outturn has been approved (coins-dimension:dataSubtype)
the accounting period: I think date range arguments take the form: .

Putting this all together, we can come up with a query along the lines of:

PREFIX rdf:
PREFIX rdfs:
PREFIX coins-dimension:
PREFIX coins-measure:
PREFIX coins-attr:
PREFIX coins-def:

SELECT ?boundaryLabel ?resCapitalLabel ?subType ?objectGroupDesc ?amount WHERE {
?transaction coins-dimension:departmentCode ?deptCode.
?transaction .
?transaction coins-dimension:dataType coins-def:outturn.
?deptCode rdfs:comment 'Department for Business Innovation and Skills'.
?transaction coins-attr:programmeObjectGroupCode ?objectGroupCode.
?objectGroupCode rdfs:comment ?objectGroupDesc.
?transaction coins-measure:amount ?amount .
?transaction coins-attr:budgetBoundary ?boundary. ?boundary rdfs:label ?boundaryLabel.
?transaction coins-attr:resourceCapital ?resCapital. ?resCapital rdfs:label ?resCapitalLabel.
?transaction coins-dimension:dataSubtype ?subType. } LIMIT 5

…which is a query that works, although I’m not sure how meaningful it is or what it actually says. (Interpretations in the comments, please

So what do we learn?

Writing queries on the COINS Linked Data is non-trivial: you need a good understanding of both SPARQL and COINS.
Exact/meaningful queries need qualifying along all sorts of dimensions (I’m not even sure if the above query is complete, or whether additional constraints/caveats need applying to turn it into a meaningful query).
When writing “exemplar”, parameterised queries, it would be useful to be able to: - comment them liberally; (can comments be added into the middle of a SPARQL query?) - identify the essential, “do not remove” elements, without which the query is, to all intents and purposes, rendered meaningless.

It’s also worth bearing in mind that the version of SPARQL supported by the COINS Linked Datastore does not support aggregation functions – which means you can’t query for transactions across departments and return totals for each department. Instead, you’d have to return all the transactions and then total them somewhere else. (For a related discussion on this, see @jenit’s SPARQL & Visualisation Frustrations: Aggregation and Projection.)

For the moment then, accessing COINS Linked Data is probably not recommended for the armchair auditor…

Where Does My Money Go? Phase 2: A Review, And Some Next Steps

Anna PS — Wed, 01 Dec 2010 16:09:06 +0000

We’re pleased to announce that Phase 2 of the Where Does My Money Go project is now complete.

This has involved a huge amount of work, completely rebuilding the application from the ground up – from the datastore to the dashboard.

Some highlights:

A complete reworking of the interactive dashboard, with a new version of our flagship financial bubbles visualisation, and new ways to comment on and share what you find.
An interactive tax calculator, showing where your taxes get spent.
A clean new design for the whole site.
An updated and improved datastore with lots of new features including a richer domain model, faceted searching, and integrated graphs.
Boxfresh new data, with UK government spending up till 2009 (2011 if you count projections!).
A new Q&A forum where you can ask and answer questions related to spending and where your tax goes.

Next comes phase 3 – and we discuss what that means for Where Does My Money Go below.

A new look for financial bubbles

Our flagship visualisation shows UK government spending by area, using a clicky, bouncy, zoomy bubble diagram that’s fun to play with.

Our first version was very simple: the latest version builds on the full expertise of design house Iconomical and David McCandless of Information is Beautiful. Here are the results:

Shiny! It’s fresher and cleaner, with friendly icons, designed by David (himself a bit of an icon in the visualisation field).

Unlike the first version, which was standalone Flash, the visualisation is now part of a webpage, and each view of the data has its own unique URL.

So it’s now easier to share and comment on spending, using social media. If you’re interested in, say, the fact that last year the UK spent four times as much on ‘old age’ as on ‘family & children’…

… you can now leave comments on this view of the data, or share the view in Twitter and Facebook. Coming soon: the ability to embed the visualisation in your own page, in an iframe.

Our regional and time-series visualisations stay in place, also with social links and unique URLs: and finally, you can also now compare regional differences in spending.

And finally, a small but significant change. You can now navigate from the visualisation to the underlying data. This will become more useful as the data UI improves – but I’m getting ahead of myself.

Where your tax gets spent: Daily Bread

One of the original aims for Where Does My Money Go was to make government spending feel more real and more personal.

We decided that one of the best ways to achieve this would be through tax. This is because tax is universal, and all too easy to evaluate in real terms.

To paraphrase, rather tastelessly: a million pounds is a statistic, but ten pounds is a pizza.

There are plenty of politically motivated tax calculators out there, but we realised that Where Does My Money Go could do something unique. We could show you, in an elegant, non-partisan way, how much of your daily tax goes to health, how much to education, how much to defence.

Again, Iconomical and David brought their wizardry to bear, and here’s the results.

How did we calculate the tax numbers? Following discussion on our Ask site, we used Treasury figures for the total tax paid by households in each income decile, including indirect taxes like VAT and fuel duty.

When you enter your income, we figure out which decile you’re in, and then your tax proportionally. This calculation (with more details) is available through our API.

Obviously, there’s some handwaving here: individuals are different from households; VAT will vary with what you buy; and precise allocation of spending is always metaphorical.

But we believe this is about as meaningful as these things get. We work out how much tax you pay to central government: and we know where central government revenues (which are almost entirely tax) go.

Again, you can link to individual views of the data: here’s the breakdown for someone who earns £144,000 a year, slightly more than David Cameron.

Data updates

We’ve also made some substantial improvements to the datastore. These include:

Search: You can now search all the data in the datastore, for a supplier name or phrase, and get results quickly. And you can refine your search results quickly for a particular supplier – see the screenshot below.
Payments visualised over time: For each search result and each supplier, we also show a graph of payments over time. The screenshot below shows payments to Capita.
Data: We’ve loaded the latest Treasury data (CRA 2010), which shows actual spending up until 2009. A few weeks ago, we also added central government spending over £25k.
Technical improvements: The datastore now runs off an OLAP domain model – and we’re currently upgrading it even further to use Mongo. Both increase its power and flexibility.
Comments: And finally, you can now comment on individual entries in the datastore.

For those who aren’t sure what the datastore is – it’s the site that drives the visualisations. Currently, that’s pretty much all it does, but we have ambitions to make it much more usable and powerful.

What next for WDMMG?

So, that was phase 2 of Where Does My Money Go. We hope you like it.

What next? Broadly, we know that phase 3 will involve making the site easier to use, and getting more people involved.

Last week, we started to outline the priorities for this phase, and we decided that they included:

Datastore: UI improvements so it’s easier to search the data, and the ability to annotate items of spending with information.
Crowdsourcing: Expanding our potential dataset by allowing users to add spending items.
International work: Working with initiatives in other countries to map budget data, making our tools more resuable in the process.

But we need your views, on phase 2, phase 3 and on the future: please sign up for our discussion list to join the conversation.

In at the deep end: how to get started with COINS Linked Data

psychemedia — Fri, 26 Nov 2010 09:23:16 +0000

A tweet today from … someone?! … alerted me to the fact that COINS data has just been published as Linked Data over on the TSO Open Up Labs website. The documentation is still in the works, and as yet there aren’t any example queries available showing how to get started with querying the dataset, so I thought it might be worth trying to pull a few simple queries together to at least providing a jumping off point for exploring the data via the SPARQL interface.

I’m afraid this is all pretty much in at the deep end, but I’ve tried to construct queries that you can: - copy and paste straight into the form on the TSO OpenUpLabs website; - edit in hopefully obvious places to create your own queries; - combine together, in part, to create more complex queries.

At the moment, I’m leaving the money out of it, in favour of just getting a feeling for the structure of the data, and how to interrogate it.

So let’s get started…

The following query will generate a list of department codes and the associated department names – copy and paste it into the form at http://openuplabs.tso.co.uk/sparql/gov-coins, select the “Plain Text” output, and hit submit:

PREFIX rdf:
PREFIX rdfs:

SELECT distinct ?deptCode ?deptName WHERE {
?dept ?deptCode.
?deptCode rdfs:comment ?deptName.
} LIMIT 20

(For more, or fewer, results, change the LIMIT amount, or remove it altogether.)

After a short delay (?! – I’m not sure if I could optimise the query…? ) the results will come back looking something like this fragment:

$deptCode $deptName
“Serious Fraud Office”
“HM Revenue and Customs”
“Office of the Immigration Services Commissioner”

To find the code for a particular department, we can try searching around a word that appears in the name of the department we are looking for – such as “Business”:

SELECT distinct ?deptCode ?deptName WHERE {
?dept ?deptCode.
?deptCode rdfs:comment ?deptName.
FILTER (regex(str(?deptName), 'Business'))
}

The following query returns the names of programmes, and their codes, for the Serious Fraud Office (which has identifier ):

PREFIX rdf:
PREFIX rdfs:

SELECT distinct ?progCode ?progName WHERE {
?dept .
?dept ?progCode.
?progCode rdfs:comment ?progName.
}

We can also run more refined queries, for example search for projects to do with Nuclear topics in the Department of Energy and Climate Change?

PREFIX rdf:
PREFIX rdfs:

SELECT distinct ?progCode ?progName WHERE {
?dept .
?dept ?progCode. ?progCode rdfs:comment ?progName.
FILTER (regex(str(?progName), 'Nuclear'))
} LIMIT 5

Many of the items in COINS involve one department as counterparty to another. So how do we see which other departments are counterparty to the Cabinet Office and on which programmes?

PREFIX rdf:
PREFIX rdfs:

SELECT distinct ?v ?ccLabel ?progName ?progCode WHERE {
?dept .
?dept ?progCode.
?progCode rdfs:comment ?progName.
?dept ?v.
?v rdfs:comment ?ccLabel.
FILTER (!(?v = ))
} limit 20

So, now we can start to follow the money… Only, err, erm, I haven’t worked out how to find the money yet!

Video: a beginners guide to Where Does My Money Go?

admin — Wed, 24 Nov 2010 14:50:54 +0000

With many thanks to Daniel Dietrich who made this possible.

How to explore government spending over £25,000 on Where Does My Money Go

Anna PS — Fri, 19 Nov 2010 15:09:22 +0000

Today, the UK government published its spending items over £25,000. From now on, every month you’ll be able to see just what each central government department spent, with whom, and when.

Exciting stuff – the current government came to power promising a new era of open data, and this is probably its most significant release so far. The papers today are full of scandalous tales of £170,000 on bottled mineral water, but the real stories are still buried deep in the data.

Here at Where Does My Money Go, we were lucky enough to be given early access to the data. Rufus and I have spent a few days loading it into our data store, and we’ve made 150 raw CSV files and 200,000 spending items into something you can search and explore online.

A map of government spending

So what can you do with the data store?

Firstly, you can simply search it – here’s all payments made to IT supplier Atos Origin, for example. The data begins on 12th May 2010, so we learn that since then, 613 separate payments have been made.

On search results, we calculate the total spending for those results, draw atimeline of payments, and show which departments spent most.

Since outsourcing is clearly the business to be in, let’s check out Capita:

Capita has received a remarkable £3.35 billion, just in the five months since May 2010! (To put this in context, we know from our spending visualizations that we spent £5 billion on primary education in the whole of 2009.)

We’ve also created an individual page for each spending item, showing all the information we have about it from the files released today. The Capita figures include a huge single payment of more than £600 million, so let’s check that out.

Ah, pensions. As you can see, we’ve added the ability to make one-click Freedom of Information requests about individual items. If you click the button, it’ll take you straight to a page on WhatDoTheyKnow, pre-populated with all the information you need to make an FOI request.

That’s because spending items by themselves don’t always tell the whole story. This isn’t the gold itself: it’s just a map of where the gold might be.

Here be dragons

This is genuinely exciting data, and it can’t have been an easy task for the Cabinet Office to get it out of departments – we’re thrilled to have it.

Departments are clearly still learning how to release data in standard formats. But the fact that I’ve just spent an hour writing Python to convert 40416 into 26/8/2010 (yay, Excel serial numbers) doesn’t dim my happiness.

That said, there are some additions and changes we’d really like to see in future releases.

Unique identifiers. Suppliers have an exciting variety of names in this data, because different departments call them different things: Arval, Arval UK Ltd, Arval (UK) Ltd… If the data consistently included unique VAT registration numbers, it would be much easier to analyse, and to link it with other datasets, like Companies Open House. And that’s where it’s going to get really interesting.
Consistent classifications. The data tells us the general area of spending for each item – ‘Legal Services’ or ‘Overseas Subsistence’. However, these aren’t used consistently across departments. That’s why we haven’t visualized spending in different areas (yet) – it was just too hard to compare.
Redaction. Departments are allowed to remove items that should not be released under the Freedom of Information Act. This is legal and sensible. However, FOIA is vague, and we don’t know when items have been removed. The Local Public Data Panel warned councils that redactions should be (a) very rare and (b) justifiable. We think it’s very important that the Treasury’s guidance to departments say the same thing.

Go forth and explore

So here we have it: (almost) all spending above £25,000, opened up for the first time. Quite a day.

Here at Where Does My Money Go, we think spending data is exciting because it shows us what the big, sprawling, confusing thing called government actually does. Today, finding out got a whole lot easier.

Just browsing the data is instantly intriguing: why are we giving money to News International? What about 04 Bras Limited? And who even knew there was an International Potato Center?

But the real power of this data will become clear in the months to come, as developers and researchers – you? – start to link it to other information, like the magisterial OpenlyLocal and the exciting WhosLobbying. Please make use of our API and loading scripts to do so.

Finally, we know that our data explorer needs to become clearer and less buggy. We’ll be working on this in the coming weeks, and if you have any suggestions, please get in touch.

Government Spending – Data Explorer

psychemedia — Fri, 19 Nov 2010 11:47:09 +0000

A whole swathe of data may now out there, but how can we start to explore it? Here’s one way, using the Google spreaadsheet versions of the data, at departmental level, as republished on the Guardian DataStore:

The application allows you to select a departmental spreadsheet and then write queries on it. Queries can be bookmarked (err, most of the time – the code is really horrible, and may be a bit flakey…) and you can also get links straight from Google spreadsheets to CSV and HTML versions of the data returned from your query.

Here are a couple of examples:

Dept of Health: view suppliers and amounts where the amount is negative… [try it, HTML preview]
Department for International Development: view suppliers and amounts over a million for expense areas containing ‘Africa’; (as well as the free text search ‘contains’, ‘matches’ must match exactly) [try it, HTML preview]

As well as displaying tabular data, some queries can be visualised using simple chart types. (For example, queries that return two columns, where the first contains a label and the second contains a number can be visualised using a bar or column chart.)

Queries can do basic calculations, summing values in a particular budget area or to a particular supplier for example.

If you find an interesting query, bookmark it and tag it using wdmmg-gde10. I’ll try to add some of the most interesting queries to the examples.

If you want to run with the idea and write the code properly, please feel free to do so. On my to do list is a tool to search across all the spreadsheets using the same search terms. (As a stop gap, scraperwiki might provide one way of doing this, pulling the CSV from the spreadsheets via auto-generated queries.)

25k Departmental Spending Data in the Data Store

Rufus Pollock — Fri, 19 Nov 2010 07:32:19 +0000

The UK government has just released details of departmental spending over 25k and it has all been loaded into the data store:

25k Departmental Spending Data in the Where Does My Money Go? Data Store

As a sample of the things you can find here are some examples of what you can find:

Where Does My Money Go

Open Spending Meeting Minutes 2011-02-10

Participants

Agenda

Updates

NEXT STEPS:

Identify theme questions, questions we are interested in having answered

Find out how users navigate the site

Generality versus Specificity

Where do we spend time

Moving to Mongo: Where Does My Money Go’s new back-end

Finally Some Answers on Where My Money Goes

In the beginning

A moment of clarity

Here’s the killer

The vision for the future

How to find *the money* in COINS linked data

Where Does My Money Go? Phase 2: A Review, And Some Next Steps

A new look for financial bubbles

Where your tax gets spent: Daily Bread

Data updates

What next for WDMMG?

In at the deep end: how to get started with COINS Linked Data

Video: a beginners guide to Where Does My Money Go?

How to explore government spending over £25,000 on Where Does My Money Go

Government Spending – Data Explorer

25k Departmental Spending Data in the Data Store

How to find the money in COINS linked data