Is it possible to mash up data from Direct.gov?
There is useful data in local direct gov – but can we get at it with a web service to create mash-ups for public benefit?
<warning – slightly geeky post coming up>
Last year we tried to raise awareness of the Youth Opportunity Fund and Youth Capital Fund with the Actions Speak Louder campaign. The campaign, targetted at young people – involved a national awards ceremony and publicity campaign – but the goal was to help young people find their local Youth Opportunity Fund grant making panel.
The only place we could find a directory of Local Youth Opportunity fund websites was through Local.Direct.gov.uk, and it seems like you can only search on the Local Direct Gov orange website. However, we wanted to be able to pull a search of local Youth Opportunity Funds into a widget on the Actions Speak Louder mySpace website instead of pointing people off to Direct Gov.
As far as I knew at the time, that couldn't be done. Local Direct Gov doesn't appear to provide an XML feed, or web service API. However, here at the BarCampUKGovWeb Paul Clarke who works with Direct Gov has suggested it might just be possible. He notes:
Hantsweb gave an interesting presentation at the last Directgov Open Day about how one local authority has used the Local Directgov functionality to enhance the way it routes interested citizen to relevant local services in its area (and close surroundings outside the county).
So I'm hopefuly. But with a bit more searching I'm little further forward.
So, I still have three questions:
1) Does anyone know if its possible to query the Local Direct Gov data as a webservice? Or do I have to always direct users off to the dreaded orange pages?
2) If it is – does anyone know how?
3) If it isn't – what would it take to make it possible?
Update: follow up post here
Comments
31 Responses to “Is it possible to mash up data from Direct.gov?”
Feel free to leave a comment...
and oh, if you want a pic to show with your comment, go get a gravatar!
Tim’s Blog
This is the personal blog of Tim Davies, an independent consultant and researcher from the UK, writing about young people's participation, social media and social change.(And also now MSc Student at the Oxford Internet Institute)
I’m not clear on what you’re actually trying to do…
For example:
I’d like to have access to the Local Direct Gov data in a way that made it easy for me to create a Widget that would let a visitor to the MySpace page we used in the campaign noted above, to enter their town or postcode, and to either get taken direct to their local Youth Opportunity Fund web pages, or to have the details of their local Youth Opportunity Fund displayed back to them in the widget.
Update: This post from Simon Berry’s secondment to DCLG might show one way for my particular YOF/YCF suggestion to move forward:
http://www.web24gov.org.uk/node/11
A government data ‘widget factory’…
i read the post thrice without reading the comments and was not able to get to the head or tail of it… but why would you need a widget to get govt data…
Thanks for this work!!! Its great.
Very thanks.
Some nice questions. I would love to answer even one of those but i cant, hehe. But nevertheless, thx for the info
This may sound silly, but did you actually make an attempt to contact them over the matter?
Hi there,
Like some of the other comments here. I think your a bit unclear about what you wnated. But, I am pretty sure I get it.
There are many ways to accomplish what you seek. Perhaps one of the easiest ways maybe to get a current version of FileMaker and use it to extract the data you wish from the target website or any website. Using Filemaker, you can teach it to red and extract what you need.
However, please remember that while governemt websites are generally open for content extraction. They do often contain data that is not FREE to extract without PERMISSION.
Naturally, data from any non-government website, should be assumed not for reprint without permission. Cool.
Now frankly, for the project you seem to be dealing with, I would think that any request for re-print rights will be granted, but always offer a backlink and credit. Also, before investing in Filemaker, you may find that upon request, most source will be happy to make it easy for you to use RSS or some means that they provide to do what you wish.
The moral of the story, asking the website’s webmaster is always a great place to start.Â
Thanks and Best wishes with your project
John
I have a feeling this is not possible but good luck
I read John London’s comment and I think he’s right about contacting the webmaster, put i think there is a way to parse what you need.
Hey there,
It’s an interesting possibility–although scraping XML is much easier than trying to scrape something else. It’ll be interesting to see if you can figure it out…:P
Hey all
Thanks for the many comments. There is a follow up post from around the time of this one here: http://www.timdavies.org.uk/2008/01/29/getting-data-out-directgov
(Not quite sure why this particular post is gaining so much interest now – what brought you all here?)
For the bigger picture on government data – take a look at http://www.showusabetterway.com
There are two issues here:
1) The technical issue – how to get the data out. Parsing and getting all the data into a separate database is less than idea – as it relies on regular parsing to keep the data up-to-date, and is not a stable long-term solution.
2) The issue of principle – Government data should be collected for public benefit – and so should be provided in free and open ways that maximise public benefit from its collection. At present, Local Direct Gov does not make that easy.
“(Not quite sure why this particular post is gaining so much interest now – what brought you all here?)”
You’re such a nice guy!
And your site has good backlinks for commentators. Who wouldn’t love that
I came acrossed your blog while searching for an article I had read about an international gesture to thank soldiers for their services. So many people are a little too shy to walk up to and thank a soldier. So a gesture was created to say thank you without actually having to walk up and initiate a conversation. Anyhow, I was searching for actions speak louder than words and followed a few links and here I am.
This was interesting reading. What would one do when they found their local grant making panel ?
The issue of principle might be the biggest issue.
Well done! Good article.
On the lines of what John London suggested above… I think you may try using a web data extractor application such as webscraper plus which basically takes data from web and puts it into a spreadsheet or database. You may then use that data to build the widget.
As long as you have the government permission to use the data off a .gov site… I think it could be done.
I am pretty sure anything on a US .gov site is free distribution, but only state websites need permission. Has to do with the funding sources for compiling the original information.
As per the whole copyright comments, the following link states “may be reproduced provided it is reproduced accurately and not used in a misleading context”, except material that is “identified as being the copyright of a third party”.
http://www.direct.gov.uk/en/SiteInformation/DG_020460
Providing links directing users to the orange pages at the same time can never hurt
Many states will allow you to use their copyrighted data if you give them credit.
You should try and contact the webmaster … he’ll help you for sure if you give them credit.
The moral of the story, asking the website’s webmaster is always a great place to start.
Thanks and Best wishes with your project
well tim, i think its fully depend on the type of the data, whether its allowed to be publish or its a private .gov data
Hi, Tim
I am a researcher of Institute of Education University in London and we are doing some project about young bloggers.
Would you be please so kind and write me:
what audience would you like to address by your blog : young people, your friends, collegues…
Thank you very much for your cooperation
have a nice day
Andrea Miklovicova
andreafmaatgmaildotcom
Has anyone had any luck making heads ort ails of this?
For developments with getting data from Direct Gov check out the new Direct Gov Innovation site: http://innovate.direct.gov.uk/
wow, thanks for sharing..
Hi,
I thought you might be interested in OpenPSI, a collaboration between the University of Southampton and the UK government, lead by the National Archive, to trial a new form of community provisioned information service.
we are trying to spark interaction between government information providers, academic researchers and information intermediaries, specifically to bridge the gap between those researchers who may not have all
the technical skills or data knowledge to answer important research questions.
We are exposing UK government data in the Semantic Web standards, RDF. We have SPARQL end point so data mashers can issue via query requests. This is a form of open API that allows mashups to be created quite quickly.
There is a current request for information dealing with all forms of data about youth activities which we may try and satisfy.
Hey John
Thanks for pointing out OpenPSI. Looks very interesting in deed – I shall investigate more soon.
Re: the request around Youth Activities – where can I find more on that? Is that activities as in ‘Positive Activities’ – i.e. the sort of things the http://www.plings.net project is looking at?
Tim,
Yes it is about the sort of things the http://www.plings.net project is listing but from a different direction. It is not about the immediate, what can a person do today but about understanding the long term effect that access has. I think what the social science community is trying to understand is impact. They suffer from small survey issue. If we can use technology to build a systematic picture of opportunities for youth and maintain it over time then we can understand how it’s provision helps social need.
So a good research question might be, in a recession if we cut spending on services what is the social impact. We could use the UK government admin data as a backbone for understanding this.
This is just one of the types of question we could investigate with a linked data web of UK government admin data as a backbone.
Many other questions could be investigated.
Hey John
I’d be really interested to talk more about anything you are doing on youth-activity related questions. It would be interesting to see what datasets might help here – as, so far as I’m aware, Plings and Parent Know How are the only government (owner or sponsored) data sources with any reasonable coverage of data on activities – and then the coverage is still problematic…