For me, the strongest messages to come out of the day were:

• Evaluating projects is about demonstrating success and learning from the experience therefore the first thing to do when planning a project evaluation is to decide who it is for and what it is supposed to do. Once you know the audience that the evaluation is aimed at and what you want them to do as a result of seeing the results then planning what you are going to do becomes much easier.
• Focus your evaluation. Making it tight and focused will make it easier to plan and undertake. A good way to focus an evaluation is to use your project objectives, these can give you a few high level questions that your evaluation can seek to answer.

These are just the stand out points of a very rich discussion. The afternoon session was a general discussion of the concerns that people had about project evaluation and it threw up a wealth of good advice and ideas for evaluating projects. I thought it would be useful to write up the notes from this session so that others could use the advice.

Effort

The first concern we discussed was: how do you know how much effort should be directed at a project evaluation?

Unfortunately there is not an easy answer to this and it really depends on the type of project you are managing and what it aims to do. If, for example, your project is trying to prove that a certain approach is a good idea then a robust evaluation is probably quite important, however if you are developing some new technology then perhaps evaluation needs to focus on whether user requirements are met. So to decide what is required you first need to decide what role the evaluation is going to play in your project. JISC programme managers are there to support projects with this kind of decision so it is a good idea to give them a call to discuss it. It is also important to realise that the role of the evaluation in your project may change over time.

Timings

The second concern was about when should a project evaluate? The group all agreed that gathering data throughout the project was essential and wherever possible evaluation should be built in to project activities such as events so that data can be gathered.

We also talked about how you evaluate projects when the real impact and benefits will occur some time after the end of the project. We decided that this was a built in limitation of innovation projects and it needs to be recognised that it is not realistic to prove impact a few years down the line in project evaluations. However it may be possible to gather evidence that shows that this future impact has a chance of occurring and to make informed speculations about its likelihood.

Impact also depends on the type of project. Some projects are assessing an unproven technology whereas others are seeking to embed use of a proven technology in an institution. For embedding projects, impact should be a concern as it shows the success of the project, however for projects assessing unproven technology, impact is probably less important.

What and how

The group discussed practical ways to gather different types of evidence for projects:

• Set project checkpoints to assess progress but ensure you think through why you are doing this and who you are doing it for.
• Consider the use of some kind of daily/weekly log or diary, blogs are very useful for this but you may also need to record some things that are not for public consumption.
• Plan in reflective time to allow yourself to record and consider the events that have occurred in the project. Making this a regular habit will make it more useful. To ensure you keep the habit up, it is a good idea to make it a part of project meetings or calls. 
• Collect interesting anecdotal evidence that you pick up from events, twitter, meetings, emails etc. This could all be useful when writing up evaluation results.
• Think about embedding evaluation in each project output to try and get a regular flow of evidence rather than trying to gather it all at the end.
• Think about arranging controlled experiments to judge whether your project has made a difference to end user experiences.

Audience

Deciding who you are evaluating for is a vital step in planning the evaluation and should be done right at the start. If this audience is defined then it makes it easier to decide how you need to present the information gathered as a result of the evaluation.

JISC is one possible audience for the results of a project evaluation, however, it is also worth considering who else needs to know what the project has produced, what was learned and how successful it all was. For example, if your project is focused on implementing an institutional repository then perhaps you need to think about demonstrating to senior managers that the repository is an essential piece of institutional infrastructure so that steps are taken to ensure its sustainability once the project funding has ended. Similarly, if you are seeking to embed a repository in user workflows then perhaps the evaluation needs to focus on gathering evidence that persuades researchers that the repository offers them tangible benefits.

Meaningful

Ensuring that the project evaluation is meaningful is also important. This is an issue that is also dependent on defining an audience for the evaluation, once you know who they are then ensuring that the evaluation means something to them is easier.

Another important point about making an evaluation meaningful is to ensure that things that don’t work are recorded as well as successes. This evidence is really important as it can save other people time and effort. The recent JISC rapid innovation projects used blog posts entitled Small Wins and Small Fails to record things that worked and things that didn’t and I think this is a model that could work for other projects.

User and stakeholder engagement

User and stakeholder engagement was another of the dominant themes of the day and the group had loads of good ideas for engaging users:

• Consider using independent facilitators for meetings with end users and stakeholders, this can prevent meetings dividing into stakeholders vs project staff. 
• Poll websites such as Polldaddy can be a useful alternative to surveys. Polls can be embedded in blogs or project websites and can be used to ask a drip feed of questions throughout the project. They require much less effort on behalf of the user than surveys so may increase participation. Another possible way to use them is to present the results of a survey or meeting and let users vote on the desirability of options suggested by meeting or survey participants.
• Using feedback sites like Uservoice can be useful. I wrote about them in a previous blog post.

Project evaluation can be a tricky area since there is very rarely a one size fits all methodology that can be applied to a project. This blog post is reporting on the information that came up in the seminar and I’d be very interested to hear any additions or disagreements with this post. I have reported the collected wisdom of the group who attended the seminar here so thanks are due to everyone who attended.

All in all it was a very useful meeting. One general point raised was that it would be really useful to share more good practice in the area of evaluation and project management. At the moment JISC are experimenting with using the tag jiscpm to aggregate useful project management related content. More information will be available about this soon but for the moment you can see the use of the jiscpm tag jiscpm netvibes site. There is also a mailing list and a question and answers site that you can use to discuss evaluation and related project management issues as part of our experiments with the jiscpm tag.  

I don’t propose to review the event because you can get an impartial review from some of the many blogs written by people who attended the event. This post is meant to collect the interesting things that were done at and after the event into one place so people can easily find out more about what the event produced. 

Despite this not being a review, reviews are an excellent way to get an overall flavour of the event. So here are a few of the thoughtful reviews produced by people who attended dev8D.

• Chris Keene 
• Tom Smith
• Chuck Severance
• Mark Johnson
• Steve Coppin
• Steve Lee

As well as these reviews you can also see the feedback that delegates left about the event on the wiki.  

One of the benefits of dev8D is the networking. Dave Challis of Southampton has used the twitter accounts of people attending the event to illustrate how people’s networks grew during the event

• Part 1
• Part 2

If you have a hankering then you can access all the dev8D tweets from before during and after the event.

The dev8D blog also talks about networking and why it’s important.  

We were lucky at this year’s dev8D to have a really diverse bunch of attendees, here are some examples of the people who attended:

• Katie Pekacar - MLA 
• Ian Mulvany - Nature
• Chuck Severance - University of Michigan  
• Keiran Marron - Eduserv
• Alex Bilbie - 2nd year undergraduate- University of Lincoln

Adrian Stevenson has also posted some video interviews with some people at dev8D over on the eFragments blog. 

The event was jam packed with opportunities to learn. These came in the form of guided sessions to learn new languages, quick 15 minute intros to topics, freeform workshops and ad hoc meetings.

All this learning activity is neatly summed up in Milly Shaw’s  post on the dev8D blog. You can also get a flavour of how the delegates felt about the training from the review posts linked above.

What does this kind of event produce? Well, not finished software but demonstrators and new ideas abound. This year at the event a number of organisations offered prizes for developers who came up with an interesting solution to a problem or did something interesting with their technology. There were 9 of these prizes offered by people as diverse as Microsoft Research, MLA, IMS, Edina, Memento and the Internet Archive.

A description of all the entries to these competitions is available on the dev8D blog The prototypes produced for these challenges are often worth much more than the prizes offered. Sam Adams, the developer who won the Memento challenge, is going out to visit the Memento team in the US as a result of his entry and Rob Sanderson who ran the Memento challenge commented to me that Sam’s entry was likely to have a real impact on the work of the Memento team.

In a similar vein, the winner of the Microsoft Research challenge has been asked to do a show and tell on his entry at the Open Repositories conference in Madrid.

The ideas weren’t limited to those entering the competition for prizes. There were fascinating ideas people bought along to the event or worked on while they were there:

• Overview of achievements at dev8D
• Reprap - the astonishing 3D printer
• Ben O’Steen’s experiments with the format of books
• Emma Tonkin and team’s DIY electronic whiteboard

For a complete listing of event outputs see the happy stories page of the dev8D wiki which collects interesting ideas, experiments, thoughts, etc.  

In summary, it was an amazing event, so much happened and I missed far more than I managed to see. The enthusiasm and energy that organisers and attendees put into the event was astonishing and I am still digesting a lot of the things I saw and learned. The devCSI project who organised this event as part of their remit to support a community of developers in UK HE did a fantastic job and keep an eye on their blog for more events like this. 

[For a general report on the day Doug Clow liveblogged the event.]

The Digital Scholarship project at the OU is led by Professor Martin Weller and Dr Nick Pearce and is focused on researching just what Digital Scholarship is but also promoting the uptake of Digital Scholarship within the OU. This obviously leads to a slightly schizophrenic project as on one hand they need to be slightly sceptical about things whereas on the other they are acting as cheerleaders. That said it seemed to me that the two of them were a good balance in this respect as martin was obviously a committed cheerleader whereas Nick demonstrated a little more scepticism!

The main aim of Digital Scholarship is to find a way to give academic activity on the open web the same weight as outputs in more traditional scholarly outputs (book chapters, conference speaking, journal articles etc). In order to do this it is closely following the principles of Boyers Scholarship - 2MB PDF here and mapping digital activity against them.

Discovery
Integration
Application
Teaching

A standard issue that they have been facing with the evangelism element of the project is that there remains a lack of trust in the stability and sustainability of many of the social web tools. Will they be around in a month, year, decade, longer. I think in alot of ways this argument isn’t quite as relevant these days - there is rarely any need for a single point of failure. A presentation has every chance of getting considerably more views on Slideshare but that doesn’t mean that for preservation reasons it might not be best to have a version in another location (i.e. an institutional respository). One of the strengths of the UKOER programme was the insistence that OERs were released to a minimum of two locations - one of which had to be JorumOpen. This allowed projects to release resources out into the wilds of the open web if they wanted - greatly increasing the possibility of takeup but also having the peace of mind that the resources would be properly preserved (though perhaps digital preservation is more of a preoccupation for JISC than the creator of the resources.) Currently this does increase the workload for staff tasked with depositing resources and maintaining them but tools that allow deposit in multiple locations via one interface are now more than possible and hopefully just around the corner.

A case for the speed of digital outputs against the slower processes of traditional publishing methods was made when Nick discussed the fact that he research the use of technology (particularly web tools) by academics in 2007 and he has a paper due to be published based on that work later this year. However the original research does not feature one mention of Twitter as it was barely a blip at the time of the research whereas now it has a much higher profile. That fact will immediately date the paper.

There are alot of parallels between this activity and many of the ideals of openness that JISC supports across our work. It certainly seems to be close to both the Open Access and Open Education agendas. [I actually like the idea of the Open Scholar as defined in this quote.]

“the Open Scholar is someone who makes their intellectual projects and processes digitally visible and who invites and encourages ongoing criticism of their work and secondary uses of any or all parts of it–at any stage of its development.”
http://www.academicevolution.com/2009/08/the-open-scholar.html

The OU is seeking to build this kind of concept into the ‘personality’ it portrays externally. It has already taken the (unusual?) step of adding participation in the open web to the promotion criteria for academics alongside more traditional measures (though they admitted they were still working on appropriate measures for this kind of activity.)

Currently they are working hard on developing appropriate metrics for contributions to the social web and are asking questions like:

What makes a good blog?
Can you map traditional scholarly outputs against digital work? (i.e. Podcast vs Keynote)
How can a young academic at the start of their career make an impact in the digital space?
How can this work make an impact if the REF does not currently even acknowledge it?

It was briefly mentioned that perhaps what was needed is something like the famous Google 20% time (or the less famous BBC Radio Labs 10% time ) that allowed academics the time to participate and experiment on the open web to find their voices and their communities.

Alot of the talk around creating metrics and perhaps build up an ‘online reputation’ score (despite fears that this could be ‘gamed’) made me think of some of the work around ‘whuffie’ that Cory Doctorow wrote about and the upcoming book from O’Reilly. It is an interesting and hugely difficult area of work that many people are trying to crack for different reasons on the web (in a previous job it was a constant topic of conversation in the office). Finding a reliable way of doing this for the academic community would be really very impressive and potentially important. Again I think it is the sort of thing that would be of equal interest to the OER community as the digital scholarship one. Recognition and reputation should never be underestimated as a driver for contributing to the open web.

It was pointed out that one side-effect of a high profile in the world of digital scholarship was increased opportunities to take part in more traditional scholarship activities! Well respected bloggers are often invited to speak at leading conferences and contribute to books & journals.

Not surprisingly Micheal Wesch was pointed to as the poster child for Digital Scholarship. The success of his YouTube videos has been amazing and massively enhanced not only his own reputation but also that of Kanses State. To some extent though this is a dangerous comparison as the huge success of Wesch is unlikely to be something that can replicated any time soon.

Throughout the presentation(s) a list kept popping up that covered the most common issues/barriers that the project had identified for academics to really make the jump to digital scholarship. One of the things that immediate struck me was that it could be exactly the same issue for anyone looking to take part in the OER movement - or in fact just become an active participant on the web with any kind of work related focus. None of these issues are insurmountable and for many of them the perception is a bigger problem than the reality but they do continue to be identified as obstacles time after time;

Recognition
Rights
Skills
Plagarism
Time
Quality
Exposure
Sustainability

This brought to mind the work Forrester did around identifying how and what people actually contributed to the web and actually how few of them were actually creators rather than consumers.

I found it to be an interesting day that left me with a great deal to think about - not to mention a couple more blogs to read. I’ll be following this work closely in the months to come particularly looking out for opportunities to join it up with both OER and open access work.

• Strand A: Deposit of research outputs (tag = jiscDEPO)
• Strand B: Exposing digital content for education and research (tag = jiscEXPO)

These two strands of projects will be separate in their project remits, however there is significance in our suggestion that the Web can and should act as the primary place where open content should reside.  It is essential that both the process of deposit and then expose is as simple and as straightforward for the user as possible.

To help clarify each of these strands I’ve had my JISC colleagues quiz me on the Call document for each strand so that we can give a basic overview of how to read the Call document.

To note, the following resources will be referred to in both podcasts so please make sure to have the following on hand while listening:

• “The Call Document” which is available either on the JISC website as a Word document or on the writetoreply.org site as a set of HTML links (we would encourage comments on the call to be placed on writetoreply or if via another site to please use the appropriate tag).
• “The Briefing Paper” which is published alongside the “The Call” document on the JISC website as a Word Document.
• “The IE Team Blog” which is what you are reading now!  All updates to this call and set of projects as they are funded will be announced here, so subscribe to the RSS feed now!
• The “jiscDEPO” podcast and the “jiscEXPO” podcast.  These are two separate podcasts which are below and are intended to address each strand of work described in “the Call Document”.

12 minute podcast discussion between David and Andy to give an overview of Strand A (Deposit) of the Call document (jiscDEPO):

Podcast on jiscDEPO Call

12 minutes podcast discussion  between David and Frederique on an overview of Strand B (Expose) of the Call document (jiscEXPO):

Podcast on jiscEXPO Call

Please don’t hesitate to call or email me (David F. Flanders) if there are any further questions with regards to both strands.

The focus at Repcloud was discussions about how the leading Digital Repository vendors (represented here by Duraspace, EPrints and Microsoft Zentity) were integrated ‘cloud’ capabilities into their offerings to the community and also how ready and interested the community was for those options.

The three speakers in the morning covered their, mainly beta or upcoming, strategies for ‘cloud’ services and introduced us to some interesting projects and ideas.

Eduserv have already released the videos and presentations online as well as some ‘voxpox’ videos and a link to the Twitter archive on Twapperkeeper so I will just pick out a few highlights.

Duraspace, via its DuraCloud pilot projects, is pushing ahead in this area. The work they are doing seems particularly focused on [multi]media objects rather than traditional papers and they are utilising the ‘cloud’ to not only provide storage but also large computational jobs offering filetype conversion (TIFF to JPEG2000) and video encoding and streaming. They did identify the fact that bandwidth to and from the ‘cloud’ remains an unreliable and costly process though.

Alex Wade from Microsoft did something of a whistle-stop tour through the offerings of Microsoft research and the MS ‘cloud’ service Azure. Some particular points of interest here were the relationship Microsoft has formed for the NSF in the US around offering ‘cloud’ capability, the Research Information Centre which they developed in collaboration with the British Library which includes Sword based push to repository capability and Entity Cube which maps relationships between people (and places) in a visual manner by querying the open web.

The final take of the morning was from Les Carr of EPrints. The focus here was on the changes in EPrints 3.2 that allows for a ‘hybrid storage solution’ mixing local and ‘cloud’ offerings as most appropriate and using the new Storage Controller to write rules that would make decisions on what to do with uploaded content on the fly ensuring the optimal performance of the repository. Something I found particularly interesting (and wrote about elsewhere) was the idea of EPrints offer a ‘Blogger’ like service that would allow users to fire up their own repository on-demand. I thought this sounded like a very clever idea and was surprised it wasn’t discussed further.

The afternoon opened with Terry Harmer from the Belfast e-Science Centre who gave a hugely interesting talk from a perspective of someone who has been using ‘cloud’ capabilities as the backbone of their operations since before the term ‘cloud’ became popular. What was particularly interesting here (to me at least) was how they had used this ‘cloud’ based strategy to circumvent the limitations and bureaucracy at their host institution and as such were much more flexible with the options they could offer.

The rest of the afternoon was split into separate technical and policy sessions (though judging by the discussion after there was more than a little cross-over.)

Issues around geo-location and legal issues (particularly IP) were raised in both sessions. SLAs were also discussed in both sessions with differences from one provider to another identified as a bit of a problem.

Issues around the true costs of using ‘cloud’ services also came up in both sessions - with the issue of bandwidth costs coming up again in the Technical session.

The day ended with Andy Powell and Rachel Bruce closing things by saying both JISC and Eduserv would continue the conversation to see what the next steps might be for work in this area.

I found it a useful day and it seemed that it was a topic that many of the attendees were thinking about and moving towards to some extent or another.

[As I attended the Technical session the following are my notes from the Q&A in that session] (more…)

London, University of London Student Union.  Today at the #Linkeddata #London meetup JISC announced that a Call will be issued in March requesting bids for projects to make content available on the Web working using linked data approaches.

To note, the above announced call was originally labelled on the JISC funding Roadmap as “Innovative ways to Deposit, Expose and Discover Web Resources”.  The original grant funding call was planned for issue in early January but due to the funding letter from DBIS to HEFCE on 22 December 2009 it was put on hold whilst funding implications were assessed.  At the HEFCE Board meeting in late January the decisions regarding the  final impacts on current capital budgets were agreed.   As a result of this the grant funding Call will now go ahead, albeit with a changed scope and a lower budget, it’s emphasis will still be upon deposit and exposure of content.  The re-scoped Call is summarised below.  We aim to have this call out in March and there will be a six week bid preparation period.

As soon as this call is approved by policy we will announce that the call is live and that we welcome proposals.  If you have any queries in the meantime please contact d.flanders@jisc.ac.uk for further information.

This Call strand will also be released with the Deposit strand of the Call.  Please see separate post for further information about the Deposit Call.

To accompany Andy’s post I thought I’d share a selection of links to information that might be useful to those attending the event and to people interested in cloud computing and repositories.

Repository specific links:

• The JISC Fedorazon project looked at setting up a repository on Amazon’s cloud offering
• The ePrints team are investigating the use of the cloud 
• Information about DuraSpace’s DuraCloud project

General cloud information:

• The national institute of Science and Technology produced a good definition of cloud computing
• There’s a JISC CETIS activity group looking at cloud computing in institutions which has produced some really useful documents
• The Tower and the Cloud, an Educause ebook about cloud computing is a detailed overview of the cloud and Higer Education
• Cornell University has a useful overview on outsourcing and cloud computing for universities
• Berkeley published an interesting, academic view of the cloud

JISC links:

• Cloud computing in the JISC strategy
• Technical review of cloud computing for research
• Using cloud computing for research
• Review of the environmental impacts of the cloud
• My colleague James Farnhill collects links to resources about cloud computing that he’s found useful on his delicious account

That’s the list we’ve got so far, what else do you think is worth reading in this area? Leave a comment and I’ll add it in.

For those of you who are not able to attend the event, we will be writing up the event to ensure there are useful outputs from the day and you can follow and join discussion at the event on Twitter by using the #repcloud tag.

Proposals are now with markers. As you know from the Call, we hope to get back to proposers with outcomes during week beginning March 22 (the intervening time is to allow for marking and then for the marks to be collated and the Evaluation Panel to meet etc.).

The project plan is available from the JISC Web site, with further information available from the project Web site. The primary partners in this project are MIMAS, Cranfield University, COUNTER, CrossRef and Oxford University Press - which means that it is well placed to consider the many issues to which the collection, aggregation and use of article level statistics gives rise.

PIRUS2 is not alone in considering these issues and is in contact with the Open Access Statistik and SURFSure projects in Germany and the Netherlands respectively which are also working on collecting article level usage data from repositories. The projects are taking similar technical approaches. One key decision - which is in line with a recommendation of the JISC usage statistics review of 2008 - has been to format log data as OpenURL context objects. One explanation of OpenURL context objects can be found on the SURF Web site. Other standards being used are OAI-PMH and SUSHI for harvesting the usage data.

PIRUS2 continues to the end of 2010.

They have provided the data about each session as a dynamic dataset that will continue to evolve in the lead up to the event. This dataset is available to anyone who is interested from the dev8D website and is provided in three semantic web formats: RDF, n3, and nt . The dataset includes details of session titles, timings, and locations.

The organisers have used this dataset to build a timetable for the event which includes useful links to add events to your calendar. However they have also issued a challenge to people attending the event to design a better way to view or use the data. The best examples will be provided along with the existing timetable.

All sorts of interesting tools were designed at dev8D in 2009. So I am very excited to see the innovative ways that this year’s delegates choose to use this data both before the event and during it.

Congratulations to Chris Gutteridge, Dave Challis (both of ECS at Southampton University) and Mahendra Mahey of UKOLN for their great work on this.

There are still a few places left for dev8D, it should be really useful for all HE developers and for people interested in their work. Registration is free and each day should stand on its own so no need to feel you have to attend the entire event. You can register on the event website.