Tips for evaluating JISC projects
Yesterday we held a small seminar to look at project evaluation for some people involved in JISC projects in the information environment programme. The seminar was designed to be very practical and to recognise that there is no one size fits all evaluation methodology for projects.
For me, the strongest messages to come out of the day were:
- Evaluating projects is about demonstrating success and learning from the experience therefore the first thing to do when planning a project evaluation is to decide who it is for and what it is supposed to do. Once you know the audience that the evaluation is aimed at and what you want them to do as a result of seeing the results then planning what you are going to do becomes much easier.
- Focus your evaluation. Making it tight and focused will make it easier to plan and undertake. A good way to focus an evaluation is to use your project objectives, these can give you a few high level questions that your evaluation can seek to answer.
These are just the stand out points of a very rich discussion. The afternoon session was a general discussion of the concerns that people had about project evaluation and it threw up a wealth of good advice and ideas for evaluating projects. I thought it would be useful to write up the notes from this session so that others could use the advice.
Effort
The first concern we discussed was: how do you know how much effort should be directed at a project evaluation?
Unfortunately there is not an easy answer to this and it really depends on the type of project you are managing and what it aims to do. If, for example, your project is trying to prove that a certain approach is a good idea then a robust evaluation is probably quite important, however if you are developing some new technology then perhaps evaluation needs to focus on whether user requirements are met. So to decide what is required you first need to decide what role the evaluation is going to play in your project. JISC programme managers are there to support projects with this kind of decision so it is a good idea to give them a call to discuss it. It is also important to realise that the role of the evaluation in your project may change over time.
Timings
The second concern was about when should a project evaluate? The group all agreed that gathering data throughout the project was essential and wherever possible evaluation should be built in to project activities such as events so that data can be gathered.
We also talked about how you evaluate projects when the real impact and benefits will occur some time after the end of the project. We decided that this was a built in limitation of innovation projects and it needs to be recognised that it is not realistic to prove impact a few years down the line in project evaluations. However it may be possible to gather evidence that shows that this future impact has a chance of occurring and to make informed speculations about its likelihood.
Impact also depends on the type of project. Some projects are assessing an unproven technology whereas others are seeking to embed use of a proven technology in an institution. For embedding projects, impact should be a concern as it shows the success of the project, however for projects assessing unproven technology, impact is probably less important.
What and how
The group discussed practical ways to gather different types of evidence for projects:
- Set project checkpoints to assess progress but ensure you think through why you are doing this and who you are doing it for.
- Consider the use of some kind of daily/weekly log or diary, blogs are very useful for this but you may also need to record some things that are not for public consumption.
- Plan in reflective time to allow yourself to record and consider the events that have occurred in the project. Making this a regular habit will make it more useful. To ensure you keep the habit up, it is a good idea to make it a part of project meetings or calls.
- Collect interesting anecdotal evidence that you pick up from events, twitter, meetings, emails etc. This could all be useful when writing up evaluation results.
- Think about embedding evaluation in each project output to try and get a regular flow of evidence rather than trying to gather it all at the end.
- Think about arranging controlled experiments to judge whether your project has made a difference to end user experiences.
- Baselining the project can be an interesting approach and doesn’t have to be metrics based. Scott Hill of the I-Wire project at Cardiff University has developed a blueprint of the current state of the research lifecycle in his institution as a way of assessing whether the project has made significant changes. Scott has blogged about this in detail on the I-Wire blog.
Audience
Deciding who you are evaluating for is a vital step in planning the evaluation and should be done right at the start. If this audience is defined then it makes it easier to decide how you need to present the information gathered as a result of the evaluation.
JISC is one possible audience for the results of a project evaluation, however, it is also worth considering who else needs to know what the project has produced, what was learned and how successful it all was. For example, if your project is focused on implementing an institutional repository then perhaps you need to think about demonstrating to senior managers that the repository is an essential piece of institutional infrastructure so that steps are taken to ensure its sustainability once the project funding has ended. Similarly, if you are seeking to embed a repository in user workflows then perhaps the evaluation needs to focus on gathering evidence that persuades researchers that the repository offers them tangible benefits.
Meaningful
Ensuring that the project evaluation is meaningful is also important. This is an issue that is also dependent on defining an audience for the evaluation, once you know who they are then ensuring that the evaluation means something to them is easier.
Another important point about making an evaluation meaningful is to ensure that things that don’t work are recorded as well as successes. This evidence is really important as it can save other people time and effort. The recent JISC rapid innovation projects used blog posts entitled Small Wins and Small Fails to record things that worked and things that didn’t and I think this is a model that could work for other projects.
User and stakeholder engagement
User and stakeholder engagement was another of the dominant themes of the day and the group had loads of good ideas for engaging users:
- Consider using independent facilitators for meetings with end users and stakeholders, this can prevent meetings dividing into stakeholders vs project staff.
- Poll websites such as Polldaddy can be a useful alternative to surveys. Polls can be embedded in blogs or project websites and can be used to ask a drip feed of questions throughout the project. They require much less effort on behalf of the user than surveys so may increase participation. Another possible way to use them is to present the results of a survey or meeting and let users vote on the desirability of options suggested by meeting or survey participants.
- Using feedback sites like Uservoice can be useful. I wrote about them in a previous blog post.
Project evaluation can be a tricky area since there is very rarely a one size fits all methodology that can be applied to a project. This blog post is reporting on the information that came up in the seminar and I’d be very interested to hear any additions or disagreements with this post. I have reported the collected wisdom of the group who attended the seminar here so thanks are due to everyone who attended.
All in all it was a very useful meeting. One general point raised was that it would be really useful to share more good practice in the area of evaluation and project management. At the moment JISC are experimenting with using the tag jiscpm to aggregate useful project management related content. More information will be available about this soon but for the moment you can see the use of the jiscpm tag jiscpm netvibes site. There is also a mailing list and a question and answers site that you can use to discuss evaluation and related project management issues as part of our experiments with the jiscpm tag.
Comments
3 Responses to “Tips for evaluating JISC projects”
Leave a Reply
This all seems very reasonable. SOme comments that I think build upon what’s said here:
1. an agile approach (which is also not a one size fits all proposition) encourages data collection and evaluation as a project proceeds. Indeed, each iteration becomes a project checkpoint.
2. The JISCRI approach to reporting has been very enlightening. I found that recording stuff as I went along, which I did with some enthusiasm as I knew it meant less work to do at the end of project, resulted in much more detailed records than a large end-of-project report. My final report took about half a day.
Really useful day. Would be very beneficial to run a similar event for each programme start-up to help get the focus and effort right from the start. Thanks again.
[...] McGregor has put togther some really interesting notes called “Tips for evaluating JISC Projects” after a small seminar for JISC IET projects. The tips and discussion have a more wider relevance [...]