Performance and Measurement in Libraries

In his article in the New York Times, Robert Crease wrote:

We look away from what we are measuring, and why we are measuring, and fixate on the measuring itself.

For libraries, so used to collecting, managing and analysing various sets of data and metrics, this is a critical point.

It is also a sentiment that kicked off the 10th Northumbria conference on Performance Measurement in Libraries held in York earlier this week.

Elliot Shore from ARL (Association of Research Libraries) spoke about the need for libraries to take heed of this advice: To focus on the ‘fit’ of what we’re measuring. 

This fit, as Shore calls it, has been evolving over the past 10 years as the role and presence of the library has changed. The digital environment and changing technologies and expectations of users means that what was once important to measure and capture may no longer have the same urgency. 

This focus on what should be measured – and how it impacts on the role and shape of the library – was developed in a great talk by Margie Jantti at the University of Wollongong in Australia.

Margie talked about the constant flow of information and data that her staff (relationship managers) get from the researchers and academic staff, which is used to tailor services and focus resources on priority services. This has seen the library develop expertise in publication support for researchers by the library.

The large knowledgebase of data the library collects on its users enables it to punch far above it’s weight: helping develop a fast; agile and world-class library team.

Finally, one thing that emerged from a majority of the presentations during the conference was the increasing recognition that data and metrics from inside or about the library were no longer enough. The field from which the data and metrics is harvested is growing, and reaching further beyond the library. Into the teaching and learning space through to research, registry and student services and beyond.

The idea that library performance and measurement requires only data from the library – or within the immediate vicinity of the library – is no longer an option.

So, it was against this background that the Library Analytics and Metrics Project (LAMP) presented at the conference.

We provided some of the background to the project (where it has come from and the work that has led us to this point) and provided an overview of the work so far and how you can get involved and follow the progress of the project.

For me, what’s really interesting, is that LAMP has the potential to bring in data from across the institution (and beyond) to help inform decision making and how and where resources are allocated. It also takes away the burden of collecting the data and provides the space for libraries to act on the data, and to think strategically about what they want and should be measuring and analysing.

The conference was also useful in bringing to my attention LibQual, and the potential for LAMP to work with that data too (although this may be something for further down the development pipeline).

You can find a link to our presentation here. At the end are some ways that you and your library can get involved – so do feel free to get in touch.

lamp

Library Systems Workshop

On Monday this week the Library Systems Programme held a one-day workshop in London.

I’ll talk more about some of the things that cam out of the workshop in later posts – for now I just wanted to share some of the presentations which were given during the day.

You can also see what people were saying about the event on Twitter with this storify created by Helen Harrop from the LMS Change project:
[View the story "Jisc Library Systems Programme Event" on Storify]

The workshop was a chance for the projects that made up the programme to talk about the work they had done and the tools and resources they have created, and a chance for the community to discuss some of the issues and challenges that the sector currently faces.

The workshop was opened by Rachel Bruce of Jisc and Ann  Rossiter of SCONUL and introduced some of the main themes of the day.

The workshop had three main strands that explored:

The workshop was then drawn to a close with a panel, chaired by Suzanne Enright of the university of Westminster, which explored what would be on your LMS wishlist.

 

The panel began with three short ‘provocations’ from Martin Myhill (Exeter), Andrew Preater (Senate House Libraries), and Owen Stephens (consultant).

 

Andre Preater at Senate House Libraries has also done a fantastic job of writing about the event, and you can find a copy of his presentation on his blog too.  The provocations were rich in ideas and arguments, for example:
In summing up the panel discussion and the day overall, Suzanne did a superb job of drawing out some of the main discussion themes and issues that had been surfaced during the day. An overview of these can be found on some slides she kindly put together:

A number of important themes emerge from Suzanne’s slides, and importantly there is a clear recognition that many of the challenges libraries face are not technological in nature. Rather they are about cultures and people.

 

So,what follows is a short overview of each session from the workshop and the presentations given (where available).

 

Collaborative Services and Systems

This session included presentations from projects exploring the potential to develop shared library systems and services. These were projects by SCURL in Scotland, WHELF in Wales and the Bloomsbury Consortium in London.


This project has contributed towards a new vision for library systems by investigating the following question: “How would a shared library management system improve services in Scotland?”


Building on the work of the earlier ‘WHELF: Sharing a Library Management System’ feasibility report the project has explored the potential benefits and pain points inherent in a move from distributed to centralised hosting and infrastructure model for a suite of library systems software, while building a possible overall business case for such a move by the HEIs within the WHELF consortium.

The Bloomsbury Library Management Consortium is building on the strengths of the Bloomsbury Colleges and Senate House Library and their track record for sharing and collaboration. The group undertook a study of the landscape of the 21st century Library Management Systems (LMSs) – and evaluating the options for building, commissioning or procuring a Bloomsbury Library Management System (BLMS) as a shared-service.

The presentation from the Bloomsbury consortium can be found here: 2013-07-15_JISC-Event-BLMS-for-circulation.

The group have made a decision in principle to go with KualiOLE open source /community library system.

Transforming workflows and processes 

This session included a number of presentations exploring the impact of new systems and technologies on traditional library workflows and processes.


HIKE is exploring the integration of next generation library systems (specifically Intota) at the University of Huddersfield with Knowledgebase+ and the impact on traditional workflows and processes.


EBASS25 in a collaborative project, led by Royal Holloway, University of London, to develop shared models of ebook procurement using Patron-driven acquisition approaches.

[presentation to be added]

The Collaborative collections management project saw King’s College London and Senate House libraries collaborate on above campus initiatives around collection management for the benefit of students and researchers, and the use of the Copac collection management tool. 

Tools and techniques for systems change

The LMS Change project took on the entire burden of this session themselves, showcasing the tools and approaches they have developed during the project and getting participants introduced to some of the tools. The LMS change presentation is below, and Ken Chad’s presentation on the business case for change can also be found here: Business_case_for_change_Jisc_LMSchange_wkshop_KenChad_July2013


Preparing for Mobile: Supporting institutions in a mobile world.

I recently facilitated a workshop at the Institutional Web Managers Workshop (#IWMW12) entitled: Preparing for Mobile. This is a slightly delayed (by holiday) write up of the event and some of the things I took away from the session.

The aims of the session were:

  1. to share some of the resources JISC (and others) have developed for institutions attempting to understand, and adapt to ubiquitous mobile devices;
  2. give delegates an opportunity to share their current practice, experiences and tips, and finally;
  3. provide JISC with a perspective on how mobile is being addressed in institutions and its impact.

I began the session with a presentation of some of the resources and work JISC has done on mobile within a teaching, learning and research context.


Delegates were then invited to discuss the issues and experiences they have had in groups. The groups were arranged around four themes: content (this was a self-forming group by the delegates themselves); strategy; design and delivery; tools and techniques.

After a period of discussion, each group was asked to present back for 3 minutes on their discussions to the whole group, outline their top 3 tips for their area and their top disruptor for HE in terms of ‘mobile’.

Content (is King)

Content remains the bedrock of what web managers do in terms of their day-to-day job: The technologies and strategies may change, but delivering the content does not.

Content tips:

Content disruptor

The seduction of delivery distracts us from the content. Content is King.

Strategy

It was generally agreed that any strategy development should look beyond just mobile to a broader e-strategy or similar. But there is an urgency in institutions getting this right – access via mobile devices is growing exponentially.

The strategy group were also very clear about the need to engage and understand users. It was clear any strategy needs to begin with an understanding of the user and work from there.

Strategy tips

Strategy disruptor(s)

Unhappy being restricted to just one, the group came up with three!

  1. HTML CSS future web techs will remove need for most native apps.
  2. Network speeds, cost will change things
  3. Apples standards become the de facto standards; watch it carefully

Design and Delivery

It was clear that for most of the delegates mobile was something they were ‘dipping their toes into’. This meant that few had useful strategies for adapting to the growing importance of mobile access and most mobile delivery was being done on an ad hoc basis.

Design and Delivery tips

Design and delivery disruptor

Devices beyond mobile (the new desktop?): Impact of users accessing institutional content via plasma screens, Google tv (already get Xbox, PS3). What’s a TV anyway….?

Tools and Techniques

Much of the decisions being made on tools for delivering content are based upon the CMS that’s currently employed by the institutions.

The group also made it clear that as a sector we borrow from each other a lot (and there is always the need for those outliers and cutting-edgers). So it’s important to look out beyond the institution for support and advice, especially via blogs.

Tools and techniques tips

Tools and techniques disruptor

There will be something cross-platform that will disrupt all our current tools – something finally to end the debates and disagreements about formats etc.

  Many thanks to my colleague Amber for helping facilitate the session and taking wonderful notes and photos!

Putting the User at the Heart of Education

This is an update on a recent Usability workshop held by Ben Showers and Torsten Reimer as part of their work on the Usability and Adaptability of User-Interfaces.

Background

As universities and academic institutions increase the focus and investment on improving the ‘student experience’, so the ‘user’ needs to find their way into the heart of everything the institution does, not just the teaching and learning.

Usability and user experience (UX) have become important considerations in the design and creation of new websites, software and systems for universities. With increasing investments in digital infrastructure and content, addressing the needs of the users is one of the best ways to ensure the uptake of these investments – be it in research, administration or learning.

UX should become a critical tool in the sector’s response to the challenges of rising student fees, the need to make every investment count in the light of reduced budgets and increasing expectations from students and staff who are used to new sleek gadgets and web 2.0 environments.

In light of this context, Torsten and I recently held a two day workshop exploring usability in higher education.

The first day focused on JISC funded projects that were exploring the Usability and Adaptability of User-Interfaces. The second day involved a broad range of delegates from both within and outside universities, including the library, institutional web managers, systems managers, usability consultants and academics.

The workshop resulted in some very rich discussions from the delegates, and we have attempted to capture some of the themes and main topics of conversation below.

Emerging themes

We have grouped the themes under some broad headings:

Affecting the project/development culture

Embedding Usability

It was clear that usability and considerations of the user are usually a ‘bolt-on’ to institutional developments and projects. The workshop reiterated the need for usability to be part of the project from the start – furthermore it should be the framework from within which you undertake your project. This point was echoed by a number of the projects that took part in the JISC usability programme (an example being the British History Online project).

Indeed during a ‘programme design’ session it was suggested that future usability project funding should aim to embed usability into the institution. This could be achieved by a kind of ‘pay it forward’ idea whereby the successful project then initiates its usability lessons into another of the institutions developments.

Strategic buy-in

Connected to the above poin is the need to make strong arguments to the senior managers who make the decisions about resources as there may not always be a strong appreciation of the benefits of a user-centred approach.

An appeal to the usability of the product is a first step, but more may be required. One suggestion was exposing senior managers to the user’s pain – let them watch the UX sessions you undertake. Let them see every grimace and hear all the expletives!

Training and Skills

It was clear that there are a number of significant skills gaps within the current training and teaching of developers and project managers for usability practice and methods. JISC programmes of work are often addressing the skills gaps in their area specifically, and this may be something that any future work in the UX environment should confront.

However, there was agreement that it’s important to be able to share best practice, and enable institutions to have conversations with each other and experts to ensure they’re able to get hold of the right skills externally if necessary (one of the projects: UsabilityUK, has exactly this remit).

Demonstrating Impact

One of the most interesting discussions at the event was the difficulties involved in demonstrating the impact of usability. Functional requirements are easy to quantify (it either does or doesn’t do as requested), non-functional requirements (like usability) are harder to measure.

What potential does including metrics in the National Student Survey, as an example, have to demonstrating the impact usability could have within HE? Maybe more fundamentally having a clearer idea of how you measure impact more generally could help clarify how the user experience is evidenced.

Communication

Usability by any other name…

I am guilty of it in this post, and I even started day two of the workshop with an admission of guilt: I would use the phrase usability without unpacking it. Worse, I used it as a synonym for a group of similar terms (user experience, user-centred design, human computer interaction).

However, this may well be one of the issues that contributes to a misunderstanding of usability, and confusion around the different terms. A number of the delegates felt it would be beneficial to ensure terms were well defined. If you’re communicating with non-experts, such as developers, you need to be sure they’re able to understand you and your requirements.

On reflection, I do wonder whether this is a reflection of usability as a research discipline. Is the real message that the user needs to go at the heart of everything the academy does; it needs to become truly user-centred. Does the concern for semantics get in the way of this goal?

Inter-communication

Connected to the above is the difficulty of communicating with the members of your team or project. While you might be able to capture the requirements of your users, it is essential you’re able to then communicate these directly and precisely to your developer.

One potential answer is a very agile approach to development that sees the developer coding with the user(s). This was an approach that the ALUIAR project took at Southampton, and it worked well for the project.

Not always talking to the converted

One of the interesting aspects of the workshop was that we had a pretty good mix of professional UXers, researchers, institutional system people, librarians and learning technologists. However, most had a very good appreciation of the benefits and importance of UX to their institutional mission.

Indeed, a number of delegates made the point that it was important that we addressed the un-converted.

Usability is dead, long live usability

Finally, two points seem to provide interesting conclusions to the workshops discussions:

In its practical application within an HE environment, usability is closely meshed with other similar issues: impact; accessibility; sustainability. The aim is simply to make software easier for users.

Usability should be the driving principle behind projects within HE (especially in JISC funded ones); but this doesn’t that the only way to address this is through a usability programme. Rather, usability should pervade projects without defining them.

What’s next…?

The workshop was incredibly useful as a way to start thinking about how JISC might continue to help support usability practice within HE.

Indeed, it gave us a clear message that usability/accessibility is one area of the HE picture that is in real need of some focused activity. Torsten and I already have a few concrete ideas we’d like to start developing – if all goes well nthrough JISC-funded activities later this year.

Watch this space for more news!

—–

Many thanks to the delegates and projects who made the workshop so successful and the discussions so rich. And to Addy Pope of Edina and the USED project for the use of his camera and pictures.

Shared Academic Knowledge Base (KB+) – Library Directors event

Yesterday saw the shared academic knowledge base (KB+) briefing day for approx. 60 library directors and senior managers take place in London, at the Wellcome Trust.

The project, known as KB+, is developing a shared community service that will improve the quality, accuracy, coverage and availability of data for the management, selection, licensing, negotiation, review and access of electronic resources for UK HE.

The aims of the day were to:

  • Provide an update on the progress of the shared academic knowledge base project;
  • Surface and share some of the questions, concerns and ideas participants may have about the project and the management of electronic resources in general;
  • To let participants know what will be happening next with the project and how you can get involved if you would like.

The day began with Ben Showers (JISC Programme Manager) providing some context to the work and situating the project within the wider subscriptions management landscape. The presentation can be found here: Shared Academic Knowledge Base: Context and Landscape

Liam Earney, the project lead from JISC Collections, then went on to outline the vision and approach that the project will be adopting, as well as providing the participants with an idea of what will happen next and how, if institutions wish, they can get involved.
Liam’s presentation can be found here:  shared academic knowledge base: Approach and Vision

The meeting engendered a large amount of discussion about the project, with participants freely sharing concerns, ideas and possible solutions to some of the issues that surfaced.

Extensive notes were taken from the Q&A sessions to help inform the project, but instead of repeating verbatim the questions and answers I have tried to highlight some of the themes that emerged during the meeting below.

Themes

A number of themes emerged during the day and, while this is not an exhaustive list, these are some of the recurring or critical issues that were surfaced:

Transformation of current practice

It was acknowledged that this project was potentially transformative; it has the potential to change what might be termed the bread and butter of library work.  Therefore its impact on the community, and how it works, could be significant.

This means that the community, from senior managers to practitioners and beyond will be keenly interested in the developments and the project will need to build trust and facilitate the involvement of the whole library community. Which brings me on to another of the days themes:

Communication

This was a theme that seemed to surface at regular intervals during the day. There was a clear message that the project needs to be able to communicate regularly with the library community on both progress and developments as they take place.  This might manifest itself in a newsletter such as that employed by the Discovery programme, or utilising existing communication channels from JISC, JISC Collections and other sector bodies (or indeed a combination).

The combination of communications channels is also important given the range of stakeholders interested in the developments, from commercial vendors and publishers to librarians in the UK and internationally.

Under this theme there were issues surrounding how the communication channels would allow for more interactivity than might otherwise be usual in a JISC funded project given both the high profile nature of the project, as well as the need for ongoing community engagement in the work.

Engagement

Closely related to communications was the topic of engagement.

Specifically a lot emerged on how the community, especially ERM librarians and similar, could be engaged in the project in a useful and meaningful way.

In his presentation Liam made it clear that the project hopes to ‘recruit’ a number of embedded librarians where the project will pay for a proportion of their time to work on the project.  It was made clear by the participants, however, that it would need to be made clear what expectations any involvement might have, from the skill levels and expertise of the person, through to the time length they might be involved.

Clarity on these issues would be key to maintain sector engagement.

It was also suggested there might be the need for something like an ‘advocacy pack’ so that library directors had the arguments to convince senior staff of the benefits of engaging with the project.

An interesting sub-theme within engagement was the power of the institutions themselves to help engage with the commercial companies and organisations they work with to put pressure on them to both work with the project as well as implement the recommendations and standards the project might recommend.

The message was clearly that this was a partnership.

Collaboration and leveraging other work

It was expressed a number of times how important it will be for the project to leverage this work and funding with other initiatives and projects that can help the KB+ project deliver its outputs.

It was acknowledged how much work was currently taking place around this area, such as national projects such as KBART , TERMs and JISC funded projects such as the journal usage statistics portal and e-journal archiving work including Peprs and the entitlement registry, as well as international projects such as the Open Library Environment.

This helped reinforce the projects own ambitions of engaging with, and where possible working with these complimentary projects and initiatives.

Sharing problems

This is a shared service, but it will be important that when issues are surfaced by an individual institution, or indeed a problem is resolved by someone, that the whole community can be made aware of this.

What tends to happen now is a problem will be reported to a supplier and that problem is then normally resolved, but no one other than the originating institution knows about this.

Further points of discussion

There were a number of other points of discussion including:

  • The potential conflict between aiming for quality of data and ensuring its timeliness.  It’s essential that quality doesn’t impact on the ability of libraries to deliver services to users as and when they want them.
  • Print subscriptions: The briefing day concentrated largely on electronic resources, but it was clear that participants wanted to see print incorporated in the work.  The project is taking a unified approach so this won’t be an issue, although electronic will remain the focus for much of the work.
  • The Identifiers elephant! It was clear that participants also felt that the issue of how the project deals with identifiers (be those institutional, journal title etc) will be a critical.
  • Decision making and workflows will be two potential aspects of the final service, but it is important to recognise that a focus on the decision making components that the service will deliver could help strengthen potential business models, and demonstrate real value to institutions.

As the event demonstrated, there will be a lot more work going on over the next few months to get the project into a position where it can successfully transfer to a service.

In the meantime, this won’t be the last you’ll hear of the project, with plans already in place to start communicating and engaging with the community over this important shared service.

If you would like to find out more about the project, or have any questions them please feel free to contact Liam Earney at JISC Collections.

Open Edge: ‘Mashed Libraries’ and JISClms

Tomorrow I am flying up to Edinburgh to attend the Mashed Libraries: Open Edge-Open Source event which runs Tuesday and Wednesday.  This particular mashed libraries event (Haggis and Mash) is being supported by SCONUL and JISC, and has been developed with the Enhancing  Library Management Systems programme (JISClms) which recently came to an end.

The JISClms projects investigated a number of issues that libraries currently face and the systems they use, including: The usability and enhancement of interfaces; management of electronic resources (ERM) and the feasibility and implementation of open source library systems.

The open source library strand of the JISClms work was relatively small, yet it has been extremely prescient with recent developments of institutions adopting open LMS alternatives (such as Staffordshire University).

But the event also ties in with broader issues that form part of JISC’s strategy, and specifically the aims of the Digital Infrastructure programme (the IE Team and eRsearch Team are now working together as a single programme under this new name).  Some of the more obvious examples include:

This isn’t meant to be a very comprehensive list, instead I wanted to highlight some of the connections between the mashed libraries event and work in JISC and the Digital Infrastructure team.

With so many links I intend to write another post upon my return and flesh out some of these connections, and add new and more interesting ones!

Notes and presentations from the briefing day.

We held a briefing day for our current grant call for funding on Monday. The presentations from the day are available to download and Owen Stephens made notes on most sessions for us:

In the afternoon we focused on discussion sessions. One of the sessions was dedicated to the activity data strand and the presentations from that session are available.

Owen also made notes on these presentations and the discussion that followed:

Some people may be interested in browsing the twitter stream from the event.

We plan to post a set of frequently asked questions on this blog later in the week.

Gaining business intelligence from user activity data

Business intelligence resulting from ‘user activity data’ could help universities to manage resources more efficiently, budget more effectively, make smarter purchasing decisions, improve their services and demonstrate impact.  The likes of Amazon and Tesco use activity data to make business decisions and to also provide recommender services on top of this data.  They know a lot about thier customers’ purchasing habits. How does this translate to the academic environment? Are we able to utilise this use data for our benefit?  This is a very timely and exciting area of investigation for the sector.

We invite you to attend a workshop exploring the potential of this data which is derived from services such as library systems, virtual learning environments and student registries.

As well as informing people of the potential of this data, we are looking to generate ideas and use cases to help plan for future work in this area.  So please come ready to participate – I promise it will be an interesting day!

This workshop is suitable for senior managers and practitioners working in libraries, teaching and research. It will be chaired by Professor David Baker, Deputy Chair of JISC, with contributions from practitioners who have practical experience of using user activity data in higher education.

Date: 14 July 2010 

Venue: The Hatton, 51-53 Hatton Garden, London, EC1N 8HN

For more information and registration go to: http://www.jisc.ac.uk/events/2010/07/businessintelligence.aspx

Survive or Thrive Conference

On the 8th and 9th June JISC and UKOLN hosted the Survive or Thrive conference in Manchester.  This event hadn’t been without its challenges in the planning stages but it came together brilliantly over the two days (give or take the occasional AV problem) with an agenda full of interesting talks and an engaged and knowledgeable audience eager to debate the issues.

I’m not going to report on the event here but rather point to the quite incredible live blog that Owen Stephens produced during the event (I swear sometimes he had published his posts before the audience had finished clapping the speaker!).

You can also check out an analysis of the Twitter tag on the Eduserv Labs Summarizr site (a bit of a disclaimer here is that we did end up clashing tags with a Spanish fishing event of all things but it doesn’t seem to have made much difference to the analysis!) .

We also videoed all the talks and have started to make them available on the Survive or Thrive site.

If you attended the event I hope you enjoyed it as much as I did and if not I do urge you to check out the videos.

Halfway through the US Blue Ribbon Meeting

Greetings from Blue Ribbon Meeting again. http://brtf.sdsc.edu/index.html

Some interesting angles emerging from a variety of participants. Heard from Thomas Kalil this morning who works as a policy staffer at the White House in the Office of Science and Technology. (Is he perhaps what they call a ‘policy wonk’ … I don’t know. I should have paid closer attention to the West Wing). Anyway – he talked about how the preservation community might be able to get this whole area of digital sustainability onto the Presidential radar. What we don’t do is to present this as a problem that needs to be tackled in ten different ways, at different levels, by diverse stakeholders. We need to realise that the White House is in the business of saying ‘no’ most of the time, so we need to make it as easy as possible for them to say ‘yes’. We need to be realistic about what we’re asking for and we need to be very directed about who in the Presidential office we approach to champion the cause.

What we probably do is get a message to the President that person x at trusted organisation y, has a low cost/high benefit measure that the President really needs to hear, which fits with his broad agenda, and has the backing of many thought leaders across the expert community.

I’m assuming we can transpose that to Downing Street.

 Right … we’re back in session. More later.

#brtf

Next Page →