You can now measure the impact of your online resource
The Toolkit for the Impact of Digitised Scholarly Resources (TIDSR), developed for JISC by the Oxford Internet Institute, is now available online for everybody to use.
If you have been struggling with making sense of hits, visitors numbers, log analysis, users feedback, wondering how to interpret all this data, how to gather it in the first place and how to assess whether your resource is being used and how, the toolkit will provide a clear, concise and easy to use framework for carrying out an assessment of the impact an online resource is having.
As the toolkit web site emphasises:
There are a number of challenges in assessing the use and impact of online digital resources: these include new methods, shifts in the way that people access resources, new audiences, and new forms of information-seeking behaviour among different audiences.
The evaluation of online scholarship is a moving target, and therefore a flexible set of measures and practices will be used. The toolkit consists not of a single software solution, but a set of recommendations for best practices.
The toolkit inlcudes useful information, related articles, tools and guidance on how to use a range of quantitative and qualitatives measures (including webometrics, analytics, content analysis of media coverage, focus groups, resource surveys, user feedback analysis and more) and is open for submission of additional relevant resources, or comments, by the community.
The toolkit was piloted through case studies of five diverse JISC Digitisation projects funded under Phase 1 of the Digitisation programme.
Toolkit for the Impact of Digitised Scholarly Resources
In a previous post earlier in the year, Measuring the impact of digitised resources (12/6/2008), we announced the work that the Oxford Internet Institute (OII) was about to embark upon of identifying use and usage patterns of five JISC-funded online resources and devising meaningful metrics for the measurement of the impact of digitised scholarly resources.
This is a crucial tool for those interested in digitisation, providing much needed evidence and analysis of how digital resources are actually making a difference
At a recent JISC Digitisation programme meeting, Eric T Meyer and Katherine Eccles provided some background on how the OII is planning to carry out the work and the mixture of quantitative and qualitative measures they will take into consideration to gain some understanding of the use of such resources.
Quantitative Measures include:
• Webometrics
• Analytics
• Log file analysis
• Scientometrics / bibliometrics
• Content analysis of media coverage
Qualitative Measures include:
• Stakeholder interviews
• Resource surveys
• User feedback analysis
• Focus groups
• Questionnaires
One of the interesting things that emerged from their presentation was the need not to “obsess” too much about any particular indicator at any given time (eg, no need to look at web stats every month), but to consider a range of indicators collectively at regular intervals in time, in order to identify patterns over a longer duration of time.
The work will culminate in the creation of a Toolkit for the the Impact of Digitised Scholarly Resources, which will be disseminated in Spring 2009.
Presentation (PDF) on the Toolkit for the Impact of Digitised Scholarly Resources.
The Impact of Digitizing Special Collections on Teaching and Scholarship
A recent report from OCLC on The Impact of Digitizing Special Collections on Teaching and Scholarship. Reflections on a Symposium about Digitization and Humanities highlights the main recommendations that emerged from the symposium held in June 2008.
The symposium brought together both primary users of (digitised) primary sources as well as “custodians”, such as librarians, archivists, museum professionals and senior managers. Participants discussed, from their own particular view points, strategies to maximise the impact of digitisation of special collections on teaching and research.
The report calls for specific directions for libraries and archives to take in the near future:
- work with faculty to understand current research methods and materials
- go outside the library or archive to build collections and work with faculty
- continue to build digital and material collections for both teaching and research.
Other important issues that emerged were:
- licencing and third-party agreements: the need for common principles in negotiating licensing contracts in order to ultimately guarantee open access to content
- Metrics: the need for more evidence of the impact of digitization and the acknowledgement that quantitative web stats on their own are just not enough.
On this last point: JISC is currently conducting an Impact study of the projects funded under Phase 1 of its Digitisation Programme. The project, carried out by the Oxford Internet Institute, will have as one of its key outcome the production of a Toolkit for the Impact of Digitised Resources, which will provide a framework for useful metrics to consider when assessing impact - see blog post on Measuring the impact of digitised resources.
The Long Tail of Usability
The Stormont Papers resources makes available the debates from the parliament of Northern Ireland (Stormont) from creation in 1921 until the end of Home Rule in 1971.
It’s been available since 2006 and some statistics from the website are available. Of most interest is graph showing the spread of search terms entered by users
There are two interesting points from this
1) The bulk of your users may not be looking for the things you expect them to be looking for.
2) Pre-arranged hyperlinks on your home page can provide a user-friendly way of letting users get to know a resource’s contents.
Measuring the impact of digitised resources
Measuring the use and impact of digitised resources is no easy exercise. This is not only because of the changing nature of information seeking behaviour of different audiences, which has an effect on how users engage with digital resources. It is also due to the challenge in establishing appropriate metrics and criteria for measuring the impact and use of digitised collections.
When can a digital resource be considered a well-used resource? When and how do we know a resource has had an impact on its target audience? Are visitors’ numbers a useful performance indicator for judging the success of an online scholarly resource or should we dwell more on investigating what visitors actually do on a web site?
Yet, measuring impact is an important step in ascertaining if we’re reaching our audiences and responding to their needs. Impact has also been recognised as “the key factor in the potential for achieving long-term sustainability” for online academic resources in a recent report on Sustainability and Revenue Models for Online Academic Resources - an Ithaka Report commissioned by the Strategic Content Alliance.
Such are the issues, among many others, that the Oxford Internet Institute (OII) will be grappling with over the next year by leading on the JISC study on “Usage and Impact of Digitised Resources Funded Under the JISC Phase One Digitisation Programme”.
In addition to focusing on the assessment of five online collections created as part of the JISC Phase One Digitisation Programme, the OII will also produce a Toolkit for the Impact of Digitised Resources which aims to provide a set of approaches and tools available to measure and potentially improve the impact of current and future digitisation projects.
The Toolkit will be of value not only to JISC within the context of its digitisation programme, but also to the wider community. The OII study is due to terminate in Spring 2009 and both the final report and Toolkit will be made publicly available by JISC.
Usage and impact of digital resources
The JISC invites tenders to conduct a study on the usage and impact of a selection of online digital resources which were produced as part of the JISC Phase One Digitisation Programme, 2003-2007.
The digital collections created as part of the programme are aimed at enhancing the provision of e-content for teaching, learning and research purposes primarily in UK Higher and Further Education Institutions and to respond to the specific needs of users within this sector.
This study intends to investigate the level of impact, usage and take-up that such resources have had on teaching, learning and research within relevant subject areas and the degree to which they respond to users’ needs.
Funding of between £40,000 and £50,000, inclusive of VAT and expenses, is available for this study.
The deadline for proposals is 12 noon on Monday 28 April 2008. The study is expected to begin in June 2008 and be completed by March 2009.