Evaluation Criteria

1.  High-level evaluation criteria

Overall, there is consensus on some high-level criteria that institutional repositories (IRs) should meet. Based on the work of Genoni (2004), Johnson (2002), and Lynch (2003), Jones et al. (2006, pp.53-4) summarise these and add defining criteria as follows:

• Institutionally defined (no extra-institutional issues to be resolved, as in subject repositories, easily integrated into existing system, in terms of style, semantics and technology)
• Scholarly (not necessarily publishable but of value to academics)
• Cumulative and perpetual (the collection is expanding, and items are preserved in perpetuity)
• Open and interoperable (access to the collection, and its content freely available)
• Digitally capture and preserve many aspects of campus life (dependant on decisions made by the institution on scope, discussed on the Implementation Considerations section, and types of material to be collected)
• Search without constraints (software employed able to answer queries from human and automated users, effective user and web interface)

Jones et al. (2006, pp.55-76) provide more detailed criteria for interpreting these elements, providing example use cases to show how such criteria can be developed to define user needs. There is useful information here to help institutions assess how well they are meeting goals such as ‘open’ and ‘interoperable’, as well as in the RLG and US National Records and Archives Association (NARA) checklist.

Genoni (2004, p.304) additionally propose that IRs should be evaluated based on the ways that print collections have been evaluated - size, meeting needs of users, levels of use, and satisfaction. Based on these four elements, he suggests an evaluation should consider which groups of users are accessing the repository and the degree to which different categories of content are being used. It is acknowledged that this form of evaluation needs a closer definition of the terms ‘user’ and ‘use’, making a distinction between external and internal user groups. While the goals of a repository may be to encourage high rates of external use, scholars within the institution may find the repository useful as a mechanism for the storage and retrieval of their own work.

Westell (2006) developed a set of indicators for evaluating the success of selected institutional repositories in Canada, based on “their demonstrated integration with other research initiatives”. Finding little guidance in the literature, she developed her own set that include:

• mandate
• integration into institutional planning
• funding model
• relationship to digitisation centres
• interoperability
• measurement
• promotion
• preservation strategy

Westell (2006) comments “a further gauge of IR success, user acceptance, is outside the scope of this study, however, further research, particularly usability studies, would do much to complete the picture of a successful IR”.

Shearer (2003) describes a study being conducted by the Canadian Association of Research Libraries to determine some success factors of institutional repositories. Defining success as ‘use’, the project investigates if levels of input activity impact on use and develops a set of critical success factors that include input activity, disciplines, advocacy activities, archiving policies, copyright policies, content type, staff support, quality control policies, software, and use. In the initial phase (the study is long-term and does not claim to have produced any reliable data as yet) a key factor to emerge as impacting on ‘use’ was interoperability with other repositories and participation in some form of metadata harvesting activity.

Thibodeau (2006) coming from a records management perspective at NARA identifies orientation, coverage, and collaboration as three key axes on which to base evaluation and develop criteria.

2.  Trusted and Trustworthy Digital Repositories

Moore and Smith (2006) show how the Research Libraries Group (2005) Trusted Digital Repository Requirements can be built into an assessment process, in particular focusing on the extent to which management policies ensure compliance with the requirements.

The Research Libraries Group and National Archives and Records Association (NARA) Audit Checklist: The Research Libraries Group (2005) Trustworthy Repositories Audit and Certification: Criteria and Checklist (TRAC) is the most comprehensive system currently being advocated for the evaluation of a repository and covers both technical and management issues in considerable detail. It is rapidly becoming a de facto standard (an ISO standard has been applied for and is itself based on ISO 9000 and ISO 17799 covering data security and information management systems).

The checklist is based on four domains:

1. organisation
2. repository functions, processes and procedures
3. designated community and intended uses of the information
4. technology and technical infrastructure

Metrics addressing each of these elements are grouped into further sections. Where possible international standards are referred to as criteria against which to measure success.

Implementing the Checklist: Kaczmarek (2006) reports on the implementation of this checklist at the NDIIPP-sponsored ECHO DEPository project, further illuminating several issues and experiments with a scoring system, in order to carry out inter-institutional comparisons. Moore and Smith (2006) state that work on the checklist is ongoing and other institutions who have trialed it as a measurement tool are invited to contribute experiences.

3.  OAIS Reference model

The OAIS (2007) reference model can also be used as a basis for evaluating an open access archive system. Ball (2006) provides arguments why it is not fully appropriate as an evaluation tool and describes the development of the RLG/NARA Checklist to better fulfil this purpose.

4.  Other Approaches to Evaluation

Xia and Sun (2007) suggest that ‘depositorship’ (whether the item is deposited by the author(s) or not) and ‘availability of full-text’ are key criteria in assessing repositories. An issue raised by Probets and Jenkins (2006) that should be included in a set of evaluative criteria is the quality of documentation related to an individual repository. They argue that a key criterion in any successful repository is that the “purpose and aims of the IR (should be) clearly defined and that the IR documentation itself should be concise and easy to understand, with the rights and responsibilities of stakeholders clearly presented”.

5.  Download Statistics

An alternative approach to evaluating repositories is to investigate download statistics, both of individual repositories and making comparisons between them (benchmarking). Organ (2006) reports how download and usage statistics were used to assess the effectiveness of the University of Wollongong Repository, focusing on access via Internet search engines and the relationship between cover page hits only and full-text downloads.

6.  Software Selection and Testing

There is a variety of repository hosting software available to choose from. An international survey conducted by the Primary Research Group (2007) of 53 repositories gives an indication of the distribution of use of different hosting software. While overall DSpace was the most commonly used (37%), followed by Fedora (17%) and EPrints (13%), nearly 28% of repositories in the sample used a type of repository software not identified on the survey form. Software selection varied markedly between different regions in the world such as the United States and Europe, and between developed and developing countries.

Jones et al. (2006, pp. 72-83) provide valuable advice on the topic of software selection, but many of the accounts of individual institutional repositories that have been published will contain useful examples of how choices were made (see for example Jones (2004), What software should I choose? published by the Australian Partnership for Sustainable Repositories, and Nixon’s comparison of EPrints and DSpace in the University of Glasgow’s DAEDALUS repository).

OARiNZ commissioned a report on the Technical Evaluation of Selected Open Source Repository Solutions. For more information, see the Technical Evaluations section.

Goh (2006) developed an alternative set of criteria for evaluating open source software for repositories from a survey of the literature and tested them on four software packages. They provide a full checklist of criteria assessing each of these categories, with weightings dependent on the critical nature of each. Their criteria are:

• content management
• content acquisition
• metadata
• search
• access control and security
• report and inquiry
• preservation
• interoperability
• user interface
• standards compliance
• automatic tools and support.

7.  User studies

Although few repositories are established with the primary goal of meetings users needs, this remains an important criterion for evaluation. As noted above, there are two key classes of user that should be investigated; internal stakeholders and external users in the research community. Given the short development period of institutional repositories there have not been a large number of user studies to date, although there are some user needs analysis studies available. Examples include reports to JISC by Zuccala (2006) and Shipsey (2005) that focus on user needs for version control in repositories.

As user studies are completed and become available, especially within the IR community in New Zealand, authors are invited to post abstracts and links here.