In higher education settings the peer review process is highly valued and used for evaluating the academic merits of grant proposals, journal submissions, academic promotions, monographs, text books, PhD thesis and a variety of other academic products. The purpose of this thesis was to evaluate the peer review process for awarding research grants used by the Australian Research Council (ARC) Large Grants Program and to propose strategies to address potential shortcomings of the system. This study also evaluated psychometric properties such as the reliabilities of various ratings that are part of the assessment process of the ARC Large grants Program. Data for the all grant applications submitted for the 1996 round of the Large Grants Program were provided by the ARC. In a variation to the typical peer review process, applicants were given an opportunity to nominate assessors to review their proposals. The results indicated that global ratings given by the researcher-nominated assessors were systematically higher and less reliable than those by panel-nominated external reviewers chosen by the ARC. The reliability of peer reviews is not adequate by most standards. A critical direction for future research is considering what strategies need to be put in place to improve the quality of the reviews. To improve the reliability it is recommended that researcher-nominated reviewers should not be used; that there should be more reviews per proposal and a smaller more highly selected core of reviewers should perform most of the reviews within each sub-discipline providing a greater control over error associated with individual reviewers / Doctor of Philosophy (PhD)

In higher education settings the peer review process is highly valued and used for evaluating the academic merits of grant proposals, journal submissions, academic promotions, monographs, text books, PhD thesis and a variety of other academic products. The purpose of this thesis was to evaluate the peer review process for awarding research grants used by the Australian Research Council (ARC) Large Grants Program and to propose strategies to address potential shortcomings of the system. This study also evaluated psychometric properties such as the reliabilities of various ratings that are part of the assessment process of the ARC Large grants Program. Data for the all grant applications submitted for the 1996 round of the Large Grants Program were provided by the ARC. In a variation to the typical peer review process, applicants were given an opportunity to nominate assessors to review their proposals. The results indicated that global ratings given by the researcher-nominated assessors were systematically higher and less reliable than those by panel-nominated external reviewers chosen by the ARC. The reliability of peer reviews is not adequate by most standards. A critical direction for future research is considering what strategies need to be put in place to improve the quality of the reviews. To improve the reliability it is recommended that researcher-nominated reviewers should not be used; that there should be more reviews per proposal and a smaller more highly selected core of reviewers should perform most of the reviews within each sub-discipline providing a greater control over error associated with individual reviewers / Doctor of Philosophy (PhD)

Thesis (Ph.D.) -- University of Western Sydney, 2003. / "A thesis submitted to the University of Western Sydney in fulfilment of the requirements for the degree of Doctor of Philosophy" Bibliography : leaves 350-371.

Thesis (MPhil)--Stellenbosch University, 2001. / ENGLISH ABSTRACT: Peer review as an institutional mechanism for certifying knowledge and allocating resources dates back as far as 1665. Today it can with confidence be stated that it is one of the most prominent evaluative tools used in science to determine the quality of research across all scientific fields. Given the transformation within the processes of knowledge production, peer review as an institutionalised method of the evaluation of scientific research has not been unaffected. Peer reviewers have to act within a system of relevant science and find themselves responsible to the scientific community as well as to public decision-makers, who in turn are responsible to the public. This dual responsibility of reviewers led to the development of criteria to be used in the evaluation process to enable them to measure scientific excellence as well as the societal relevance of science. In this thesis peer review in science is examined within the context of these transformations. In looking at the conceptual and methodological issues raised by peer review, definitions of peer review, its history and contexts of application are examined followed by a critique on peer review. Peer review in practice is also explored and the evaluation processes of four respective funding agencies are analysed with regards to three aspects intrinsic to the peer review process: the method by which reviewers are selected, the review criteria by which proposals are rated, and the number of review stages within each review process. The thesis concludes with recommendations for possible improvements to the peer review process and recommended alternatives to peer review as an evaluative tool. / AFRIKAANSE OPSOMMING: Portuurgroep-evaluering as 'n geïnsitutsionaliseerde meganisme in die sertifisering van kennis en die toewys van hulpbronne dateer terug so ver as 1665. Huidiglik kan dit as een van die mees prominente metingsinstrumente van die kwaliteit van navorsing in alle wetenskaplike velde beskou word. Die transformasies wat plaasgevind het binne die prosesse waar kennis gegenereer word, het ook nie portuurgroep-evaluaring as 'n geïnstitusionaliseerde metode van evaluering ongeraak gelaat nie. Portuurgroep-evalueerders bevind hulself binne 'n sisteem van relevante wetenskap. Binne hierdie sisteem het hulle 'n verantwoordelikheid teenoor die wetenskaplike gemeenskap sowel as die publiekebesluitnemers wat op hul beurt weer verantwoordelik is teenoor die publiek. Hierdie dubbele verantwoordelikheid het tot gevolg die saamstel van kriteria waarvolgens evalueerders wetenskaplike uitmuntendheid sowel as relevansie tot die breër samelewing kan meet. Hierdie tesis ondersoek portuurgroep-evaluering teen die agtergrond van hierdie transformasies. Die konseptueie en metodologiese aspekte van portuurgroepevaluering word ondersoek deur eerstens te kyk na definisies van portuurgroepevaluering, die geskiedenis daarvan en kontekste waarbinne dit gebruik word. Tweedens word gekyk na kritiek gelewer op portuurgroep-evaluering. Portuurgroep evaluering binne die praktyk word ook ondersoek waar vier onderskeie befondsingsagentskappe se evaluerings prosesse geanaliseer word. Hierdie analise word gedoen in terme van drie essensiële aspekte binne portuurgroep- evaluering. Hierdie drie aspekte is as volg: 1) die wyse waarop evalueerders geselekteer word, 2) die evalueringskriteria waarvolgens navorsingsvoorstelle gemeet word en 3) die hoeveelheid evalueringsfases binne die protuurgroep-evaluerings proses. Laastens word aanbevelings ter verbetering van die portuurgroep-evaluerings proses as ook voorstelle tot moontlike alternatiewe tot portuurgroep-evaluering as 'n evaluerings instrument gebied.