Performance-based Research Fund - Implications for Research in the Social Sciences and Social Policy

Jonathan Boston
Victoria University of Wellington and the Tertiary Education Commission

Brenden Mischewski
Tertiary Education Commission

Roger Smyth
Ministry of Education

Abstract

This paper discusses the outcome and likely implications of the new Performance-Based Research Fund (PBRF) in New Zealand, with particular reference to the results for, and possible impact upon, the social sciences. The paper begins with a brief outline of the rationale for, and design of, the PBRF and then examines the outcome of the first Quality Evaluation of research in the tertiary education sector, conducted in 2003. The paper includes consideration of the relative performance of the eight main social science subject areas, together with an analysis of how these subject areas compare with the other 33 disciplinary groupings (drawing upon a number of data sources in addition to the PBRF results). Attention is also given to what the 2003 Quality Evaluation reveals about the demographic structure of New Zealand’s academic community, with particular reference to the social sciences. The paper concludes by exploring the possible implications of the PBRF for the funding and conduct of research in the social sciences. The analysis indicates that although the research performance of the social sciences is generally superior to the average across the 41 subject areas assessed, the introduction of the PBRF is likely to result in the social sciences suffering a small net loss of funding. Nevertheless, the best performing of the eight social science subject areas, such as psychology, stand to gain in relative terms, as do the two universities with the highest quality scores – Auckland and Otago.

Introduction

This paper discusses the design, results and likely implications of the new Performance-Based Research Fund (PBRF) in New Zealand, with particular reference to the results for, and possible impact upon, the social sciences. The paper begins with a brief outline of the rationale for the PBRF and a description of the three components of the fund – the Quality Evaluation, the Research Degree Completions (RDC) measure and the External Research Income (ERI) measure. Consideration is then given to the outcome of the first Quality Evaluation of research in the tertiary education sector, which was conducted in 2003.

Having examined the overall results for the tertiary sector as a whole, the paper analyses the performance of the social sciences. This includes comparisons of the relative performance of the various social science disciplines, together with an analysis of how the social sciences compare with other disciplinary groupings (drawing upon a number of data sources in addition to the PBRF results). Attention is also given to what the 2003 Quality Evaluation reveals about the demographic structure of New Zealand’s academic community, with particular reference to the social sciences. The paper concludes by exploring the possible implications of the PBRF for the funding and conduct of research in the social sciences. Significant attention to the RDC and ERI measures within the PBRF is outside the scope of this paper.

The Design and Implementation of the PBRF

Internationally, there has been growing recognition of the importance to a nation’s economic and social advancement of a high performing research sector. That recognition has led governments in many countries to reform the systems for managing and funding the research activities of higher education institutions.¹ The public funding for research in the tertiary education sector in New Zealand has for many decades been largely delivered as a component of student tuition subsidies; it has thus been significantly dependent upon the structure and volume of student demand. During the 1990s, various critics argued that funding research on such a basis did not encourage research excellence; nor did it ensure that high-calibre researchers received adequate resources.² In 1998, the government proposed the establishment of a separate contestable research pool for the tertiary sector, but the idea received a mixed reception and was not implemented.

Subsequently, in late 2001, the Tertiary Education Advisory Commission (TEAC), created by a new government to advise it on tertiary education matters, proposed the establishment of a Performance-Based Research Fund (PBRF). Under the proposed model, the research “top-up” component of tuition subsidies would be placed in a separate fund and allocated to eligible tertiary education organisations (TEOs) via a new performance-based funding formula. In accordance with this formula:

• 50% of the available funding would be allocated on the basis of the results of periodic assessments by expert panels of the quality of the research produced by eligible staff in participating TEOs
• 25% would be based on the volume of RDC (with cost weightings and research component weightings for different types of research degrees)
• 25% would be based on the volume of ERI.

The suggested combination of peer review and performance indicators led to the PBRF being referred to as a “mixed model”. In this context, the proposed approach differed from the “pure” indicator models used in Australia and Israel and the “pure” peer review models employed in Britain and Hong Kong. Equally important, unlike the British Research Assessment Exercise (RAE) where the unit of assessment is a discipline-based department or school, TEAC proposed that the unit of assessment be individual staff members (as in Hong Kong) (see Boston 2002).

The government endorsed the broad concept of a PBRF in mid-2002. But in December 2002, on the advice of a sector working group, it made a number of changes to the scheme envisaged by TEAC. For instance, the weighting placed upon the peer assessment component was increased to 60%, while the weighting of the ERI component was reduced to 15%. Also, in keeping with the Working Group’s recommendations, a much more comprehensive approach to the assessment of research quality was approved. This entailed an evaluation of Evidence Portfolios (EP) prepared by each eligible staff member, with each EP providing details of the author’s research outputs, peer esteem and contribution to the research environment during the preceding six years. It was agreed, in accordance with the working group’s advice, that the first so-called “Quality Evaluation” of research in the tertiary education sector would be conducted in 2003, with a second planned for 2006. Beyond this, Quality Evaluations would be held every six years. Under the new policy, funding via the PBRF would be phased in during 2004–2007, reaching a total of about $175 million in 2007. The newly established Tertiary Education Commission (TEC) was given the task of implementing the PBRF.

Under the assessment regime developed for the first Quality Evaluation, participating TEOs conducted an initial internal evaluation of the EPs of their respective PBRF-eligible staff members – this included most academic staff and a limited number of non-academic staff (e.g. post-doctoral fellows) (see Hall et al. 2003). Each EP was assigned one of four possible Quality Categories (“A”, “B”, “C” and “R”). Those nominated an “A”, “B” or “C” were submitted, in late September 2003, to TEC for assessment by one of the 12 peer review panels. These panels, in turn, made their own judgement concerning the quality of each submitted EP and assigned an appropriate Quality Category. Such judgements were based on clear standards and guidelines established by TEC. The decisions of each panel were subject to the oversight of a Moderation Panel consisting of the 12 panel chairs and an independent chair.

In the event, 22 of the 45 PBRF-eligible TEOs participated in the 2003 Quality Evaluation, including all of New Zealand’s eight universities. Of the 8,013 PBRF-eligible staff in these 22 TEOs, 5,771 had their EPs assessed by a peer review panel. The remainder were automatically assigned an “R”. Although 23 eligible TEOs did not participate, it is very likely – given the level of research output in the non-participating TEOs and given the results of academic researchers in those that did – that the 2003 Quality Evaluation included the vast majority of research-active staff within New Zealand’s tertiary education sector.

Overall Results of the 2003 Quality Evaluation

The published results of the 2003 Quality Evaluation provided a research “quality profile” for each participating TEO, subject area and nominated academic unit (see TEC 2004a). The profile furnished data on the number of PBRF-eligible staff – on both a headcount and a full-time equivalent (FTE) basis – in the relevant “grouping”, the number and proportion of As, Bs, Cs and Rs, and a quality score (out of a maximum possible of 10). In order to calculate the quality score, weightings were assigned to the four Quality Categories based on the PBRF funding formula: “A” (5), “B” (3), “C” (1) and “R” (0). The weighted scores were then multiplied by 2 and divided by the total number of eligible staff in the relevant grouping. To secure the maximum score of 10, all the members of the relevant grouping would need to have been assigned an “A” Quality Category

The overall results of the 2003 Quality Evaluation are outlined in Table 1. Significantly, the peer review panels assigned relatively few high Quality Categories, with only 5.7% of PBRF-eligible staff receiving an “A” and 23.2% a “B”. Altogether, almost 40% were assigned an “R”. As might be expected, the results of the internal assessment conducted by TEOs were somewhat more favourable, with 11.9% of staff nominated an “A” and only 26.9% an “R”.

Table 1 Overall Results of the 2003 Quality Evaluation

The overall quality score for the tertiary sector as a whole (i.e. the 22 participating TEOs) was 2.59 (FTE-weighted). This was lower than many expected, and lower than the results of the internal TEO assessment undertaken (which yielded a quality score of 3.49). The low average quality score and the relatively low proportion of staff assigned an “A” or “B” reflected a variety of factors, including:

• the exacting nature of the criteria for achieving a high Quality Category
• the need for those hoping to secure a “C” to achieve either a reasonably good score for the research output component of their EP or an adequate score across all three components
• the decision by the 12 peer review panels to apply the relevant assessment criteria in a rigorous manner
• the eligibility criteria for staff participation in the PBRF (see below).

The results for each of the 41 subject areas covered by the Quality Evaluation are outlined in Figure 1. The quality scores – which vary from 4.74 for Philosophy to 0.34 for Nursing – reveal very substantial differences in the relative research performance of the many different subjects taught within New Zealand TEOs – differences that appear to be significantly greater than occur in Britain (as measured by the RAE). ³

Considerable care is needed when interpreting the results for particular subject areas. First, some of the subject areas cover two or more separate disciplines, and hence the results may mask major variations in the relative performance of these disciplines.

Figure 1 - Subject Area Ranking – All Subject Areas

Numbers alongside bars indicate FTE-weighted quality scores
Numbers in parentheses indicate total number of PBRF-eligible FTE-weighted staff

Second, many PBRF-eligible staff undertake research that crosses a number of subject-area boundaries. Accordingly, the results for some subject areas (especially those with small numbers of researchers) may have been influenced by the choices made by staff (or subsequently by TEC) concerning the subject area and panel under which they were assessed. For instance, a health economist could potentially have been assessed by one of three panels (Business and Economics; Medicine and Public Health; and Social Sciences), and the person’s results could have been reported under at least four different subject areas (Economics; Political Science, International Relations and Public Policy; Public Health; and Sociology, Social Policy, Social Work, Criminology and Gender Studies).

Third, although TEC instituted a relatively comprehensive and rigorous moderation process, there can be no guarantee that the panels applied the relevant quality standards in an absolutely consistent manner (whether on an inter-panel or intra-panel basis). Accordingly, the results for certain subject areas may have been influenced, at least to some degree, by the relative generosity or otherwise of the relevant panel assessors.

Fourth, the nature of the staff eligibility criteria influenced, at least to some degree, the quality scores for particular subject areas (and also TEOs and nominated academic units). For instance, unlike the British RAE, people who were not members of the academic staff but who made a significant contribution to degree-level teaching and/or research (e.g. post-doctoral fellows, and some senior tutors, technicians and librarians) were eligible for inclusion in the PBRF. A very high proportion of such staff received a “C or an “R” in the 2003 Quality Evaluation. Hence, subject areas with a disproportionately large number of PBRF-eligible non-academic staff and/or post-doctoral fellows, other things being equal, secured lower quality scores than those with a low number of such staff (see Wiltshire 2004). Similarly, subject areas with a disproportionately large number of new and emerging researchers secured lower quality scores, other things being equal, than those with a low number of such staff. Hence, the quality score achieved by a department with large external research grants – and thus able to afford a significant number of post-doctoral fellows and other new researchers – may well have been the same as a department with very little external research income.

Subject areas with significant numbers of part-time academic staff (e.g. clinicians) also tended to fare less well because such staff, for understandable reasons, typically secured lower Quality Categories. This affected, at least to a small degree, the quality scores of areas like clinical medicine and dentistry, and the creative and performing arts.

Overall, the combined impact of the PBRF staff-eligibility criteria, the structure of employment in different disciplines and the formula for calculating quality scores is likely to have reduced the quality scores of some subject areas (especially those in engineering, the natural and physical sciences, medicine, and the creative and performing arts) relative to those in other subject areas (especially the humanities, social sciences and commerce). Although the precise magnitudes are difficult to determine, they are not likely to have been large.

The Results of the 2003 Quality Evaluation for the Social Sciences

Assessing the research performance of the social sciences under the PBRF poses the question of which of the 41 subject areas should be included in any such analysis. The Social Sciences and Other Social/Cultural Studies Panel assessed EPs in six subject areas:

1. Anthropology and Archaeology
2. Communications, Journalism and Media Studies
3. Human Geography
4. Political Science, International Relations and Public Policy
5. Psychology
6. Sociology, Social Policy, Social Work, Criminology and Gender Studies.

However, the subject area of Economics, which is generally regarded as a social science, was the responsibility of a separate panel – Business and Economics. Further, some of the other 41 subject areas also included significant numbers of social sciences, most notably Public Health.

For the purposes of this analysis, the social sciences will be defined to include all those staff whose results were reported under the following eight subject areas:

1. Anthropology and Archaeology
2. Communications, Journalism and Media Studies
3.  Economics
4. Human Geography
5. Political Science, International Relations and Public Policy
6. Psychology
7. Public Health
8. Sociology, Social Policy, Social Work, Criminology and Gender Studies.

It is recognised that some of the staff whose results were reported under these eight subject areas may not regard themselves as social scientists (e.g. journalists). Conversely, there are likely to be many social scientists who had their results reported under other subject areas (e.g. Accounting and Finance; Management, Human Resources, Industrial Relations, International Business, and Other Business; Māori Knowledge and Development; and Marketing and Tourism). Nevertheless, while accepting that the boundaries are rather arbitrary, the results for the eight subject areas in question are likely to provide a reasonably reliable guide to the overall research performance of the social sciences, as well as a good indication of the relative performance of the key social science disciplines.

Table 2 Results of the 2003 Quality Evaluation for the Social Sciences

Tables 2, 3 and 4 summarise the key results of the 2003 Quality Evaluation for the social sciences within participating TEOs (see also TEC 2004a, 2004b). As highlighted in Table 2, the overall quality score (FTE-weighted) for the eight subject areas was 3.10, roughly comparable to the quality score for the six subject areas covered by the Social Sciences Panel (3.15).

Both scores place the social sciences well above the average score for all 41 subject areas (2.59), and just ahead of the average score for the subject areas assessed by the Humanities and Law Panel (3.09). Against this, the overall quality score for the social sciences was significantly lower than the average scores for the physical sciences (4.1), the biological sciences (3.55) and medicine excluding Public Health (3.69) – suggesting that the quality of research in the social sciences in New Zealand is rather lower than in the “hard” sciences (even before allowance is made for the impact of the eligibility criteria on quality scores, as discussed above).

Altogether, 1,181 (or 14.7%) of the 8,013 PBRF-eligible staff in 2003 were assessed within the eight social science subject areas. Of these, the overwhelming majority were located within the country’s universities, with three universities – Auckland, Massey and Otago – accounting for close to 53% of the total. On an FTE-weighted basis, 82.3 (or 7.5%) of the 1,095.1 social scientists were rated “A”, 289.3 (or 26.4%) were rated “B”, 419.3 (or 38.3%) were rated “C”, and 304.4 (or 27.8%) were rated “R”. Accordingly, more than a third of social scientists received either an “A” or “B”, indicating a significant depth of research capability within the tertiary education sector.

Against this, it is evident that there are large differences in research quality across the various social science disciplines. The subject area of Anthropology and Archaeology achieved a remarkably high quality score of 4.55 (second only to Philosophy), while the subject area of Communications, Journalism and Media Studies achieved a much more modest quality score of 1.59 (placing it 35th in the subject-area ranking). It is notable that one of the largest social science disciplines, Psychology (with 217.5 PBRF-eligible staff), secured a quality score of close to 4, and that nearly half of those assessed in this area were assigned either an “A” or “B” Quality Category. The quality score for the largest subject area in the social sciences – Sociology, Social Policy, Social Work, Criminology and Gender Studies – was 2.4, somewhat under the sector-wide average. Within this subject area it is evident (e.g. on the basis of the results for the relevant nominated academic units) that the average quality score for sociologists was higher than for those in the area of social work.

Overall, the quality scores for the social sciences are consistent with the pattern in other research fields: generally speaking, the newer disciplinary areas and those with a higher concentration of practice-based researchers performed less well than long-established disciplines and those with proportionately fewer “practitioners”.

Table 3 Quality Scores by Universities for the Social Sciences (FTE-weighted)*

*Subject areas included are: Anthropology and Archaeology; Communications, Journalism and Media Studies; Economics; Human Geography; Political Science, International Relations and Public Policy; Psychology; Public Health; and Sociology, Social Policy, Social Work, Criminology and Gender Studies.

As revealed in Tables 3 and 4, there are significant differences at the TEO level in the research performance of the social sciences. The University of Auckland, with a quality score of 4.25 (FTE-weighted), is plainly the pre-eminent social science research institution in the country. Its nearest rival, the University of Otago, achieved a score of 3.75, while at the other end of the spectrum the Auckland University of Technology secured a score of only 0.83 – this was close to its average institutional score for all subject areas (0.77), but a long way behind the seventh-ranked university, Massey University, with a score of 2.6. In all but two of the eight universities (the University of Canterbury and the University of Waikato), the average quality scores for the social sciences was higher, on average, than the overall results for each respective institution. Significantly, of the 82.3 (FTE-weighted) social scientists to achieve an “A”, 50 (60.8%) were based in just two institutions – Auckland and Otago. These universities also accounted for 46.8% of those assigned a “B”.

The relatively high quality score achieved by the University of Auckland, coupled with its disproportionately large number of As and Bs, is reflected in the fact that it received the best quality score for three of the eight subject areas in the social sciences. Moreover, when the results of the relevant nominated academic units are considered, the University of Auckland achieved the highest scores in all but one disciplinary area, namely Psychology (where the University of Otago ranked first). In the broad subject area of Communications, Journalism and Media Studies, the 14 staff in Massey’s Department of Communications and Journalism achieved the best result in these particular sub-fields (with a quality score of just 1.6), but the 15 staff in Auckland’s nominated academic unit Film, TV and Media Studies achieved a much higher score of 4.9.

Table 4 Results of the 2003 Quality Evaluation for the Social Sciences – Relative TEO Performance

The Relative Performance of the Social Sciences

As noted earlier, the results of the 2003 Quality Evaluation suggest that the research performance of the social sciences in the tertiary education sector is superior to the average for all 41 subject areas but significantly below the results for medicine and the biological and physical sciences. An important question arising from such data is whether this finding is consistent with other available evidence.

Unfortunately, there is not a wealth of alternative data sources, and much of the other data that can be used for making such comparisons is of limited relevance and/or reliability. For instance, one can draw upon the available bibliometric data (see Ministry of Research, Science and Technology et al. 2003). Table 5 provides data on the number of research papers (in different disciplinary groupings) published during 1997⁴  and the number of times indexed research papers were cited or referred to by other researchers in their publications during the five years from 1997 to 2001.

Such data suggest that New Zealand social scientists not only produce a relatively low volume of publications by comparison with researchers in the broad fields of science and health, but also that their work is much less well cited (and thus of lower impact). Such conclusions, however, are open to serious question. First, a much lower proportion of the research published in the social sciences than in the biological and physical sciences appears in journal articles (or at least in the journals included in the Thomson ISI database). This necessarily reduces the apparent volume of research output in the social sciences. Second, and related to this, the citation rates reported in Table 5 are based on citations in journals that form part of the Thomson ISI database, and do not include citations in conference papers, reports, books and other types of research outputs. Again, the differences in the publication conventions of the different disciplines help explain why the citation rates for the social sciences (not to mention the humanities) appear to be so low. Thirdly, a significant proportion of the research in the social sciences in New Zealand is focused on local audiences and publications that are less likely to be considered in the citations database. Fourthly, research in the social sciences is more likely than research in the natural sciences to appear in books and hence be overlooked in citations indices.

Table 5 Citations per Research Paper by Research Field, 1997–2001

Note: In this table, social sciences includes economics and education.
Source: Ministry of Research, Science and Technology et al. 2003.

Another possible way of assessing the relative quality of research in the social sciences in New Zealand’s tertiary education sector is to consider the number of doctoral completions. As shown in Table 6, in 2003 more than 21% of all doctoral completions in New Zealand universities were in the social sciences (including Education), the highest of any broad field. By contrast, the biological sciences were responsible for 18.8% of completions, the physical sciences 13.7% and the humanities 13.2%. Furthermore, the evidence suggests that the proportion of completions in the social sciences is increasing, unlike the situation in the biological and physical sciences, thus suggesting that the social sciences are in a relatively healthy state.

Table 6 Doctoral Degree Completions at New Zealand Universities, 1998 to 2003

Source: Ministry of Education 2004.

Yet another approach to assessing the research quality of the social sciences is to compare the performance of New Zealand social scientists with their counterparts in other countries. One obvious option is to use bibliometric data. But for a variety of reasons – including the fact that a significant proportion of the research outputs of New Zealand social scientists are not published in the form of journal articles and are thus not reported in the relevant databases – such data are unlikely to provide a very reliable guide to relative performance levels.

A potentially more fruitful approach would be to compare the results for the social sciences in the 2003 Quality Evaluation with those of the British RAE. For instance, it is possible to rank the 68 subject areas assessed in the most recent RAE (in 2001) using a method like that employed by McLeay (2004a, 2004b). The results of such an exercise could then be employed to undertake a non-parametric comparison of the New Zealand subject-area rankings based on the results of the first Quality Evaluation.⁵

Demographic Patterns

Tables 7–12 provide data on the ethnicity, gender and age for all PBRF-eligible staff (based on the PBRF census (staffing return) conducted in late July 2003), together with a breakdown of equivalent data for staff in the social sciences. In addition, data are provided on the distribution of Quality Categories (“A”–“R”), disaggregated on the basis of ethnicity, gender and age. It is important to note that the relevant demographic data are incomplete because submission of the information was voluntary and in many cases it was not provided.

Table 7 First Ethnicity of PBRF-Eligible Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes 2,052 (1,181.64 FTE) staff who did not report their ethnicity to their TEO. The analysis incorporates the priority recording system for ethnicity data used by Statistics New Zealand.
* Weighted on FTE basis

Table 8 Quality Categories by First Ethnicity for Social Science Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes 307 (286.3 FTE) staff who did not report their ethnicity to their TEO. The analysis incorporates the priority recording system for ethnicity data used by Statistics New Zealand. Subject areas included are: Anthropology and Archaeology; Communications, Journalism and Media Studies; Economics; Human Geography; Political Science, International Relations and Public Policy; Psychology; Public Health; and Sociology, Social Policy, Social Work, Criminology and Gender Studies.
* Weighted on FTE basis.

Table 9 Gender of all PBRF-Eligible Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes excludes 31 (28.6 FTE) staff whose gender was not reported by their TEO.
* Weighted on FTE basis.

Table 10 Gender for Social Sciences Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes 1 (1 FTE) staff member whose gender was not reported by their TEO. Subject areas included are: Anthropology and Archaeology; Communications, Journalism and Media Studies; Economics; Human Geography; Political Science, International Relations and Public Policy; Psychology; Public Health; and Sociology, Social Policy, Social Work, Criminology & Gender Studies.
* Weighted on FTE basis.

Table 11 Age Ranges for all PBRF-Eligible Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes 1,755 (1,603.17 FTE) staff whose age was not reported by their TEO.
* Weighted on FTE basis.

Table 12 Age Ranges for Social Sciences Staff (all TEOs)

Notes: The table includes all TEOs who participated in the 2003 Quality Evaluation. It excludes 267 (240.27 FTE) staff whose age was not reported by their TEO. Subject areas included are: Anthropology and Archaeology; Communications, Journalism and Media Studies; Economics; Human Geography; Political Science, International Relations and Public Policy; Psychology; Public Health; and Sociology, Social Policy, Social Work, Criminology & Gender Studies.
* Weighted on FTE basis.

The key findings can be summarised as follows.

• In terms of ethnicity, almost three-quarters of PBRF-eligible staff in mid-2003 were Pakeha/European, about 6% Māori, just over 1% Pacific and almost 3% Chinese. The pattern for PBRF-eligible staff in the social sciences was broadly comparable with that of all PBRF-eligible staff, although the proportions of Pakeha/European, Maori and Chinese were slightly lower than the overall average while the proportions of Pacific and people of other ethnic backgrounds was slightly higher.⁶
• The proportion of Pakeha/European staff in the social sciences receiving an “A” or “B” Quality Category (35.56%) was higher than for any other ethnic group, at least where there was more than one person in the ethnic group in question. The proportion of Maori social scientists receiving a high Quality Category was relatively low. Note, however, that these results may not include all social scientists. For instance, there may have been some (possibly many) social scientists assessed by the Maori Knowledge and Development Panel, and the results for such staff cannot be readily distinguished from those of the Panel as a whole. Hence, the overall performance of Māori social scientists cannot be fully ascertained.
• As might be expected, the proportion of women in the social sciences (45.78%) was higher than the average across all subject areas (40.72%). Also, in line with other relevant data, the research performance of women in the social sciences was better than the average across all subject areas, with almost a quarter of women in the social sciences receiving an “A” or “B” compared with only 17.57% of all PBRF-eligible women staff. Similarly, whereas 52.60% of all women were assigned an “R”, the figure for the social sciences was 36.57%. Nevertheless, the proportion of women in the social sciences receiving a high Quality Category was significantly lower than the proportion of men. There are, doubtless, many reasons for this, including the fact that proportionately fewer women than men hold senior academic positions.
• In terms of age structure, the results show that the average age of PBRF-eligible staff is in the high 40s, with social scientists, on average, being slightly younger than the average for all eligible staff. The results also highlight that very few PBRF-eligible staff under 40 secured an “A” and relatively few a “B”. The pattern for the social sciences was broadly similar to that for all subject areas. Having said this, a much lower proportion of social scientists aged 60 and older received high Quality Categories than those aged 50–59, whereas for all PBRF-eligible staff the proportions for the two age groups in question were roughly the same. Why the pattern for the social sciences is different in this regard is not clear.

Implications of the PBRF for Social Science Research in New Zealand

It is only possible in this paper to consider very briefly how the PBRF might affect the social sciences in New Zealand. Nevertheless, the implications are likely to be wide ranging and significant – certainly if judged by the experience of the British RAE (see Goldfinch 2003, Roberts 2003, TEAC 2001:92-94).

At a sector-wide level, the PBRF can be expected to bring about a variety of behavioural changes, and have implications for:

• the allocation of research funding (both between and within TEOs)
• the nature, form, quantity and quality of research activity undertaken within the academic community
• human resource management, including performance management and remuneration policies
• the way PBRF-eligible staff allocate their time
• the nature and quality of the teaching-research nexus (both at the undergraduate and post-graduate levels)
• the relationships between TEOs and between TEOs and other organisations (e.g. other research providers and research funders).

In many respects, the impacts of the PBRF on the social sciences are likely to be similar to the general sector-wide impacts. However, there are a number of specific matters that deserve attention.

Turning first to the implications for research funding, it is hard to determine exactly how the PBRF will affect the quantum of financial resources available to the social sciences over the medium-to-longer term whether in overall terms or in relation to particular disciplines. There are various reasons why precise calculations are difficult to make, including:

• the problem of determining how much funding the social sciences will lose as a result of the removal of the current research “top-ups”
• uncertainty over possible changes to the three PBRF cost categories arising from the recent Funding Category Review
• uncertainty over the future structure of the subsidies for taught post-graduate programmes
• uncertainty over how much “new” money will be allocated by the government to the PBRF over the medium term and the extent to which the PBRF will be inflation adjusted
• uncertainty over how the PBRF results may affect the level and pattern of student demand
• the fact that PBRF funding is allocated to TEOs as an untied block grant, thus giving individual institutions discretion over how they distribute the funds internally.

Having said this, it is evident that the social sciences will generate a relatively small proportion of the 15% of total PBRF funding allocated on the basis of the ERI measure; this is because only a modest proportion of external research income is currently generated by social scientists and this pattern seems unlikely to change much in the near future (especially given the dearth of local public and private funding for research projects in the social sciences, except perhaps in Public Health).

Equally significant, six of the eight social science subject areas are funded at the lowest of the three PBRF cost categories – the exceptions being Psychology (which is funded at twice the lowest rate) and Public Health (which is funded at two-and-a-half times the lowest rate). This has significant implications for the funding allocated to the social sciences via the Quality Evaluation and the RDC measure. Against this, the social sciences received a quality score significantly above the sector-wide average, and almost 800 social scientists secured a funded Quality Category (i.e. “A”, “B” or “C”). These latter two factors will help mitigate the fact that most of the social sciences (and close to 60% of social scientists) are funded at the lowest rate.

We have attempted to ascertain more precisely how the introduction of the PBRF (and the eventual removal of research “top-ups”) will affect the share of Vote Education research funding that is allocated to the social sciences and to particular social science disciplines. Given the way data on enrolments are classified in tertiary education, however, it is not readily possible to separate the funding for the social sciences from that for the arts and humanities under the former research “top-up” system. Therefore, the following analysis compares the research funding earned by the arts, humanities and social sciences under the old and new funding systems.⁷

The government has put additional funding into the PBRF over the phase-in period 2004–2007. In order to separate the effects of the phase-in and the additional funding from the move to research performance funding, the analysis below considers the impact of the PBRF by comparing:

• the percentages of the 2003 research funding generated by the arts, humanities and social sciences
• the corresponding percentages of the 2004 contestable PBRF funding.

In effect, this analysis explores what would have happened had the PBRF been implemented without the additional funding and without a three-year phase-in.

It is not possible from the data collected for the PBRF to assign all of the components of PBRF funding to disciplines or subject areas. Nevertheless, the funding generated by the quality component of the PBRF can be assigned to broad fields of study. It is also possible to make a fairly good estimate of the share of the RDC funding attributable to the arts, humanities and social sciences.⁸

Table 13 shows the proportion of the PBRF quality funding generated by the arts, humanities and social sciences and compares this with the 2003 research “top-up” funding. It also includes the estimated percentage of quality and RDC funding attributable to the arts, humanities and social sciences. In 2003, under the research “top-up” funding system, the arts, humanities and social sciences generated nearly 16.6% of all of the research funding in the tertiary education sector. In 2004, those fields generated 17.2% of the PBRF quality funding. If the estimate of the RDC funding attributable to the arts, humanities and social sciences is also included, the proportion falls to 16.5%. Bear in mind that the data in Table 13 make no allowance for the way the ERI component of the PBRF is distributed across the disciplines.⁹

Table 13 Estimated Share of Total Research Funding for the Arts, Humanities and Social Sciences, 2003 and under PBRF

Notes: RDC funding attributed to classification #03 disciplines is estimated using the number of #03A4-funded EFTS as a proportion of all A4-funded EFTS.
In this table “Arts, Humanities and Social Sciences” excludes public health, economics and psychology. It includes education, but excludes teaching.
Source: Tertiary Education Commission 2004a.

Another approach is to consider the proportion of all PBRF quality-component funding earned by the staff in the eight social sciences disciplines. Those disciplines account for 14.7% of the full-time equivalent PBRF eligible staff. Weighting the staff according to the funding differentials in the PBRF gives better sense of the performance of staff in the social sciences in attracting quality funding.¹⁰  On a weighted basis, the staff in the eight social science disciplines represent 13.3% of all FTE PBRF eligible staff. Yet Table 14 shows that those staff generated 15.6% of all quality funding. That they attract a higher proportion of the total available funding reflects the fact that the assessment of their research standing in the 2003 Quality Evaluation was, on average, above the mean for all staff.

The research funding generated by the social sciences may be relatively less affected by the introduction of the PBRF than other disciplines funded at lower rates. The PBRF is expected to reduce the share of research funding for the lower-funded fields of study as a whole (including business and teaching, as well as the arts, humanities and social sciences). Table 15 indicates that the lower-funded subject areas are likely to experience a reduction in their share from 29% to 24% – even without taking into account the uneven allocation of the ERI component of the PBRF. To a considerable extent, this shift results from the fact that some of the lower-funded subject areas – management and marketing are good examples – are likely to generate relatively limited PBRF funding, yet earned substantial research “top-up” funding in the past because of large undergraduate degree classes.

Table 14 Share of Research Quality Funding for the Social Sciences, based on the 2003 Quality Evaluation

Note: In this table, “The Social Sciences” includes economics, psychology and public health, and excludes education.
Source: Tertiary Education Commission 2004a.

Table 15 Estimated Share of Total Research Funding for all Lower Funded Disciplines, 2003 and under the PBRF

Note: Funding categories A and I include arts, humanities, social sciences, education, teaching, economics and commerce, and exclude psychology and public health.
Source: Tertiary Education Commission 2004a.

Overall, the preceding analysis suggests that the introduction of the PBRF is likely to result in a small shift of research funding from the arts, humanities and social sciences to higher-funded subject areas (especially those with high quality scores and high ERI).

But what of the impact of the PBRF on individual subject areas? This is even more difficult to calculate, and of course much will depend on how individual TEOs choose to allocate their share of PBRF funding. Nevertheless, Table 14 shows, based on the results of the 2003 Quality Evaluation, the share of the quality funding generated by each of the eight social science subject areas. The data highlight that two subject areas – Psychology and Public Health – generated over 60% of the total quality funding for the social sciences, and about 9.4% of all quality funding (i.e. across the 41 subject areas).

In sum, while the social sciences as a whole may suffer a modest reduction in their share of research funding under the PBRF, the highest-rated of the eight social science subject areas (i.e. Anthropology and Archaeology, and Psychology) should fare moderately well (and may even generate greater funding in relative terms). Similarly, the TEOs with the highest quality scores for the social sciences (i.e. Auckland and Otago) are likely to gain, other things being equal, relative to those with the lowest scores.

But while the PBRF can be expected, over time, to contribute to a greater concentration of resources in the highest-scoring subject areas and TEOs, its impact will be much less dramatic than the British RAE. This is because the PBRF has a much flatter funding “gradient”. Hence, whereas departments rated less than “4” in the RAE generate little or no funding for their institution, and departments rated 5* generate many times the funding of departments (of a similar size and subject area) rated “4”, even very weak academic units in New Zealand have the potential to generate PBRF funding – assuming that they have staff rated at least “C” and/or some research students and/or some ERI.

The implications of the PBRF for individual social scientists are likely to vary depending on their particular subject area, institution and assigned Quality Category, as well as any future changes to the design of the Quality Evaluation. Overall, highly rated social scientists (and those able to attract external research income) are likely to be in greater demand and, over time, might be expected to command somewhat higher salaries. Conversely, those rated an “R” in the 2003 Quality Evaluation are likely to be less highly valued (other things being equal). Note that the “R” category includes a large number of new and emerging social scientists. Accordingly, unless the design of the PBRF is altered for the 2006 round, there is a risk that TEOs will become more reluctant to appoint young social scientists with recently completed doctorates and few publications. Interestingly, the experience of the British RAE suggests that the motivational and behavioural effect of the exercise has been “greatest for individuals with a moderate intrinsic interest in research” rather than those with either a low or high intrinsic interest (Goldfinch 2003:45). Whether the pattern is similar in New Zealand is one of many questions deserving further inquiry.

British experience under the RAE suggests that social scientists (and other PBRF-eligible staff) are likely to take the PBRF into account in determining how they allocate their time between their many and varied responsibilities. A key issue will be whether a particular activity or output is thought likely to “count” under the assessment methodology used in the Quality Evaluation. Activities deemed to “count” will become more attractive; those that do not “count” will become less attractive. For instance, supervising research students “counts”, whereas teaching large undergraduate classes does not. Other things being equal, therefore, academics might be expected to devote more attention to their postgraduate students and less to undergraduates. Equally, it is likely that active researchers (or those seeking to build a stronger research profile) will be even more reluctant to undertake demanding administrative responsibilities within their TEOs and somewhat less eager to be involved in professional or community service (or undertake their “critic and conscience” role) – unless this takes the form of activities that can be legitimately included within the “peer esteem” and “contribution to the research environment” components of their EPs.

At another level, there is the question of how the PBRF may affect the kind of social science research undertaken (e.g. basic, strategic and applied), the focus of the research undertaken (e.g. local, regional, national and international), and the overall research culture (e.g. whether more, or less, research is undertaken and published individually or jointly). In this regard, there is a widespread perception that the relevant PBRF panels tended to give greater weight to certain kinds of research outputs (e.g. sole-authored books and articles in overseas-refereed journals) and correspondingly lesser weight to others (e.g. chapters in books and reports for government agencies) (see WEB Research 2004). Similarly, there is a perception that locally-published and locally oriented research fared less well than research published overseas and research of a more comparative or international focus. Whether or not these perceptions are well-founded, they are likely to influence behaviour. There is thus a risk that social scientists will be less inclined to pursue research of an applied nature and research with a strong New Zealand orientation, and that they will be less inclined to publish the results of their research in local journals (and other outlets). Against this, of course, there will be many other influences on the behaviour and research interests of social sciences, not least the availability of funding for locally focused research from government departments and agencies.¹¹

Finally, there is the question of what impact the PBRF will have on the overall research performance of the social sciences in New Zealand. A detailed and authoritative assessment of this will not be possible for many years. Despite the differences between the RAE and the PBRF, British experience suggests that the PBRF is likely to increase the overall volume and quality of research output – although part of the reported improvement (e.g. as measured via the periodic RAEs) appears to be due to “grade inflation” (Boston 2002). It is highly probable that the quality scores of most, if not all, the social sciences will increase at the second and third Quality Evaluations, but part of this “improvement” is bound to be the product of academics learning the art of EP writing and selecting their nominated research outputs with greater care.

Conclusions

To sum up, the research performance of the social sciences in New Zealand, as judged by the 2003 Quality Evaluation, appears to be reasonably solid – although there are plainly large variations in the average quality of the research being conducted in different disciplinary areas and within different TEOs. The results suggest that the best-quality social science research, on average, is being conducted in two subject areas – Anthropology and Archaeology, and Psychology. By contrast, two subject areas – Sociology, Social Policy, Social Work, Criminology and Gender Studies, and Communications, Journalism and Media Studies – had quality scores either below or significantly below the average for the 41 subject areas assessed. This, of course, is not to suggest that such subject areas lack academic researchers of high distinction, but the outstanding researchers tend to be a lower proportion of the academics in these fields. The reasons for the large differences in research performance require further analysis, but are likely to be related, amongst other things, to differences in research culture and traditions and the structure of the academic workforce (e.g. the proportion of new and emerging researchers).

At the institutional level, the results of the 2003 Quality Evaluation reveal major differences in research performance. Within the social sciences, the University of Auckland is clearly New Zealand’s leading research organisation, followed at some distance by the University of Otago. It is also of some significance that the two top universities contain over 60% of those assigned an “A” and almost 50% of those assigned a “B”. In part, this result is related to the fact that these universities are both relatively large and are the primary providers of research in one of the eight social science subject areas, namely Public Health. Nevertheless, it is of concern that some universities that have substantial teaching programmes in the social sciences have relatively few senior and highly respected researchers in these disciplinary areas.

It will be many years before the full implications of the PBRF for the tertiary education sector, and the social science community within it, become fully apparent. In the meantime, the experience of other jurisdictions – most notably Britain – provides an indication of the likely impact that the PBRF will have on the behaviour of institutions and academic staff. This experience suggests generally positive outcomes with respect to overall research quantity and quality but more questionable implications for teaching quality (especially at the undergraduate level) and community service by academic staff. It is to be hoped that any negative impacts in New Zealand will be modest and that such impacts are properly evaluated and, where necessary, addressed in an appropriate manner.

References

Boston, J. (1999) “The Funding of Research in the Tertiary Sector” Access: Critical Perspectives on Cultural and Policy Studies in New Zealand, 18(2) 103-119.

Boston, J. (2002) Designing a Performance-Based Research Fund for New Zealand, Report for the Transition Tertiary Education Commission, Wellington.

Bryson, L. (2004) “Further Comment on PBRF”, Sociological Association of Aotearoa (NZ) Newsletter, August, pp.6-8.

Goldfinch, S. (2003) “Investing in Excellence? The Performance-Based Research Fund and its Implications for Political Science Departments in New Zealand” Political Science, 55(1):39-54.

Hall, C., K. Morris and T. Sawicka (2003) “Performance-Based Research Fund (PBRF): Policy and Practice”, New Zealand Annual Review of Education, 13:79-104.

McLeay, S. (2004a) “Reconciling RAE 2001 with RAE 2008” unpublished paper, Bangor University, April.

McLeay, S. (2004b) “Comparing the RAE 2001 and PBRF 2003 – a British Perspective” New Zealand Education Review, May 19 – 25, p.7.

Ministry of Education (2004) New Zealand’s Tertiary Education Sector: Profile and Trends, 2003, Ministry of Education, Wellington.

Ministry of Research, Science and Technology, Foundation for Research, Science and Technology, Health Research Council and Royal Society of New Zealand (2003) National Bibliometric Report, 1997 to 2001: International Benchmarking of New Zealand Research, Ministry of Research, Science and Technology, Wellington.

Roberts, Sir Gareth (2003) Joint Consultation on the Review of Research Assessment, Higher Education Funding Council for England, London.

Tertiary Education Advisory Commission (TEAC) (2001) Shaping the Funding Framework, Fourth Report, Wellington.

Tertiary Education Commission (TEC) (2004a) Performance-Based Research Fund: Evaluating Research Excellence – The 2003 Assessment, Wellington.

Tertiary Education Commission (TEC) (2004b) Performance-Based Research Fund: Report of the Social Sciences and Other Cultural/Social Studies Panel – The 2003 Assessment, Wellington.

WEB Research (2004) Phase 1 Evaluation of the Implementation of the PBRF and the Conduct of the 2003 Quality Evaluation, Centre for Research on Work, Education and Business Limited, Wellington.

Wiltshire, D. (2004) “How PBRF averages penalise departments with research grant success” New Zealand Education Review, May 19-25, p.6

Footnotes

1 Examples of countries that have moved to performance-linked research funding systems in higher education are the United Kingdom, Australia and Hong Kong.

² Criticisms of this nature were advanced by a number of leading academics, such as Professor Gary Hawke and Professor Peter Gluckman, and were also evident in the responses of the tertiary education sector to the government’s green and white papers on tertiary education reform during 1997–98. For a fuller account, refer to Boston (1999).

³ It should be noted that the extent of that difference may be exacerbated by the inclusion in the PBRF Quality Evaluation of a wider range of non-university TEOs and by the broader criteria for inclusion of staff in the assessment in New Zealand. In other words, the PBRF takes greater account of the work of academic researchers at the lower end of the scale than the RAE.

⁴ The figures are based on the Thomson ISI New Zealand National Citation Report database, which indexes publications from more than 8,700 research journals.

⁵ It should be noted, however, that the PBRF has a number of features that distinguish it from the UK RAE and other similar systems. Therefore, any such comparison would need to be treated with caution. It should be noted that a comparison of the results of the PBRF and those of the RAE was developed by TEC but was withheld following legal action taken by two universities.

⁶ There was a high incidence of non-response to the ethnic group question in the PBRF staff census. Around 25% of respondents did not declare their ethnic group. The figures quoted here are percentages of those with a declared ethnicity.

⁷ The fields of study in this analysis do not precisely match those used elsewhere in this paper. In order to achieve comparability between the broad area of social science under the old and new research funding systems, public health, psychology and economics have been excluded from the 2003/2004 comparative analysis. Further data on the funding share for the social sciences as more broadly defined is included in Table 15.

⁸ This can be calculated using the number of wholly research (level 4) EFTS in #03 as a proportion of total Category A-funded wholly research EFTS.

⁹ It should be noted that much of the Vote Research, Science and Technology contestable research funding is directed towards higher cost fields of study such as science, technology and health. There are, however, opportunities within the government sector to win research funding in the social sciences; the Foundation for Research, Science and Technology and the Health Research Council both fund some social science research, while developments such as the Building Research Capability in the Social Sciences initiative offer further opportunities.

¹⁰ This calculation weights each eligible staff member by the funding differential of the discipline to which he or she was assigned. The social science disciplines have a lower-than-average weighting. The average weighting for all eligible staff in the 2003 Quality Evaluation was 1.60 whereas in the social sciences it was 1.44.

¹¹ There is evidence of a much higher volume of research grant applications being made to research funding bodies over recent years, and some of this is almost certainly attributable to the introduction of the PBRF.