As a first step in developing a more effective and robust approach to faculty research performance assessment, this study analyzed the existing faculty research assessment methods by comparing Korean universities’ research evaluation standards, National Research Foundation of Korea’s (KRF) standard for assessing research proposals, and various bibliometric measures. The study data included research evaluation standards of KRF along with 27 Korean universities with Library and Information Science (LIS) department, and the publication data of 183 LIS faculties in Korean universities from 2001 to 2015, which consisted of 3,863 papers 16,978 citations. After the comparative analysis of research evaluation standards, the study examined the rankings of institutions and faculty produced by applying the collected standards and bibliometric metrics to the publication data. The bibliometric measures used for the study were publication count, citation count, journal impact factor, and h-index, each weighted by author contribution formulas of Inflated, Fractional, Harmonic, Harmonic+, which adjusts the Harmonic method by increasing the first author contribution to 1, and school+, which refers to the most common author contribution weight found among the university standards.
The findings of the study are as follows: First, in evaluating research performance, most Korean universities classify journals into indexed and non-indexed categories where internationally indexed journals are typically given twice or more scores as domestic indexed journals; as for author contribution, the university standards appeared most similar to Harmonic method and most different from Fractional method. Second, at the institutional level, there were no statistically significant differences between evaluation measures, most likely due to the even mixture of faculty across universities; however, citation count weighted by Harmonic+ authorship contribution method showed the most consistently stable ranking patterns. Third, at the faculty level, there were statistically significant differences among evaluation methods, which were magnified in clusters of 30 faculty rankings; the differences were most prevalent and exaggerated in middle ranks, which suggests that choice of evaluation measure can produce wildly different outcomes for mid-level faculty.
The study showed that Korean university standards for research evaluation, which aim to consider both productivity and impact of faculty research, differ from similar bibliometric measures such as h-index, and the difference could significantly impact the assessment of faculty research performance. The findings of the study suggests that incorporation of more granular measures such as journal impact factor and citation counts may benefit the current standards in becoming more robust.