VOLUME 11 (2007): ISSUE 1. PAPER 6

The b index as a measure of scientific excellence.
A promising supplement to the h index

Lutz Bornmann^a, Rüdiger Mutz^a, Hans-Dieter Daniel<sup>a,b

a</sup>Professorship for Social Psychology and Research on Higher Education
Swiss Federal Institute of Technology Zurich (ETH Zurich)
Zähringerstr. 24. CH-8092 Zurich (Switzerland)
E-mail: bornmann@gess.ethz.ch

^bEvaluation Office, University of Zurich
Zurich (Switzerland)

We propose the b index as a measure of scientific excellence at the micro and meso levels, as a promising supplement to the h index and its variants (such as g index and R index).

Keywords

h index, b index, scientific excellence

Hirsch (2005) proposed the h index as a criterion to quantify the scientific output of a researcher: A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np − h) papers have no more than h citations each (Hirsch, 2005). The new measure of individual scientific achievement was quickly taken up by researchers working in the area of information sciences and evaluative bibliometrics. In an overview of the research literature, Bornmann and Daniel (2007b) found approximately 30 papers on the h index only one year after the publication of Hirsch (2005). Besides the many advantages that the new index is said to have for evaluation purposes (e.g., the robustness and the ease with which h index values can be verified, see Vanclay, 2007), the papers also point to a number of disadvantages (for an overview, see Bornmann & Daniel, 2007b; Jin et al., 2007; Liu & Rousseau, 2007). It has been pointed out, for instance, that the h index is only weakly sensitive to the level of the highly cited papers in a publication set and that it is field-dependent (Egghe, 2006a; Jin et al., 2007).

Some variants of the h index have been proposed in an effort to overcome the disadvantages, such as the m quotient (Hirsch, 2005), g index (Egghe, 2006b), h(2) index (Kosmulski, 2006), A index (Jin, 2006), R index (Jin et al., 2007), AR index (Jin et al., 2007), and h_w index (Egghe & Rousseau, in press). Findings by Bornmann et al. (in press-a) show that using the h index and its proposed variants, either the most productive core of a scientist’s output is identified and the number of papers in the core counted (h index and g index), or the (citation) impact of the papers in the core is quantified (R index and AR index). The cut-off value for including or excluding publications in the most productive core in the different indices results from the characteristics of the scientist’s publication set itself. Thus, whereas the different indexes can be calculated without great difficulty for a scientist, the determination of the cut-off value based on the set itself results in the problem of the comparability of scientists’ index values. Which publications are included in the most productive core as high-performance publications is determined differently in each case.

It is for this reason that we propose the b index, an indicator where the cut-off value for including or excluding publications in the most productive core does not result from the publication set itself but instead is determined using a field-specific reference standard for scientific excellence (the name b index refers to the use of a baseline for calculating the index value). Only comparison with a reference standard of this kind makes it possible to assign meaning to the performance of a scientist or of the performance of the scientist’s (best) publications. The Essential Science Indicators (ESI) from Thomson Scientific (Philadelphia, PA, USA), for example, offer baselines that provide expected citation rates for groups of papers (journal sets) in a specific field. The ESI percentiles table displays data on the minimum number of citations needed to meet the top 50%, 20%, 10%, 1%, 0.1%, and 0.01% of publications within scientific fields. For example, in the “Physics” table, a value of 29 in the 10% column for the year 1997 indicates that the top 10% of papers in physics journals entered into the Science Citation Index (SCI, provided by Thomson Scientific) in that year have been cited at least 29 times.

Percentile rank scores of this kind are used widely as a standard for comparison in educational and psychological testing, in order to judge a person’s test scores (intelligence test scores, for example) based on a comparison with the percentiles of a calibrated sample (see Jackson, 1996). Percentile rank scores usually involve ranking the units, here the papers, in ascending order according to a criterion, here the citation counts (see Rousseau, 2005, for an example of the use of percentiles describing journal impact). Next, the frequencies with which papers with a certain citation count are found are accumulated successively across all papers (papers with citation count 0, 1, 2, …). The percentile rank score amounts to the fraction of the cumulative frequencies of the total number of all papers.

Particularly in bibliometric analysis the use of percentile rank scores for evaluative purposes is very advantageous (see also Plomp, 1990), as (1) no assumptions have to be made as to the distribution of citations; that is, the scores are applicable also for the (usually) skewed distributions of bibliometric data; and (2) in contrast to the use of average citation counts as a cut-off value for including or excluding publications in the most productive core of a scientist, through the use of percentile rank scores the scientist’s papers can be appraised directly as to their excellence (10% of the papers with the highest citation counts, see Tijssen & van Leeuwen, 2006).

The b index tells us the number of papers in the publication set of a scientist that belong to the top 10% of papers in a field (as defined, for example, by an ESI baseline). Holding that “a measure which should indicate the overall quality of a scientist […] should deal with the performance of the top articles” (Egghe, 2006b, p. 8), only those papers by a scientist enter into the calculation of the b index that belong to the top performers in a specific field (for the characteristics of highly cited papers, see Aksnes, 2003). According to Tijssen et al. (2002) these top performance papers in a field (the top 10%) provide a useful analytic framework for identifying “world class” scientific excellence. Based on this b index definition we can say that there is no such thing as a b index (in the same way as there is no Journal Impact Factor that is provided by Thomson Scientific, Philadelphia, PA, USA). Indeed, the b index depends on the year in which it is determined, the period under consideration and the used database (for example, the citation index produced by Thomson Scientific or by Scopus, Elsevier, Amsterdam, The Netherlands, see Bornmann et al., in press-b).

As a reference standard for the calculation of the b index, not only ESI baselines are appropriate; all of the other expected citation rates that have been proposed up to now can also be used for comparison purposes, such as the citation rate of the top 10% of those papers that together with a scientist’s publications appeared in specific journals (for an overview of various reference standards, see Schubert & Braun, 1993, 1996; Van Raan, 2005; Vinkler, 1986). When choosing the reference standards for calculating the index, however, it is important always to make sure that they are appropriate baselines for a scientist’s research, or publication set (see here Bornmann et al., in press-b; Van Leeuwen, 2007).

If the b index is determined via the Web of Science (provided by Thomson Scientific) and using the ESI baselines, the calculation steps are the following: First, the publication set for a scientist is put together using the Web of Science database. Then the field-specific reference standards are chosen using the ESI. The percentiles table in the ESI displays data on the minimum number of citations needed to meet the top 10% of publications within 22 journal sets (for example “Physics” and “Biology & Biochemistry”) over the past 10 years. After selecting the reference values, papers from a specific publication year (of the last 10 years) are selected in the scientist’s publication set using the Web of Science function “refine your results.” The selected papers are sorted according to “times cited,” and the number of publications noted that received at the minimum just as many citations as the minimum number of citations received by the top 10% of papers for that year in the ESI. The procedure is repeated for all publication years. The b index, finally, is defined as the sum of all those publications that in the individual publication years belong to the top 10% of most cited papers. As a mathematical formula the b index is defined as

As examples, we calculated the b index for some highly-cited physicists for whom Hirsch (2005) calculated the m quotient and Ball (2005) the h index (see Table 1). We used the ESI “Physics” baseline as the cut-off value for including or excluding publications in the most productive core of a scientist. Although the physicists published their papers in journals that Thomson Scientific assigns to very different journal sets (e.g., “Physics, Mathematical” or “Physics, Nuclear”), we still used a common, subfield-overlapping ESI baseline as the standard of excellence in calculating the index. As the journal summary lists in Journal Citation Reports (JCR, provided by Thomson Scientific) show for the subject fields of physics, the average citation rates for the individual fields are similar: as per the 2006 JCR Science Edition, the median impact factors for the fields have values between 1.033 (for “Physics, Atomic, Molecular & Chemical”) and 1.690 (for “Physics, Mathematical”).

Table 1. b index, h index, and m quotient for some highly-cited physicists (in descending order according to the b index; calculated on September 18, 2007)

Note. ^*Spearman’s rank-order correlation is 0.46. ⁺Spearman’s rank-order correlation is 0.2.

Table 1 shows the b index, h index, and m quotient for six highly-cited physicists: Manuel Cardona, Marvin L. Cohen, Giorgio Parisi, Cumrun Vafa, Frank Wilczek, and Edward Witten. As the index values in the table reveal, the highest b index, with a value of 52, is found for Marvin L. Cohen. This means that in the last 10 years, Cohen has published 52 papers that belong to the top 10% of publications in physics. All of the physicists in Table 1 have b index values of at least 34. Whereas the h index values of the physicists fluctuate closely around a value of 30 (between 27 and 32), the b index values lie between 34 and 52. The greater range of fluctuation of the b index values as compared to the h index values could indicate that there may be an incremental contribution associated with the b index aiding measurement of the scientific performance of these physicists, in that it allows finer differentiation than the h index when performance is in the excellence range.

As the example of the physicists shows, the b index as a numerical relative for scientific excellence may well be an interesting supplement to the h index. Similar to variants developed thus far for the h index, variants of the b index are, of course, also conceivable: (1) For the calculation of the b index in specific situations (such as when comparing the scientific achievements of junior scientists), it could make sense to use as a baseline not the minimum citation rate for the top 10% of publications but rather the minimum citation rate for the top 20% of publications in a field. Papers in the top 10% of a field are expected for junior scientists only in exceptional cases. Glänzel and Schubert (1992) give three basic criteria for the selection of appropriate cut-off values in order to obtain a reliable indicator. (2) In addition, an index could be calculated that is the average number of citations in the excellent publication core of a scientist that belongs to the top 10% (or 20%) of publications in a field. The proposal to use this average number of citations as a variant of the h index was made by Jin (2006). (3) Finally, the b index could be normalized to the number of years since the first paper in a publication set appeared.

But before the b index (and variants of the b index) is actually used for evaluation and comparison purposes, its validity should be tested thoroughly on the basis of empirical data in various research fields. While the b index may be differently conceptualized than the h index (or differently than the variants of the h index proposed up to now), only comprehensive studies can clarify to what extent its development is in fact associated with an incremental empirical contribution. The studies should investigate, for one, to what extent the b index correlates with bibliometric standard indicators (that is, to what extent the index has convergent validity) and whether it permits evaluation of scientists across disciplines. Because an index for measuring scientific performance can only be a useful yardstick to compare different scientists, if the index is strongly related to peer assessments (see here Cole, 1989), future studies should, for another, examine the relationship between b index values and peer assessments (see here Bornmann & Daniel, 2005, 2007a).

The authors wish to express their gratitude to two anonymous reviewers for their helpful comments.

References

Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12(3): 159-170.

Ball, P. (2005). Index aims for fair ranking of scientists. Nature, 436(7053): 900.

Bornmann, L., Daniel, H.-D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65(3): 391-392.

Bornmann, L., Daniel, H.-D. (2007a). Convergent validation of peer review decisions using the h index: extent of and reasons for type I and type II errors. Journal of Informetrics, 1(3): 204-213.

Bornmann, L., Daniel, H.-D. (2007b). What do we know about the h index? Journal of the American Society for Information Science and Technology, 58(9): 1381-1385.

Bornmann, L., Mutz, R., Daniel, H.-D. (in press-a). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology.

Bornmann, L., Mutz, R., Neuhaus, C., Daniel, H.-D. (in press-b). Use of citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics.

Cole, S. (1989). Citations and the evaluation of individual scientists. Trends in Biochemical Sciences, 14(1): 9-13.

Egghe, L. (2006a). How to improve the h-index? The Scientist, 20(3): 14.

Egghe, L. (2006b). An improvement of the h-index: the g-index. ISSI Newsletter, 2(1): 8-9.

Egghe, L., Rousseau, R. (in press). An h-index weighted by citation impact. Information Processing & Management.

Glänzel, W., Schubert, A. (1992). Some facts and figures on highly cited papers in the sciences, 1981-1985. Scientometrics, 25(3): 373-380.

Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46): 16569-16572.

Jackson, C. (1996). Understanding psychological testing, Leicester, UK, British Psychological Society.

Jin, B. (2006). h-index: an evaluation indicator proposed by scientist. Science Focus, 1(1): 8-9.

Jin, B., Liang, L., Rousseau, R., Egghe, L. (2007). The R- and AR-indices: complementing the h-index. Chinese Science Bulletin, 52(6): 855-863.

Kosmulski, M. (2006), A new Hirsch-type index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3): 4-6.

Liu, Y., Rousseau, R. (2007). Hirsch-type indices and library management: the case of Tongji University Library. In: D. Torres-Salinas, H. F. Moed (Eds.), Proceedings of the 11th Conference of the International Society for Scientometrics and Informetrics. Madrid, Spain, Spanish Research Council (CSIC), pp. 514-522.

Plomp, R. (1990). The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19(3-4): 185-197.

Rousseau, R. (2005). Median and percentile impact factors: a set of new indicators. Scientometrics, 63(3): 431-441.

Schubert, A., Braun, T. (1993). Reference standards for citation based assessments. Scientometrics, 26(1): 21-35.

Schubert, A., Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3): 311-324.

Tijssen, R., van Leeuwen, T. (2006). Centres of research excellence and science indicators. Can 'excellence' be captured in numbers? In: W. Glänzel (Ed.) Ninth International Conference on Science and Technology Indicators. Leuven, Belgium, pp. 146-147.

Tijssen, R., Visser, M., van Leeuwen, T. (2002), Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3): 381-397.

van Leeuwen, T. N. (2007). Modelling of bibliometric approaches and importance of output verification in research performance assessment. Research Evaluation, 16: 93-105.

van Raan, A. F. J. (2005). Measurement of central aspects of scientific research: performance, interdisciplinarity, structure. Measurement, 3(1): 1-19.

Vanclay, J. K. (2007). On the robustness of the h-index. Journal of the American Society for Information Science and Technology, 58(10): 1547-1550.

Vinkler, P. (1986). Evaluation of some methods for the relative assessment of scientific publications. Scientometrics, 10(3-4): 157-177.

Received November 14th 2007
Accepted November 28th 2007

Copyright information  |    |     |