
Cite: Trishchenko N. D., Makeenko M. I., Anisimov I. V. Beyond simple calculations: Lessons from an empirical study on national integration into the global scientific landscape. (Part 1) // Scientific and technical libraries. 2025. No. 7, pp. 63–78. https://doi.org/10.33186/1027-3689-2025-7-63-78


Introduction
    In the 2010s, the Russian research community was assigned a set of state-driven objectives primarily aimed at increasing the international visibility of Russian science [1]. To ensure the achievement of these goals, a series of funding programs was launched, supporting both various research projects and Russian scientific journals, which were required to be included in international indexes Web of Science and Scopus – these indexes were intended to serve as the primary measure of the "visibility" of domestic science.
    At the same time, it is evident that there can be a significant gap between formal indicators such as the number of publications – and even their citation rates – and actual integration, which entails strengthening international connections, increasing the visibility of Russian scientific output, and fostering genuine engagement with it by foreign colleagues. This gap becomes particularly pronounced when state-funded scientific institutions in the country are primarily focused on generating reports that showcase progress in meeting government-imposed target indicators.
    We set ourselves the task of assessing the extent to which Russian research within a particular discipline actually fits into the international scientific discourse. Russian articles on media and communication were chosen as the object, which, of course, was largely due to our belonging to this academic field. This is what allowed us to more accurately assess the relevance of the results at each stage of the research and how well one actually needs to navigate the thematic field and the authors active in it in order to ensure that the results of such studies reflect reality accurately enough. 
    At the same time, the field of media and communications is also of interest because it is at the intersection of humanities and social sciences, i.e. it reflects to some extent the situation in both fields. In addition, since it is a rather young discipline, it has a relatively limited number of researchers, so the total number of publications is much smaller than in many major scientific fields, which allows for more effective investigation with attention not only to the quantitative, but also to the qualitative characteristics of the material considered.
    We also hoped that a single-discipline study would result in a comprehensive methodology that we could expect similar analyses for other disciplines and a high degree of accuracy in the results.
Literature Review
    Issues of publication activity have taken an important place in the lives of researchers, academic institutions, and authorities in many countries. Therefore, it is not surprising that over the past two decades, numerous articles have been published assessing scientometric indicators within specific scientific fields at the national level, including through interdisciplinary and cross-country comparative analyses [2–5], including in the Russian context [6–8].
    At the same time, the question remains to what extent quantitative indicators reflect qualitative changes. Researchers have repeatedly pointed out the “side effects” of a research policy focused on stimulating publication activity: instead of thoughtful development of a research program and immersion in the subject, scientists are forced to produce as many publications as they can; when selecting a new employee, preference is given to those who publish more. As a result, the constant pressure makes researchers increasingly resort to unethical practices: plagiarism, duplicate publications, falsified research results, “salami-slicing”, attributing to articles authors who are not involved in the research, etc. [9]. Another consequence of the attitude of publishing at all costs is a more rigid orientation to the framework of one's discipline and a rejection of interdisciplinary research [10]. Thus, formal indicators that can be tracked by bibliometric databases are increasing, but the quality of scientific production and, consequently, the interest in it, may even decrease.
    One of the first papers devoted to the problem of the impact of the formal system of productivity assessment on academia shows, using the example of Australian scientists, that the first reaction to a sharp “change of course” is publication in low-ranked journals with a corresponding decrease in the number of citations [11]. The issues of predatory publishing [12], and poor quality of university journals are also related to the formal evaluation system, and Russia is not spared from these problems either [13, 14].
    Since the introduction of bibliometric databases, researchers have pointed out the need to be extremely careful and cautious about indicators and to look critically at the original data sources rather than aggregated indicators [15]. As a solution, some countries and individual organizations focus on another indicator of scientific activity – article citations. This presumably more accurately reflects the “quality” of scientific articles, which is often considered in terms of impact and value.
    However, this is far from a perfect metric – the questionable accuracy of citation indicators is linked to several factors. Often, the citation of a particular work has little to do with the actual objectives of the citing research. As a result, relying solely on quantitative indicators provides a rather weak understanding of a given author’s true scientific contribution [16, 17]. 
    Nevertheless, the use of citation data is at the heart of many studies designed to determine the place of different countries in the research hierarchy [18–20]. At the same time, even small changes in the assessment methodology can lead to significant differences in conclusions, which mainly serves the diversity of ratings and approaches to reporting.
    International collaborations — the number of which typically increases sharply among authors from countries that promote publication activity — on the one hand, significantly complicate the calculation of metrics, and on the other, contribute to more active citation of publications [21].
    Thus, the number of published articles and the citations they receive often fail to reflect the actual situation. Of course, in recent years, more sophisticated approaches have been actively developed. These include, for example, the construction of composite indices based on multidimensional analysis (such as an openness index combining publication data, international collaborations, and researcher mobility statistics using principal component analysis [22]), the decomposition of the impact of international collaboration on bibliometric indicators by identifying categories (domestic, bilateral, and multilateral collaborations), and the calculation of specific indicators such as the Collaboration Category Normalized Citation Impact (Collab-CNCI) 
[23, 24]. Network analysis methods are also employed to study scientific collaboration, including the construction of adjacency matrices, network density analysis, clustering, and the identification of hierarchies within the scientific community [25, 26]. Some studies apply multivariate statistical methods and clustering techniques to investigate structural changes and trends in international cooperation [27, 28].
    At the same time, advanced bibliometric methods (indices, PCA, Collab-CNCI, network analysis) were primarily developed to work with large volumes of publications, where stable patterns and trends can be identified. In the case of smaller disciplines (and accordingly, smaller samples), the results become unstable, and the proportion of rare events becomes disproportionately high, distorting the overall picture. Moreover, in small samples, qualitative aspects (such as where and by whom the article was published, or which concepts are being adopted) become more important than aggregated metrics.
       
    Therefore, in our case, to ensure the accuracy of the research results — at least to the extent reasonably possible – it is necessary, on the one hand, to incorporate additional data and characteristics, and on the other, to differentiate the indicators of publication count and citation impact by additional categories.
Methodology development: challenges and solutions
    Due to all the described complexities associated with the application of scientometric approaches to the assessment of science, we needed to address a number of methodological and methodological questions that arose both during the creation of the study design and already at the stage of processing the collected data.
    The main source of data in our study was the international scientific citation index Web of Science. We will not discuss how justified and representative of the national science output the use of Web of Science data is – many articles have been written about the shortcomings of using this scientometric system for these purposes [29, 30]. We are ready to accept the problem of underrepresentation of publications made at the regional level: in our case, we considered that additional involvement of regional databases (in particular, the Russian Index of Science Citation or RISC and the joint RISC and WoS database called Russian Science Citation Index) is unlikely to provide us with information about international integration. At least, we would be able to suspect that this representation is actually wrong if the foreign citation rates for Russian articles in ESCI were high enough (but this did not happen). Therefore, we focused exclusively on data from the WoS Core Collection.
    Since we were interested in the results of implementing measures aimed at increasing the visibility of domestic research at the international level, the data were taken from the period 2017–2021. The choice of this particular period is justified by the fact that by 2017, five years had passed since the "May Decrees" of 2012, which set international integration as a priority for the state science policy [2]. This gave university staff and administration enough time to adapt to the new conditions and align their activities accordingly. Additionally, it was from 2017–2018 onwards that domestic journals on media and communication began to be indexed in international bibliometric databases.
    We were interested in publications by domestic researchers, so the basic selection criterion was the value “Russia” in the column with the country of affiliation of the authors.
    During the course of the work, we encountered a number of key questions, the resolution of which, as we believe, we managed fairly successfully, as well as additional questions that were generally less critical and could possibly be addressed in a more effective way.
    The key questions were to understand what is, within the scope of our study:
    international integration, 
    article in media studies, 
    international-level journal, 
    Russian author,
    and Russian article.
    Additional methodological tasks included:
    chronological clustering of the authors of cited sources;
    classification of collaborator affiliation countries into clusters appropriate to the research objectives.
    In addition, an important methodological issue worth discussing is the difference between how "leading" national scientific organizations are defined domestically and internationally, and we deemed it necessary to address this point in the present article as well.
    Let us now consider each question, the approaches to its resolution, and the results of applying the approach we selected separately.
Key questions
What is an ‘international integration’?
    The first and most fundamental methodological question was what indicators could characterize the international integration of national science. It was obvious that a comprehensive methodology was needed that went beyond counting the number of articles and citations and categorizing them into quartiles.
    From our perspective, three main aspects or levels can be identified for assessing the degree of international integration.
    The first level is directly related to publications in international journals, that is, the very publication activity on which perceptions of the degree of integration of Russian science into the international scientific process are mainly based. This is the least detailed and substantive part of our study, but its results allow for a rough understanding of the components that make up the publication activity indicators.
    The second level is the citation of Russian authors' publications by foreign colleagues, which more fully and convincingly characterizes the degree of integration of Russian research into international science. At this stage, we turned to the analysis of citation statistics based on a special report generated by WoS. Additionally, we also attempted to identify which characteristics of the articles correlate with the frequency of their citation. For this purpose, a second separate database was created within the main one, using data only from those articles that had been cited at least once by foreign authors.
    The third level is the use of foreign concepts, theories, and current research findings by Russian scientists. We examined this aspect through the analysis of reference lists, which imposed certain limitations on the results: as mentioned earlier, studies show that authors often cite certain sources without necessity. Although foreign concepts are mentioned, this could be more of a formality than a genuine integration of these concepts or theories into domestic research. In fact, the author may only be familiar with the abstract but still cites the material, and so on.
    To refine the results at all levels, we added a comparison of the statistics for articles from the entire database (or “main” database) and only from the pool of "classic" journals (the composition of the "classic" database will be discussed below – see the section “What is ‘international-level journal?’)”.
    To clearly demonstrate the importance of a comprehensive approach, we will provide a table for each section of the research report.
    Table 1 contains the main statistical data on articles and citations of Russian authors, which, in general, show a steady growth in the number of articles and a fairly high citation rate. However, the addition of even one parameter – the number of articles cited at least once – raises doubts about the homogeneity of the considered set of articles.
Table 1
Dynamics of the number of publications by year (2017–2021)
Year
Main database (n = 2011)

Number of articles
Citations
Articles
with citations n > 1
2017
281
289
98
2018
362
316
106
2019
448
329
124
2020
474
341
118
2021
446
123
63
Total
2011
1398
509
    
    If we proceed further (to the second level of analysis) and divide the citing sources into Russian and foreign (Table 2), the difference becomes even more impressive – in fact, out of 2011 articles from the database, only 214 were cited by foreign authors at least once. And the articles written by Russian authors without any interaction with foreign authors turn out to be only 145, which fundamentally changes the whole picture.

Table 2
Summary data on articles by authors from different groups: number of authors, number of articles, citation rates by foreign authors (main base)
Author group
Total articles
Total 
citations
Average 
number of citations per article
Av. number of citations 
including Russian citations
Russian authors
145
260
1,8
2,8
Collaborations
36
269
7,5
8,6
Foreign authors
33
178
5,4
5,4
Total
214
707
3,3
4,1
           
    For more details on the composition of author groups, see the section “What is ‘Russian author’?”
           
    As a result, we came to the conclusion that we are dealing with, 
in fact, two completely different clusters of articles, although all of them constitute national output. And this is manifested not only in the authority of journals or the visibility of articles for foreign colleagues, but also 
in their internal content. As can be seen from Table 3 (this is the third level of the study), the authors of articles from different clusters focus 
on a significantly different set of literature – note that if we exclude “classic journals” from the entire database, this difference becomes even more evident (here we should also take into account that the reference lists in “classic journals” are much longer).
Table 3 
Distribution of the number of sources cited by authors of Russian scientific articles and their citations by authors' affiliation countries (cited at least five times
Country 
of affiliation 
of the author
Number of cited authors
Number of citations

Main database
Classic database
Main database
Classic database

numb.
%
numb.
%
numb.
%
numb.
%
Foreign countries
666
65
128
91
6519
55
940
89
Russia
350
34
12
9
5136
43
110
10
Russia + foreign country
5
0
1
1
102
1
6
1
Total
12
100
141
100
11 827
100
1056
100
    The table was produced on the basis of data from the study of reference lists in publications from the main database.
    Thus, only an integrated approach, involving consideration of different aspects, allows us to get a real impression of what is actually happening.
What is ‘article in media studies’?
    The first question we had at the base formation stage was to determine what we would consider to be an article on media and communication. Often when considering individual disciplines, authors limit themselves to the WoS subject category without any additional adjustments. Thus, Bradford's law [31] is often neglected, although, as our experience shows, it should not be: working with journal subject categories requires a combination of automatic and manual approaches to data processing. Perhaps in some cases the solution would be to use the Research areas field, but, firstly, ‘communication’ is not declared in it, and secondly, a significant scatter in areas is still inevitable, especially in the case of areas with a lot of interdisciplinary research. For example, in our case, the set of Research areas contained Social Issues, History, Linguistics, Philosophy, Psychology, etc.
    Two search queries were used to form the main base: 
    “media or journalism or communication” under the ‘Communication’ WoS category (resulting in 881 publications);
    “media or journalism or "mass communication"” within 28 WoS categories of the socio-humanitarian cluster (resulting in 2140 publications).
    The two databases were then merged, then all duplicate articles were removed from the resulting set.
    The next stage was the manual screening of the database, which also resulted in the removal of articles that did not correspond to our chosen subject area (keywords were mentioned in the abstract but were not related to the content of the article, the articles dealt with interpersonal communication, technical aspects of telecommunications, etc.). The final database after removal of duplicates and manual screening of articles by subject area amounted to 2011 publications, including both journal articles and book chapters, while the initial set contained more than 3 thousand articles and less than 900 articles within the subject category. 
Thus, we observed an almost perfect realization of Bradford's law: approximately one third of the articles were present in the relevant category, and the remaining two thirds were scattered across journals in many other categories.
    In this case, only manual checking of the sample could provide us with the required accuracy of the final data. In the future, we can consider the possibility of using artificial intelligence technologies to solve this problem, but at this stage this issue seems challenging and deserves special attention.
What is ‘international-level journal’?
    The above tables already mentioned a “classic” database, which we considered necessary to form in order to better separate the indicators characterizing the presence in WoS and the real international integration.
    The thing is that when we took a closer look at the journals from the main database, we realized that it contains publications of very different kinds, which can probably greatly blur the results of our study. In particular, 686 articles in our database (a little more than 34%) are from four Russian journals: “Theoretical and practical issues of journalism” 
(228 articles), “Vestnik Moskovskogo universiteta. Seriya 10. Zhurnalistika” (172 articles), ‘Media Education’ (187 articles), and ‘Nauchniy dialog’ (99 articles). 
    In addition, there is a set of articles published in formally foreign, but actually affiliated with Russia journals (e.g., registered in Turkey, Cyprus, Colombia, but surprisingly publishing articles by predominantly Russian scientists) or published in the post-Soviet states. 
    To correct the described problems and to better understand the situation, a second database with articles from journals that we conditionally labeled as “classic” was formed. All publications mentioned in the previous paragraph (Total 34 articles) and those indexed in the Emerging Sources Citation Index (ESCI) were excluded – the chosen name for the database is primarily linked to the use of traditional indexes as a point of reference.
    As a result, the following publications made up the base of “classic” articles:
    articles from foreign journals that are categorized in any quartile (i.e., included in SSCI, AHCI indices) – 193 in total;
    chapters from books (BKCI-SSH) – 28 in total. 
    Thus, the “classic” database included 221 publications. For simplicity, in the future we will consider the word “publication” and “article” to be synonyms, including book chapters as well, since the type of publication is not crucial in our study. It should be noted that articles in formally foreign journals with impact factor, which we did not include in the “classic” database, would have accounted for 15% of publications.
    As can be seen from Image 1, during the selected period we can observe a progressive growth in the number of publications, including on the “classic” database, but still their number is not comparable with the total set.

Dynamics of the number of publications by year (2017–2021)
    The indicators of the “classic” database characterize the situation much more reliably than those of the main database (remember, we are talking about international integration), but in the following stages we continued to clarify the meaning of these statistics.

References
1. Pravitel'stvo RF. Ukaz prezidenta RF № 599 “O merah po realizacii gosudarstvennoj politiki v oblasti obrazovanija i nauki”. 2012. 
URL: http://kremlin.ru/events/president/news/15236.
2. Zhang L., Shang Y., Huang Y., Sivertsen G. Toward internationalization: A bibliometric analysis of the social sciences in Mainland China from 1979 to 2018 // Quantitative Science Studies. 2021. Vol. 2. No 1. P. 376–408. DOI 10.1162/qss_a_00102.
3. Shaposhnik S. B. Mezhdunarodnoe nauchnoe sotrudnichestvo i publikatsionnaya aktivnost' rossiyskikh uchenykh v Computer Science v 1993–2017 godakh: mezhdistsiplinarnyy i mezhstranovoy sravnitel'nyy analiz // Informatsionnoe obshchestvo. 2021. No 6. P. 39–45. URL: http://infosoc.iis.ru/article/view/153.
4. Kademani B., Sagar A., Kumar V., Gupta B. Mapping of Indian Publications in S&T: 
A Scientometric Analysis of Publications in Science Citation Index // DESIDOC Journal of Library & Information Technology. 2007. Vol. 27. No 1. P. 17–34. 
DOI 10.14429/djlit.27.1.120.
5. Cano V. Bibliometric overview of Library and Information Science Research in Spain // Journal of the American Society for Information Science. 1999. Vol. 50. No 8. P. 675–680.
6. Kirchik O. I. “Nezametnaya” nauka: patterny internatsionalizatsii rossiyskikh nauchnykh publikatsiy // Forsayt. 2011. No 5 (3). P. 34–42.
7. Murav'ev A. A. O rossiyskoy ekonomicheskoy nauke skvoz' prizmu publikatsiy rossiyskikh uchenykh v otechestvennykh i zarubezhnykh zhurnalakh za 2000–2009 gg. // Ekonomicheskiy zhurnal Vysshey shkoly ekonomiki. 2011. Vol. 15. No 2. P. 237–264.
8. Mikhaylov O. V. Tsitiruemost' i bibliometricheskie pokazateli rossiyskikh uchenykh i nauchnykh zhurnalov // Problemy deyatel'nosti uchenogo i nauchnykh kollektivov. 2017. 
No 3 (33). P. 152–170.
9. Rawat S., Meena S. Publish or perish: Where are we heading? // Journal of Research in Medical Sciences. 2014. Vol. 19. No 2. P. 87–9. 
URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3999612/.
10. Van Dalen H. P., Henkens K. Intended and unintended consequences of a publish-or-perish culture: A worldwide survey // Journal of the American Society for Information Science and Technology. 2012. Vol. 7. No 63. DOI 10.1002/asi.22636.
11. Butler L. Modifying Publication Practices in Response to Funding Formulas // Research Evaluation. 2003. Vol. 1. No 12. Pp. 39–46. DOI 10.3152/147154403781776780.
12. Beall J. Predatory publishers are corrupting open access // Nature. 2012. No 489. P. 179. DOI 10.1038/489179a.
13. Guba K. Resursnaya zavisimost' nauchnykh zhurnalov: avtorskie vs chitatel'skie zhurnaly // E'konomicheskaya sotsiologiya. 2018. No 4. P. 73–100. DOI 10.17323/1726-3247-2018-4-73-100.
14. Sokolov M. Can the Russian Research Policy be Called Neo-Liberal? A Study in the Comparative Sociology of Quantification // Europe-Asia Studies. 2021. Vol. 6. No 73. 
Pp. 989–1009. DOI 10.1080/09668136.2021.1902945.
15. Narin F., Stevens K., Anderson J., Collins P., Irvine J., Isard Ph., Martin B. On-line approaches to measuring national scientific output: a cautionary tale // Science and Public Policy. Vol. 15. No 3. P. 153–161. DOI 10.1093/spp/15.3.153.
16. Chubin D. E., Moitra S. D. Content Analysis of References: Adjunct or Alternative to Citation Counting? // Social Studies of Science. 1975. Vol. 5. No 4. P. 423–441. 
DOI 10.1177/030631277500500403.
17. Moravcsik M., Murugesan J. P. Some Results on the Function and Quality of Citations // Social Studies of Science. 1975. Vol. 5. No 1. P. 86–92.
18. Li N. Evolutionary patterns of national disciplinary profiles in research: 1996–2015 // Scientometrics. 2017. Vol. 111. No 1. P. 493–520. DOI 10.1007/s11192-017-2259-4.
19.  Patelli A., Cimini G., Pugliese E., Gabrielli A. The scientific influence of nations on global scientific and technological development // Journal of Informetrics. 2017. Vol. 11. No 4. 
P. 1229–1237. DOI 10.1016/j.joi.2017.10.005.
20. Sandström U., Van den Besselaar P. Funding, evaluation, and the performance of national research systems // Journal of Informetrics. 2018. Vol. 12. No 1. P 365–384. 
DOI 10.1016/j.joi.2018.01.007.
21. Lancho Barrantes B. S., Guerrero Bote V. P., Rodríguez Z. C., de Moya Anegón F. Citation flows in the zones of influence of scientific collaborations // Journal of the American Society for Information Science. 2012. Vol. 63. P. 481–489. DOI 10.1002/asi.21682.
22. Wagner C. S, Whetsell T., Baas J., Jonkers K. Openness and Impact of Leading Scientific Countries // Frontiers in Research Metrics and Analytics. 2018. Vol. 3. No 10. 
DOI 10.3389/frma.2018.00010.
23. Adams J., Szomszor M. National research impact is driven by global collaboration, not rising performance // Scientometrics. 2024. No 129. P. 2883–2896. https://doi.org/10.1007/s11192-024-05010-6.
24. Adams J., Szomszor M. A converging global research system // Quantitative Science Studies. 2022. Vol. 3. No 3. P. 715–731. DOI 10.1162/qss_a_00208.
25. Wagner C. S., Leydesdorff L. Mapping the network of global science: comparing international co-authorships from 1990 to 2000 // International Journal of Technology 
and Globalisation. 2005. Vol. 1. No 2. P. 185–208.
26. Wagner C. S., Park H. W., Leydesdorff L. The Continuing Growth of Global Cooperation Networks in Research: A Conundrum for National Governments // PLoS ONE. 2015. Vol. 10. No 7. DOI 10.1371/journal.pone.0131816.
27. Okamura K. A half-century of global collaboration in science and the “Shrinking World”. Quantitative Science Studies. 2023. Vol. 4. No 4. P. 938–959. DOI 10.1162/qss_a_00268.
28. Coccia M., Wang L. Evolution and convergence of the patterns of international scientific collaboration // Proceedings of the National Academy of Sciences of the United States of America. 2016. Vol. 113.  No 8. P. 2057–2061, DOI 10.1073/pnas.1510820113.
29. Vera-Baceta M., Thelwall M., Kayvan K. Web of Science and Scopus Language Coverage // Scientometrics. 2019. Vol. 121. P 1803–1813. DOI 10.1007/s11192-019-03264-z.
30. Marginson S., Xu X. Hegemony and Inequality in Science: Problems of the Center-Periphery Model // Comparative Education Review. 2023. Vol. 67. No 1. DOI 10.1086/722760.
31. Bradford S. C. Sources of information on specific subjects // Engineering. 1934. Vol. 137. P. 85–6.

Authors
Natalia D. Trishchenko – Cand. Sc. (Philology), Senior Researcher, New Media and Communication Theory Chair, Journalism Department, Lomonosov Moscow State University, Moscow, Russian Federation
trishchenko.nataliia@yandex.ru
Mikhail I. Makeenko – Cand. Sc. (Philology), Associate Professor, Mass Media Theory and Economics Chair, Journalism Department, Lomonosov Moscow State University, Moscow, Russian Federation
makeenko.mikhail@smi.msu.ru
Igor V. Anisimov – Cand. Sc. (Philology), Senior Lecturer, Photojournalism and Media Technologies Chair, Journalism Department, Lomonosov Moscow State University, Moscow, Russian Federation
igor.anisimov@gmail.com


БИБЛИОГРАФИЧЕСКОЕ ОБСЛУЖИВАНИЕ
УДК 025.5;004.65:011/016
https://doi.org/10.33186/1027-3689-2025-7-79-101
Реализация персонального библиографического менеджера 
средствами клиентских веб-технологий
И. С. Орещенков
Интернет-портал «Банки РБ», Молодечно, Республика Беларусь,
iharsw@yandex.by

