Why Research Methods Matter

Some research methods are frequently used, while others are reserved for particular problems. However, frequency tells only part of the story. A method may be popular but limited to a small number of topics, while another may be applied across many research areas.

The study therefore examined two dimensions: usage frequency, indicating how often a method was used, and application variety, indicating how broadly it was applied across different topics. Together, these dimensions provide a fuller picture of methodological development in LIS.

How the Study Was Conducted

Manually reading and classifying nearly 27,000 papers would be extremely time-consuming. The authors therefore developed an automatic classification approach.

They began with 1,977 manually coded articles and classified research methods into 16 categories, including content analysis, bibliometrics, experiments, theoretical approaches, questionnaires, interviews, ethnography, and webometrics. Because some methods had few training examples, the labeled dataset was expanded, and confident learning was used to identify potentially incorrect labels.

Several machine-learning and deep-learning models were compared. The best-performing model combined SciBERT, which was trained on scientific language, with TextCNN. It achieved Micro-F1, Macro-F1, and Samples-F1 scores of approximately 0.85. A manual examination of automatically classified papers also showed substantial agreement between machine predictions and human judgments.

Top2Vec was subsequently used to identify 18 major LIS research topics. These topics were then connected with the classified research methods.

Figure 1. Automatic classification of research methods.

What the Study Found

The first major finding was a shift from conceptual toward empirical research. The use of theoretical approaches declined, while interviews and questionnaires became increasingly common. LIS scholars are therefore placing greater emphasis on evidence collected from users, organizations, documents, and information systems.

Content analysis was the most frequently identified method, followed by bibliometrics, experiments, theoretical approaches, and questionnaires. Together, these five methods accounted for more than three-quarters of all method applications.

The second finding was a movement from system-centered toward user-centered research. Topics such as information retrieval systems, metadata, collections, and traditional information organization generally declined in relative share. In contrast, information services, health information, open-access publishing, text processing, and webometrics increased. These changes indicate growing attention to users, communities, and the contexts in which information is accessed and used.

The third finding was that frequently used methods were not necessarily equally versatile. Content analysis had the highest application-variety score, showing that it could be used to investigate a particularly wide range of LIS topics. Bibliometrics, interviews, experiments, and theoretical approaches also demonstrated broad applicability, whereas the Delphi method and research diaries were used more narrowly.

Finally, the study identified distinctive connections between research topics and methods. Bibliometrics was particularly common in science-evaluation research, while experiments were widely used in information retrieval, text processing, and information-seeking studies. Theoretical approaches were closely associated with information ethics and the philosophy of information. Content analysis and interviews were applied across many different topics.

The authors also created an interactive map showing how the relationships between 16 methods and 18 topics changed annually from 1991 to 2021:

https://chengzhizhang.github.io/research/research_methods/research_topic_method.html

What This Means for the Future

The findings can help researchers select methods that better match their research questions. They can also help LIS educators update research-methods courses by identifying which methodological skills are becoming increasingly important.

The study further demonstrates how machine learning can expand the scale of methodological analysis. Previous studies based on manual coding commonly examined selected years or relatively small collections of papers. Automatic classification makes it possible to investigate continuous changes across several decades.

Practical Implications

Machine learning is not intended to replace human methodological judgment. Human experts are still needed to define method categories, interpret ambiguous descriptions, and recognize emerging approaches. Automated techniques can nevertheless complement human analysis by processing large collections of literature more efficiently.

The approach also has limitations. Because full texts were not consistently available, research methods were mainly identified from abstracts. Broad categories such as content analysis and “other methods” were more difficult for the model to recognize. Future research could therefore use full-text data and develop a more detailed classification scheme.

In conclusion, LIS became increasingly empirical and user-centered between 1991 and 2021. By examining both how often methods were used and how widely they were applied, the study provides a more complete understanding of how LIS researchers investigate problems and how the discipline’s methodological toolbox continues to evolve.

For more on this, see:

Zhang, C., Tian, L., & Chu, H. (2023). Usage frequency and application variety of research methods in library and information science: Continuous investigation from 1991 to 2021. Information Processing & Management, 60(6), 103507. https://doi.org/10.1016/j.ipm.2023.103507