The Comparative Uncertainty Criterion: A New Approach to Assessing Model  Accuracy

Boris Menin

The scientific community is currently grappling with a reproducibility crisis, marked by an alarming increase in the number of published articles that have been retracted due to errors or fraudulent practices. This crisis can be attributed to various factors, including the intense pressure to publish, insufficient training in research methods, and flaws in the peer review process.

Equally interesting and important in the flood of falsified papers, however, are seemingly unimportant papers that simply point out curious observations or lay out a few important logical steps, and thereby contribute little to the dense undergrowth of science. Each explorer knows similar gems from his field.

One of the main challenges in addressing the reproducibility crisis is the lack of a universally applicable criterion for assessing model-phenomenon mismatch. Currently, there is no single way to measure the accuracy of a model, and different disciplines use different methods. This makes it difficult to compare results across disciplines and to identify potential problems with a particular model.

In this article, I propose the comparative uncertainty criterion as a new approach to quantifying model uncertainty. The comparative uncertainty criterion is grounded in the informational approach and adheres to the International System of Units (SI). The criterion can be used to quantitatively assess the model’s proximity to the object of study and to detect subtle deviations from widely accepted principles.

The comparative certainty criterion is based on the following two principles:
The principle of information transmission: A model is an information channel that transforms information about the object of study into a set of predictions.
The principle of uncertainty: The uncertainty of a model is determined by the amount of information that is lost during the transformation process.

The information quantity of a model is a measure of its uncertainty. A system with an amount of information close to optimal is considered more certain, while a system with high difference is more uncertain.

The comparative uncertainty criterion can be used to assess the accuracy of a model in a number of ways. For example, the criterion can be used to compare the accuracy of different models of the same phenomenon. The criterion can also be used to track the accuracy of a model over time.

The comparative uncertainty criterion is a valuable tool for advancing scientific rigor. The criterion can help to ensure that scientific investigations are more reproducible and reliable, and it can help to identify potential problems with existing models.

A comparative uncertainty analysis was performed on scientific and technical works, related to the investigation of “underwater electric discharge,” comparing the obtained comparative uncertainty of the model (εmod) with the theoretically justified uncertainty (εopt). When the uncertainties exhibit close values (εmod/εopt→1), it confirms the reliability and utility of utilizing either model to describe the studied process. Conversely, a substantial disparity between these uncertainties (εmod/εopt<<1) signifies a significant risk associated with the application of a specific model. A total of 800 articles were reviewed in this analysis, selected based on four simultaneous criteria to ensure a comprehensive evaluation. These criteria were applied to assess the legitimacy and practical feasibility of the presented ideas related to “underwater electrical discharge” and to facilitate comparisons with other models to identify the most suitable model for describing the process. Except for only one article (!), none of the other examined studies thoroughly explain the calculation of relative uncertainty in their experiments which can be renormalized to comparative uncertainty.