Measuring quality in science
DDS, PhD, Prof hc, Professor
Using a reproducible and valid parameter is the base of every evaluation. Quality control means evaluation, which is divided into evaluation of the process quality and evaluation of the outcome quality. Process quality means that the manufacturer is required to define all processes that yield the product. The CE certificate of a company means that the company has proven that it meticulously sticks to the processes they have defined themselves. FDA basically does the same, when it comes to medical products. The outcome quality makes sure that the product meets its specifications and serves the intended purpose without showing negative, not tolerable side effects. So far so good for manufacturers. When it comes to dental products, of course the dentist has a big influence on the outcome and we should not forget his influence on the patient’s behavior and life style.
You can consider publications or journals as products as well. So having some sort of quality control seems reasonable. The process control in publication is for sure a very stringent editorial process and peer review. With this, as done with the Stomatology Education Journal, the reader may know that the paper is sound and that they may be able to find out what was done, how it was done and how valid it is, so they can make up their own mind. But how to evaluate the quality of a journal? Eugene Garfield, the founder of the Institute for Scientific Information (ISI) came up with the idea to look at citations from a journal to evaluate its quality. The idea is, the more citations, the better the quality of the journal. The outcome is the impact factor (IF), which is calculated for every given year as the number of citations, received in that year of articles published in that journal during the two preceding years, divided by the number of articles published in that journal during the last two years. At first glance it seems OK and logical, however, the IF has become very problematic over time, for many reasons. Since it looks at a mean of citations, one article may generate a lot of citations, while other articles may get very little or even none. So much about validity. I used to say “create a scientific scandal with your publication and you get a high IF, though the published thing may not be true at all”. The second reason why it is problematic is that you cannot compare journals just based on the impact factor alone, since the number of citations is highly dependent on the size of the scientific community in the given specialty. This reduces the IF dramatically for journals reporting science of very small specialties such as ophthalmology or dentistry. When I was a member of the scientific committee of the Charité in Berlin, we rated journals according to their rank in their respective specialty in order to have comparable factors within the medical sciences. This leads to the third reason why it has become problematic. A new journal depends on the ISI to be listed to enter evaluation at all. Furthermore the IF is used by many Universities, against the advice of its inventor to evaluate the scientific output of individuals for their carrier or departments for the allocation of funds. The very negative effect is that it gets extremely difficult for a new journal to obtain good manuscripts. The most productive individuals in the scientific community are usually young scientists working on their academic carrier. Therefore they will try to publish in journal with the highest possible IF. Since IF is used to allocate funds as well, experienced scientists also are usually not willing to submit a manuscript to a non IF journal, because it may hurt financially.
Finally the most critical point about IF is that it can be manipulated by Editors or Publishers. Here are some examples: Editors may choose to publish more reviews, hoping that they get more citations. Editors may decrease the number of published citable manuscripts, thus decreasing the number in the denominator of the equation to calculate the IF and thus increasing the IF. Furthermore papers with a higher probability to be quoted are published early in the year, because they have more time to generate citations.
There are some more methods to increase the IF, which I consider unethical. One is that the editor actively promotes some papers from his journal as interesting and citable to his/her peers. Another step is that some editors tend to “help” authors to improve the article and have it published with the Editor ending on the author line in the hope to increase the IF of the journal as well the personal cumulative IF of the Editor. Finally some Editors and/or publishers practice something that is called “coercive citation”. At the end of the review process they confront the authors with a list of papers that have been published in “their” journal and require the authors to add them to their reference list as a prerequisite for publication!
The newest thing that has appeared is fake impact factor, used by so called predatory publishers. They create an IF with publications that are not listed with ISI. Therefore, dear readers, think twice when you look at an IF! It is obvious that the IF is NOT a number indicating the quality of a paper. It is much better to look at the content to make your judgement!