PakMediNet Discussion Forum : Pakistani Medical Journals : Do we review literature properly in “Reviews”?
Do we review literature properly in “Reviews”?
Explosion in the number of papers published in medical journals makes it difficult to keep pace with the primary research. Experts synthesize this accumulating knowledge in a summary for the benefit of a busy clinician. We find these reviews in all medical journals published in Pakistan.
This traditional method of reviewing literature, commonly known as narrative review, has several disadvantages. One obvious problem is that, reviewers rarely began with an open mind and review can be biased by their professional opinions. Further, reviewers may include only those studies that agree with their own opinions and may completely ignore studies that have reached to a different opinion.
A better way of reviewing medical literature is to develop a search strategy in order to identify all the relevant clinical trials and systematically review all these trials. Such a review is generally known as a ‘Systematic Review’, and is very hard to find in Pakistani medical journals. This approach should eliminate the bias resulting from selective inclusion of studies. Although a systematic review is better than a narrative review, it also has one major limitation. A systematic review is, generally, unable to reach to a conclusion without ignoring sample size, effect size, and research design of the clinical trial.
The limitation of systematic review can be dealt by statistically combining results of all relevant clinical trials. This method of reviewing literature is known as ‘Meta-analysis’ and is never found in Pakistani journals. Such a review not only evaluates all the available literature on a particular topic, but also provides a summary estimate of the effect size after taking into consideration sample size, effect size and study design.
I would encourage anyone who is planning to write a review to consider doing a meta-analysis as it is superior to other two types of reviews.
Posted by: rqayyumPosts: 199 :: 20-11-2005 :: | Reply to this Message
Re: Do we review literature properly in “Reviews”?
Very rightly said, rqayyum. I think, writing a meta-analysis is really a quite hard job, esp for those clinicians who are already scared of biostatistics. I want to know is there any easier way to do it?
2ndly, systematic-review is also difficult in a sense that most of the time, you dont find enough literature on it. For example, if you search any on PakMediNet, the results doesnt mean that they are the conclusion of all medical journals of Pakistan. Unfortunately, still some of the journals are either not indexed in PakMediNet or not cooperating at all. If you take example of PubMed, some of the journals journals from third world, are not indexed at all.
Posted by: yasirPosts: 90 :: 21-11-2005 :: | Reply to this Message
Re: Re: Do we review literature properly in “Reviews”?
As far as pure statistical work is concerned, there is a freely available software program for meta-analysis "RevMan" from Cochrane Collaboration. Anyone can download it. It is user-friendly, and deals with main types of data input. It has limitations, for example, it can not calculate an effect size if only mean and p-value/95% CI are given. One has to calculate standard error/deviation. These are not very difficult or labour-intensive calculations. Ofcourse, like in anyother software, if data is incorrect, results will be incorrect too - "Garbage in Garbage out".
There is a lot of discussion in literature about what is an adequate search strategy. Some argue that if you identify every published article, it completes your search. Others also want to include abstracts presented at conferences. Still others want to include thesis submitted for doctrate/fellowship degrees. To further complicate the problem, some advocate to include unpublished studies, as studies with negative results are unlikely to be published. This is theoratical, in practice there are many limitatiosn to meet these ideals. PubMed indexes only 33% of the medical journals. EMBASE is better, about 60%. But still there are more than 30% journals that are not indexed by these two. None of these services index conference abstracts or thesis. So, practically speaking what could be an adequate search strategy? I think, it depends on the question being asked. Lets say, if you are asking whether Statin-use lowers the risk of CAD, stroke, peripheral arterial disease, abdominal arotic anuerysm or increases the risk of cancer, a PubMed search complemented by hand-search of the references of retrieved articles will suffice. However, if question is relatively less well-explored, it might not be sufficient. Actually, there is an additional advantage of meta-analaysis, one can statistically evaluate for "publications bias". Although, there are no robust tests for evaluation of publication bias, it still gives a good indication of an adequate search strategy.
Posted by: rqayyumPosts: 199 :: 21-11-2005 :: | Reply to this Message
Limitations of Meta-analysis
Having said that meta-analysis is better than systematic review which is better than narrative reivew, I should also discuss major limitations of meta-analysis (Nothing is perfect in this mortal world).
First and the foremost, meta-analysis can't be done on every topic. For example, meta-analysis can't substitute for a review paper on a hypothesis.
Second, meta-analysis is an observational study of clinical trials and, therefore, has limitations similar to any other observational study.
Third, a meta-analysis is as good as its component studies. If studies have biases or flaws, meta-analysis will also be flawed.
Fourth, publication bias is always a real possibility in every meta-analysis. Similarly, inclusion and exclusion criteria for trials, and methods for meta-analysis may produce baised results.
Fifth, trials in a meta-analysis may differ in the composition of study population, thus making it difficult to generalize the results.
Having said that, meta-analysis is still a better way of aggregating and reviewing results of clinical trials.
"We can never arrive at infallible truth because, firstly, that is simply not the nature of scientific knowledge and, secondly, in clinical medicine we are dealing with complex, ever changing units of analysis that are people with illnesses." (TOSHI FURUKAWA)
Posted by: rqayyumPosts: 199 :: 05-12-2005 :: | Reply to this Message
Re: Limitations of Meta-analysis
1. Do we review literature properly? The short answer is NO. To review an article, one has to understand the 'Materials and Methods' part of the article and that is where the flaw is.
2. Is most of the research honest? The answer is No. Most of the research is funded by third parties and the authors, in order to maintain the funding source, always manipulate the results.
3. But there is some hope. After reading for a few years, one develops a critical eye to differentiate between genuine and junk authors and that can make things a bit easy. Just because an article has been published, it doesn't mean that it is worthy.
Posted by: chameedPosts: 173 :: 01-01-2006 :: | Reply to this Message
Re: Re: Limitations of Meta-analysis
I appreciate your interest in this topic. I agree that Methods section is a very important part of the paper, although I give importance to Results, Introduction, and Discussion parts as well.
Going back to my question, which was 'do we review literature properly in "Reviews"?' I would like to hear what do you think.
I would not comment on the 'honesty' of research. It will distract current discussion.
However, I will add a comment to the 'critical eye'. It is true that there is no substitute for experience, however, knowledge is essential for the development of 'critical eye'. Our eyes can't see what our mind does not know.
Posted by: rqayyumPosts: 199 :: 01-01-2006 :: | Reply to this Message
Re: Re: Re: Limitations of Meta-analysisI appreciate your “appreciative” nature. How’s life in WV? As I said earlier, the answer to your original question is “NO”. You can give importance to whatever you like but the “Materials and Methods” section of a paper must be understood clearly before one can make any comments. What one needs to look at is whether the study design is appropriate for the particular question that the paper is trying to answer. And then there is a question of honest data collection and proper statistics, and this is the red herring. So you draw your own conclusions. “Critical eye” comment was applicable to me and I don’t need any elaboration on that.
Posted by: chameedPosts: 173 :: 26-01-2006 :: | Reply to this Message
Re: Re: Re: Re: Limitations of Meta-analysis
Dr. Hameed, I am glad to know that you also have an appreciative nature. About WV, as they say, it is Wild and Wonderful. I am enjoying every moment of my stay here. People are poor, but straightforward. Most of all, I am enjoying the Muslim and Pakistani Community. There are quite a few wonderful Pakistani physicians here, including APPNA's new president, Dr. Piracha. I have lived in Connecticut and Illinois, but the sense of community I found here is the best. I am leaving WV in next few months to join Johns Hopkins University as Assistant Professor of Medicine and I am sure I will miss WV, and Pakistani community.
It is interesting that you talked about honest data-collection. For a reader, there is no way of knowing the honesty in data collection if author(s) are experts in their field. Experts know how to manipulate data. Recent scandal about an article published by a Korean scientist in the journal Science (on stem cells) is a good example. Even after rigorous peer-review before and after publication it took more than a year to discover (quite accidently) that it was not the true representation of the data.
To understand whether statistics are proper or not, one needs formal or informal training in statistics, which unfortunately most of us don’t have. We probably know about t-test and chi-square test, but not about the complexities of more complex tests. For example, if someone has performed a multiple linear regression, we have to take his word that he/she did it right. Otherwise, how often you have seen in articles comments about co-linearity, or leverage, or other issues related to regression being discussed in the methods section, even though all these should have been looked at and addressed before drawing a conclusion about the validity of regression. And I can give other examples of complex but commonly used tests.
Posted by: rqayyumPosts: 199 :: 13-02-2006 :: | Reply to this Message
Re: Do we review literature properly in “Reviews”?Good progress. From WV to Maryland. LOL
Posted by: chameedPosts: 173 :: 17-02-2006 :: | Reply to this Message
Re: Re: Do we review literature properly in “Reviews”?
Thank you.
I feel sorry for people who don't progress and get stuck at one place for decades.
[Edited by rqayyum on 18-02-2006 at 02:03 AM GMT]
Posted by: rqayyumPosts: 199 :: 17-02-2006 :: | Reply to this Message