Beyond the Single Experiment
So far, we’ve considered the many elements needed for a proper experiment. But we should also realize that the scientific process doesn’t end once a single experiment or observational study is finished (Figure 1.16). As one further step, the research must be evaluated by experts in the field to make certain it was done properly. This step is usu-ally achieved during the process of publishing the study in one of the scientific jour-nals. Specifically, a paper is published only after being evaluated and approved by other researchers who are experts in that area of investigation. These other researchers pro-vide peer review (i.e., the paper’s authors and these evaluators are all “peers” within the scientific community), and they must be convinced that the procedure was set up cor-rectly, the results were analyzed appropriately, and the conclusions are justified by the data. It’s only at this point that the study will be taken seriously by other psychologists.
Even after it’s published, a scientific study continues to be scrutinized. Other researchers will likely try to replicate the study—to run the same procedure with a new group of participants and see if it yields the same results. A successful replication assures us that there was nothing peculiar about the initial study and that the study’s results are reliable. Other investigators may also run alternative experiments in an attempt to challenge the initial findings.
This combination of replications and challenges eventually produces an accumulation of results bearing on a question. Researchers then try to assemble all the evidence into a sin-gle package, to check on how robust the results are—that is, whether the results are consis-tent even if various details in the procedure (the specific participants, the particular stimuli) are changed. Sometimes, this pooling of information is done in a published article—called a literature review—that describes the various results and discusses how they are or are not consistent with each other. In addition, researchers often turn to a statistical technique called meta-analysis. This is a formal procedure for mathematically combining the results of numerous studies—so, in effect, it’s an analysis of the individual analyses contained within each study. Meta-analysis allows inves-tigators to assess the consistency of a result in quantitative terms.
It’s only after all these steps—the result has been replicated, has survived scrutiny and challenge, and has been corroborated through other studies brought together in a review or meta-analysis—that we can truly consider the original results persuasive and the conclu-sions justified. Now we can say that the original hypothesis is confirmed—that is, well sup-ported by evidence. Notice, however, that even after all these steps, we do not claim the hypothesis is proven. That’s because scientists, in an open-minded way, always allow for the possibility that new facts will become available to challenge the hypothesis or show that it’s correct only in certain circumstances. On this basis, no matter how often a scientific hypothesis is confirmed, it is never regarded as truly “proven.” But, of course, if a hypothe-sis is confirmed repeatedly and withstands a range of challenges, scientists regard it as extremely likely to be correct. They then conclude that, at last, they can confidently build from there.
We should also mention the other possible outcome: What if the study is properly done and the data aren’t consistent with the authors’ original prediction? In that case, the hypothesis is disconfirmed, and the scientist must confront the contrary findings. Often, this means closely scrutinizing these findings to make certain the study that is challenging one’s hypothesis was done correctly. If it was, the researcher is obliged to tune the original hypothesis—or set that hypothesis aside and turn instead to some new proposal. What the scientist cannot do, though, is simply ignore the contrary find-ings and continue asserting a hypothesis that has been tested and found wanting.
Finally, with all of these safeguards in place, what about our earlier example? Are recordings containing subliminal suggestions an effective way to give up smoking or to increase your attractiveness? Several carefully designed studies have examined the effects of this type of recording, and the results are clear: The messages do seem to work, but this effect almost certainly involves a placebo effect—that is, an effect
produced by the participants’ positive expectations for the procedure and not the procedure itself (Figure 1.17). Once the investigator controls for the participants’ expec-tations about the recordings, the subliminal messages themselves produce no benefit (Greenwald, Spangenberg, Pratkanis, & Eskenazi, 1991).