Within Statistics
What Would Count as Strong Replication?
The Utts-Hyman dispute turns on what kind of independent replication would make remote-viewing evidence hard to dismiss.
On this page
- Why Utts saw accumulated evidence as persuasive
- Why Hyman wanted independent confirmation
- What hostile replication would need to control
Page outline Jump by section
Introduction
The debate between Jessica Utts and Ray Hyman shifted the remote-viewing discussion away from a simple question of statistical significance towards a more demanding one: what kind of replication would justify concluding that a genuine anomalous phenomenon exists? Both reviewers accepted that some laboratory findings deserved serious attention, but they disagreed sharply about whether the existing body of evidence had already crossed the threshold from intriguing statistics to persuasive scientific proof.[National Security Archive]nsarchive2.gwu.eduIn this review, they were to cover four general topicsNational Security ArchiveAn Evaluation of Remote Viewing: Research and…March 13, 2015 — by MD Mumford · 1995 · Cited by 76 — Utts and…
That disagreement has shaped nearly every subsequent discussion of remote-viewing research. Utts argued that repeated positive findings across experiments and laboratories had already demonstrated a reliable effect. Hyman maintained that the decisive test had not yet been performed because the strongest claims still lacked sufficiently independent, sceptical, and methodologically transparent replication. The dispute therefore became less about individual experiments than about the standards of evidence appropriate for extraordinary claims.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaProfessor Utts concludes that "psychic functioning has been well…
Why Utts considered the accumulated evidence persuasive
Utts’ position was not based on one dramatic experiment but on the cumulative statistical record. Reviewing the government-sponsored SRI and SAIC programmes alongside broader parapsychology research, she argued that the observed effects appeared consistently above chance and were comparable to effect sizes accepted in many areas of behavioural science. She further argued that criticisms based solely on hypothetical methodological flaws became increasingly difficult to sustain if similar results continued appearing in multiple studies using different investigators and protocols.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaProfessor Utts concludes that "psychic functioning has been well…
An important feature of her argument was that replication did not require every individual experiment to produce identical outcomes. Instead, she emphasised long-run reproducibility: small but statistically reliable effects emerging repeatedly under properly controlled conditions. From this perspective, variability between experiments was expected rather than fatal, provided the overall pattern remained positive.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaProfessor Utts concludes that "psychic functioning has been well…
Utts therefore concluded that future work should focus less on asking whether an effect existed and more on understanding its characteristics, boundary conditions, and practical limitations. In her view, scientific progress would come from refining knowledge about when the phenomenon appeared strongest rather than repeatedly attempting to prove its existence from the beginning.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaProfessor Utts concludes that "psychic functioning has been well…
Why Hyman insisted on stronger independent confirmation
Hyman accepted that some of the later SAIC experiments appeared more carefully designed than earlier remote-viewing work. He explicitly acknowledged that the reported statistical effects were too consistent simply to dismiss as random fluctuations. However, he argued that this observation did not justify concluding that paranormal functioning had been demonstrated.[Wikipedia]WikipediaRay HymanRay Hyman
His central concern was independence. Much of the positive evidence came from a relatively small research community using related methods, overlapping investigators, similar judging procedures, and common analytical traditions. Hyman argued that genuine scientific acceptance requires successful reproduction by investigators who are not invested in the original claims and who may even expect failure.[ResearchGate]researchgate.netEvaluation of Program on 'Anomalous Mental Phenomena'PDF | Jessica Utts and I were commissioned to evaluate the research on r…
He also distinguished replication from repetition. Repeating similar procedures within closely connected research programmes does not provide the same evidential weight as successful reproduction by laboratories that independently develop, implement, and analyse the protocol. Without that broader confirmation, he regarded conclusions about psychic functioning as premature.[ResearchGate]researchgate.netEvaluation of Program on 'Anomalous Mental Phenomena'PDF | Jessica Utts and I were commissioned to evaluate the research on r…
Finally, Hyman argued that the absence of a well-developed theoretical framework complicated interpretation. Without a positive theory specifying when remote viewing should or should not work, researchers faced an open-ended problem of deciding which methodological variables mattered after results were already known. That made it harder to separate genuine effects from subtle experimental artefacts.[Wikipedia]WikipediaRay HymanRay Hyman
What hostile replication would need to control
The standards implied by Hyman’s critique have become an informal benchmark for evaluating later remote-viewing claims. A genuinely persuasive hostile replication would require more than another statistically significant experiment.
Key features would include:
- Independent investigators with no professional involvement in previous positive remote-viewing programmes.
- Pre-registered protocols specifying hypotheses, scoring methods, exclusion criteria and analyses before data collection begins.
- Strict blinding so that viewers, experimenters, judges and analysts cannot unintentionally exchange information.
- Random target generation using procedures that eliminate predictable patterns.
- Independent judging performed without knowledge of experimental conditions.
- Complete reporting, including unsuccessful sessions and negative experiments, to reduce publication bias.
- Open data and analysis code, allowing outside researchers to verify every statistical calculation.
- Successful replication across multiple laboratories, rather than isolated positive studies from one research group.
Most of these practices have since become standard expectations across psychology and other experimental sciences, particularly following wider concerns about reproducibility. They are now viewed as general safeguards rather than requirements unique to parapsychology.[National Security Archive]nsarchive2.gwu.eduIn this review, they were to cover four general topicsNational Security ArchiveAn Evaluation of Remote Viewing: Research and…March 13, 2015 — by MD Mumford · 1995 · Cited by 76 — Utts and…
Why the disagreement remains important
The Utts-Hyman exchange illustrates that scientific disagreements often concern evidential standards rather than simple acceptance or rejection of data. Both reviewers recognised that some experiments deserved serious statistical attention. Their disagreement centred on how much evidence should be demanded before accepting an explanation that would challenge established scientific understanding.[National Security Archive]nsarchive2.gwu.eduIn this review, they were to cover four general topicsNational Security ArchiveAn Evaluation of Remote Viewing: Research and…March 13, 2015 — by MD Mumford · 1995 · Cited by 76 — Utts and…
This distinction has influenced later evaluations of remote-viewing research. Supporters frequently point to cumulative meta-analyses and repeated above-chance findings as evidence that the phenomenon continues to replicate statistically. Critics respond that the decisive issue is not whether some positive studies exist, but whether independent laboratories operating under the strongest modern transparency standards can repeatedly obtain comparable results while excluding ordinary explanations such as methodological artefacts, selective reporting, sensory leakage or subjective interpretation.[PMC]pmc.ncbi.nlm.nih.govPMCFollow‐up on the U.SCentral Intelligence Agency's (CIA…by Á Escolà‐Gascón · 2023 · Cited by 10 — Reports on the declassified SRI and SAIC experiments were…
How replication fits into the broader proof debate
Within the wider discussion of statistical significance versus practical proof, the replication question serves as the bridge between an interesting experimental result and an accepted scientific claim. Statistical significance may indicate that something unexpected occurred under specific experimental conditions. Replication asks whether the same result survives when different researchers, different laboratories and increasingly stringent controls attempt to reproduce it.
The enduring significance of the Utts-Hyman debate is therefore not simply whether remote viewing produced above-chance scores in particular experiments. It is the continuing disagreement over what level of independent confirmation should be considered sufficient before treating those scores as evidence for a genuine anomalous human capability rather than an unresolved experimental finding.[National Security Archive]nsarchive2.gwu.eduIn this review, they were to cover four general topicsNational Security ArchiveAn Evaluation of Remote Viewing: Research and…March 13, 2015 — by MD Mumford · 1995 · Cited by 76 — Utts and…
Amazon book picks
Further Reading
Books and field guides related to What Would Count as Strong Replication?. Use these as the next step if you want deeper reading beyond the article.
The Signal and the Noise
Highlights why reliable prediction requires strong, repeatable signals.
Statistics Done Wrong
First published 2015. Subjects: Missing observations (Statistics), Methodology, Statistics.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
1.
Source: pmc.ncbi.nlm.nih.gov
Title: PMCFollow‐up on the U.S
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10275521/
Source snippet
Central Intelligence Agency's (CIA...by Á Escolà‐Gascón · 2023 · Cited by 10 — Reports on the declassified SRI and SAIC experiments were...
2.
Source: Wikipedia
Title: Ray Hyman
Link:https://en.wikipedia.org/wiki/Ray_Hyman
3.
Source: researchgate.net
Link:https://www.researchgate.net/publication/267978941Evaluation_of_Program_on%27Anomalous_Mental_Phenomena%27
Source snippet
Evaluation of Program on 'Anomalous Mental Phenomena'PDF | Jessica Utts and I were commissioned to evaluate the research on r...
4.
Source: Wikipedia
Title: Remote viewing
Link:https://en.wikipedia.org/wiki/Remote_viewing
Source snippet
Remote viewingThe program ran from 1975 to 1995 and ended after evaluators concluded that remote viewers consistently failed to produc...
5.
Source: youtube.com
Title: Ray Hyman
Link:https://www.youtube.com/watch?v=-8VAOzcXGc4
Source snippet
An Open-Ended Conversation with Paul H. Smith...
6.
Source: nsarchive2.gwu.edu
Title: In this review, they were to cover four general topics
Link:https://nsarchive2.gwu.edu/NSAEBB/NSAEBB438/docs/doc_57.pdf
Source snippet
National Security ArchiveAn Evaluation of Remote Viewing: Research and...March 13, 2015 — by MD Mumford · 1995 · Cited by 76 — Utts and...
Published: March 13, 2015
7.
Source: ics.uci.edu
Link:https://www.ics.uci.edu/~jutts/hyman.html
Source snippet
UC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaProfessor Utts concludes that "psychic functioning has been well...
8.
Source: psi-encyclopedia.spr.ac.uk
Title: spr.ac.uk Remote Viewing
Link:https://psi-encyclopedia.spr.ac.uk/articles/remote-viewing/
Source snippet
Viewing - Psi Encyclopedia13 Jan 2017 — Hyman and Utts were each asked by AIR to produce an independent report by a fixed date. Utts comp...
Additional References
9.
Source: facebook.com
Title: do you believe in remote viewing or any other forms of parapsychology shawnryans
Link:https://www.facebook.com/shawnryan762/posts/do-you-believe-in-remote-viewing-or-any-other-forms-of-parapsychology-shawnryans/1511586156155460/
Source snippet
Do you believe in remote viewing or any other forms...Ray Hyman, 1995) “Because my report will emphasize points of disagreement between...
10.
Source: facebook.com
Link:https://www.facebook.com/historyoasis/posts/from-1972-to-1995-the-united-states-military-invested-over-20-million-in-one-of-/790173214116954/
Source snippet
From 1972 to 1995, the United States military invested over...The final AIR report concluded that no remote viewing report ever provided...
11.
Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180005-5.pdf
Source snippet
, they were to cover four general topics: Was there a...
12.
Source: youtube.com
Title: [Stargate]({{ ‘stargate/’ | relative_url }}): The PARANORMAL at the Pentagon
Link:https://www.youtube.com/watch?v=3tgzIm5IZ4I
Source snippet
These videos are relevant because they feature key figures and experts involved in or commenting on the Stargate Project, the scientific...
13.
Source: youtube.com
Title: Ex-CIA Scientist Created a Test To Find Out If You’re Psychic — Take It Now
Link:https://www.youtube.com/watch?v=edWrufOe55Q
Source snippet
Correlates of Anomalous Cognition with Edwin C. May...
14.
Source: youtube.com
Title: An Open-Ended Conversation with Paul H. Smith
Link:https://www.youtube.com/watch?v=QhWOt2q_xnU
Source snippet
Ex-CIA Scientist Created a Test To Find Out If You're Psychic — Take It Now...
15.
Source: metabunk.org
Title: page 3
Link:https://www.metabunk.org/threads/claim-remote-viewing-is-a-scientifically-proven-technique-that-utilizes-a-natural-human-ability-to-enable-access-to-hidden-information.13057/page-3
Source snippet
Claim: Remote Viewing is a Scientifically Proven...24 Jul 2023 — The resulting AIR report contained two major reviews of the research (U...
16.
Source: youtube.com
Title: Correlates of Anomalous Cognition with Edwin C. May
Link:https://www.youtube.com/watch?v=Y5X4_DXGX60
Source snippet
Stargate: The PARANORMAL at the Pentagon...
Topic Tree