Why New Judges Mattered So Much

Introduction

One of the most influential outcomes of the 1995 review of the U.S. government’s remote-viewing programme was a shift in emphasis from whether the reported results were statistically unusual to whether they would survive genuinely independent evaluation. Within the later SAIC (Science Applications International Corporation) experiments, blind judging was already a central feature of the methodology. However, reviewers argued that this was not sufficient if the same experienced judge, scoring conventions and laboratory practices continued to dominate the evidence. The practical question became straightforward: would different judges, working independently and under comparable conditions, reach the same conclusions? Until that question was answered through successful replication, the reported statistical effects could not be treated as robust evidence for remote viewing or as a reliable basis for continued government investment.[CIA]cia.govAN EVALUATION OF REMOTE VIEWING: RESEARCH…Utts and Hyman were asked to prepare independent reports based on their review. In this r…

Why One Familiar Judge Was Not Enough

The SAIC experiments relied heavily on free-response tasks. Participants described an unknown target, after which a judge compared the description against several possible targets and ranked the best match. This design attempted to reduce subjectivity by forcing explicit rankings rather than allowing open-ended impressions.

Even so, the judging stage remained the point where human interpretation entered the experiment. Unlike a multiple-choice test with objectively correct answers, free-response remote-viewing sessions produced descriptions that could be vague, symbolic or partially accurate. A judge therefore had to decide which details mattered and which were irrelevant.

Ray Hyman argued that this dependence on human judgement created an important vulnerability. If the same judge repeatedly evaluated sessions, that individual’s interpretation style might unintentionally become part of the experimental effect. Even without fraud or deliberate bias, subtle preferences for particular descriptive features, recurring target characteristics or familiar scoring habits could influence outcomes. As a result, apparent statistical success might reflect the interaction between one judge and one experimental system rather than a reproducible phenomenon.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaBoth Professor Utts and I agree that the very first consideration…

The concern was not simply that a judge might consciously favour successful results. Scientific methodology routinely assumes that observers bring expectations and habits to complex classification tasks. Independent assessment is therefore used in many fields to determine whether conclusions survive changes in personnel rather than depending upon one individual’s judgement.

How Cross-Judge Agreement Would Strengthen the Claim

The AIR evaluation identified an obvious way to test this possibility: use independent judges who had no previous involvement with the programme and compare their decisions.

If several independent judges consistently ranked the correct targets highly, confidence in the results would increase for several reasons:

• Individual scoring preferences would become less influential.
• Laboratory-specific expectations would be less likely to affect outcomes.
• Statistical significance would become harder to explain as an artefact of one evaluator’s methods.
• Other researchers would gain confidence that the reported effect reflected the data rather than the scoring process.

This principle extends beyond remote viewing. In psychology, medicine and other behavioural sciences, agreement among independent raters is often treated as evidence that a finding is not merely the product of one observer’s interpretation. High inter-rater reliability does not prove that a phenomenon is genuine, but low agreement immediately raises questions about whether the measurement itself is stable.

Hyman therefore argued that independent judging represented one of the simplest and most informative next tests. If remote-viewing transcripts genuinely contained information about their targets, new judges should continue to identify the correct targets at above-chance rates without requiring familiarity with the original research team or scoring traditions.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaBoth Professor Utts and I agree that the very first consideration…

Why Replication Became the Decisive Next Step

The debate ultimately moved beyond judging itself to the broader scientific standard of independent replication.

Jessica Utts accepted that the accumulated statistical evidence across multiple studies suggested an anomalous effect worthy of serious consideration. Hyman did not primarily dispute the reported statistical calculations. Instead, he argued that extraordinary claims require demonstrations that remain successful when reproduced by independent investigators using independent personnel, including judges. In his own summary of the AIR review, he stated that the first priority was determining whether the SAIC results would remain statistically significant with independent judges and independent laboratories.[UC Irvine Bren School]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaBoth Professor Utts and I agree that the very first consideration…

This distinction became central to the programme’s evaluation. Statistical significance answers one question: whether the observed pattern is unlikely under a specified chance model. Replication answers a different question: whether the same pattern reliably appears when different people repeat the work under comparable conditions.

For government sponsors, replication carried particular importance because operational intelligence systems require dependable performance across time, personnel and settings. A result that depended upon one laboratory, one scoring team or one experienced judge would have limited practical value regardless of its statistical strength.

The Policy Impact of the AIR Review

The 1995 AIR evaluation helped shift discussion away from isolated successful experiments towards governance standards familiar across mainstream science.

Rather than recommending further operational use based on the existing evidence, reviewers highlighted methodological priorities that would have to be satisfied before stronger conclusions could be justified. Independent judging became one component of a broader expectation that findings should survive changes in researchers, laboratories and experimental implementation.[CIA]cia.govAN EVALUATION OF REMOTE VIEWING: RESEARCH…Utts and Hyman were asked to prepare independent reports based on their review. In this r…

This emphasis reflected a wider scientific norm rather than a requirement unique to remote viewing. Across experimental science, confidence grows when results persist despite changes in investigators and analytical procedures. If an effect disappears once different judges score the same type of data, the credibility of the original finding is substantially weakened. Conversely, consistent cross-judge agreement across independent laboratories would provide much stronger evidence that the observed phenomenon was not an artefact of a particular research group.

Why the Standard Still Matters

The debate over independent judges illustrates why the SAIC programme remains significant in discussions of research methodology. The central issue was never simply whether some sessions appeared impressive. It was whether those impressions represented a repeatable empirical finding that could withstand independent scrutiny.

The AIR review effectively reframed the discussion from asking, “Were statistically unusual results reported?” to asking, “Would those results survive new judges, new laboratories and successful independent replication?” That change in emphasis transformed independent judging from a procedural detail into one of the decisive tests by which later remote-viewing claims would be assessed.[UC Irvine Bren School+2CIA]ics.uci.eduUC Irvine Bren SchoolEvaluation of Program on Anomalous Mental PhenomenaBoth Professor Utts and I agree that the very first consideration…

Amazon book picks

Further Reading

Books and field guides related to Why New Judges Mattered So Much. Use these as the next step if you want deeper reading beyond the article.

Book

Bad Science

By Ben Goldacre

Helps readers understand why independent verification matters.

See on Amazon

Book

Research methods and statistics in psychology

By Hugh Coolican

First published 2009. Subjects: Methodology, Psychology, Psychometrics, Research, Psychology, research.

See on Amazon

Book

The conscious universe

By Dean I. Radin, Radin

First published 1997. Subjects: Parapsychology, Case studies, Cas, Études de, Paranormale verschijnselen, Parapsychologie.

See on Amazon

Book

Parapsychology

By Etzel Cardeña, John Palmer et al.

First published 2015. Subjects: Parapsychology.

See on Amazon

Browse more on Amazon:Bad ScienceResearch methods and statistics in psychologyThe conscious universe

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

Canyon View Through Arch Outside Mo Framed Wall Art Poster Canvas Print Picture

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

Playstation 4 Gaming Console 3d Smashed View Wall Sticker Poster Decal A834

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

Photo:Steele Missouri 1938 Children Viewing Movie Posters Outside Theater

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

Boulevard Of Lights At Dusk Eiffel Remote View Wall Home Decor - POSTER 20x30

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: cia.gov
Link:https://www.cia.gov/readingroom/document/cia-rdp96-00791r000200180006-4

2. Source: cia.gov
Link:https://www.cia.gov/readingroom/document/cia-rdp96-00791r000200180005-5

3. Source: ics.uci.edu
Link:https://www.ics.uci.edu/~jutts/hyman.html

4. Source: Wikipedia
Title: Remote viewing
Link:https://en.wikipedia.org/wiki/Remote_viewing

5. Source: psi-encyclopedia.spr.ac.uk
Title: spr.ac.uk Remote Viewing
Link:https://psi-encyclopedia.spr.ac.uk/articles/remote-viewing/

Additional References

6. Source: facebook.com
Link:https://www.facebook.com/historyoasis/posts/from-1972-to-1995-the-united-states-military-invested-over-20-million-in-one-of-/790173214116954/

7. Source: researchgate.net
Title: 374881423 Remote Viewing A 1974 2022 Systematic Review and [Meta Analysis]({{ ‘meta-analysis/’ | relative_url }})
Link:https://www.researchgate.net/publication/374881423_Remote_Viewing_A_1974-2022_Systematic_Review_and_Meta-Analysis

8. Source: academia.edu
Link:https://www.academia.edu/95285973/The_Star_Gate_Operational_Remote_Viewing_Program_A_Human_Intelligence_HUMINT_Collection_Platform

9. Source: pmc.ncbi.nlm.nih.gov
Title: Follow‐up on the U.S
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10275521/

10. Source: youtube.com
Title: Russell Targ on the Science of Remote Viewing
Link:http://www.youtube.com/watch?v=IDaFwLS6goU

11. Source: youtube.com
Title: Remote Viewing Training, Part One: The Initial Phases, with Paul H. Smith
Link:http://www.youtube.com/watch?v=Y2Ryc196s0I

12. Source: koestlerunit.wordpress.com
Title: wiseman milton 1998
Link:https://koestlerunit.wordpress.com/wp-content/uploads/2015/06/wiseman-milton-1998.pdf

13. Source: youtube.com
Title: The Atheist Experience
Link:https://www.youtube.com/watch?v=AoUCHtGyoFg

14. Source: youtube.com
Title: Remote Viewing From the Inside Out with Russell Targ
Link:http://www.youtube.com/watch?v=V6Pn6WGahYI

15. Source: youtube.com
Title: Principles of Remote Viewing with Paul H. Smith
Link:http://www.youtube.com/watch?v=Smo1aP0v4xg