Within Hyman Review
Could One Judge Make the Effect Look Real?
The use of one embedded judge made it hard to know whether the matches would survive outside the original research culture.
On this page
- Why judging free response sessions is unusually subjective
- How one familiar judge can shape apparent accuracy
- What independent judging would need to test
Page outline Jump by section
Introduction
One of Ray Hyman’s most specific methodological concerns about the later Science Applications International Corporation (SAIC) remote-viewing experiments was not simply that judging was subjective, but that it relied heavily on a single experienced judge. In free-response remote viewing, participants produce sketches and verbal descriptions rather than selecting from fixed answers. Someone must later compare those descriptions with possible targets and decide which match is best. That judgement is an essential part of the experiment rather than an incidental step. Hyman argued that when one embedded judge performs this task throughout a research programme, it becomes difficult to know whether the apparent effect reflects remote viewing itself or the particular habits, expectations and expertise of that individual. This concern became one of the central implementation questions in the American Institutes for Research (AIR) review of the Stargate programme.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review…
Could one judge make the effect look real?
The issue was not an accusation of fraud or deliberate bias. Instead, it concerned whether the experimental design allowed the results to generalise beyond the original laboratory.
Unlike a multiple-choice experiment, free-response remote viewing asks judges to compare often ambiguous descriptions with several candidate targets after the viewing session has ended. The judge must decide whether broad features such as “a large structure”, “water”, “vertical shapes” or “movement” fit one target better than the alternatives. Because these judgements involve interpretation, experience inevitably plays a role.[CIA]cia.govAN EVALUATION OF REMOTE VIEWINGA judge then examines the viewer's report and determines if this report matches the target or, alternat…
In the SAIC studies reviewed by AIR, the principal judge was also the programme’s lead investigator, Edwin May. Hyman noted that May believed he obtained the best results because he had become familiar with each viewer’s reporting style. From the perspective of producing consistent internal scores, that may have seemed reasonable. From the perspective of experimental validation, however, it created an important uncertainty: would an equally knowledgeable but independent judge reach the same conclusions?[UC Irvine Bren School]ics.uci.eduThis difference is even more extreme in the SAIC…Read more…
Why judging free-response sessions is unusually subjective
Free-response protocols differ fundamentally from experiments with objectively correct answers.
Instead of asking participants to identify one option from several possibilities, remote-viewing sessions typically generate pages of sketches, adjectives and impressions. A judge must then:
- decide which features deserve emphasis;
- determine which similarities are meaningful rather than coincidental;
- balance accurate details against incorrect ones;
- compare the response against several potential targets.
Each of those decisions involves discretion. Two careful judges can legitimately disagree about which target is the closest match without either acting improperly.
Hyman argued that this subjectivity is precisely why judging procedures require especially strong safeguards. Statistical significance after subjective interpretation is not automatically equivalent to objective evidence for paranormal perception. The reliability of the interpretation itself also has to be demonstrated.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review…
How one familiar judge can shape apparent accuracy
The concern was less about conscious bias than about learned expertise that cannot easily be separated from the reported effect.
Over years of working with the same viewers, a judge may begin to recognise recurring patterns:
- particular ways individual viewers describe buildings, landscapes or water;
- consistent drawing styles;
- habitual omissions or exaggerations;
- idiosyncratic symbolic language.
Such familiarity may genuinely improve that judge’s ability to identify the intended target. However, it also means the measured success could partly depend on a relationship between specific viewers and a specific judge rather than on a broadly reproducible phenomenon.
Hyman highlighted exactly this possibility. The AIR review notes that the single SAIC judge believed familiarity with the viewers improved judging performance. Hyman regarded that explanation itself as a reason for caution, because it implied that the effect might depend upon local laboratory expertise rather than a universally observable process.[UC Irvine Bren School]ics.uci.eduThis difference is even more extreme in the SAIC…Read more…
This concern also complicated comparisons with other areas of parapsychology. Hyman observed that ganzfeld researchers often favoured different judging arrangements, sometimes finding stronger results when participants judged their own responses. The contrasting practices suggested that different laboratories might be measuring different procedural effects rather than demonstrating one clearly understood phenomenon.[UC Irvine Bren School]ics.uci.eduThis difference is even more extreme in the SAIC…Read more…
What independent judging would need to test
The obvious response to the single-judge problem is not merely to recruit additional judges but to structure judging so that agreement itself becomes part of the evidence.
A stronger implementation would include:
- several judges working independently;
- judges who have never interacted with the viewers;
- blinding so judges cannot infer target identities or session order;
- predefined scoring rules established before judging begins;
- measurement of inter-rater agreement rather than relying on one person’s rankings.
If independent judges consistently identify the same target while remaining blind to the experimental conditions, confidence increases that the apparent correspondence lies in the responses themselves rather than in one individual’s interpretation.
Conversely, if apparent success disappears when new judges replace the original one, the original findings become harder to interpret as evidence for remote viewing.
These are standard methodological principles across psychology and behavioural science whenever human judgement forms part of the measurement process, and Hyman argued they were particularly important for remote viewing because subjective matching is central to the protocol.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review…
Why this issue mattered beyond one experiment
The single-judge question became important because the AIR review was intended to determine whether decades of government-sponsored research had demonstrated a genuine phenomenon suitable for intelligence applications.
Hyman accepted that the later SAIC experiments addressed several criticisms directed at earlier Stanford Research Institute work. Nevertheless, he argued that improvements in laboratory controls did not resolve the question of whether the reported effects depended on features unique to one research group. Independent replication requires more than repeating the same protocol with the same investigators and the same evaluation culture. It requires showing that different researchers, different judges and different laboratories obtain comparable outcomes.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review…
Later critics, including Richard Wiseman and Julie Milton, similarly argued that important aspects of the SAIC methodology needed more transparent reconstruction and independent evaluation before strong conclusions could be drawn. Their critiques addressed several methodological issues beyond judging alone, but they reinforced Hyman’s broader point that robust evidence must survive examination outside the original research team.[Hertfordshire Research Profiles]researchprofiles.herts.ac.ukexperiment one of the saic remote viewing program a critical re eHertfordshire Research ProfilesExperiment One of the SAIC Remote Viewing Programby R Wiseman · 1999 · Cited by 24 — This paper first outl…
The lasting significance of the single-judge problem
The single-judge issue remains a methodological rather than ideological criticism. It does not claim that the SAIC judge intentionally influenced outcomes, nor does it prove that the reported statistical effects were artefacts.
Instead, it highlights a straightforward scientific question: if one experienced insider consistently produces the strongest evidence, can the same level of performance be achieved by independent judges unfamiliar with the viewers?
For Hyman, answering that question was essential before interpreting statistically significant matches as convincing evidence for remote viewing. Until judging could be shown to remain reliable across different evaluators and research settings, he argued that the observed effects could not confidently be separated from the particular research culture in which they were produced.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review…
Amazon book picks
Further Reading
Books and field guides related to Could One Judge Make the Effect Look Real?. Use these as the next step if you want deeper reading beyond the article.
Research methods in psychology
First published 2012. Subjects: Textbooks, Methodology, Psychology, Experimental Psychology, Research.
Blindspot
First published 2013. Subjects: Discrimination, Prejudices, Psychological aspects, nyt:relationships=2016-07-10, New York Times bestseller.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
1.
Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180005-5.pdf
Source snippet
AN EVALUATION OF THE REMOTE VIEWING PROGRAMThe first component was a review of the research program. The second component was a review...
2.
Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180006-4.pdf
Source snippet
AN EVALUATION OF REMOTE VIEWINGA judge then examines the viewer's report and determines if this report matches the target or, alternat...
3.
Source: ics.uci.edu
Link:https://www.ics.uci.edu/~jutts/hyman.html
Source snippet
This difference is even more extreme in the SAIC...Read more...
4.
Source: researchprofiles.herts.ac.uk
Title: experiment one of the saic remote viewing program a critical re e
Link:https://researchprofiles.herts.ac.uk/en/publications/experiment-one-of-the-saic-remote-viewing-program-a-critical-re-e
Source snippet
Hertfordshire Research ProfilesExperiment One of the SAIC Remote Viewing Programby R Wiseman · 1999 · Cited by 24 — This paper first outl...
Additional References
5.
Source: researchgate.net
Link:https://www.researchgate.net/publication/333228024_An_Assessment_of_the_Evidence_for_Psychic_Functioning
6.
Source: richardwiseman.com
Link:https://www.richardwiseman.com/resources/SAICreply.pdf
Source snippet
In our original paper (Wiseman & Milton, 1999), we described a number of potential information [leakage]({{ 'leakage/' | relative_url }}) pathways in Experiment One of the...
7.
Source: koestlerunit.wordpress.com
Link:https://koestlerunit.wordpress.com/wp-content/uploads/2015/06/wiseman-milton-1998.pdf
Source snippet
(Utts, 1995a; Hyman, 1995a) and a concluding section that outlined the main points of.Read more...
8.
Source: scribd.com
Link:https://www.scribd.com/doc/92017954/Air-Report
Source snippet
or all experiments. This judge, who was also the principal...Read more...
9.
Source: youtube.com
Title: Edwin May, [Psychic Spying]({{ ‘psychic-spying/’ | relative_url }}) (Remote Viewing, Star Gate Program)
Link:https://www.youtube.com/watch?v=EPvrOsEXBbE
Source snippet
CLASSIC REBOOT: America's Psychic Spy Program with Edwin C. May...
10.
Source: youtube.com
Title: Dr. Edwin May, Psychic Research (Remote Viewing, Telekinesis)
Link:https://www.youtube.com/watch?v=m66dX1NQ9k4
Source snippet
Classic Reboot: Training Anomalous Cognition with Edwin C. May...
11.
Source: youtube.com
Title: Classic Reboot: Training Anomalous Cognition with Edwin C. May
Link:https://www.youtube.com/watch?v=6vbb7E7UD30
Source snippet
Edwin May, Psychic Spying (Remote Viewing, Star Gate Program) - Jimmy Akin's Mysterious World...
12.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10275521/
Source snippet
Central Intelligence Agency's (CIA...by Á Escolà‐Gascón · 2023 · Cited by 10 — Additionally, Hyman (1996) and other skeptical researcher...
13.
Source: youtube.com
Title: Researching Anomalous Cognition with Edwin C. May
Link:https://www.youtube.com/watch?v=sT_tbmB4J5I
Source snippet
Dr. Edwin May, Psychic Research (Remote Viewing, Telekinesis) - Jimmy Akin's Mysterious World...
14.
Source: youtube.com
Title: CLASSIC REBOOT: America’s Psychic Spy Program with Edwin C. May
Link:https://www.youtube.com/watch?v=WBrGgWwghcY
Topic Tree