Within Judging
When a First Place Match Is Not Enough
Rank-order judging turns vague impressions into numbers, but small design choices can change what looks like a hit.
On this page
- How five target ranking works
- Why chance depends on the target pool
- Where early scoring controls broke down
Page outline Jump by section
Introduction
Rank-order judging was introduced to make remote-viewing experiments more objective. Instead of asking whether a transcript contains anything that resembles the target, a blind judge compares one genuine target with several decoys and ranks them from best to worst match. In principle, this converts an open-ended description into a measurable outcome. In practice, however, the method is highly sensitive to design choices. The composition of the target pool, the number and similarity of decoys, the way targets are reused, and even the order in which judging takes place can all influence the apparent success rate. As a result, a first-place ranking is not always as compelling as it appears unless the underlying judging procedure has been carefully controlled and its statistical assumptions are satisfied.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
How five-target ranking works
Many remote-viewing studies have used a simple rank-order procedure. After producing a free-response description, the viewer’s transcript is presented to a blind judge alongside five candidate targets: one genuine target and four decoys. The judge ranks all five according to how well each appears to fit the transcript.
This system deliberately avoids a simple yes-or-no judgement. Even if every target shares some features with the transcript, the crucial question becomes whether the real target consistently receives a better rank than would occur by chance. If all five candidates are equally plausible, random guessing would produce a first-place rank about one time in five, with average ranks clustering around the middle of the distribution. Statistical analysis is therefore based on the pattern of ranks across many independent trials rather than isolated “hits”.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
The attraction of this approach is that it acknowledges the ambiguity of remote-viewing transcripts. Rather than rewarding every possible resemblance, judges must make comparative decisions. However, comparative decisions introduce their own sources of variation, because the meaning of “best match” depends heavily on the alternatives provided.
Why chance depends on the target pool
A common misunderstanding is that achieving first place automatically represents strong evidence. In reality, the difficulty of obtaining first place depends on the construction of the target pool.
If the four decoys are visually and conceptually very different from the actual target, the genuine target may naturally stand out. Conversely, if several decoys share major features with the correct target—for example, several coastal scenes, multiple bridges or similar industrial structures—the ranking task becomes substantially harder.
Several characteristics of the target pool affect the probability of success:
- Similarity among targets. Closely related targets reduce the distinctiveness of any single image.
- Target diversity. Extremely diverse collections can make broad descriptions such as “water”, “buildings” or “people” appear unusually diagnostic.
- Visual complexity. Highly detailed scenes offer more possible matching points than simple geometric or minimalist images.
- Repeated themes. Collections containing many related locations or objects change the likelihood that general descriptions will appear meaningful.
Because of these effects, two experiments reporting identical first-place percentages may not represent equally difficult tests. Apparent performance can change even if the viewers’ descriptions remain the same simply because the target pool has been altered. This is one reason later evaluations placed considerable emphasis on standardising target selection and randomisation procedures.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
Where early scoring controls broke down
One of the best-known criticisms of early remote-viewing research involved the interaction between ranking and target selection rather than the judges themselves.
Some early experiments selected targets without replacement from a common pool. Once a target had already appeared during earlier trials, both experimenters and judges could potentially infer that it would not appear again. This meant that not every target retained an equal probability of receiving every possible rank, violating assumptions built into the statistical analysis. The American Institutes for Research review commissioned by the CIA identified this as a genuine statistical flaw in some early studies.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
Related criticisms focused on what statisticians call serial dependence. Decisions made during one judging session could subtly influence later sessions because information accumulated across the experiment. Later analyses argued that correcting for these dependencies reduced the apparent strength of some early statistical findings, although not every reported effect disappeared after recalculation.[ResearchGate]researchgate.netResearch Gate(PDF) Remote viewing as applied to futures studiesAIR). The resulting report, Enhancing Human Performance: Issues, Theories, and Techniques, was. predominately negative concerning the ut…
Researchers also identified opportunities for unintended cueing. If transcripts contained references that indirectly revealed previous targets, or if experimental logistics provided clues about likely locations, judges could achieve above-chance rankings without any anomalous information transfer. These concerns prompted later studies to adopt stricter double-blind procedures and better isolation between trials.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
Why a first-place match can still be misleading
A genuine first-place ranking is informative only within the context of the entire judging system.
Several situations can inflate the apparent importance of a single successful match:
- A transcript may contain many broad descriptors that fit numerous targets, with the judge selecting the least imperfect option rather than recognising a uniquely accurate description.
- Different judges may produce different rankings from the same transcript, demonstrating that similarity judgements are partly subjective.
- A target set containing one unusually distinctive image may naturally attract first-place rankings even from vague descriptions.
- Reporting isolated successful trials while ignoring weaker rankings across the full experiment creates a distorted impression of accuracy.
For these reasons, remote-viewing researchers generally analyse aggregate rank distributions rather than celebrating individual dramatic matches. The statistical question is whether the complete collection of rankings departs from the distribution expected under random target assignment.
What later experiments tried to improve
Later laboratory protocols attempted to reduce these weaknesses through more rigorous experimental design rather than different scoring formulas.
Common improvements included:
- random target selection with replacement where appropriate;
- fully double-blind judging so neither judges nor experimenters knew the correct target;
- carefully defined target pools, often using large collections of standardised photographs;
- independent judges working separately;
- statistical analyses planned before data collection rather than after notable matches emerged.
Even with these improvements, the 1995 American Institutes for Research review concluded that methodological quality had generally improved while also emphasising that laboratory statistical effects did not necessarily translate into reliable or operationally useful remote viewing. The review highlighted that judging procedures had become more defensible than in the earliest experiments but continued to examine whether the remaining evidence justified claims of anomalous information transfer.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological…
Amazon book picks
Further Reading
Books and field guides related to When a First Place Match Is Not Enough. Use these as the next step if you want deeper reading beyond the article.
The seventh sense
First published 2003. Subjects: Military intelligence, American Espionage, Military aspects of Parapsychology, Remote viewing (Parapsycho...
Remote viewing secrets
First published 2000. Subjects: Remote viewing (Parapsychology), Parapsychology, Prophecies (occultism), Astral projection.
Mind-Reach
First published 2005. Subjects: Consciousness, Parapsychology, Case studies.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
1.
Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180005-5.pdf
Source snippet
AN EVALUATION OF THE REMOTE VIEWING PROGRAMEarly research on remote viewing was plagued by a number of statistical and methodological...
2.
Source: researchgate.net
Title: Research Gate(PDF) Remote viewing as applied to futures studies
Link:https://www.researchgate.net/publication/248497861_Remote_viewing_as_applied_to_futures_studies
Source snippet
(AIR). The resulting report, Enhancing Human Performance: Issues, Theories, and Techniques, was. predominately negative concerning the ut...
3.
Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180006-4.pdf
Source snippet
of these problems are described in the National...Read more...
4.
Source: Wikipedia
Title: Remote viewing
Link:https://en.wikipedia.org/wiki/Remote_viewing
Source snippet
Remote viewingIn 1995, the CIA hired the American Institutes for Research (AIR)... Marks in his book The Psychology of the Psychic (2...
Additional References
5.
Source: scribd.com
Link:https://www.scribd.com/doc/92017954/Air-Report
Source snippet
Many of these problems are described in the National Research...Read more...
6.
Source: pmc.ncbi.nlm.nih.gov
Title: PMCFollow‐up on the U.S
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10275521/
Source snippet
Central Intelligence Agency's (CIA...by Á Escolà‐Gascón · 2023 · Cited by 10 — Researchers addressing these issues are positioned in two...
7.
Source: facebook.com
Link:https://www.facebook.com/historyoasis/posts/from-1972-to-1995-the-united-states-military-invested-over-20-million-in-one-of-/790173214116954/
Source snippet
From 1972 to 1995, the United States military invested over...However, this illustration also shows one of the problems that must be dea...
8.
Source: ciaotest.cc.columbia.edu
Title: sa jan02srm01
Link:https://ciaotest.cc.columbia.edu/olj/sa/sa_jan02srm01.html
Source snippet
Remote Viewing: The US Sponsored Psychic...This paper deals with experiments conducted in USA in which certain individuals were trained...
9.
Source: acadintuition.com
Link:https://acadintuition.com/wp-content/uploads/2018/07/Applications.pdf
Source snippet
The resulting report, Enhancing Human Performance: Issues, Theories, and Techniques, was predominately negative concerning the utility of...
10.
Source: youtube.com
Title: [Statistics]({{ ‘statistics/’ | relative_url }}) in Parapsychology with Jessica Utts
Link:https://www.youtube.com/watch?v=WmYGtKB9EEA
Source snippet
Classic Reboot: Training Anomalous Cognition with Edwin C. May...
11.
Source: youtube.com
Title: New Directions in Remote Viewing with Chase from Social-RV
Link:https://www.youtube.com/watch?v=lv0gXL7RerQ
Source snippet
Remote Viewing and Statistical Validation...
12.
Source: youtube.com
Title: Remote Viewing Psychology with Charles T. Tart
Link:https://www.youtube.com/watch?v=jPZwaicuiek
Source snippet
New Directions in Remote Viewing with Chase from Social-RV...
13.
Source: youtube.com
Title: Remote Viewing and Statistical Validation
Link:https://www.youtube.com/watch?v=YrwAiU2g5RU
Source snippet
Statistics in Parapsychology with Jessica Utts...
14.
Source: youtube.com
Title: Classic Reboot: Training Anomalous Cognition with Edwin C. May
Link:https://www.youtube.com/watch?v=6vbb7E7UD30
Topic Tree