When Blind Judges Do Not Agree

Introduction

In associative remote viewing (ARV), the most important step often comes after the viewing session itself: deciding which of two or more possible targets best matches the viewer’s transcript. This decision is usually made by a blind judge who does not know which target corresponds to the real future outcome. If different judges consistently reach the same conclusion from the same transcript, the method appears more objective. If they frequently disagree, the apparent predictive success may depend more on who performed the judging than on any underlying signal.

For this reason, rater reliability has become one of the central methodological issues in ARV research. Even researchers who are broadly sympathetic to remote-viewing claims have acknowledged that inconsistent judging can undermine practical forecasting. Critics likewise argue that subjective interpretation is one of the largest remaining sources of bias, especially when transcripts contain vague imagery that could plausibly fit several targets. The debate therefore focuses less on whether judges are honest than on whether the judging process itself produces repeatable decisions.

What rater reliability means in ARV

Unlike many laboratory tasks that produce a single numerical measurement, ARV usually generates free-form material: sketches, written descriptions and symbolic impressions. A judge compares this material with two or more target photographs associated with different future outcomes and decides which target fits best.

In a reliable system:

If those conditions are not met, the practical value of ARV falls sharply. A forecasting method cannot easily be operational if identical evidence leads equally qualified judges to opposite predictions.

Inter-rater reliability—the degree to which independent judges agree—is therefore a standard quality measure borrowed from psychology and behavioural science. High agreement suggests that the judging criteria are sufficiently objective to produce stable decisions. Low agreement suggests that subjective interpretation is dominating the outcome.

Why disagreement weakens prediction claims

The difficulty is not merely statistical. It changes what a successful ARV trial actually means.

Remote-viewing transcripts often contain broad descriptors such as “water”, “movement”, “metal”, “crowds”, or emotional impressions. Such descriptions may legitimately resemble more than one target image. If Judge A considers one feature decisive while Judge B emphasises another, they may reach different conclusions despite following the same written protocol.

This creates several problems.

Prediction becomes judge-dependent. If changing the judge changes the prediction, the forecast cannot be regarded as a stable property of the transcript itself.

Positive results become harder to interpret. A successful prediction may reflect one judge’s interpretation rather than information objectively present in the viewing session.

Replication becomes more difficult. Independent laboratories cannot easily reproduce findings if judging standards vary between research groups.

The 1995 evaluation commissioned by the CIA and carried out by the American Institutes for Research highlighted broader concerns about the ambiguity of remote-viewing data, concluding that reports were often too vague for dependable operational use even where laboratory effects had been reported. The review argued that subjective interpretation remained a major obstacle to practical application.[CIA]cia.govAN EVALUATION OF THE REMOTE VIEWING PROGRAMIt is unclear whether the observed effects can unambiguously be attributed to the paranorma…

What published ARV work has found

One of the clearest examinations of this issue came from David Katz’s analysis of ARV projects published in 2021. Rather than assuming that judges naturally agree, the study explicitly examined rater reliability across previously completed ARV datasets.

The review found that judging agreement was far from perfect. Different judges sometimes produced different predictions from the same viewing material, indicating meaningful variability in how transcripts were interpreted. The paper argues that judge disagreement is itself an important variable influencing ARV performance rather than merely random noise.[Koestler Parapsychology Unit]koestler-parapsychology.psy.ed.ac.ukKPU 1034 Published ResultsKoestler Parapsychology Unitassociative remote viewing projects: assessing rater reliability…by DL KATZ · 2021 · Cited by 10 — Associa…

The same review also noted that previous researchers had already recognised this weakness decades earlier. Rather than treating judging as an unavoidable human judgement, several groups attempted to redesign the evaluation process to reduce subjective discretion.

Attempts to make judging more objective

Researchers have proposed several approaches intended to reduce disagreement between judges.

Structured scoring

Instead of asking judges simply which target “feels” like the better match, structured systems require them to answer predefined questions about specific features within the transcript.

For example, a judge may separately score the presence of:

• water
• artificial structures
• motion
• colour
• geometric forms
• living organisms

The overall prediction is then based on accumulated scores rather than an overall impression.

This approach attempts to replace intuitive matching with explicit criteria that another judge could apply in the same way.

Computer-assisted figure-of-merit systems

Edwin May and colleagues proposed reducing subjective judgement even further by converting transcript features into coded variables entered into a computer.

Instead of making the final decision directly, human coders completed structured coding sheets. Software then calculated a “figure of merit” score indicating how strongly each target matched the coded features. The goal was to minimise the influence of individual judgement styles.[ResearchGate]researchgate.netassociative remote viewing projects: assessing rater…10 Nov 2021 — Another approach to improve rater reliability developed…

Although this increased procedural consistency, it did not eliminate the initial human judgement required when coding the transcript.

Multiple independent judges

Another common safeguard is to use several blind judges rather than relying on one individual.

If several judges independently reach the same conclusion, confidence increases that the transcript genuinely favours one target.

However, this approach introduces another issue: if judges frequently disagree, researchers must decide how disagreements are resolved. Majority voting, averaging and consensus discussions each introduce different methodological assumptions.

Why this remains a difficult problem

Even carefully designed judging protocols cannot entirely remove ambiguity from natural language descriptions.

Two challenges repeatedly arise.

First, many remote-viewing impressions are symbolic rather than literal. A viewer might describe “explosion”, “brightness” or “celebration”, which could plausibly correspond to multiple photographs depending on interpretation.

Second, ARV deliberately pairs future outcomes with unrelated target photographs. The photograph associated with a sporting result or market movement often has no intrinsic connection to the event itself. Judges therefore evaluate only correspondence between transcript and image, not between transcript and the future event. Small interpretive differences can therefore reverse the resulting prediction.

These characteristics make judging fundamentally different from scoring a standard psychological test with objectively correct answers.

Practical implications for forecasting

For ARV to function as a reliable forecasting tool, two independent requirements must both hold:

• viewers must generate information related to the correct future target; and
• judges must extract that information consistently.

Failure at either stage reduces predictive performance.

This explains why some ARV practitioners regard judge training as nearly as important as viewer training. Several published analyses suggest that differences between judges can materially affect overall project success, while studies examining commercial or investment-based ARV have likewise treated judge selection as a potentially influential component of forecasting performance.[Academia+2Semantic Scholar]academia.eduAcademia(PDF) Associative Remote Viewing for Profit: Evaluating…This study examined the importance of the judge and the particular inv…

From a scientific perspective, however, this dependence also creates a challenge. If success depends heavily on particular judges whose decisions are difficult to replicate independently, it becomes harder to distinguish a genuine information signal from subjective interpretation.

How judging rules could become stricter

Researchers seeking stronger evidence generally recommend moving further away from intuitive matching and towards predefined, reproducible procedures. Possible improvements include:

• Publishing judging protocols before data collection begins.
• Using multiple fully independent blind judges.
• Measuring and reporting formal inter-rater reliability statistics rather than only prediction accuracy.

These changes cannot determine whether remote viewing itself is genuine, but they can reduce one important source of uncertainty. If blind judges consistently reach the same conclusions from the same material, confidence increases that any observed effect reflects the viewing data rather than variability in human interpretation. Conversely, persistent disagreement between equally qualified judges remains a significant methodological weakness because it limits both reproducibility and the practical usefulness of ARV predictions.

Amazon book picks

Further Reading

Books and field guides related to When Blind Judges Do Not Agree. Use these as the next step if you want deeper reading beyond the article.

Book

The Signal and the Noise

By Nate Silver

Useful background on separating signal from subjective interpretation.

See on Amazon

Book

Remote Viewers: The Secret History of America's Psychic Spies

By Jim Schnabel

First published 1997. Subjects: United states, department of defense, United states, central intelligence agency, Parapsychology.

See on Amazon

Book

The Art of Statistics

By David Spiegelhalter

First published 2019. Subjects: Statistics, Popular works.

See on Amazon

Book

Statistics Done Wrong

By Alex Reinhart

First published 2015. Subjects: Missing observations (Statistics), Methodology, Statistics.

See on Amazon

Browse more on Amazon:The Signal and the NoiseRemote Viewers: The Secret History of America's Psychic SpiesThe Art of Statistics

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

Remote Viewing Framed Art Print Framed Wall Art Poster Canvas Print Picture

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

PSIONIC REMOTE VIEWING Wall Art, Psychic Telepathy Medium Empath Meditate Poster

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

PSIONIC REMOTE VIEWING Wall Art, Psychic Telepathy Medium Empath Meditate Poster

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Example eBay listing

Retro REMOTE VIEWING Wall Art, Psychic Telepathy Medium Empath UFO Occult Poster

SearcheBay.co.uk: remote viewing poster

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180005-5.pdf

2. Source: researchgate.net
Link:https://www.researchgate.net/publication/356109550_ASSOCIATIVE_REMOTE_VIEWING_PROJECTS_ASSESSING_RATER_RELIABILITY_AND_FACTORS_AFFECTING_SUCCESSFUL_PREDICTIONS

3. Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00789R002200660001-5.pdf

4. Source: academia.edu
Link:https://www.academia.edu/56242943/Associative_Remote_Viewing_for_Profit_Evaluating_the_Importance_of_the_Judge_and_the_Investment_Instrument

5. Source: cia.gov
Link:https://www.cia.gov/readingroom/docs/CIA-RDP96-00791R000200180006-4.pdf

6. Source: cia.gov
Link:https://www.cia.gov/readingroom/document/cia-rdp96-00791r000200180006-4

7. Source: academia.edu
Link:https://www.academia.edu/Documents/in/associative_remote_viewing

8. Source: academia.edu
Link:https://www.academia.edu/figures/8545451/table-5-note-this-takes-all-scores-regardless-if-the-judge

9. Source: researchgate.net
Title: 374881423 Remote Viewing A 1974 2022 Systematic Review and [Meta Analysis]({{ ‘meta-analysis/’ | relative_url }})
Link:https://www.researchgate.net/publication/374881423_Remote_Viewing_A_1974-2022_Systematic_Review_and_Meta-Analysis

10. Source: researchgate.net
Link:https://www.researchgate.net/publication/344470071_THE_ASSOCIATIVE_REMOTE_DREAMING_EXPERIMENT_A_NOVEL_APPROACH_TO_PREDICTING_FUTURE_OUTCOMES_OF_SPORTING_EVENTS

11. Source: youtube.com
Title: Learn Remote Viewing
Link:https://www.youtube.com/watch?v=i62l3aktXpo

12. Source: youtube.com
Title: Associative Remote Viewing (ARV)
Link:https://www.youtube.com/watch?v=0yJxTpW6HAQ

13. Source: koestler-parapsychology.psy.ed.ac.uk
Title: KPU 1034 Published Results
Link:https://www.koestler-parapsychology.psy.ed.ac.uk/Documents/KPU_1034_Published_Results.pdf

14. Source: semanticscholar.org
Link:https://www.semanticscholar.org/paper/Associative-Remote-Viewing-for-Profit%3A-Evaluating-Kruth/91382b756c8e6f59f25f2d3b497eda53507168c0

15. Source: Wikipedia
Title: Remote viewing
Link:https://en.wikipedia.org/wiki/Remote_viewing

Additional References

16. Source: reddit.com
Link:https://www.reddit.com/r/consciousness/comments/181s71r/the_cias_experiments_with_remote_viewing_and/

17. Source: parapsychologypress.org
Link:https://www.parapsychologypress.org/jparticle/jp

18. Source: facebook.com
Link:https://www.facebook.com/Central.Intelligence.Agency/posts/has-cia-ever-studied-psychic-phenomena-or-remote-viewingits-a-question-we-get-as/5058607874154178/

19. Source: e-space.mmu.ac.uk
Title: mmu.ac.uk Follow‐up on the U.S
Link:https://e-space.mmu.ac.uk/631983/3/Brain%20and%20Behavior%20-%202023%20-%20Escol%20%E2%80%90Gasc%20n%20-%20Follow%E2%80%90up%20on%20the%20U%20S%20%20Central%20Intelligence%20Agency%20s%20%20CIA%20%20remote%20viewing.pdf

20. Source: slideshare.net
Link:https://www.slideshare.net/slideshow/an-evaluation-of-remote-viewing-research-and-applications-air1995pdf/257460594

21. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10275521/

22. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10921830/

23. Source: youtube.com
Title: The Science Behind Remote Viewing Explained
Link:https://www.youtube.com/watch?v=B9fv5diIi-Y

24. Source: scribd.com
Title: RVMeta Analysis
Link:https://www.scribd.com/document/668687232/RVMetaAnalysis

25. Source: ciaotest.cc.columbia.edu
Title: sa jan02srm01
Link:https://ciaotest.cc.columbia.edu/olj/sa/sa_jan02srm01.html