Leadership Under Uncertainty
Launch paper

The Judgment Layer: From Forecasting to Strategic Understanding in AI Systems

Christopher Berzins, Visiting Professor, Graduate School of Public and International Affairs, University of Ottawa.
Draft 13, September 2026. Correspondence: cberzins@uottawa.ca.
Download the PDF
A working draft, circulated for comment. Comments and corrections are welcome at the address above.

Abstract

Leading AI systems now forecast events about as accurately as superforecasters, the best human forecasters. This is an impressive and important achievement. Forecasting is nonetheless a narrow strategic skill. A forecast is only as good as the question asked, and it is a small part of how individuals and organizations build a strategic understanding of their environment and, ultimately, take decisions based on that understanding.

Leadership Under Uncertainty, an applied research project, aims to build AI systems demonstrably capable of strategic understanding and, ultimately, of judgment, for higher-value strategy work and AI safety, since judgment is key to responsible decision-making. For millennia people have grappled with strategic thinking under radical uncertainty, perhaps most systematically in the intelligence community. There, strategic assessment and judgment form a dynamic process, of which any single determination is one moment. How information is gathered and evaluated, how individual assessments are rolled up into broader judgments and, above all, how those judgments are systematically updated on new information and assessments matter far more than the accuracy of any single forecast. The project asks whether AI systems can replicate this process, whether it enhances strategic understanding, and whether a judgment layer measurably improves AI agents’ decisions.

Components of every stage are implemented, most partly operating, and the whole process is unproven. Measured on a small scale, 13 of 40 audited citations by the system supported their claims, and structured generation made insights better grounded and more testable without gain in decision relevance. A study of decisions holding forecasting ability constant is specified for collaborators to run.

Keywords. forecasting, strategic understanding, strategic assessment, judgment, belief updating, intelligence analysis, analytic standards, agentic AI, AI safety, decision-making under uncertainty

How to cite

Berzins, C. (2026). The Judgment Layer: From Forecasting to Strategic Understanding in AI Systems. Draft 13. Leadership Under Uncertainty research project, Graduate School of Public and International Affairs, University of Ottawa. https://leadershipunderuncertainty.org/papers/the-judgment-layer.pdf
Companion manuscripts

Six manuscripts the launch paper draws on

Each is at manuscript stage and available to collaborators on request. They will be listed here, with identifiers, as they are released.

The philosophical account
From Information to Judgment: A Judgment-First Account of Cognition in Bounded Agents
Manuscript, 2026.
Why the minimal unit of consequential cognition for an agent acting under uncertainty is a situated, revisable commitment: a claim held with a stated confidence, resting on attributed evidence, standing against represented alternatives and answerable to an outcome.
The shared representation
The Strategic Cognition Ontology: A Validation-Oriented Application Ontology for AI-Assisted Strategic Analysis
Manuscript, 2026.
The nine-element vocabulary for judgments under uncertainty, its strategic extensions, the machine-readable schema and the public validator against which artifacts are checked.
The evaluation instrument
SCB: A Two-Layer Benchmark for Strategic Cognition in AI Systems
Manuscript, 2026.
Outcome-scored forecasting alongside tradecraft-scored analysis, with the judge-reliability audit, the capped composite and the ladder of four rungs that grades what a judgment record can show.
The insight architecture
SIP: A Substrate-Grounded Architecture and Evaluation Methodology for Strategic Insight Generation in AI Systems
Manuscript, 2026.
The closed vocabulary of search patterns, the gates between generation, criticism and audit, and the evaluation design under which the architecture is compared with a single request to a model.
The organizational theory
AI for Judgment-Centric Organizations: A Research Program for Representing, Evaluating, and Governing Strategic Judgment
Manuscript, 2026.
The three conditions that define judgment-centric work, live and frozen judgment, and twenty-one pre-specified propositions, including the predictions that would test the objection that better models will make a judgment layer redundant.
The board application
SCO-Board: An Evidence-Gated Application Profile and Pre-Registered Evaluation Protocol for AI-Assisted Board Oversight
Manuscript, 2026.
The application profile for board and committee documents, the comparison against filter-matched prompting, the citation-support audit and a specified study of board oversight, designed to run with a participating board.
Research materials

What is public now

  • The vocabulary: JSON Schema, JSON-LD context, OWL ontology, SHACL shapes and concept vocabulary
    /sco/v1Implemented
  • The reference validator, which checks a record against the project’s full schema
    POST /api/v1/sco/validateImplemented
  • The sufficiency test, a structural check of whether a record carries a field for each of the nine elements
  • The agent handoff format
  • The anchors repository: daily roots, a sample seal and a script that walks the chain
  • Checking a single record against its seal
    /credibilityVerifier: ImplementedSealed record: Operating

Available on request until public release: the twenty-five-task starter evaluation, the sixty-instance conformance corpus, the handoff test dataset, the governance walkthrough and the queries of the operating audit in Appendix A.8 of the launch paper. Write to cberzins@uottawa.ca.