Arborist API Reference

Generated from docstrings. Replaces the static modules.md.

Contents:

Summary pages

• arborist — answers your AI can prove
• arborist — Merkle-Providence Reverse RAG, reference implementation
• Solution: a RAG pipeline for 10M+ docs with zero hallucination

Getting started

• Quickstart
  • Install
  • Path A — Wikipedia 2003 (canonical bootstrap dataset)
  • Path B — Crawl any live website and query it
  • After the answer
  • The query path
  • Where to next
• Concepts
  • What arborist is
  • The Merkle commitment
  • Federation across peers
  • The 8-dim cache key
  • The audit chain
  • The trichotomy and the four-rung ladder
  • The verifier (lexical, layered)
  • Falsification state
  • Sidecar diagnostics
  • Where to go next
• Cookbook
  • Re-crawl a website to detect changes
  • Falsify a wrong answer (audit-preserving)
  • Promote your own past answers into the corpus
  • Query across mixed corpora
  • Override the LLM endpoint
  • Verify shard integrity after a bulk operation
  • Run the QA bench and read the results
  • Tune retrieval per-question
• Use arborist as a Python library
  • Open a store and ingest documents you already hold
  • Define a custom Source for stateful corpora
  • Walk and verify the audit chain
  • Round-trip a Merkle inclusion proof
  • Run a Q&A from Python and read the result programmatically
• Benchmark surface
  • Quick reference
  • The four batteries
  • Cross-modality discipline
  • ForkScore consumes battery output
  • Authoring new fixtures

API Modules

• Makefile reference
  • Setup
  • Fetch
  • Ingest
  • Distill
  • Query
  • Verify and inspect
  • Operations on cached records
  • Tests and benches
  • Docs
  • Clean
  • Uncategorized
• Substrate: Core data structures
  • merkle
    • hash_leaf()
    • hash_combine()
    • ProofNode
    • MerkleProof
    • MerkleTree
    • verify_proof()
    • proof_to_dict()
    • proof_from_dict()
  • document
    • Edge
    • Document
    • canonicalize()
    • Chunker
    • TokenChunker
    • SentenceChunker
    • get_chunker()
  • wikitext
    • Versioning protocol
    • Algorithm (wikitext-base-v1)
    • to_base()
    • extract_wikilinks()
• Storage: SQLite schema and audit
  • store
    • invalidate_migration_cache()
    • connect()
    • discover_shards()
    • connect_query()
    • connect_readonly()
    • transaction()
    • get_meta()
    • set_meta()
    • latest_event_hash()
    • chain_audit_events()
    • append_audit()
    • stats()
  • ingest
    • IngestStats
    • ingest_source()
    • verify_random_sample()
  • evict
    • evict_to_cold()
    • rehydrate()
• Retrieval: FTS5 search and concepts
  • search
    • AuditMode
    • Hit
    • SearchBackend
    • FTS5Backend
  • sources
    • Source
    • WikipediaSqlDump
    • WikipediaCurDump
    • WikipediaOldDump
    • parse_html()
    • HtmlPageSource
  • concepts
    • add_concept_relation()
    • concept_relations_for_token()
    • has_compare_phrasing()
    • invalidate_cache()
    • purge_by_evidence_kind()
    • rivalry_excluded()
    • synonym_expand()
• Q&A Pipeline: question → answer → verify → cache
  • keys
    • canonical_question()
    • question_hash()
    • model_profile_hash()
    • conversation_hash()
    • governance_policy_hash()
    • verifier_policy_hash()
    • cache_key()
  • runner
    • ask()
  • query
    • query()
  • verify
    • extract_quotes()
    • extract_claim_spans()
    • extract_proper_nouns()
    • verify_quotes()
    • verify_claim_lattice()
    • evidence_map_by_evidence_id_local()
    • claim_lattice_structured_output_extras()
    • verify_claim_lattice_json()
  • evidence
    • EvidenceObject
    • build_evidence_map()
    • evidence_map_root()
    • render_evidence_block()
    • render_evidence_map()
    • render_evidence_block_for_json()
    • render_evidence_map_for_json()
    • evidence_map_by_pointer_id()
    • evidence_map_by_evidence_id()
    • render_claim_lattice()
  • quantifier
    • classify_question_quantifier()
  • metacognition
    • QuestionState
    • detect_temporal_sensitivity()
    • detect_contradiction()
    • detect_false_premise()
    • detect_out_of_corpus()
    • preflight_question()
  • dag
    • localize_failure()
    • build_preflight_node_payload()
    • preflight_node_hash()
    • build_run_dag()
    • build_reject_run_dag()
    • verify_run_dag()
  • client
    • ChatClient
    • StubClient
    • OpenAICompatibleClient
• Distillation: surface → core compression
  • DistillationResult
    • DistillationResult.core
    • DistillationResult.contributing_chunk_indices
  • Distiller
    • Distiller.name
    • Distiller.distill()
  • FirstSentenceDistiller
    • FirstSentenceDistiller.name
    • FirstSentenceDistiller.distill()
  • TfidfKeywordDistiller
    • TfidfKeywordDistiller.name
    • TfidfKeywordDistiller.distill()
• Federation: multiplayer arborist
  • MeshIdentity
    • MeshIdentity.member_id
    • MeshIdentity.sign_priv
    • MeshIdentity.sign_pub
    • MeshIdentity.dh_priv
    • MeshIdentity.dh_pub
    • MeshIdentity.group_name
    • MeshIdentity.created_at
  • MeshRosterEntry
    • MeshRosterEntry.member_id
    • MeshRosterEntry.sign_pub
    • MeshRosterEntry.dh_pub
    • MeshRosterEntry.role
  • aead_decrypt()
  • aead_encrypt()
  • current_epoch()
  • ecdh_shared_secret()
  • generate_dh_keypair()
  • generate_signing_keypair()
  • init_identity()
  • is_enabled()
  • load_identity()
  • set_enabled()
  • sign()
  • verify()
• CLI: Command-line interface
  • build_parser()
  • main()

Substrate theory (advanced)

The pages below formalize the Merkle-AGI substrate that drove arborist’s design and business logic (the research line behind the tickets and code). They are optional for users — nothing in Getting started or the API depends on them. Read them only if you want the formal model.

Substrate theory (advanced)

• Merkle-AGI v7 — Logic / Math Substrate (Plastic Training)
  • § 1 Introduction & Prior Failures
  • § 2 Model & Axioms
  • § 3 Merkle-DAG Commitments (Completeness-Binding)
  • § 4 Security Core — Long-Form Proofs
  • § 5 Quantization-Aware Semantics (Deterministic, Reproducible, Implementable)
  • § 9 Plastic Merkle-DAG Training
  • § 10 Verifiable Dictionary-Learning Circuits
  • § 11 Multimodal Composition
  • § 12 Benchmark Plan & Reproducibility (No simulated results)
  • § 13 Governance & Assurance Framework
  • § 14 Dual-Use Analysis (Unredacted) & Mitigations
  • § 15 Limitations (Candid)
  • § 17 Emergent Capabilities & Newly Observed Phenomena (Truth-only)
  • § 18 Conclusion
  • Appendix A — Canonical Encoding & Hashing (A1 · A3)
  • Appendix B — Deterministic Kernels & LUTs (A2)
  • Appendix C — Proof Schemas (JSON)
  • Appendix D — Plasticity (Dual Hash) & Separation
  • Appendix E — Dictionary Verification (SAE Features)
  • Appendix F — Bench Harness (Deterministic, No Simulated Results)
  • Appendix G — Threat Cases & Mitigation Playbooks
  • Appendix H — Reproducibility & Scripts
  • A Final De-Novo Verdict (What is proved, what is not)
  • C Subtle Edge Cases & Adversarial Corners (and how v7 handles them)
  • D Why v7-Local beats FOR/FIDC/SCMT (clean comparison)
  • F Multimodal Subtleties (where ε applies cleanly)
  • H Emergent/Unexpected Findings (upsides, downsides)
  • I Phase-2 Roadmap (private verification without weakening v7)
  • J Interop & Ontology (so this doesn’t become an island)
  • K Release Engineering & Auditor Checklist (cut-and-use)
  • L Minimal Mathematical Additions (tighten the paper)
  • M What we have not said earlier but matters (quiet truths)
  • N Epilogue (landmark, under-claimed on purpose)
• Merkle-AGI v7-W — Spatial-Temporal Substrate
  • Part 1 — Introduction & motivation
  • Part 2 — Substrate definition
  • Part 3 — Theorems
  • Part 4 — Verifier kernels
  • Part 5 — Multimodal composition with v7
  • Part 6 — Adversarial corners
  • Appendix — Worked example: toy SLAM
  • Out of scope (re-stated from #000013)
  • Closure
• Merkle-AGI v8 — Selection & Consensus Protocol
  • Part 1 — Introduction & motivation
  • Part 2 — Substrate definition
  • Part 3 — Validator state machine
  • Part 4 — Acceptance protocol
  • Part 5 — Challenge protocol
  • Part 6 — Stake mechanics
  • Part 7 — Fork choice rule (GRANDPA-style)
  • Part 8 — Mesh wire format extension
  • Part 9 — BFT analysis
  • Part 10 — Worked example (appendix)
  • Part 11 — Out of scope
  • Closure
• v8 ForkScore
  • Formula
  • Verdict thresholds
  • Default weights
  • CLI
  • What’s NOT in Phase 1a
• π* domain library
  • Registry overview
  • Cross-modality discipline
  • Math π*’s — SQD §14
  • CLI surfaces
  • Composition algebra
  • Authoring a new π*

Project

• License

Indices and tables

• Index

• Module Index

• Search Page

---

Permacomputer Preamble — License: AGPL-3.0-only

This is free software for the public good of a permacomputer hosted at permacomputer.com, an always-on computer by the people, for the people. Durable, easy to repair, & distributed like tap water for machine learning intelligence.

Our permacomputer is community-owned infrastructure optimized around four values:

• TRUTH — First principles, math & science, open source code freely distributed.

• FREEDOM — Voluntary partnerships, freedom from tyranny & corporate control.

• HARMONY — Minimal waste, self-renewing systems with diverse thriving connections.

• LOVE — Be yourself without hurting others, cooperation through natural law.

NO WARRANTY. Software is provided “AS IS” without warranty of any kind. Full text: License.