Abstract
Incidental findings in abdominal CT scans pose a significant clinical challenge, demanding systematic identification and management to prevent missed diagnoses and unnecessary follow-up procedures. We introduce INFORM-CT, an agentic pipeline that integrates Large Language Models (LLMs) and Vision-Language Models (VLMs) for automated incidental findings management in abdominal CT. INFORM-CT operates in three stages: (1) structured labeling of radiology reports using LLM-based agents guided by clinical decision trees, producing validated ground-truth annotations; (2) automated generation of organ-specific clinical decision programs by parsing medical guidelines with a planner-executor framework; and (3) executing these programs on CT scans using VLMs to generate actionable clinical recommendations. We evaluate INFORM-CT across multiple abdominal organs including liver, pancreas, kidney, and lung, demonstrating strong performance on incidental findings detection and clinical recommendation generation, and outperforming vanilla VLM baselines.
Pipeline Overview
Stage 1: Data Labeling
Structured labeling of radiology reports using LLM-based agents guided by clinical decision trees. Produces validated ground-truth annotations using Pydantic-enforced schemas.
Stage 2: Program Generation
Automated generation of organ-specific clinical decision programs by parsing medical guidelines with a planner-executor framework powered by multimodal LLMs.
Stage 3: Execution
Executing generated programs on CT scans using Vision-Language Models (MERLIN) to produce actionable clinical recommendations with decision paths.
Key Features
- Multi-Organ Support — Handles incidental findings across liver, pancreas, kidney, lung, and more.
- Schema-Driven Annotations — Pydantic-enforced structured outputs guarantee 100% format compliance for ground-truth labeling.
- Planner-Executor Framework — Automatically generates organ-specific clinical decision programs from medical guideline PDFs.
- Agentic Pipeline — LLM-based agents parse, review, and refine decision trees from multimodal guideline documents.
- VLM Integration — Leverages MERLIN vision-language model for efficient CT image encoding and feature extraction.
- Comprehensive Evaluation — Includes tools for decision-path accuracy, F1 scoring, confusion matrix analysis, and VANILLA MERLIN baseline comparison.
BibTeX
@InProceedings{pmlr-v315-tankel26a,
title = {INFORM-CT: INtegrating LLMs and VLMs FOR Incidental Findings Management in Abdominal CT},
author = {Tankel, Idan and Mazor, Nir and Brada, Rafi and Lebedis, Christina and Ben-Yosef, Guy},
booktitle = {Proceedings of The 9th International Conference on Medical Imaging with Deep Learning},
pages = {2461--2473},
year = {2026},
volume = {315},
series = {Proceedings of Machine Learning Research},
month = {08--10 Jul},
publisher = {PMLR},
pdf = {https://raw.githubusercontent.com/mlresearch/v315/main/assets/tankel26a/tankel26a.pdf},
url = {https://proceedings.mlr.press/v315/tankel26a.html}
}