How to build a clinical scribe agent
Free blueprint for a clinical scribe agent. A code-first, single-agent, self-hosted stack with 14 connected components. Built around Nuance DAX, Glass Health, Vanna AI, LangChain. Copy it, export it, or change the answers to fit your project.
Healthcare scribe & intake agent
A code-first, single-agent, self-hosted stack with 14 connected components.
Architecture map
Tap a component to see its connections
Swipe sideways to see the whole map
Tool configuration
Interface & triggers
Chat interface
Where people talk to the agent
Web chat widgetor Slack or Teams bot
Orchestration
Agent framework
Agent loop, tool calling and state in code
LangChainor LlamaIndex, CrewAI
Human approval step
Pauses risky actions until a person approves
Built into your orchestratoror n8n wait node, Slack approval message
Reasoning
Open-weight model
Reasoning model that never leaves your servers
DeepSeekor Mistral, Qwen
Model server
Runs the model on your own hardware
vLLMor Ollama
Memory & knowledge
Conversation memory
Keeps track of the current chat or task
Built into your orchestratorDocument ingestion
Splits and indexes your files and pages
LlamaIndexor Firecrawl
Vector database
Finds the most relevant passages for each question
Qdrantor ChromaDB, Weaviate
Tools & integrations
Database / SQL
Answers questions by querying your data
Vanna AIor Direct SQL tool, Julius AI
Clinical note-taker
Drafts visit notes from the doctor–patient conversation
Nuance DAXClinical reference
Suggests possibilities for a clinician to check
Glass HealthHealth records
Patient records — read-only to start
Your EHR (Epic, Cerner…)
Monitoring & safety
Guardrails
Blocks unsafe input, prompt injection and data leaks
Input & output checksor Allow-list of tool actions
Tracing & evaluation
Records every step, cost and answer quality
Langfuseor LangSmith, Weights & Biases
Set up your agent, step by step
Tick each step as you finish it. Your progress is saved with this plan in your AItooling account.
0 of 24 done
AItooling never signs up for you and never asks for your passwords or keys. You create each account yourself on the tool's own website, and only your ticks are saved here.
Sign in and save this plan to keep your progress in your account.
1Create your accounts
Sign up or install it, then make a new project for this agent.
Install it on your own server. No account is needed, and your data stays with you.
Install it on your own server. No account is needed, and your data stays with you.
Sign up and create a new database or index just for this agent.
Sign up and create a new database or index just for this agent.
Sign up and connect only the apps this agent needs, with read-only access where you can.
Sign up and connect only the apps this agent needs, with read-only access where you can.
Sign up and connect only the apps this agent needs, with read-only access where you can.
Use the account you already have, and give the agent its own login with only the access it needs.
Sign up and turn on logging so you can review what the agent does.
2Build it in this order
3Launch your agent
Copy the ready-made code for this agent. It hides secrets and personal data before the AI model sees them.
Put each key in a password manager or your hosting's secret settings. Never paste keys or passwords into AItooling or a chat.
Start small, read every answer yourself, and only then let other people use it.
Want to go deeper?
Read our step-by-step guides on building agents. Browse guides