I keep a written checklist next to every release that touches this topic. The list is boring on purpose: name the owner, name the evidence artifact, and name the rollback. If a step cannot produce a file, a ticket, or a log line, it is not a step — it is a wish.
| Signal | What to record | Stop-ship? |
|---|---|---|
| Unexpected write or send | Tool name, args, identity | Yes until gated |
| Missing confirmation on destructive action | Call path and bypass | Yes |
| Untrusted text treated as instruction | Source surface and resulting action | Yes for agent reach |
| Scanner clean, judgment open | Deferred items + owner | Only if core path safe |
medguard helps with the measurable first pass. Architecture choices, legal sign-off, and customer-facing claims still need a human owner. Treat every checklist row as decision support, not a certificate.
When a prospect asks whether the product "covers everything," I answer with the boundary: automated checks catch known patterns; novel chains and policy judgment still need review. That honesty keeps support load down and keeps the blog aligned with what the product actually does.
If you need a second opinion, pair this write-up with the references below and with your own staging reproduction. Do not skip the reproduction step just because a model wrote a confident summary.
directly after? The instruction: "Keep the provided opening
EXACTLY as the first element. Do not delete the direct answer." Wait they said opening block (KEEP VERBATIM as first content):
. The earlier instruction: "Keep the provided opening
EXACTLY as the first element. Do not delete the direct answer." But the opening block they gave is
from the existing thin body? Let's read: "Opening block (KEEP VERBATIM as first content):
maybe? The existing thin body had a
I built MedGuard after seeing too many patient‑facing AI tools give advice that had not been checked for medical accuracy. The product works as a gate‑keeper that reads the output of a health AI before it reaches a user.
When a developer submits a model or a chatbot, MedGuard asks for the exact text that the AI would show to a patient. It then runs a series of checks on that text.
The goal is simple: catch any statement that could mislead a person about their health, such as an unverified claim about a treatment, a missing disclaimer about limitations, or a suggestion that conflicts with consensus guidelines.
I designed the evaluation as a pipeline that moves from raw text to a numeric risk score. Each step adds a layer of scrutiny and produces an intermediate artifact that the next step consumes.
The developer pastes the exact patient‑facing text into a form field or uploads a plain‑text file. MedGuard stores the string unchanged and computes a hash so we can detect if the same input is submitted twice.
Here the system scans the text for statements that look like medical advice. Patterns include imperative verbs (take, use, avoid) followed by a noun phrase that names a drug, supplement, device, or procedure. The extractor also looks for conditional language such as “if you have X, then Y might help.”
To guard against manipulation, I consulted the OWASP LLM01 prompt‑injection guide https://genai.owasp.org/llmrisk/llm01-prompt-injection and added rules that ignore text hidden inside code blocks or markdown comments.
Each extracted claim is sent to a lightweight lookup service that checks against a curated set of sources: FDA labels, NIH clinical trials registry, and major society guidelines. The service returns a boolean indicating whether the claim appears in at least one source with a level‑A evidence rating.
MedGuard looks for phrases that limit liability, such as “this is not medical advice” or “consult a clinician.” If the text lacks any of these phrases, the system flags a missing disclaimer. The wording of the disclaimer is not judged for legal sufficiency; only its presence is recorded.
I aligned this check with the NIST AI Risk Management Framework https://www.nist.gov/itl/ai-risk-management-framework which recommends explicit communication of system limits to users.
| , | . Provide maybe three rows: Factor, Weight, Example impact.
We'll produce:
Required inputs and formatMedGuard expects a single UTF‑8 encoded text file. The file must contain only the patient‑facing copy that the AI would show. No code, no HTML, no JSON wrappers. Accepted formats
Optional metadataYou may add a header line that starts with # followed by key‑value pairs. MedGuard ignores these lines when scoring but stores them for audit.
What this is NOThonesty. We need to state limitations: does not guarantee safety, does not replace clinical review, does not catch every possible error, etc. We must avoid phrase "more than". Use "not just". Also avoid "guarantee"? The word "guarantee" is not banned. It's okay. But we must not say "100%" maybe not banned but we can avoid. We must not use "useful change", "current", "in today's". Avoid. Let's write.What this is NOTMedGuard is a screening aid, not a substitute for a qualified clinician’s review. It does not promise that every unsafe slip will be caught; some nuanced misstatements may pass through if they resemble legitimate phrasing. The tool does not evaluate the underlying model weights or training data. It only looks at the static output you give it, so a model that changes its answer based on context could still produce harmful text that MedGuard never sees. MedGuard does not provide legal advice about disclaimer wording. It merely checks for the presence of a short phrase; a lawyer must decide if that phrase meets local regulations. What is MedGuard? paragraphs:
Paragraph 1: "I built MedGuard after seeing too many patient‑facing AI tools give advice that had not been checked for medical accuracy. The product works as a gate‑keeper that reads the output of a health AI before it reaches a user."
Check words: "built", "after", "seeing", "too", "many", "patient‑facing", "AI", "tools", "give", "advice", "that", "had", "not", "been", "checked", "for", "medical", "accuracy", "The", "product", "works", "as", "a", "gate‑keeper", "that", "reads", "the", "output", "of", "a", "health", "AI", "before", "it", "reaches", "a", "user". No banned.
Paragraph 2: "When a developer submits a model or a chatbot, MedGuard asks for the exact text that the AI would show to a patient. It then runs a series of checks on that text."
Check: "When", "a", "developer", "submits", "a", "model", "or", "a", "chatbot", "MedGuard", "asks", "for", "the", "exact", "text", "that", "the", "AI", "would", "show", "to", "a", "patient", "It", "then", "runs", "a", "series", "of", "checks", "on", "that", "text". No banned.
Paragraph 3: "The goal is simple: catch
|
|---|