How does MedGuard check health AI?

Operator checklist for How does MedGuard check health AI?

I keep a written checklist next to every release that touches this topic. The list is boring on purpose: name the owner, name the evidence artifact, and name the rollback. If a step cannot produce a file, a ticket, or a log line, it is not a step — it is a wish.

Before you change production

- Freeze the branch or config revision you will measure against. - Record the current failure mode with one concrete input and one concrete output. - Confirm which identity the job runs as, and whether that identity can reach data it should not. - Open the primary references you will cite: https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/europe-fit-digital-age_en, https://genai.owasp.org/llmrisk/llm01-prompt-injection.

While you run the check

- Prefer one reproducible path over ten anecdotal screenshots. - Write the expected safe outcome before you trigger the tool or query. - Capture the exact command, prompt, schema, or regex under test — not a paraphrase. - If the system can act (send, write, delete, transfer), require a confirmation boundary in the same pass.

After you finish

- File residual risk with an owner and a date, even when the scan is clean. - Do not convert a green check into sales language that claims completeness. - Re-run the same checklist when the model, tool schema, or validation rules change.

Signal | What to record | Stop-ship? Unexpected write or send | Tool name, args, identity | Yes until gated Missing confirmation on destructive action | Call path and bypass | Yes Untrusted text treated as instruction | Source surface and resulting action | Yes for agent reach Scanner clean, judgment open | Deferred items + owner | Only if core path safe

medguard helps with the measurable first pass. Architecture choices, legal sign-off, and customer-facing claims still need a human owner. Treat every checklist row as decision support, not a certificate.

How I explain limits to buyers

When a prospect asks whether the product "covers everything," I answer with the boundary: automated checks catch known patterns; novel chains and policy judgment still need review. That honesty keeps support load down and keeps the blog aligned with what the product actually does.

If you need a second opinion, pair this write-up with the references below and with your own staging reproduction. Do not skip the reproduction step just because a model wrote a confident summary.

What is MedGuard?

I built MedGuard after seeing too many patient-facing AI tools give advice that had not been checked for medical accuracy. The product works as a gate-keeper that reads the output of a health AI before it reaches a user.

When a developer submits a model or a chatbot, MedGuard asks for the exact text that the AI would show to a patient. It then runs a series of checks on that text.

The goal is simple: catch any statement that could mislead a person about their health, such as an unverified claim about a treatment, a missing disclaimer about limitations, or a suggestion that conflicts with consensus guidelines.

How MedGuard evaluates health AI

I designed the evaluation as a pipeline that moves from raw text to a numeric risk score. Each step adds a layer of scrutiny and produces an intermediate artifact that the next step consumes.

Step 1: Input capture

The developer pastes the exact patient-facing text into a form field or uploads a plain-text file. MedGuard stores the string unchanged and computes a hash so we can detect if the same input is submitted twice.

Step 2: Claim extraction

Here the system scans the text for statements that look like medical advice. Patterns include imperative verbs (take, use, avoid) followed by a noun phrase that names a drug, supplement, device, or procedure. The extractor also looks for conditional language such as “if you have X, then Y might help.”

To guard against manipulation, I consulted the OWASP LLM01 prompt-injection guide https://genai.owasp.org/llmrisk/llm01-prompt-injection and added rules that ignore text hidden inside code blocks or markdown comments.

Step 3: Evidence lookup

Each extracted claim is sent to a lightweight lookup service that checks against a curated set of sources: FDA labels, NIH clinical trials registry, and major society guidelines. The service returns a boolean indicating whether the claim appears in at least one source with a level-A evidence rating.

Step 4: Disclaimer check

MedGuard looks for phrases that limit liability, such as “this is not medical advice” or “consult a clinician.” If the text lacks any of these phrases, the system flags a missing disclaimer. The wording of the disclaimer is not judged for legal sufficiency; only its presence is recorded.

I aligned this check with the NIST AI Risk Management Framework https://www.nist.gov/itl/ai-risk-management-framework which recommends explicit communication of system limits to users.

Step 5: Risk scoring

Required inputs and format

MedGuard expects a single UTF-8 encoded text file. The file must contain only the patient-facing copy that the AI would show. No code, no HTML, no JSON wrappers.

Accepted formats

- Plain .txt file - Copy-and-paste into the web textarea (max 10 KB) - Email attachment with mime type text/plain

Optional metadata

You may add a header line that starts with # followed by key-value pairs. MedGuard ignores these lines when scoring but stores them for audit.

- # version: 1.2.0 - # audience: adults with diabetes - # language: en-US

What this is NOT

MedGuard is a screening aid, not a substitute for a qualified clinician’s review. It does not promise that every unsafe slip will be caught; some nuanced misstatements may pass through if they resemble legitimate phrasing.

The tool does not evaluate the underlying model weights or training data. It only looks at the static output you give it, so a model that changes its answer based on context could still produce harmful text that MedGuard never sees.

MedGuard does not provide legal advice about disclaimer wording. It merely checks for the presence of a short phrase; a lawyer must decide if that phrase meets local regulations.

Frequently asked questions

How does MedGuard check health AI?

It reviews the exact patient-facing text a health AI would show, runs safety and evidence checks, and returns a visible risk signal before that message reaches a person.

References

- Plain language / decision-support framing - FTC advertising guidance

Try MedGuard

Related reads

- What is MedGuard? - The LX AI Micro-SaaS Topic Matrix