How can I reliably identify AI-generated content in a resident's clinical note?

Look for specific linguistic patterns, structural anomalies, and discrepancies between the note and the resident's verbal presentation. Language Patterns: AI tends to use formal, complete sentences, excessive synonyms (e.g., "the patient is a pleasant gentleman"), and avoids institutional shorthand, abbreviations, or the terse phrasing typical of busy residents. Verbal Disconnect: Ask the resident to summarize the patient's story verbally without looking at the note. If they struggle, hesitate, or contradict what is written, the AI likely did the heavy lifting. See more information on whether your residents are over-relying on AI for notes.

Is it safe to let residents use AI for differential diagnoses during high-acuity on-call shifts?

Yes, but strictly as a second opinion after the resident commits to their independent differential, never as a primary decision-maker during time-critical or unstable patient encounters. Escalation Protocol: AI is useful for jogging memory on well-known guidelines, but it should never substitute for escalating clinical concerns directly to the senior attending or rapid response team. Best Practice: Supervising attendings should review these AI-assisted differentials the next morning as a dedicated teaching case, analyzing specifically what the AI caught versus what clinical intuition dictated, and why the discrepancy occurred.

What main elements must be included in a residency program's AI use policy?

An effective policy emphasizes transparency, specific clinical use, mandatory verification protocols, and clear consequences for undisclosed or unsupervised AI reliance. Transparency: Residents must disclose when and how AI was used (e.g., a mandatory checkbox in the EHR, a sentence in the note stating "AI used for formatting/literature review only," or a dedicated disclosure field). Accountability Framework: Formalize that residents are personally and professionally accountable for all content submitted under their name. AI output is a draft, never a final sign-off without independent validation and editing. Educational Accountability: Include AI literacy (teaching residents how to critique, verify, and safely integrate AI output) as a core competency during orientation, monthly didactics, and semi-annual evaluations. Best Practice: Treat undisclosed, unsupervised AI use as a serious transgression requiring immediate remediation and clear educational consequences, not merely a casual warning. See more information on supervised AI use in medical education.

Supervising Resident AI Use Without Losing the Learning

For the first time in medical education, residents can utilize an AI clinical scribe tool that generates comprehensive notes and differentials in seconds, yet efficiency does not equal expertise. As clinical artificial intelligence becomes more common in the wards, supervisors face a dilemma: how to harness AI's productivity gains without sacrificing the critical thinking that builds clinical judgment skills. This article presents a framework for transforming AI clinical notes from a shortcut into a supervised teaching partner.

Understanding the Dual Nature of Clinical AI: A Quick Overview

The integration of AI clinical notes presents two opposing sides that educators must actively balance:

The Efficiency Promise:

Offloads clerical burden (charting, note formatting, literature searches).
Frees mental capacity for direct patient interaction and complex problem-solving.
Accelerates access to rare differentials and updated guidelines.

The Threat to Learning:

Automation Bias: Residents who accept AI suggestions without independent verification weaken their critical thinking skills.
Illusion of Competence: A polished AI-generated note masks gaps in the resident's actual understanding.

A Framework for Resident Supervision

To transform AI clinical notes from a shortcut into a learning tool, supervision must shift from monitoring access to a monitoring process:

1. Establish an Educational Contract

Clearly communicate that AI is permitted only as a confirmation tool, never as an initiator of clinical decisions.
Define specific phases where AI is prohibited versus where it is encouraged.
Mandate Transparency: Residents must disclose when and how AI was used in their workflow.

2. Implement the "Think-Aloud" Protocol

Residents verbalize their clinical reasoning before consulting AI output.
Attendings assess the quality of the resident's independent thought, then compare it against AI suggestions.
This creates a "teachable discrepancy" moment, revealing whether the AI adds value or merely reinforces existing biases.

See more in-depth information on AI bias.

3. Shift Evaluation Criteria

Judge residents on their process (how they questioned the AI, what they verified) rather than the product (the final note).
Reward skepticism and verification as core clinical competencies.

The "First Draft" Rule: Preserving the Differential

What this rule entails is that the resident must write the first draft of the medical decision‑making (MDM), and AI may only edit, format, or challenge it afterward. This preserves the clinical reasoning essential for medical care.

The "Draft"

Residents write the full Assessment and Plan section manually before consulting any AI tool.

AI is then permitted only for:

Formatting or grammar improvement.
Generating a structured differential to compare against their own.
Retrieving literature or guideline references to support their existing plan.
Prohibited: Asking AI to generate the MDM from scratch or to "suggest" a diagnosis before the resident has committed to their own.

The Attending’s Role in the Review

Once the resident has produced their independent draft and consulted AI for comparison, the attending's supervision shifts from observation to active review:

1. Deconstruct the AI Output

Do not accept the final note at face value. Ask questions:

"The AI suggested [X diagnosis]. Did you consider that? Why or why not?"
"What clinical clues in this patient's presentation support your diagnosis over the AI's suggestion?"

This forces the resident to defend their reasoning while learning to critique AI‑generated content.

2. Identify "Hallucinations" as Teaching Moments

AI frequently invents medication dosages, misinterprets lab values, or invents plausible‑but‑wrong clinical details, also known as hallucinations.

When an error is spotted, reframe it as a teaching opportunity:

"This prescription doesn't exist; how would you verify it?"
"The AI missed this physical exam finding. Why is that finding critical here?"

This builds habits of source verification and clinical skepticism.

Conclusion

The question facing medical educators and attendings is no longer whether residents will use AI, but how well they will be supervised in doing so. By implementing structured frameworks like the "First Draft" Rule and emphasizing transparent, critical engagement, programs can utilize AI's efficiency without sacrificing clinical reasoning. The ultimate goal remains unchanged: producing physicians who think deeply and prioritize the best care for their patients.

References

China, C. R. (2023, September 1). What Are AI Hallucinations? IBM.

Shaked, J. (2026, March 16). After-Hours Electronic Health Record Use Associated With Resident Burnout. Yale School of Medicine.

Thompson, R., Shah, Y., Aquirre, F., Stewart, C., Lallas, C., & Shah, M. (2025, May 29). Artificial Intelligence Use in Medical Education: Best Practices and Future Directions. Springer, 26(1).

FAQ

Frequently asked questions

How can I reliably identify AI-generated content in a resident's clinical note?
Look for specific linguistic patterns, structural anomalies, and discrepancies between the note and the resident's verbal presentation.
- Language Patterns: AI tends to use formal, complete sentences, excessive synonyms (e.g., "the patient is a pleasant gentleman"), and avoids institutional shorthand, abbreviations, or the terse phrasing typical of busy residents.
- Verbal Disconnect: Ask the resident to summarize the patient's story verbally without looking at the note. If they struggle, hesitate, or contradict what is written, the AI likely did the heavy lifting.
See more information on whether your residents are over-relying on AI for notes.
Is it safe to let residents use AI for differential diagnoses during high-acuity on-call shifts?
Yes, but strictly as a second opinion after the resident commits to their independent differential, never as a primary decision‑maker during time‑critical or unstable patient encounters.
- Escalation Protocol: AI is useful for jogging memory on well-known guidelines, but it should never substitute for escalating clinical concerns directly to the senior attending or rapid response team.
- Best Practice: Supervising attendings should review these AI-assisted differentials the next morning as a dedicated teaching case, analyzing specifically what the AI caught versus what clinical intuition dictated, and why the discrepancy occurred.
What main elements must be included in a residency program's AI use policy?
An effective policy emphasizes transparency, specific clinical use, mandatory verification protocols, and clear consequences for undisclosed or unsupervised AI reliance.
- Transparency: Residents must disclose when and how AI was used (e.g., a mandatory checkbox in the EHR, a sentence in the note stating "AI used for formatting/literature review only," or a dedicated disclosure field).
- Accountability Framework: Formalize that residents are personally and professionally accountable for all content submitted under their name. AI output is a draft, never a final sign-off without independent validation and editing.
- Educational Accountability: Include AI literacy (teaching residents how to critique, verify, and safely integrate AI output) as a core competency during orientation, monthly didactics, and semi-annual evaluations.
- Best Practice: Treat undisclosed, unsupervised AI use as a serious transgression requiring immediate remediation and clear educational consequences, not merely a casual warning.
See more information on supervised AI use in medical education.