Specialist Review of Harm Ratings for Clinical Responses Acronym: NOHARM

The purpose of this study is to determine how closely potential-harm ratings assigned using the NOHARM rubric agree with independent ratings from specialty-matched physicians. The study uses reconstructed clinical consultation cases and previously generated responses written by artificial intelligence systems or physicians. The materials contain no patient identifiers or protected health information. Board-certified attending physicians from 10 medical specialties will review assigned case-response pairs through a secure website. For each response, they will indicate whether it contains an error, describe the error when applicable, and rate the greatest potential patient harm as none, mild, moderate, or severe. Reviewers will not be shown the response source, existing rubric rating, or other reviewers' ratings. Two specialists will initially review each response; a third may review it if the first two ratings differ. The study includes a development stage, possible rubric revision, and a separate validation stage.

Inclusion Criteria: * Licensed or registered physician in good standing * Attending- or consultant-level physician actively practicing in allergy and immunology, cardiology, dermatology, endocrinology, gastroenterology, hematology, infectious disease, nephrology, neurology, or pulmonology * Practicing in the United States, Canada, United Kingdom, or Australia * Assigned clinical materials match the physician's specialty * Able to read and respond in English * Access to a computer with a stable internet connection * Willing and able to complete at least one remote specialist-rating engagement of approximately three hours Exclusion Criteria: * Does not meet the professional, career-level, specialty, geographic, or credential-verification requirements above * Prior involvement with assigned cases, responses, or rubric information that could compromise independent rating * Unable to complete the remote rating activities in English * Unable or unwilling to follow the independent-rating procedures * For validation-stage assignments, participation in the rubric-revision process