We evaluate the Code's Principles and Commitments for their clarity, relevance, completeness, conciseness, robustness and adaptability by Ann Novakowski, Megan Doerr, Drew Duglan, and Susheel Varma
Published on Jun 19, 2024. DOI 10.21428/4f83582b.4693dec3
There are significant trade-offs with any disruptive technology.
The explosion in artificial intelligence is being increasingly leveraged in biomedical discovery, clinical decision-making and patient care. But with these benefits come significant potential risks to human safety, privacy, and justice.
To help us realize the benefits and mitigate the associated risks, we need a variety of voices at the table who can offer different perspectives on these pressing issues.
In an attempt to address these issues, the National Academy of Medicine’s Steering Committee on AI in Health, Health Care, and Biomedical Science recently drafted a Code of Conduct framework, which they have since shared for public comment. After responding officially to their request for stakeholder input, we are now summarizing our perspectives here and inviting further community discussion.
What are your perspectives on the following framework? We’d love to hear from you in the discussion thread below.
The framework is split into a proposed Code of Principles and a proposed Code of Commitments:
The Code of Principles embeds the essential values underpinning responsible behavior in AI development, use and ongoing monitoring.
The Code of Commitments is intended to support the application of these Principles in practice, promoting governance at every level.
Each of the Principles and Commitments were evaluated for their clarity, relevance, completeness, conciseness, robustness and adaptability.
Image by vectorjuice on Freepik
Code of Principles
The framework contains 10 stated Principles:
Engaged: Understanding, expressing, and prioritizing the needs, preferences, goals of people, and the related implications throughout the Al life cycle.
Our Thoughts: It will be important to define “who” the people are in this statement as different individuals will have different needs and motivators. The authors highlighted the criticality of inclusive collaboration, and the principle could embody this more directly.
Safe: Attendance to and continuous vigilance for potentially harmful consequences from the application of Al in health and medicine for individuals and population groups.
Our Thoughts: While the Code of Conduct is excellent at highlighting some of the potential harms, the rubric for ongoing vigilance needs to be further expanded.
Effective: Application proven to achieve the intended improvement in personal health and the human condition, in the context of established ethical principles.
Our Thoughts: Further definition of “the intended improvement” is needed to increase the robustness and completeness of this statement.
Equitable: Application accompanied by proof of appropriate steps to ensure fair and unbiased development and access to Al-associated benefits and risk mitigation measures.
Our Thoughts: Who would be defining “by proof of appropriate step?” We do not want a “reasonable person” standard as guidance for such ethically fraught technologies.
Efficient: Development and use of Al associated with reduced costs for health gained, in addition to a reduction, or at least neutral state, of adverse impacts on the natural environment.
Our Thoughts: While some data centers may be working toward being carbon-free by 2030, the broader data ecosystem is struggling under sustainability issues related to data volume, processing power, and the cost of egress/compute. Creating trusted repositories, where data may be stored and interoperability achieved, can help lighten this burden on the research community. We are eager to see criteria for evaluating how cost reductions compare to improvements in health outcomes.
Accessible: Ensuring that seamless stakeholder access and engagement is a core feature of each phase of the Al life cycle and governance.
Our Thoughts: While it’s important to ensure seamless access, it’s equally important to ensure that models developed from sensitive health data do not pose individual privacy risks or broader privacy risks to the general population (i.e., group harm). We need to consider gated models (in addition to gated data) that strike a balance between openness and utility. “Stakeholders” also need to be clearly defined.
Transparent: Provision of open, accessible, and understandable information on component Al elements, performance, and their associated outcomes.
Our Thoughts: This principle specifies “understandable,” but understandable to whom? Again, we have learned many lessons on understandability (or lack thereof) from informed consent and more.
Accountable: Identifiable and measurable actions taken in the development and use of Al, with clear documentation of benefits, and clear accountability for potentially adverse consequences.
Our Thoughts: Would we be waiting on the courts to define for us “clear accountability for potentially adverse consequences”, or could we specify here a clear chain of funding, ownership and liability?
Secure: Validated procedures to ensure privacy and security, as health data sources are better positioned as a fully protected core utility for the common good, including use of Al for continuous learning and improvement.
Our Thoughts: Again, the concept of “common good” must go beyond the standard of a “reasonable person” to avoid perpetuating existing and institutionalized bias and/or harm. It will be critical to have validated procedures that are robust and can be applied throughout the learning process.
Adaptive: Assurance that the accountability framework will deliver ongoing information on the results of Al application, for use as required for continuous learning and improvement in health, health care, biomedical science, and, ultimately, the human condition.
Our Thoughts: As machines continue to learn from new data, populations, and/or environment, this continuous learning and improvement cycle is more critically important than ever. This principle is paramount, but lacks punch in its current state. How are we meaningfully going to do this?
Image by vectorjuice on Freepik
Code of Commitments
The framework then contains 6 stated Commitments:
Focus: Protect and advance human health and human connection as the primary aims.
Our Thoughts: Our only concern here is determining who decides if an application is consistent with this Commitment. This should be specified, even in general terms (e.g., regulator, oversight board, professional societies) to ensure the efficacy of this guidance.
Benefits: Ensure equitable distribution of benefit and risk for all.
Our Thoughts: It will be important to include a broad range of “beneficiaries.” In determining benefits, who should be invited, how are they selected and engaged? Whose voices are deemed important? And what are the criteria for evaluation?
Involvement: Engage people as partners with agency in every stage of the life cycle.
Our Thoughts: “People as partners with agency” needs further specification. In what ways are they partners and to whom (e.g. AI developers, funders, or healthcare providers using these tools)?
Workforce well-being: Renew the moral well-being and sense of shared purpose to the healthcare workforce.
Our Thoughts: Who defines “moral well-being”? The workforce is broad and varied, with very different expectations for how well-being is renewed and/or achieved. This Commitment will be difficult to measure and will need to be more forthright about its intention.
Monitoring: Monitor and openly and comprehensibly share methods and evidence of Al's performance and impact on health and safety.
Our Thoughts: Who will act as monitors - those using the AI application, those developing, or those purchasing for use? Would “openly” mean via a central audit repository, through a private website, or through written documentation that can be requested? And who would be able to comprehend all this (e.g. industry, healthcare providers, patients)?
Innovation: Innovate, adopt, collaboratively learn, continuously improve, and advance the standard of clinical practice.
Our Thoughts: Indeed, this is a shared goal and the hope for our future.
Image by vectorjuice on Freepik
In Summary
Overall, we compliment the authors on the relevance, brevity and adaptability of the principles articulated in this framework. However, many, if not most, of the principles should be more robust and complete, which would then also cement their clarity.
We recognize this is a tough balance to strike between all criteria. But we have learned valuable lessons about the necessity of completeness from initiatives such as the revision of the Common Rule. This emphasized the importance of diligent and continuous oversight when it comes to human protections in research and healthcare.
These lessons should loom large in the authors’ minds as they revisit this draft.
Finally, we should learn from the insights of others who have already attempted to create similar frameworks. For example, the intergovernmental Group of Seven (G7) and the UK’s National Health Service have produced analogous principles, codes of conduct and implementation guidance that can inform the National Academy of Medicine and help refine their current draft.
If these governance frameworks remain incomplete, so too will the vision of AI-enabled health for all.