This blog post expands on ideas discussed in an earlier series of blog posts on ethics in social science research. Read the introduction to that series here and read about 3ie's Transparent, Reproducible, and Ethical Evidence framework here.
There is a widely agreed upon norm in clinical research ethics that participants with non-trivial medical conditions should not be subject to interventions that are reasonably expected to be inferior to the ‘standard of care.’ Originally adopted from a legal term used to identify the level of care healthcare providers must meet to avoid negligence, the standard of care is the level and type of care to which research participants with a particular condition have a moral claim. The ‘standard of care’ norm operates to ensure that participants in clinical trials do not receive care expected to be inferior to that which they are morally entitled.
In this post, we ask whether policy research and evaluation should be subject to an analogous ‘standard of care’ norm. Focusing on research and evaluation that is conducted or authorized by government agencies, we argue that it should. Governments have duties of justice to protect people’s rights and provide them access to certain goods and services. Therefore governments may not conduct or authorize research or evaluations in which these duties go unfulfilled.
To organize our thinking and see what’s at stake, consider an example of an evaluation related to Medicaid, a joint federal and state program in the United States with the primary objective of providing health coverage to low-income Americans and other vulnerable populations. In this case, a state-level proposal would have changed eligibility requirements for Medicaid. Regardless of geography, this example is relevant for any policymaker, evaluator, and researcher who has considered how to adjust programs and policies based on evolving objectives.
On January 12, 2018, the U.S. federal government approved a waiver under the Section 1115 Medicaid demonstration program for a new policy in the state of Kentucky called Kentucky HEALTH. Section 1115 of the Social Security Act gives the Secretary of Health and Human Services the authority to waive states’ compliance with requirements of the Act in order to conduct demonstration projects that are likely to promote the objectives of Medicaid.
Among other changes, Kentucky HEALTH introduced a new eligibility requirement for Medicaid coverage: all ‘able-bodied’ adults needed to satisfy a community engagement requirement. To satisfy this requirement, recipients would need to engage in one or more of the following activities for 80 hours per month: employment, job search, volunteering, school, caregiving, or participating in treatment for substance use disorders. The chief program goals of Kentucky HEALTH were (1) to increase transition of current beneficiaries to employer-sponsored health insurance; (2) to improve beneficiaries’ engagement in preventive healthcare and encourage appropriate use of medical services; (3) to improve health behaviors among beneficiaries; (4) to foster socioeconomic advancement among beneficiaries; and (5) to improve health outcomes among beneficiaries.
State policymakers worked with academic researchers to rigorously evaluate Kentucky HEALTH, formulating a plan to conduct a randomized controlled trial (RCT) in which 90 percent of currently enrolled, ‘able-bodied’ Medicaid beneficiaries would be randomly assigned to Kentucky HEALTH, and 10 percent randomly assigned to traditional Medicaid. The randomization was carried out and baseline surveys were conducted, but Kentucky HEALTH was ended by newly-elected Governor Andy Beshear’s executive order on December 16th, 2019 after it was struck down twice in federal court.
In both cases, the court vacated the approval of the demonstration project on the grounds that the Secretary never adequately considered whether Kentucky HEALTH would promote coverage for all recipients, the central objective of Medicaid. For example, the Secretary never considered Kentucky’s own estimate that Kentucky HEALTH would lead to 95,000 recipients leaving the program over 5 years, with many leaving for reasons of non-compliance with community engagement requirements. Although most Medicaid recipients in Kentucky would have satisfied the community engagement requirements, there were good reasons to expect that many would lose coverage for not properly complying with monthly reporting requirements. This outcome happened later in Arkansas which experienced significant coverage losses when community engagement requirements were implemented. Most losses were likely due to a failure to comply with reporting requirements rather than a failure to fulfill community engagement requirements.
In reasoning about the Kentucky HEALTH case, the court identifies a baseline treatment to which Medicaid recipients are legally entitled, such that state governments may not conduct Medicaid demonstration projects reasonably expected to be inferior to that treatment. We can recognize the court’s ‘baseline’ as a standard of care. Recipients are legally entitled to a Medicaid program that promotes the objectives of Medicaid – primarily the provision of health care coverage – at least as effectively as traditional Medicaid. Therefore, recipients must not be subject to Kentucky HEALTH in so far as it is reasonably expected to be inferior in this respect.
In this case, the standard of care norm for Medicaid demonstration projects is part of U.S. law: the Social Security Act. But there is a further question whether government agencies, evaluators, and social science researchers should be subject to an ethical standard of care norm, that is, a norm they should follow even though doing so is not legally required. For example, suppose the Social Security Act did not require Medicaid demonstration projects to be likely to further the objectives of Medicaid, would federal and Kentucky state officials, as well as the academic researchers, have acted wrongly by conducting the Kentucky HEALTH RCT?
One initial reason for skepticism regarding the development of a standard of care norm for policy research is that the traditional arguments for this norm in clinical research don’t apply. Clinical research ethicists often ground this norm by appealing to the fiduciary obligation of physicians to their patients. Since physicians must promote the health-related interests of their patients, they may not offer them an intervention reasonably expected to be inferior to established treatments - the standard of care. But social scientists have no such fiduciary obligation to research participants, and so it would seem to follow that they should not be bound by any standard of care norm.
This fiduciary obligation is not the only basis for the standard of care norm in clinical research ethics, however, and many scholars reject it for ignoring the crucial role of health systems in delivering care. Instead, they argue that governments have duties of justice to ensure that their residents have access to the standard of care, and that clinical researchers funded by public resources inherit these duties. Importantly, this argument implies that policy research should be governed by a standard of care norm too, for governments have duties of justice to ensure access to adequate housing, high-quality education, nutritious food, among other goods and services, not merely medical treatments.
One of us has developed this argument for a standard of care norm in policy research more fully elsewhere. Briefly, I claim first that governments have duties of justice to realize certain outcomes for their residents and provide them with certain goods and services. I introduce the term ‘target outcomes’ to refer to these outputs, defining them as “specifications of the type and amount of goods or services governments have a duty to provide, and the outcomes they have a duty to realize.” Target outcomes may include certain levels of personal liberty and security, specific levels of health insurance coverage, certain literacy outcomes or a minimal standard of living.
I argue, second, that governments have a duty to implement evidence-based policies to realize target outcomes. Because governments face resource constraints, must realize multiple target outcomes, and must respect people’s rights, they must implement policies that are (1) evidence-based, (2) consistent with people’s rights, and (3) consistent with the realization of other target outcomes. As such, they have an obligation to implement policies that are best proven, respectful of people’s rights, and attainable and sustainable given limited financial and human resources. I call these “best proven attainable (BPA) policies.”
This account explains why governments have a moral obligation to fund, conduct, and authorize policy research. Policy research is necessary if governments are to fulfill their obligations to realize target outcomes. But it also provides a justification for a standard of care norm for policy research. Because governments have a duty to implement BPA policies, they may not authorize, conduct, or fund policy experiments or pilot projects in which people are assigned to interventions that are reasonably expected to be inferior to them. I define the standard of care norm as follows:
Standard of Care: Government agency or institution A may assign participants to an intervention only if it is not reasonably expected to be inferior with respect to the realization of target outcomes than the policy A has a duty to implement, that is, the most effective policy which it has the authority and resources to sustainably implement.
My account of this norm offers what Alex John London calls a “de jure” rather than “de facto” conception of the standard of care. The standard of care for a specific policy sphere is the policy that ought to be implemented, not necessarily the policy that is implemented. In addition, my account is sensitive to resource constraints, both fiscal and bureaucratic. The BPA policy for a particular policy area should not be understood as an abstract ideal that applies across time and space. Instead, it is likely to be highly context specific, depending on a society’s resources, administrative capacity, political culture, and existing policy portfolio, among other factors. The BPA policy will thus often be the status quo policy since, for this not to be the case, there must be a policy that is reasonably expected to be more effective in realizing the relevant target outcomes in the context in question, and that can be sustainably implemented.
To return to the above case, the Kentucky HEALTH RCT is arguably problematic from an ethical as well as a legal perspective. There are strong reasons to think that governments have a duty of justice to ensure that all residents have access to affordable health care coverage without conditions, and traditional Medicaid is an effective and sustainable program by which Kentucky and the federal government can jointly discharge this duty to low-income and other vulnerable Kentuckians. Since traditional Medicaid is therefore the BPA policy and the Kentucky HEALTH intervention is reasonably expected to be inferior to it in the provision of affordable health care coverage, the planned RCT violates the above standard of care norm.
One challenge to this norm is that it leaves too much open to difficult and controversial value judgments. People reasonably disagree about the requirements of justice and so will not agree on which target outcomes governments have a duty to realize. As such, government agencies, evaluators, and researchers may have no non-controversial way to determine what the BPA policy is and so to interpret and apply the standard of care norm. Regarding the Kentucky HEALTH RCT, while progressives may argue that governments have an obligation to ensure unconditional access to affordable health coverage for all residents, conservatives may argue that there is no obligation to do so for people unwilling to contribute to society by satisfying community engagement requirements.
Reasonable disagreement regarding the central purposes of government is a real challenge, but it does not mean the norm should be abandoned or that there are no better or worse ways of interpreting and applying it.
First, this problem of reasonable disagreement does not challenge the standard of care norm itself but rather raises a problem regarding its interpretation. With the exception of anarchists who hold that all governments are illegitimate, people agree that governments have duties of justice. They just disagree on the content of these duties, and so on the target outcomes governments should realize.
Second, while reasonable disagreement is prevalent, there is also widespread support for many responsibilities of government, including protecting basic liberties and property rights, ensuring safety and public order, resolving market failures, and ensuring access to high-quality education, among others.
Third, not all claims regarding the responsibilities of government are equally sound. As political philosophers teach us, such claims are subject to rational reflection. We may reject certain claims regarding the duties of governments if they conflict with our considered judgments, have unacceptable implications, or if they offend basic principles of political morality, for example, those concerning the basic equality of all persons. In the case of Kentucky HEALTH, we might ask why it is just to implement a policy likely to take health care coverage away from people not because they fail to comply with community engagement requirements, but because they fail to satisfy reporting requirements.
Finally, in societies that are sufficiently democratic, researchers and evaluators may appeal to existing law and policy (and rational reconstructions thereof) to resolve reasonable disagreements and identify target outcomes. As theorists of deliberative democracy rightly argue, the views of the public, when adequately informed and formulated through a reflective and deliberative process, have normative authority. So where existing laws and policies are the result of representative and deliberative processes and institutions, they deserve respect and may be relied upon to resolve reasonable disagreements. In many cases, the target outcomes for a policy sphere will be identified in the text of existing legislation. Similarly, international declarations and agreements formulated through representative and deliberative processes, including international human rights treaties or the UN Sustainable Development Goals, also offer authoritative sources for identifying the responsibilities of governments.
In the case of Kentucky HEALTH, the court distinguishes the provision of healthcare coverage to states’ Medicaid populations as a target outcome for Section 1115 demonstration program waivers because it is listed explicitly in U.S. Medicaid law. Other potentially relevant policy goals cited by Kentucky state officials are not. While there is still disagreement in the U.S. regarding whether state and federal governments have a duty to provide this healthcare coverage to low-income people without community engagement requirements, federal analysis and policymaking since the Kentucky case have fairly resolved current disagreements in favor of such an obligation.
Our aim in this post has been to defend the applicability of a ‘standard of care’ norm to policy research and evaluation. This norm applies directly to governments since it is government agencies and institutions that have the chief responsibilities for realizing target outcomes, but this does not mean that other research and evaluation stakeholders, including nonprofit institutions, international organizations, and academic researchers, may ignore it. International organizations and nonprofit institutions should consider the nature of their own responsibilities to the government agencies with which they partner as well as the research participants themselves. Academic researchers should be mindful of their degree of involvement in the planning and design of research and evaluation programs and the responsibilities they thereby take on
As evaluators, we have a role to play in examining intervention ethics. The evaluator’s role in intervention design and implementation varies – sometimes the evaluator is completely independent from intervention design and implementation, sometimes the evaluator plays a supporting role, and sometimes the evaluator leads intervention design and implementation alongside evaluation design and implementation. Regardless of role, if we theorize ex-ante an intervention may result in inferior outcomes for beneficiaries, we have a role in documenting that ex-ante assessment and determining whether we believe it is ethical (and/or legal) to proceed. Some may argue the intervention will continue with or without the evaluation and the evaluation is an opportunity to produce data and evidence to inform future decisions and adjustments in program design. Others may argue it is not ethical to evaluate interventions that may result in inferior outcomes. Assessing the standard of care and whether it is preserved is a good place to start with documenting this ethical decision-making and the evaluator’s role in deciding to evaluate or not. An example of such documentation is this recent Ethics Appendix.
Given the wide range in resource constraints across governments, evaluators and other stakeholders can rely on the concept of ‘best proven attainable (BPA) policies’ to assess the ethics of the intervention. For many who prescribe to ‘de-facto’ standards, it is easy to point to policies or programs that may result in less desirable outcomes and claim they are unethical. But view this dismisses the context and resources that inform decisions by governments and the BPA policy. Understanding the BPA policies and how a proposed pilot or new intervention compares to the BPA is therefore a useful step for evaluation practice and requires engagement with a range of local stakeholders to ensure a shared understanding prior to study design. More resources are needed for evaluators and stakeholders to identify the BPA policy.
If confronted with a program that does not appear to meet BPA policy threshold, evaluators may have an opportunity to present alternatives that aim to preserve the standard of care while potentially improving outcomes. Reflecting on the example presented in this case study, there are alternatives for using Medicaid eligibility for working age adults to target voluntary and customized workforce development programs, such as the Montana HELP-link program. In discussions around policy questions, intervention options, and factorial designs, evaluators may point to alternatives for which more rigorous impact evaluation may inform policy.
