Cannabis Policy Impacts Public Health and Health Equity (2024)
Chapter: Appendix D: Evidence Review: Methods and Approach
Appendix D
Evidence Review: Methods and Approach
This appendix describes the approach and methods that the committee used to evaluate the literature on the public health consequences on changes in the cannabis policy landscape. Specifically, the task asks the committee to “Review what is known about whether these outcomes have changed in states and localities that have changed their regulatory approach to cannabis and cannabinoids.”
The committee conducted an overview of systematic reviews to assemble and review the evidence to determine whether health outcomes have changed due to changes in cannabis policy. Unlike traditional reviews that synthesize primary research, overviews of systematic reviews employ rigorous methods to identify and analyze existing systematic reviews on a specific topic. Rather than summarizing individual studies, overviews assemble evidence and compare results across multiple reviews to paint a broader picture of the evidence landscape (Pollock et al., 2020).
EVIDENCE IDENTIFICATION
Evidence identification consisted of standard steps of literature searching, abstract screening, and full-text review.
Literature Search
A trained medical librarian in the National Academies of Sciences, Engineering, and Medicine Research Center conducted searches in three databases: Ovid Embase Update, Medline, and Scopus. Search terms included
cannabis, cannabis use, and public policy. The only limits on the searches were human studies, classification as review papers, and publication in the English language. The databases were searched on April 25, 2023. The committee also amended the search by including systematic reviews on cannabis policy that were identified subsequently by committee members.
Ovid Embase Update Search Terms
| 1 | Exp Cannabinoid/ | 89262 |
| 2 | Cannabis smoking/ OR Cannabis use/ OR Medical cannabis/ | 17905 |
| 3 | exp cannabis smoking/ or exp “cannabis use“/ | 4022 |
| 4 | 1 OR 2 OR 3 | 96485 |
| 5 | Drug legislation/ or Government/ or Health care policy/ or Law/ or Legal aspect/ or “Legislation and jurisprudence”/ or Pharmacovigilance/ or Public policy/ | 658175 |
| 6 | “systematic review*”.mp. | 545660 |
| 7 | 4 AND 5 AND 6 | 100 |
| 8 | limit 7 to English language | 97 |
Medline Search Terms
| 1 | Exp Cannabinoids/ OR Marijuana smoking/ OR Marijuana/ OR Medical marijuana/ | 32016 |
| 2 | “systematic review*”.mp. | 318864 |
| 3 | Exp government/ OR Health policy/ OR Jurisprudence/ OR Law enforcement/ OR Legislation as topic/ OR Exp Legislation, drug/ OR Pharmacovigilance/ OR Public health surveillance/ OR Public policy/ | 321953 |
| 4 | 1 AND 2 AND 3 | 17 |
| 5 | limit 4 to English language | 16 |
Scopus Search Terms
TITLE-ABS-KEY ( ( ( ( ( cannabidiol OR cannabinoid* OR cannabinol OR cannabis OR dronabinol OR marijuana OR tetrahydrocannabinol OR thc ) W/6 ( “drug narcotic control” OR “Drug control” OR “drug legislation” OR government* OR “Government regulation*” OR “Health polic*” OR jurisprudence OR “Law enforcement” OR law OR laws OR legal* OR legislati* OR pharmacovigilance OR policy OR policies OR “public health surveillance” OR “Public polic*” OR regulat* OR statute* OR surveillance ) AND ( “systematic review*” ) ) ) ) )
After the literature searches were conducted, retrieved data was imported into Endnote and then uploaded to PICO Portal (https://picoportal.net/). PICO portal is a web-based tool that helps research teams collaborate on systematic reviews. It uses machine learning and artificial intelligence (AI) to order the abstracts reviewed during abstract screening so that the abstracts most likely to be selected will be read first. The literature search identified 248 potentially relevant systematic reviews.
The review used the following population exposure comparison and outcome (PECO) statement:
Population: Review paper
Exposure/Intervention: Evaluates cannabis policy
Comparison: Any comparison group, including internal comparisons
Outcome: Any outcome
The inclusion and exclusion criteria related to the PECO statement were as follows:
Inclusion Criteria: Reviews literature related to the public health impacts of cannabis policy
Exclusion Criteria: Did not review public health impacts of cannabis policy; reviewed only animal or mechanistic studies; reviewed cannabis as medical treatment or impacts of cannabis on individuals
Abstract Screening
One National Academies staff member completed title and abstract screening, with decisions reviewed by the Study Director. Five of 248 relevant abstracts were excluded as duplicates, leaving 243 abstracts for screening. Among the abstracts screened, 183 were excluded, leaving 60 studies for full-text review (Figure D-1).
Full-Text Screening
The full-text screening was completed in PICO Portal using a method similar to the abstract stage. One National Academies staff member completed title and abstract screening, with decisions reviewed by the Study Director. Articles were excluded at the full-text stage for several reasons. Many did not evaluate cannabis policy and thus were evaluating the wrong intervention (n = 40), one was not evaluating a health outcome, and two others were commentaries or other types of papers. Several studies were not systematic reviews; thus, they did not meet population requirements (n = 3). A total of 14 studies met the final inclusion criteria (Figure D-1).
SOURCE: Committee generated using an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams Haddaway et al., 2022.
DATA EXTRACTION
We extracted data from the systematic reviews with one PICO Portal methodologist and verified the data with a second methodologist. The data tables were also reviewed by the Committee. The data abstraction form included the following:
Reference Information: Citations were migrated from PICO Portal with the addition of the two reviews identified after the database searches were completed.
Setting: The place and time for the studies included in the review.
Policy Changes Evaluated: A description of the cannabis policy changes that were reviewed in the study.
Outcomes Evaluated: Description of the outcomes evaluated.
Inclusion/Exclusion Criteria: The inclusion and exclusion criteria.
Study Quality/Risk-of-Bias Assessment Method Used: A description of the method used to assess the quality of the individual studies identified in the review.
Study Designs Identified: A description of the study designs that were found in the study.
Number of Included Studies: The number of papers included after full-text screening in the study.
Findings: A summary of the major findings of the review.
Certainty of the Evidence: A summary of how confident the authors were in the overall conclusions.
EVIDENCE EVALUATION
The 15 systematic reviews were then evaluated for risk of bias using the Risk Of Bias In Systematic Reviews (ROBIS) tool (Whiting et al., 2016). ROBIS is a specifically designed instrument for evaluating the risk of bias in systematic reviews. It has been shown to have adequate internal consistency and strong evidence for measuring the intended construct (Bühn et al., 2017). The ROBIS tool helps to identify potential biases within the systematic review process. The ROBIS tool uses three phases: (1) assesses relevance, (2) identifies concerns with the review process, and (3) judges the risk of bias in the review. Concerns with the systematic review process are captured by considering four key domains: study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings. Signaling questions are used to guide the judgments in each domain, and they are each answered “Yes,” “Probably Yes,” “Probably No,” “No,” and “No Information,” with “Yes” indicating low concerns about the domain contributing to risk of bias. The subsequent level of concern about bias associated with each domain is then judged as “low,” “high,” or “unclear” (Whiting et al., 2016).
The domains are then used to guide judgments on the overall risk of bias for each included systematic review, categorized as low, high, or unclear. This assessment uses a similar structure as the process used to assess bias within the key domains. Signaling questions and information are used to support the overall judgment of the risk of bias. The three signaling questions used to judge the overall risk of bias in the review relate to the interpretation of the review findings. The signaling questions cover important areas where bias can be introduced into the review. The first question asks whether interpreting the study findings addresses all the identified concerns with the systematic review process. The second is whether the studies included in the review apply to the research question’s goals. The third signaling question is whether the results are balanced and avoid emphasizing only those statistically significant results (Whiting et al., 2016).
STRENGTH-OF-EVIDENCE DETERMINATION
To assess the strength of evidence regarding the association of changes in cannabis policy and impacts on public health, the committee categorized the strength of the association using categories used by other reports of the National Academies (NASEM, 2022). The strength of the evidence is based on the strength and the certainty of the overall body of evidence in the reviews (Figure D-2).
In determining the strength of the evidence, the committee considered the risk of bias in the review, the strength of the evidence as determined by the authors, and other aspects of study quality, such as reporting quality.
NOTES: The categories of association only describe how strong the evidence is between exposure, change in cannabis policy, and the changes in the health outcome.
Many systematic reviews covered overlapping topics; the committee judged the most recently published review to be the most up-to-date assessment of the evidence and relied less on the older reviews in making judgments.
Categories of Association
Sufficient Evidence of an Association
For effects in this category, a positive association between cannabis policy change and the outcome must be observed in studies where chance, bias, and confounding can be ruled out with reasonable confidence. For example, the committee might regard sufficient evidence of association from several small studies that are unlikely to be due to confounding or to be otherwise biased and that show an association that is consistent in magnitude and direction. Experimental data supporting biologic plausibility strengthen the evidence of an association but are not a prerequisite, nor are they sufficient to establish an association without corresponding epidemiologic findings.
Limited or Suggestive Evidence of an Association
In this category, the evidence must suggest an association between exposure to change in cannabis policy and the outcome in studies of humans. Still, the evidence can be limited by an inability to rule out chance, bias, or confounding with confidence. One high-quality study may indicate a positive association, but the results of other studies of lower quality may be inconsistent.
Inadequate or Insufficient Evidence to Determine an Association
If there was not enough reliable scientific data to categorize the potential association with an outcome as “sufficient evidence of an association,” “limited or suggestive evidence of an association,” or on the other end of the spectrum, “limited or suggestive evidence of no association,” the outcome was placed in the category of “inadequate or insufficient evidence to determine an association” by default. In this category, the available human studies may have inconsistent findings or be of insufficient quality, validity, consistency, or statistical power to support a conclusion regarding the presence of an association. Such studies may have failed to control for confounding factors or may have had inadequate assessment of exposure.
Limited or Suggestive Evidence of No Association
A conclusion of “no association” is inevitably limited to the conditions, exposures, and observation periods covered by the available studies, and the
possibility of a small increase in risk related to the magnitude of exposure studied can never be excluded. However, a change in classification from inadequate or insufficient evidence of an association to limited or suggestive evidence of no association would require new studies that corrected for the methodologic problems of previous studies and that had samples large enough to limit the possible study results attributable to chance.
REFERENCES
Bühn, S., T. Mathes, P. Prengel, U. Wegewitz, T. Ostermann, S. Robens, and D. Pieper. 2017. The risk of bias in systematic reviews tool showed fair reliability and good construct validity. Journal of Clinical Epidemiology 91:121–128. https://doi.org/10.1016/j.jclinepi.2017.06.019.
Haddaway, N. R., M. J. Page, C. C. Pritchard, and L. A. McGuinness. 2022. PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Systematic Reviews 18(2):E1230.
NASEM (National Academies of Sciences, Engineering, and Medicine). 2022. Guidance on PFAS exposure, testing, and clinical follow-up. Washington, DC: The National Academies Press.
Pollock, M., R. M. Fernandes, L. A. Becker, D. Pieper, and L. Hartling. 2020. Chapter V: Overviews of reviews. In Cochrane Handbook for Systematic Reviews of Interventions, version 6.
Whiting, P., J. Savović, J. P. Higgins, D. M. Caldwell, B. C. Reeves, B. Shea, P. Davies, J. Kleijnen, and R. Churchill. 2016. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology 69:225–234. https://www.sciencedirect.com/science/article/pii/S089543561500308X?via%3Dihub (accessed February 8, 2024).
