Appendix F

AMSTAR 2 Tool

During the data extraction process of the umbrella review, the methodological quality of each systematic review was evaluated using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) quality assessment tool, with some minor adaptations for clarity. The tool consists of the following series of 15 questions. Alterations to or interpretations of the tool made by the committee are noted in italic text.

1. Did the research questions and inclusion criteria for the review include ALL the components of PICO?
   • Yes: Population, Intervention, Comparator, Outcome, AND Follow-up duration described fully and adequately.
   • Partial: Described, but not adequately to sufficiently understand eligibility criteria [Partial was added by the committee].
   • No: Not all PICO elements
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review, and did the report justify any significant deviations from the protocol?
   • Partial: Protocol included “just” (1) review questions, (2) search strategy, (3) eligibility criteria, AND (4) risk of bias assessment [and/or PROSPERO or other registry].
   • Yes: Protocol included all Partial, PLUS (5) meta-analysis or synthesis plan, (6) plan to investigate heterogeneity, AND (7) justifications for deviation from protocol.

2. • No: Not all criteria met (for Partial) or no mention of a protocol
     • Study authors were given the benefit of the doubt in edge cases.
3. Did the review authors explain their selection of the study designs for inclusion in the review?
   • Yes: Provided explanation for selecting study designs (and for not selecting excluded study designs)
   • No: No explanation
     • Study authors were given the benefit of the doubt in edge cases. Implicit explanations were acceptable.
4. Did the authors use a comprehensive literature search strategy?
   • Partial: (1) at least two relevant databases, (2) provided key words or search strategy, AND (3) justified restrictions (e.g., language) OR arbitrarily excluded studies (e.g., based on perceived risk of bias) [latter item added by the committee]
   • Yes: all Partial PLUS (4) searched reference lists, (5) searched study registries, OR (6) consulted content experts [Note: AMSTAR 2 says “AND” here]
   • No: Not all criteria met (for Partial) OR used a clearly inadequate search strategy. [The latter two reasons were added by the committee.]
5. Did the review authors perform study selection in duplicate?
   • Yes: At least two independent screeners, plus a method for reconciling conflicts OR double screening of a sample with at least 80% agreement, followed by single screening [Note: Committee combined original Yes and Partial into simply Yes.]
   • No: Less stringent method used.
6. Did the review authors perform data extraction in duplicate?
   • Yes: Double independent with reconciliation process OR Single with review by experienced systematic reviewer
   • No: Less stringent method used
     • We gave researchers the benefit of the doubt regarding their systematic review experience.
7. Did the review authors provide a list of excluded studies and justify the exclusions?
   • Yes: Listed and provided exclusion for each OR reported available access to such a list [Note: Committee added the second option here.]
   • Partial: Listed but did not explain each exclusion
   • No: Did not list
8. Did the review authors describe the included studies in adequate detail?

   Partial: Described “just” each PICOD element OR the studies were described, but with some limitations for the needs of the committee [Note: Committee added the second option here.]

8. • Yes: Described PICOD elements in detail, including setting and follow-up time OR the descriptions of the studies were sufficient for the needs of the committee [Note: Committee added the second option here.]
   • No: Partial not met
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?
   • Yes: Used a standard risk-of-bias tool (e.g., Cochrane Risk of Bias tool for randomized trials, ROBINS-I for nonrandomized studies) or equivalent tool that addresses relevant issues related to randomization/allocation concealment, confounding bias, selection bias, outcome ascertainment, analytic method
     • This framework is based on the concepts described by AMSTAR 2.
   • Partial: Used an appropriate tool but applied an arbitrary point system to determine level of quality/risk of bias
     • This revision was added, post hoc, upon reviewing eligible systematic reviews.
   • No: No or inadequate risk-of-bias tool applied
10. Did the review authors report on the sources of funding for the studies included in the review?
    • Yes
    • No
11. If meta-analysis was performed, did the review authors use the appropriate methods for the statistical combination of results?
    • Yes: (1) Justified combining in meta-analysis, (2) used random effects model (or equivalent), AND (3) analyzed heterogeneity
      • Committee removed concepts related to whether unadjusted analyses were included and whether studies of different designs were combined.
    • No: (1) Used fixed-effect model (or equivalent) based on heterogeneity measures OR (2) conducted meta-regression or subgroup analysis subject to ecological fallacy (i.e., regressed across the mean value for the sample, such as BMI)

      If there were concerns regarding fixed-effect models or ecological fallacy, the relevant analyses were highlighted. The committee did not derive conclusions based on analyses subject to ecological fallacy.

12. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?
    • Yes
    • No

13. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?
    • Yes: Heterogeneity was assessed AND, if present, assessed causes and included as part of their interpretation of findings.
    • No: Did not assess heterogeneity OR only enumerated without assessing the impact on findings
14. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?
    • Yes: Reported and carried out plan to assess publication bias
      [Note: The committee required reporting of a plan to assess publication bias.]
    • No: Did not report plan to assess publication bias
15. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?
    • Yes
    • No

OVERALL “QUALITY”

The system for determining the quality, or methodological adequacy, of the systematic reviews was constructed by the committee based on concepts and terminology from AMSTAR 2. While all AMSTAR 2 questions were answered, not all impacted the overall quality.

Well-done/reported systematic reviews

• (1) Adequately reported PICO [Yes, not Partial], (2) had a protocol [Yes, Partial], (4) used a comprehensive literature search [Yes, not Partial], (5) selected studies in duplicate [Yes], (6) extracted studies in duplicate or equivalent [Yes], (8) adequately described studies [Yes], (9) assessed risk of bias [Yes, Partial], (11) used appropriate meta-analysis techniques if applicable [Yes], (12) accounted for risk of bias [Yes], (13) assessed heterogeneity [Yes], and (15) reported sources of conflict of interest [Yes]
• Did not require (3) explanation of selected study designs, (7) listing of excluded studies, (10) reporting of sources of funding for included studies, OR (14) assessment of publication bias.

Partially well-done/reported systematic reviews

Studies were downgraded to Partially well-done/reported if

• (1) Partially adequate reporting of PICO [Partial], (2) did not report a protocol [No], (4) partially adequate literature search [Partial], (6) did not extract in duplicate or equivalent [No/Unclear], (8) did not adequately describe studies [Partial/No], (13) did not adequately assess heterogeneity [No], OR (15) did not report conflicts of interest.

Not well-done/reported systematic reviews

Studies were downgraded to Not well-done/reported if

• (1) Inadequate reporting of PICO [No], (4) inadequate literature search [No], (5) did not select studies in duplicate [No], (9) did not assess risk of bias [No], (11) used inappropriate meta-analysis technique [No; this may apply only to selected conclusions/findings within the systematic review], OR (12) did not account for risk of bias [No].

This page intentionally left blank.