Appendix C

Committee’s Assessment of the Agency for Healthcare Research and Quality Systematic Review

In accordance with its Statement of Task (see Chapter 1, Box 1-1), the committee was asked to consider the Agency for Healthcare Research and Quality (AHRQ) systematic review, Sodium and Potassium Intake: Effects on Chronic Disease Outcomes and Risks (AHRQ Systematic Review) (Newberry et al., 2018), in its derivation of the Dietary Reference Intake (DRI) values for potassium and sodium. The AHRQ Systematic Review included both the selection of literature and the investigators’ assessment of the strength of evidence for each indicator.

Prior to using the AHRQ Systematic Review, the committee assessed its overall quality and methodology. As anticipated in the Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease (Guiding Principles Report) (NASEM, 2017), the committee reassessed the evidence for some relevant indicators in the AHRQ Systematic Review. The details of the additional data analyses conducted by the committee for the purposes of the expanded assessment are included in Chapters 6 and 10. This appendix includes the committee’s approach to reviewing the quality of the AHRQ Systematic Review and to expanding the assessment of the evidence as the fundamental basis for the deliberations regarding establishing Chronic Disease Risk Reduction Intakes for potassium and sodium.

ASSESSMENT OF OVERALL QUALITY

The committee assessed the overall quality of the AHRQ Systematic Review using the AMSTAR 2 tool (Shea et al., 2017).¹ The committee determined the AHRQ Systematic Review met the majority of the 16 domains and that it was of overall moderate quality.² Domains that the AHRQ Systematic Review did not adequately cover related to investigating and explaining the causes of heterogeneity in the results, which in some cases is essential in order to interpret the results of meta-analysis.

ASSESSMENT OF METHODOLOGICAL APPROACH

As prescribed in the AHRQ guidance, a protocol was prepared and published for the AHRQ Systematic Review used in this study (AHRQ, 2017). The committee reviewed the protocol and determined that the PICO questions³ and the inclusion/exclusion criteria for each indicator included were complete, clear, and appropriate. The committee also reviewed the strength-of-evidence domains and their definitions in the AHRQ guidance (AHRQ, 2014) and determined that they were complete, clear, and appropriate.

The protocol specified the tools and criteria used for assessing the evidence. The AHRQ guidance scores the body of evidence separately for randomized controlled trials and observational studies and provides guidance for randomized controlled trials. However, the guidance provides flexibility and directs the evidence-based practice centers conducting the systematic review to specify risks of bias specific to the content area. Accordingly, the AHRQ Systematic Review protocol is particularly detailed at describing its constructs and implementation in the risk-of-bias (or study limitations) domain for randomized controlled trials and observational studies separately. Judging risk of bias in an objective and standardized manner is essential to the interpretation and weighing of a study (or a body of evidence) (for explanations of the importance of the risk-of-bias tool, see Chapter 2).

In spite of using assessment tools that are objective and formally accepted by the scientific community, the committee recognizes that all assessments regarding determination of risk of bias for individual studies and strength of the evidence for the body of evidence for a specific outcome entail a certain

___________________

¹ AMSTAR stands for A Measurement Tool to Assess Systematic Reviews.

² The AMSTAR 2 tool is not intended to result in an overall score. Instead, the tool can be used for a qualitative assessment, where different factors can be weighted differently, depending on the importance or relevance to the research question(s).

³ PICO is a mnemonic device for the important parts of a well-built clinical question. PICO stands for population (or problem or patient), intervention, comparison, and outcome.

degree of interpretation and judgment. With this in mind, the committee performed two tasks to explore its degree of agreement in the application of these assessment tools with the decisions and judgments of the AHRQ Systematic Review. The committee performed spot checks of the AHRQ Systematic Review’s risk-of-bias assessment and strength-of-evidence rating. The committee understood that it is its prerogative to perform additional analyses and to potentially reach a different strength-of-evidence determination, as long as there is transparency and a scientific basis in its rationale for doing so.

Spot Check of the Risk-of-Bias Assessment

The committee generally agreed with the risk-of-bias tools criteria for both randomized controlled trials and observational studies, as defined in the AHRQ Systematic Review (for the criteria, see Annex C-1). To check the application of the risk-of-bias criteria, six studies were selected at random; the selected studies were determined by the AHRQ investigators to have low, moderate, and high level of risk of bias (one of each risk-of-bias level for randomized controlled trials and one of each risk-of-bias level for observational studies). Two members of the committee independently assessed the risk of bias for each study by following the AHRQ risk-of-bias criteria. Discrepancies were minor between the committee members’ and the AHRQ Systematic Review’s risk-of-bias rating for each study. Given previous reports regarding the inconsistent application of the risk-of-bias tools and the large discrepancies in how risk of bias is being evaluated for some specific domains (Jordan et al., 2017), the committee accepted these minor discrepancies as typical and determined that based on this limited spot check, the application of the risk-of-bias tools in the AHRQ Systematic Review was appropriate.

Assessing the Application of Strength-of-Evidence Domains

The committee conducted a number of checks related to the AHRQ Systematic Review, particularly for outcomes that would likely be relevant for setting DRI values (e.g., blood pressure, cardiovascular disease). In that regard, the committee noted that the conclusions in some relevant recent systematic reviews differ from the conclusions in the AHRQ Systematic Review. For example, the strength of the evidence for an effect of a reduction of sodium intake on reducing blood pressure, an outcome for which a substantial body of evidence exists, was determined as high in past systematic reports (Graudal et al., 2017; NHLBI, 2013; WHO, 2012a). Even if using the same strength-of-evidence domains (e.g., risk of bias, inconsistency, imprecision, and indirectness), the AHRQ Systematic Review rated the strength of evidence as moderate. This discrepancy might

give the appearance that the strength of the evidence for the relationship between sodium intake and blood pressure has changed over the past few years. However, the reason for this discrepancy could lie in various other factors, including the strength-of-evidence assessment.

To understand this discrepancy, the committee examined salient systematic reviews on sodium and blood pressure and cardiovascular disease outcomes (NHLBI, 2013; WHO, 2012a,b) and an additional systematic review on sodium and blood pressure (Graudal et al., 2017). The committee found a number of differences in the systematic reviews, such as those related to the approach in the literature search, the populations of interest, and various inclusion/exclusion criteria compared to the AHRQ Systematic Review. A major difference that could have led to differences in the final determination, however, was in the application of the inconsistency domain, which refers to the unexplained heterogeneity or variability of study results in a body of evidence, or the imprecision domain.

For example, the World Health Organization’s 2012 systematic review concluded that randomized controlled trials on the relationship between sodium intake and blood pressure did not show a serious inconsistency (based on inconsistency in the direction or the size of the effect), which led to a determination of high quality (WHO, 2012a). Conversely, the meta-analysis of the relationship between sodium reduction and systolic and diastolic blood pressure conducted by the AHRQ Systematic Review resulted in high inconsistency owing to heterogeneity in the meta-analysis. The AHRQ Systematic Review did not perform further analyses and downgraded the strength of the evidence to moderate based on the existence of inconsistency (for how the AHRQ Systematic Review defined inconsistency, see Box C-1). The committee decided that, in order to understand the nuances and have more clarity in interpreting the evidence, it was essential to explore the sources of heterogeneity in the body of evidence on the relationship between sodium intake and blood pressure. The committee performed sensitivity analyses to investigate sources of heterogeneity. These analyses informed the committee’s assessment of the strength of the evidence for a relationship between sodium intake and systolic and diastolic blood pressure, which it rated as high (for additional details, see Chapter 10).

The above example presents one case where analyses beyond those conducted in the AHRQ Systematic Review were helpful in resolving an important question for assessing the evidence for an indicator of interest. The committee conducted additional analyses on select results of the AHRQ Systematic Review to clarify its interpretation of the results as needed in order to complete the committee’s task; the additional analyses are described in Chapters 6 and 10.

REFERENCES

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Guiding principles for developing Dietary Reference Intakes based on chronic disease. Washington, DC: The National Academies Press.

Newberry, S. J., M. Chung, C. A. M. Anderson, C. Chen, Z. Fu, A. Tang, N. Zhao, M. Booth, J. Marks, S. Hollands, A. Motala, J. K. Larkin, R. Shanman, and S. Hempel. 2018. Sodium and potassium intake: Effects on chronic disease outcomes and risks. Rockville, MD: Agency for Healthcare Research and Quality.

NHLBI (National Heart, Lung, and Blood Institute). 2013. Lifestyle interventions to reduce cardiovascular risk. Systematic evidence review from the Lifestyle Work Group. https://www.nhlbi.nih.gov/sites/default/files/media/docs/lifestyle.pdf (accessed January 14, 2019).

Shea, B. J., B. C. Reeves, G. Wells, M. Thuku, C. Hamel, J. Moran, D. Moher, P. Tugwell, V. Welch, E. Kristjansson, and D. A. Henry. 2017. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008.

WHO (World Health Organization). 2012a. Effect of reduced sodium intake on blood pressure, renal function, blood lipids and other potential adverse effects. Geneva, Switzerland: World Health Organization.

WHO. 2012b. Effects of reduced sodium intake on cardiovascular disease, coronary heart disease and stroke. Geneva, Switzerland: World Health Organization.

ANNEX C-1
RISK OF BIAS CRITERIA USED IN THE AGENCY FOR HEALTHCARE RESEARCH AND QUALITY SYSTEMATIC REVIEW

The two sections that follow are the risk-of-bias criteria used in the Agency for Healthcare Research and Quality systematic review, Sodium and Potassium Intake: Effects on Chronic Disease Outcomes and Risks (Newberry et al., 2018). These criteria were developed independent of this committee.

RISK OF BIAS ASSESSMENT FOR RANDOMIZED CONTROLLED TRIALS

Random Sequence Generation (Selection Bias)

For randomized controlled trials, is the sequence generation (recruitment) described as being random?

For controlled clinical trials, is the allocation described in such a way that it appears to be free of obvious (intentional) bias?

For crossover trials, was the order of receiving treatments randomized adequately?

• Low risk: The investigators describe a random component in the sequence generation process such as: referring to a random number table; using a computer random number generator; coin tossing; shuffling cards or envelopes; throwing dice; drawing of lots; minimization (minimization may be implemented without a random element, and this is considered to be equivalent to being random).
• High risk: The investigators describe a non-random component in the sequence generation process. Usually, the description would involve some systematic, non-random approach; for example, sequence generated by odd or even date of birth; sequence generated by some rule based on date (or day) of admission; sequence generated by some rule based on hospital or clinic record number. Other non-random approaches happen much less frequently than the systematic approaches mentioned above and tend to be obvious. They usually involve judgment or some method of nonrandom categorization of participants, for example: allocation by judgment of the clinician; allocation by preference of the participant; allocation based on the results of a laboratory test or a series of tests; allocation by availability of the intervention.

• Unclear risk: Insufficient information about the sequence generation process to permit judgment of “Low risk” or “High risk.”

Allocation Concealment

Was the group allocation concealed (such that assignments could not be predicted)?

• Low risk: Use of a third party and opaque envelopes or their equivalent are low risk. Participants and investigators enrolling participants could not foresee assignment because one of the following, or an equivalent method, was used to conceal allocation: central allocation (including telephone, Web-based, and pharmacy-controlled randomization); sequentially numbered drug containers of identical appearance; sequentially numbered, opaque, sealed envelopes.
• High risk: Participants or investigators enrolling participants could possibly foresee assignments and thus introduce selection bias, such as allocation based on: using an open random allocation schedule (e.g., a list of random numbers); assignment envelopes were used without appropriate safeguards (e.g., if envelopes were unsealed or nonopaque or not sequentially numbered); alternation or rotation; date of birth; case record number; any other explicitly unconcealed procedure.
• Unclear risk: Insufficient information to permit judgment of “Low risk” or “High risk.” This is usually the case if the method of concealment is not described or not described in sufficient detail to allow a definite judgment; for example, if the use of assignment envelopes is described, but it remains unclear whether envelopes were sequentially numbered, opaque and sealed.
• Not applicable: Study is a controlled clinical trial.

Blinding of Participants and Personnel

Were participants and key study personnel blinded to their intervention or exposure status?

• Low risk: Any one of the following: no blinding or incomplete blinding, but the review authors judge that the outcome is not likely to be influenced by lack of blinding; blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken.
• High risk: Any one of the following: no blinding of outcome assessment, and the outcome is likely to be influenced by lack of blinding;

• blinding of participants and key study personnel attempted, but likely that the blinding could have been broken, and the outcome is likely to be influenced by lack of blinding.

• Unclear risk: Any one of the following: insufficient information to permit judgment of “Low risk” or “High risk”; the study did not address this outcome. Just mentioning “placebo” = “unclear.”

Blinding of Outcome Assessment

• Low risk: Any one of the following: no blinding of outcome assessment, but the review authors judge that the outcome measurement is not likely to be influenced by lack of blinding; blinding of outcome assessment ensured, and unlikely that the blinding could have been broken. Apply same criteria as for patients.
• High risk: Any one of the following: no blinding of outcome assessment, and the outcome measurement is likely to be influenced by lack of blinding; blinding of outcome assessment, but likely that the blinding could have been broken, and the outcome measurement is likely to be influenced by lack of blinding.
• Unclear risk: Any one of the following: insufficient information to permit judgment of “Low risk” or “High risk”; the study did not address this outcome; “double blind” and no further information on assessor (e.g., external assessor).

Incomplete Outcome Data (Attrition Bias)

For randomized controlled trials and clinical controlled trials, could high attrition or uneven attrition across study arms have contributed to bias?

For crossover studies, only, was outcome reporting complete for all phases?

• Low risk: Similar loss to follow-up across groups OR analyses took loss to follow-up into account (e.g., by intent-to-treat [ITT] analysis, “censoring,” imputing missing data [e.g., by carrying the last observation forward] or qualitatively or quantitatively comparing the characteristics of people who dropped out with those who remained in the analysis).
• High risk: Differential loss to follow-up across groups with no attempt to take into account or to assess differences between dropouts and retained participants.
• Unclear risk: Nothing mentioned about evaluating impacts or taking into account in analysis.

Selective Reporting of Outcome Data

For studies that purport to be reporting the prespecified study outcomes, do the outcomes reported match those listed in the Methods section under “Outcomes,” or does the article state that some of the prespecified outcomes will be reported in subsequent articles?

• Low risk: Any of the following: the study protocol is available and all of the study’s prespecified (primary and secondary) outcomes that are of interest in the review have been reported in the prespecified way; the study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were prespecified (convincing text of this nature may be uncommon).
• High risk: Any one of the following: not all of the study’s prespecified primary outcomes have been reported; one or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g., subscales) that were not prespecified; one or more reported primary outcomes were not prespecified (unless clear justification for their reporting is provided, such as an unexpected adverse effect); one or more outcomes of interest in the review are reported incompletely so that they cannot be entered in a meta-analysis; the study report fails to include results for a key outcome that would be expected to have been reported for such a study.
• Unclear risk: Insufficient information to permit judgment of “Low risk” or “High risk.” It is likely that the majority of studies will fall into this category.
• Not applicable: Article reports on post-hoc analysis of data from a study initially published elsewhere.

Other: Adherence

Did the investigators describe rates of adherence to the intervention or some measure of adherence?

• Low risk: Adherence was described as high (e.g., based on biomarkers) or as greater than or equal to 80 percent.
• High risk: Adherence was described as low.
• Unclear risk: Nothing about adherence was mentioned.

Other: Unequal Distribution Among Groups of Potential Confounders at Baseline

Was distribution of demographics (e.g., age, gender, race/ethnicity), comorbidities, and other potentially critical confounding factors (e.g., blood pressure, use of antihypertensives) similar across study arms at baseline (or if not, does the analysis control for baseline characteristics)?

• Yes (Low risk): No significant difference between arms in demographic and important comorbidity characteristics (e.g., blood pressure) according to table and/or described by investigators, or difference(s) taken into account in analysis.
• No (High risk): Significant difference between arms in age, race, gender, important comorbidity with no attempt to control for the differences.

Demonstration That Outcome of Interest Was Not Present at Start of Study for All Participants

For example, analysis of the number of people with stroke includes patients that had a stroke before start of study not after the intervention or had a recurring stroke. Note: Incidence versus recurrence. This item is a trigger for excluding individual studies from analyses. Please specify for which outcome this is an issue.

• Low risk: No problem.
• High risk: People with an outcome of interest were not excluded (specify the outcome, and exclude study for that analysis).

Other: Valid Method of Exposure Assessment

Was exposure to intervention assessed using a valid method?

• Low risk: For sodium or sodium-to-potassium ratio, exposure was assessed with at least one 24-hour urinary analysis with reported quality control measure. For potassium, exposure was assessed using at least one 24-hour urinary analysis with reported quality control measure, chemical analysis of diet or food diary with intervention/exposure adherence measure, or composition of potassium supplement with intervention/exposure adherence measure.

• High risk: For sodium or sodium-to-potassium ratio or potassium, exposure was assessed with chemical analysis of diet, composition of salt substitute, or food diaries. Exposure was assessed less than 24 hours or through a published food frequency questionnaire.
• Moderate risk or unclear: For sodium, 24-hour urinary analysis without reported quality control measure. Chemical analysis of diet without intervention/exposure adherence measure, or composition of potassium supplement without intervention/exposure adherence measure. For potassium, use of food diaries without quality control.

Other: Valid Method of Outcome Assessment

Were outcomes assessed using valid methods?

• Low risk: Definitions of outcomes are provided by investigators, outcomes are not self-reported, and method of ascertainment is described.
• High risk: Definitions are not provided (e.g., for cardiovascular disease morbidity); one or more outcomes is described as self-reported.
• Unclear risk: No description of outcome definitions, no mention of method of ascertainment.

Other: Valid Statistical Assessment (for Crossover Trials Only)

Did the authors report how they did their analysis, and did they do the correct analysis for a crossover (a paired analysis of some type)?

• Yes, they report how they analyzed the data, and they report a paired analysis of some type.
• No, they report an analysis but it was not paired.
• Unclear: They do not report how the analysis was done and the outcomes do not appear to have come from pairing people with themselves.

RISK OF BIAS CRITERIA FOR OBSERVATIONAL STUDIES

Representativeness of the Exposed Cohort

• Low risk: Truly representative of the average named cohort in the community.
• High risk: Select group (e.g., only doctors).

• Moderate risk or unclear: Somewhat representative of average named population or no description of the derivation of the cohort.

Selection of the Nonexposed Cohort

• Low risk: The recruitment or allocation strategy was similar across exposure groups (drawn from the same community as exposed cohort).
• High risk: Drawn from a different source.
• Unclear risk: No description.

Ascertainment of Sodium and Potassium Exposure (Dietary Assessment/Urinary Assessment)

• Low risk of bias:
  • Multiple days (more than 4 on average, preferably nonconsecutive) 24-hour urines with reported quality control measures (e.g., instructions given and measure of completeness of collection such as creatinine, urine volume, questionnaire)
• Moderate risk of bias:
  • Two to four 24-hour urine specimens with reported quality control measures or correction for regression dilution bias with repeated 24-hour urine collection on a sample of participants
  • Multiple days of food diaries
  • Multiple nonconsecutive days (more than 4) 24-hour diet recalls or food records or correction for regression dilution bias with repeated (nonconsecutive) 24-hour diet recalls for a sample of participants
• High risk of bias:
  • 24-hour urine without any reported quality control measures
  • A single 24-hour urine collection (high random error)
  • Timed-urine collection of less than 24 hours
  • Food frequency questionnaire
  • Single-day food diaries/records or 24-hour diet recalls
  • Spot urine with or without use of a prediction equation for estimating 24-hour excretion

Potassium Exposure Assessment

• Low risk of bias:
  • Multiple nonconsecutive days (more than 4) 24-hour diet recalls or food records

• • Multiple (more than four, preferably nonconsecutive) 24-hour urines with reported quality control measures (e.g., instructions given and measure of completeness of collection, such as creatinine, urine volume, questionnaire)
• Moderate risk of bias:
  • Two to four 24-hour urine specimens or correction for regression dilution bias with repeated 24-hour urine collection on a sample of participants
  • Two to four nonconsecutive 24-hour recalls/food records or correction for regression dilution bias with repeated (nonconsecutive) 24-hour diet recalls for a sample of participants
  • Food frequency questionnaire validated for potassium intake within a subset of the study population against duplicate diets or multiple 24-hour urine collections
• High risk of bias:
  • Single 24-hour urine specimen
  • Use of more than one 24-hour urine specimen without any reported quality control measures
  • Timed-urine collection of less than 24 hours
  • Food frequency questionnaire other than that specified above under “Moderate risk of bias”
  • Single-day food records
  • Single day of 24-hour recall
  • Spot urine specimen(s) with or without use of an equation for estimating 24-hour excretion

Demonstration That Outcome of Interest Was Not Present at Start of Study for All Participants

For example, analysis of the number of people with stroke includes patients that had a stroke before the start of the study, not after the intervention, or had a recurring stroke (incidence versus recurrence). This item is a trigger for excluding individual studies from analyses. Please specify for which outcome this is an issue.

Comparability

Comparability of cohorts on the basis of the design or analysis (Was distribution of health status, demographics, and other critical confounding factors similar across study groups at baseline or did the analysis control for baseline differences between groups?)

• Low risk: Study provides explanation for and controls for the most important factors likely to affect outcomes, including blood pressure for non–blood pressure studies or body mass index.
• High risk: Study does not control for blood pressure or other important factors (e.g., demographics).
• Moderate risk or unclear: Study does not describe the exact factors controlled for in analysis.

Assessment of Outcome

Ascertainment of outcome should be appropriate for the type of outcome.

• Low risk: The authors describe independent or blind assessment or confirmation of the outcome by reference to secure records (e.g., X-rays, medical records) or use of record linkage (e.g., identification of outcome through the International Statistical Classification of Diseases and Related Health Problems [ICD] codes on database records).
• High risk: Outcomes are described as being self-reported.
• Moderate risk or unclear: No description.

REFERENCE

This page intentionally left blank.