Cochrane Handbook for Systematic Reviews of

Part One Core methods 1

1 Starting a review 3

1.1 Why do a systematic review? 3

1.2 What is the review question? 4

1.3 Who should do a systematic review? 5

1.4 The importance of reliability 7

1.5 Protocol development 8

1.6 Data management and quality assurance 11

1.7 Chapter information 12

1.8 References 12

2 Determining the scope of the review and the questions it will address 13

2.1 Rationale for well-formulated questions 13

2.2 Aims of reviews of interventions 15

2.3 Defining the scope of a review question 16

2.4 Ensuring the review addresses the right questions 21

2.5 Methods and tools for structuring the review 24

2.6 Chapter information 29

2.7 References 29

3 Defining the criteria for including studies and how they will be grouped for the synthesis 33

3.1 Introduction 33

3.2 Articulating the review and comparison PICO 35

3.3 Determining which study designs to include 51

3.4 Eligibility based on publication status and language 60

3.5 Chapter information 61

3.6 References 61

4 Searching for and selecting studies 67

4.1 Introduction 68

4.2 General issues 68

4.3 Sources to search 70

4.4 Designing search strategies 79

4.5 Documenting and reporting the search process 90

4.6 Selecting studies 92

4.7 Chapter information 99

4.8 References 99

5 Collecting data 109

5.1 Introduction 109

5.2 Sources of data 110

5.3 What data to collect 114

5.4 Data collection tools 125

5.5 Extracting data from reports 130

5.6 Extracting study results and converting to the desired format 136

5.7 Managing and sharing data 136

5.8 Chapter information 137

5.9 References 137

6 Choosing effect measures and computing estimates of effect 143

6.1 Types of data and effect measures 143

6.2 Study designs and identifying the unit of analysis 145

6.3 Extracting estimates of effect directly 148

6.4 Dichotomous outcome data 150

6.5 Continuous outcome data 156

6.6 Ordinal outcome data and measurement scales 168

6.7 Count and rate data 170

6.8 Time-to-event data 172

6.9 Conditional outcomes only available for subsets of participants 173

6.10 Chapter information 174

6.11 References 174

7 Considering bias and conflicts of interest among the included studies 177

7.1 Introduction 177

7.2 Empirical evidence of bias 180

7.3 General procedures for risk-of-bias assessment 185

7.4 Presentation of assessment of risk of bias 188

7.5 Summary assessments of risk of bias 188

7.6 Incorporating assessment of risk of bias into analyses 190

7.7 Considering risk of bias due to missing results 192

7.8 Considering source of funding and conflict of interest of authors of included studies 193

7.9 Chapter information 199

7.10 References 199

8 Assessing risk of bias in a randomized trial 205

8.1 Introduction 205

8.2 Overview of RoB 2 206

8.3 Bias arising from the randomization process 212

8.4 Bias due to deviations from intended interventions 214

8.5 Bias due to missing outcome data 217

8.6 Bias in measurement of the outcome 220

8.7 Bias in selection of the reported result 221

8.8 Differences from the previous version of the tool 225

8.9 Chapter information 226

8.10 References 227

9 Summarizing study characteristics and preparing for synthesis 229

9.1 Introduction 229

9.2 A general framework for synthesis 230

9.3 Preliminary steps of a synthesis 231

9.4 Checking data before synthesis 238

9.5 Types of synthesis 238

9.6 Chapter information 240

9.7 References 240

10 Analysing data and undertaking meta-analyses 241

10.1 Do not start here! 242

10.2 Introduction to meta-analysis 242

10.3 A generic inverse-variance approach to meta-analysis 245

10.4 Meta-analysis of dichotomous outcomes 246

10.5 Meta-analysis of continuous outcomes 251

10.6 Combining dichotomous and continuous outcomes 254

10.7 Meta-analysis of ordinal outcomes and measurement scales 255

10.8 Meta-analysis of counts and rates 255

10.9 Meta-analysis of time-to-event outcomes 256

10.10 Heterogeneity 257

10.11 Investigating heterogeneity 265

10.12 Missing data 272

10.13 Bayesian approaches to meta-analysis 276

10.14 Sensitivity analyses 277

10.15 Chapter information 279

10.16 References 280

11 Undertaking network meta-analyses 285

11.1 What is network meta-analysis? 285

11.2 Important concepts 287

11.3 Planning a Cochrane Review to compare multiple interventions 293

11.4 Synthesis of results 297

11.5 Evaluating confidence in the results of a network meta-analysis 304

11.6 Presenting network meta-analyses 309

11.7 Concluding remarks 315

11.8 Chapter information 316

11.9 References 316

12 Synthesizing and presenting findings using other methods 321

12.1 Why a meta-analysis of effect estimates may not be possible 321

12.2 Statistical synthesis when meta-analysis of effect estimates is not possible 324

12.3 Visual display and presentation of the data 330

12.4 Worked example 333

12.5 Chapter information 345

12.6 References 346

13 Assessing risk of bias due to missing results in a synthesis 349

13.1 Introduction 350

13.2 Minimizing risk of bias due to missing results 351

13.3 A framework for assessing risk of bias due to missing results in a synthesis 354

13.4 Summary 369

13.5 Chapter information 370

13.6 References 370

14 Completing ‘Summary of findings’ tables and grading the certainty of the evidence 375

14.1 ‘Summary of findings’ tables 375

14.2 Assessing the certainty or quality of a body of evidence 389

14.3 Describing the assessment of the certainty of a body of evidence using the GRADE framework 398

14.4 Chapter information 399

14.5 References 399

15 Interpreting results and drawing conclusions 403

15.1 Introduction 403

15.2 Issues of indirectness and applicability 405

15.3 Interpreting results of statistical analyses 408

15.4 Interpreting results from dichotomous outcomes (including numbers needed to treat) 411

15.5 Interpreting results from continuous outcomes (including standardized mean differences) 416

15.6 Drawing conclusions 422

15.7 Chapter information 427

15.8 References 428

Part Two Specific perspectives in reviews 433

16 Equity and specific populations 435

16.1 Introduction to equity in systematic reviews 435

16.2 Formulation of the review 437

16.3 Identification of evidence 441

16.4 Appraisal of evidence 443

16.5 Synthesis of evidence 443

16.6 Interpretation of evidence 444

16.7 Concluding remarks 445

16.8 Chapter information 445

16.9 References 445

17 Intervention complexity 451

17.1 Introduction 451

17.2 Formulation of the review 461

17.3 Identification of evidence 468

17.4 Appraisal of evidence 469

17.5 Synthesis of evidence 469

17.6 Interpretation of evidence 472

17.7 Chapter information 473

17.8 References 474

18 Patient-reported outcomes 479

18.1 Introduction to patient-reported outcomes 479

18.2 Formulation of the review 480

18.3 Appraisal of evidence 482

18.4 Synthesis and interpretation of evidence 485

18.5 Chapter information 488

18.6 References 489

19 Adverse effects 493

19.1 Introduction to issues in addressing adverse effects 493

19.2 Formulation of the review 496

19.3 Identification of evidence 500

19.4 Appraisal of evidence 502

19.5 Synthesis and interpretation of evidence 503

19.6 Chapter information 504

19.7 References 505

20 Economic evidence 507

20.1 Introduction 507

20.2 Formulation of the review 512

20.3 Identification of evidence 517

20.4 Appraisal of evidence 519

20.5 Synthesis and interpretation of evidence 519

20.6 Chapter information 521

20.7 References 522

21 Qualitative evidence 525

21.1 Introduction 525

21.2 Designs for synthesizing and integrating qualitative evidence with intervention reviews 526

21.3 Defining qualitative evidence and studies 527

21.4 Planning a qualitative evidence synthesis linked to an intervention review 528

21.5 Question development 529

21.6 Questions exploring intervention implementation 530

21.7 Searching for qualitative evidence 531

21.8 Assessing methodological strengths and limitations of qualitative studies 532

21.9 Selecting studies to synthesize 533

21.10 Selecting a qualitative evidence synthesis and data extraction method 534

21.11 Data extraction 534

21.12 Assessing the confidence in qualitative synthesized findings 537

21.13 Methods for integrating the qualitative evidence synthesis with an intervention review 537

21.14 Reporting the protocol and qualitative evidence synthesis 538

21.15 Chapter information 539

21.16 References 539

Part Three Further topics 547

22 Prospective approaches to accumulating evidence 549

22.1 Introduction 549

22.2 Evidence surveillance: active monitoring of the accumulating evidence 550

22.3 Prospectively planned meta-analysis 554

22.4 Statistical analysis of accumulating evidence 561

22.5 Chapter information 564

22.6 References 565

23 Including variants on randomized trials 569

23.1 Cluster-randomized trials 569

23.2 Crossover trials 576

23.3 Studies with more than two intervention groups 585

23.4 Chapter information 590

23.5 References 591

24 Including non-randomized studies on intervention effects 595

24.1 Introduction 595

24.2 Developing criteria for including non-randomized studies of interventions 601

24.3 Searching for non-randomized studies of interventions 606

24.4 Selecting studies and collecting data 608

24.5 Assessing risk of bias in non-randomized studies 610

24.6 Synthesis of results from non-randomized studies 611

24.7 Interpretation and discussion 614

24.8 Chapter information 617

24.9 References 617

25 Assessing risk of bias in a non-randomized study 621

25.1 Introduction 622

25.2 Biases in non-randomized studies 623

25.3 The ROBINS-I tool 626

25.4 Risk of bias in follow-up (cohort) studies 632

25.5 Risk of bias in uncontrolled before-after studies (including interrupted time series) 635

25.6 Risk of bias in controlled before-after studies 638

25.7 Chapter information 640

25.8 References 640

26 Individual participant data 643

26.1 Introduction 643

26.2 Collecting IPD 647

26.3 Managing and checking IPD 650

26.4 Analysis of IPD 652

26.5 Reporting IPD reviews 655

26.6 Appraising the quality of IPD reviews 655

26.7 Chapter information 655

26.8 References 655

Index 659