Journal for Reproducibility in Neuroscience


Supplementary Files

Reviewer's Comments


Systematic Review

How to Cite

Bolzan, J., & Lino de Oliveira, C. (2022). A compact guide to the systematic review and meta-analysis of the literature in neuroscience. Journal for Reproducibility in Neuroscience, 2, 1669.


Critical appraisals of the literature may help to increase reproducibility in neuroscience. Systematic reviews and meta-analyses are tools for neuroscientists to critically evaluate a large amount of knowledge in the research field. These methods involve individually simple decisions, which may become complex when considering the whole process. Strategies to organize the planning and implementation of the protocols minimize the workload. Here, we prepared a compact guide to assist neuroscientists willing to perform a systematic review and meta-analysis of the literature in neuroscience.


Gopalakrishnan S, Ganeshkumar P. Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare. Journal of family medicine and primary care. 2013 2(1), 9–14.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997, Mar 1;126(5):376-80.

Moller AM, Myles PS. What makes a good systematic review and meta-analysis? BJA: British Journal of Anaesthesia, 2016, v.17, n.4, Oct. P. 428–430.

Wormald R, Evans J. What Makes Systematic Reviews Systematic and Why are They the Highest Level of Evidence? Ophthalmic Epidemiology. 2018 25:1, 27-30.

Delgado-Rodríguez M, Sillero-Arenas M. Systematic review and meta-analysis. Med Intensiva (Engl Ed). 2018, Oct;42(7):444-453.

Bashir Y, Conlon KC. Step by step guide to do a systematic review and meta-analysis for medical professionals. Ir J Med Sci. 2018 187, 447–452.

Linares-espinós E, Hernández V, Domínguez-Escrig JL, Fernández-Pello S, Hevia V, Mayor J, Padilla-Fernández B, Ribal MJ. Methodology of a systematic review. Actas Urol Esp (Engl Ed). 2018 Oct;42(8):499-506.

Hooijmans CR, Rovers M, de Vries RB, Leenaars M, Ritskes-Hoitinga M. An initiative to facilitate well-informed decision-making in laboratory animal research: report of the First International Symposium on Systematic Reviews in Laboratory Animal Science. Laboratory Animals. 2012 46(4), 356–357.

Hansen H, Trifkovic N. Systematic Reviews: Questions, Methods and Usage. 2013.

Vesterinen HM, Sena ES, Egan KJ, Hirst TC, Churolov L, Currie GL, Antonic A, Howells DW, Macleod MR. Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods. 2014 Jan 15;221:92-102. Erratum in: J Neurosci Methods. 2016 Feb 1;259:156.

Bannach-Brown A, Hair K, Bahor Z, et al. Technological advances in preclinical meta-research. BMJ Open Science. 2021;5:e100131.

Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan; 4;70:747-770.

Muka T, Glisic M, Milic J. et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol. 2020 35, 49–60.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P). Systematic reviews. 2015 4(1), 1.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.

Dekkers OM, Vandenbroucke JP, Cevallos M, Renehan AG, Altman DG, Egger M. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology. PLoS Med. 2019;16(2):e1002742.

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014 May;114(5):49-56.

Schardt C, Adams MB, Owens T. et al. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007;7, 16.

Cooke A, Smith D, Booth A. Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis. Qualitative Health Research. 2012;22(10):1435-1443.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. J Med Libr Assoc. 2018 Oct;106(4):420-431.

Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evidence Based Library and Information Practice, 2011, 6(2), 75-80.

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration. 2013.

Furukawa T, Ogawa Y, Takeshima N, Hayasaka Y, Chen P, Cipriani A, Barbui C. Bupropion versus other antidepressive agents for depression [Cochrane Protocol]. PROSPERO 2015 CRD42015017732 Available from:

Casarotto P, Cannarozzo C, Rubiolo A. A systematic review of the effect of voluntary exercise in brain plasticity in rats and mice. PROSPERO 2021 CRD42021250561 Available from:

Hooijmans CR, Rovers MM, de Vries RB, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE's risk of bias tool for animal studies. BMC medical research methodology. 2014 14, 43.

Huang W, Percie du Sert N, Vollert J, Rice ASC. General Principles of Preclinical Study Design: Good Research Practice in Non-Clinical Pharmacology and Biomedicine. Handbook of Experimental Pharmacology. 2019, 257.

De Vries RBM, Hooijmans CR, Langendam MW, Luijk JV, Leenaars M, Ritskes-Hoitinga M, Wever KE. A protocol format for the preparation, registration and publication of systematic reviews of animal intervention studies. Evidence-based Preclinical Medicine. 2015 August, 2(1).

Lino de Oliveira C, Bolzan JA. Surget A, Belzung C. Do antidepressants promote neurogenesis in adult hippocampus? A systematic review and meta-analysis on naive rodents. Pharmacology & Therapeutics. 2020 Jun, v. 210.

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February, 2021) [Internet]. 2021. Available from:

Arroyave WD, Mehta SS, Guha N, Schwingl P, Taylor KW., Glenn B, Radke EG, Vilahur N, Carreón T, Nachman RM, Lunn RM. Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health. Journal of exposure science & environmental epidemiology. 2021 31(1), 21–30.

Hooijmans CR, Leenaars M, Ritskes-Hoitinga M. A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim. 2010 38 (2): 167-182.

Ramos-Hryb AB, Bahor Z, McCann S, et al. Protocol for a systematic review and meta-analysis of data from preclinical studies employing forced swimming test: an update. BMJ Open Science. 2019;3:e000035.

Bolzan JA, Lino de Oliveira C. Protocol for systematic review and meta-analysis of the evidence linking hippocampal neurogenesis to the effects of antidepressants on mood and behaviour. BMJ Open Science. 2021;5:e100077.

Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci. 1993 Dec 31; 703:125-33.

Pollock A, Campbell P, Brunton G, Hunt H, Estcourt L. Selecting and implementing overview methods: implications from five exemplar overviews. Systematic Reviews 2017a; 6: 145.

Hohls JK, Konig H, Quirke E, Hajek A. Association between anxiety, depression and quality of life - a systematic review of evidence from longitudinal studies. PROSPERO 2018 CRD42018108008 Available from:

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng H-Y, Corbett MS, Eldridge SM, Hernán MA, Hopewell S, Hróbjartsson A, Junqueira DR, Jüni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT. RoB 2: a revised tool for assessing risk of bias in randomized trials. BMJ. 2019; 366: l4898.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews thatinclude randomized or non-randomized studies of healthcare interventions, or both. BMJ. 2017 Sep 21;358: j4008.

Pérez J, Díaz J, Garcia-Martin J, Tabuenca B. Systematic literature reviews in software engineering—Enhancement of the study selection process using Cohen’s kappa statistic. Journal of Systems and Software, 2020 168, 110657.

Galindo L, Oliveira Neto OB, Pinheiro IL, Silva RPB, Correia L. Effects of environmental enrichment as strategy for ameliorate behavioral effects of social isolation: preclinical systematic review and meta-analysis. PROSPERO 2020 CRD42020200881 Available from:

Van Praag D, Maas A, Wilson L, Polinder S, Cnossen M, Synnot A. Posttraumatic stress disorder after civilian traumatic brain injury: a systematic review and meta-analysis of prevalence rates. PROSPERO 2016 CRD42016029956 Available from:

Eckert F, Triches F, Costa J, Lino de Oliveira C. Antidepressant treatment and behavior responses in Drosophila melanogaster: a systematic review and metanalysis. PROSPERO 2020 CRD42020225423 Available from:

Durlak JA. How to Select, Calculate, and Interpret Effect Sizes, Journal of Pediatric Psychology. 2009 Oct, 34:9; 917–928.

Soflau R, Fodor L, Georgescu R, Cuijpers P, Cristea I. A protocol for an individual patient data meta-analysis comparing cognitive behavioral therapy with control conditions for anxiety disorders. PROSPERO 2020 CRD42020178759 Available from:

Husain I, Cullen C, Umer M, Bonato S. Efficacy and acceptability of pharmacological treatments of anxiety symptoms in bipolar disorder: a systematic review and meta-analysis. PROSPERO 2020 CRD42020188875 Available from:

Haidich AB. Meta-analysis in medical research. Hippokratia. 2010 14(Suppl 1), 29–37.

Valentine JC, Pigott TD, Rothstein H. How Many Studies Do You Need? A Primer on Statistical Power for Meta-Analysis. Journal of Educational and Behavioral Statistics. 2010 June 35(2):215-247.

Griffin JW. MetapoweR: an R package for computing meta-analytic statistical power. R package (Version: 0.2.2) [Internet]. 2020. Available from:

Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to Meta-Analysis. Chichester, West Sussex, UK: John Wiley & Sons. 2009.

McFadzean J, Monson JP, Watson JD, Coakley JH. The dilemma of the incapacitated patient who has previously refused consent for surgery. BMJ. 1997; 315:1530–2.

DeVito NJ, Goldacre B. Catalogue of bias: publication bias. BMJ evidence-based Medicine, 2019 24(2), 53-54.

Duval S, Tweedie R. A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis, Journal of the American Statistical Association. 2000 95:449, 89-98.

Shi L, Lin L. The trim-and-fill method for publication bias: practical guidelines and recommendations based on a large database of meta-analyses. Medicine. 2019 98(23), e15987.

Dias S, Caldwell DM. Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed. 2019, Jan;104(1):F8-F12.

Pozza A, Veale D, Marazziti D, Albert U, Delgadillo J, Grassi G, Prestia D, Dèttore D. “Sex and the OCDity”: sexual dysfunction and satisfaction in obsessive-compulsive disorder: protocol for a systematic review and meta-analysis. PROSPERO 2019 CRD42019132264 Available from:

Santos JF, Brito IRR, Melo IS, Bueno NB, Araujo LA, Castro O. "Psychoactive drugs and the underlying mechanisms that affect the normal development of Drosophila melanogaster: A Systematic Review and meta-analysis". PROSPERO 2020 CRD42020146823 Available from:

Did a change in Nature journals’ editorial policy for life sciences research improve reporting? BMJ Open Science 2019;3:e000035.

Álvarez-Bueno C, Pesce C, Cavero-Redondo I. Association of physical activity with cognition, metacognition and academic performance in children and adolescents: a protocol for systematic review and meta-analysis. BMJ Open 2016;6: e011065.

Suurmond R, van Rhee H, Hak T. Introduction, comparison, and validation of Meta-Essentials: A free and simple tool for meta-analysis. Res Syn Meth. 2017; 8: 537– 553.

Lajeunesse MJ. Facilitating systematic reviews, data extraction and meta‐analysis with the metagear package for R (Version: 0.7). Methods in Ecology and Evolution, 2016 7(3), 323-330.

Rinker T, Kurkiewicz D, Hughitt K, Wang A, Aden-Buie G, Burk L. Package Management Tool, package pacman (Version: 0.5.1). 2019.

Qi X, Yang M, Ren W, Jia J, Wang J, Han G, Fan D. Find duplicates among the PubMed, EMBASE, and Cochrane Library Databases in systematic review. PLoS One. 2013 Aug 20;8(8):e71838.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016 Dec 5;5(1):210.

Elmagarmid A, Fedorowicz Z, Hammady H, Ilyas I, Khabsa M, Ouzzani M. Rayyan: a systematic reviews web app for exploring and filtering searches for eligible studies for Cochrane Reviews. In: Evidence-Informed Publich Health: Opportunities and Challenges. Abstracts of the 22nd Cochrane Colloquium. 2014 Sep 21-26.

Olofsson H, Brolund A, Hellberg C, Silverstein R, Stenstrom K, Osterberg M, Dagerhamn J. Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Research Synthesis Methods. 2017;8(3):275–80.

Wang Q, Liao J, Lapata M, Macleod M. Risk of bias assessment in preclinical literature using natural language processing. Res Synth Methods. 2021 Oct 28. doi: 10.1002/jrsm.1533. Epub ahead of print. PMID: 34709718.

Li TJ, Vedula SS, Hadar N, Parkin C, Lau J, Dickersin K. Innovations in data collection, management, and archiving for systematic reviews. Annals of Internal Medicine 2015; 162: 287-294.

Pick JL, Nakagawa S, Noble DW. “Reproducible, flexible and high throughput data extraction from primary literature: The metaDigitise R package.” BioRxiv. 2018 1–25.

Viechtbauer W. Conducting meta-analyses in R with the metafor package (Version 2.4-0). Journal of Statistical Software, v.36, n.3, p.1-48, 2010.

Schwarzer L, Carpenter JR, Rücker L. Meta-análise com R, use R! Cham: Springer International Publishing. 2015.

Cochrane Informatics and Knowledge Management Department. In: RevMan 5.3. [Internet]. 2014. Available from:

Wallace BC, Lajeunesse MJ, Dietz G, Dahabreh IJ, Trikalinos TA, Schmid CH, Gurevitch J. OpenMEE: Intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods in Ecology and Evolution, 2016.

Schwarzer L, Carpenter JR, Rücker L. Meta-análise com R, use R! Cham: Springer International Publishing. 2015.

Palmer TM, Sterne JAC. Meta-analysis in Stata: An Updated Collection from the Stata Journal. Stata Press. 2016.

Cortese S, Kelly C, Chabernaud C, Proal E, Di Martino A, Milham MP, Castellanos FX. Toward systems neuroscience of ADHD: a meta-analysis of 55 fMRI studies. American Journal of Psychiatry, 2012 169(10), 1038-1055.

Itoh Y, Arnold AP. Are females more variable than males in gene expression? Meta-analysis of microarray datasets. Biology of sex Differences, 2015, 6(1), 1-9.

Becker JB, Prendergast BJ, Liang JW. Female rats are not more variable than male rats: a meta-analysis of neuroscience studies. Biology of sex differences, 2016 7(1), 1-7.

Watzlawick R, Rind J, Sena ES, Brommer B, Zhang T, Kopp MA, Schwab JM. Olfactory ensheathing cell transplantation in experimental spinal cord injury: effect size and reporting bias of 62 experimental treatments: a systematic review and meta-analysis. PLoS biology, 2016 14(5), e1002468.

Heinzel JC, Nguyen MQ, Kefalianakis L, Prahm C, Daigeler A, Hercher D, Kolbenschlag J. A systematic review and meta-analysis of studies comparing muscle-in-vein conduits with autologous nerve grafts for nerve reconstruction. Scientific reports, 2021 11(1), 1-12.

McGuinness LA, Higgins JPT. Risk-of-bias VISualization (robvis): An R package and Shiny web app for visualizing risk-of-bias assessments (Version 0.3.0). Res Synth Methods. 2021 Jan;12(1):55-61. doi: 10.1002/jrsm.1411. Epub 2020 May 6. PMID: 32336025.

Ried K. Interpreting and understanding meta-analysis graphs: a practical guide. Australian family physician, 2006 35(8).

Harbord RM, Higgins JPT. Meta-Regression in Stata. The Stata Journal. 2008;8(4):493-519.

Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Higgins JP. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomized controlled trials. Bmj, 2011 343.

Sedgwick P. Meta-analyses: how to read a funnel plot. Bmj, 2013 346.

Cohen J. The statistical power of abnormal-social psychological research: a review. J Abnorm Soc Psychol. 1962, Sep;65:145-53.

Sawilowsky RM. New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 2009 8, 26.

Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology. 2016, 31(4), 337–350.

Sena ES, van der Worp HB, Bath PMW, Howells DW, Macleod MR. Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy. PLoS Biol 2010 8(3): e1000344.

Ramos-Hryb AB, Harris C, Aighewi O, Lino-de-Oliveira C. How would publication bias distort the estimated effect size of prototypic antidepressants in the forced swim test? Neuroscience & Biobehavioral Reviews. 2018, 92, 192-194.

Creative Commons Attribution Share Alike License Logo

This work is licensed under a Creative Commons Attribution Share Alike 4.0 International License.