Replicability in psychological research: a reflection
La replicabilidad en la investigación psicológica: una reflexión
Sharon Centeno-Leyva 1 and Sergio Dominguez-Lara 1 *
1 University of San Martín de Porres, Lima, Peru.
* Correspondence: Sergio Dominguez-Lara. Tomás Marsano 242 (5th floor), Lima 34, Peru. Mail:firstname.lastname@example.org
Received: June 07, 2020 | Revised: August 02, 2020 | Accepted: September 07, 2020 | Published Online: September 16, 2020
CITE IT AS:
Background: In recent years, psychological science has suffered a crisis of confidence that has been marked by the low rate of replicability demonstrated in collaborative projects that attempted to quantify this problem, evidencing the difficulty in making replications and the existence of a possible excess of false positives published in the scientific literature. Methods: This opinion article aimed to review the panorama of the replicability crisis in psychology, as well as its possible causes. Conclusions: It began from the state of the replicability crisis, then some possible causes and their repercussions on the advancement of psychological science were highlighted, discussing various associated issues, such as individual biases on the part of researchers, the lack of incentives to replicability studies and the priority standards that journals would currently have for novel and positive studies. Finally, the existing alternatives to reverse this situation are mentioned, among them the opening to new statistical approaches, the restructuring of incentives and the development of editorial policies that facilitate the means for replication.
Keywords: Replicability; Psychological Research; Scientific Publication.
Introducción: En los últimos años, la ciencia psicológica ha sufrido una crisis de confianza que ha sido marcada por la baja tasa de replicabilidad demostrada en proyectos colaborativos que intentaron cuantificar esta problemática, evidenciando la dificultad para realizar replicaciones y la existencia de un posible exceso de falsos positivos publicados en la literatura científica. Método: Este artículo de opinión tuvo como objetivo realizar una revisión del panorama de la crisis de replicabilidad en psicología, así como también a sus posibles causas. Conclusiones: Se inició desde el estado de la crisis de replicabilidad, luego se destacaron algunas posibles causas y sus repercusiones en el avance de la ciencia psicológica, debatiendo sobre diversos temas asociados, como los sesgos individuales por parte de los investigadores, la falta de incentivos a los estudios de replicabilidad y los estándares de prioridad que actualmente tendrían las revistas por los estudios novedosos y positivos. Finalmente se mencionan las alternativas existentes para revertir esta situación, entre ellas la apertura a nuevos enfoques estadísticos, la reestructuración de incentivos y el desarrollo de políticas editoriales que faciliten los medios para las replicaciones.
Palabras clave: Replicabilidad; Investigación psicológica; Publicación científica.
Brief overview of the replicability of studies
The publication in high-impact journals of a large number of questionable scientific studies heightened a controversy over the quality criteria of scientific production that was already pre-existing (Gelman & Loken, 2014). These controversies in psychology have stimulated conversations about the nature and quality of psychological research.(Makel et al., 2012). Therefore, a large number of psychologists are concerned about a possible excess of false positives in the scientific literature(Maxwell et al., 2015) and this concern is not recent (Agnoli et al., 2017; John PA Ioannidis, 2005). Simmons et al. (2011) would affirm that the probability that a researcher finds evidence that there is an effect is higher than the probability of correctly finding the evidence that it does not, which leads to the conclusion that there is a crisis of confidence in psychological science (Pashler & Wagenmakers, 2012) which has generated a repeated discussion about the importance of replication (Earp & Trafimow, 2015).
To a large extent, the credibility of a science depends on the possibility of replicating its findings, that is, reproducing the effects that are reported in original studies. In this sense, replicability becomes an essential procedure and one of the criteria used in scientific research to guarantee the validity of knowledge.(Blanco et al., 2018). For example, a replication experiment that demonstrates that the same results can be obtained elsewhere or with another researcher is conceived as the operationalization of objectivity, and although replication is of great importance(Schmidt, 2009), replication studies rarely appear in psychology journals(Association for Psychological Science, 2013), since only a minimum percentage (1.07%) corresponds to replicated studies (Makel et al., 2012). This situation is due to a lower assessment of replication studies compared to original (or novel) studies by the editorial board of the journals, generated by an emphasis on novelty (Stevens, 2017).
In view of this situation, one of the largest projects that attempted to quantify replicability problems in psychology was the Reproducibility Project: Psychology(Open Science Collaboration, 2015)where they tried to replicate 100 studies drawn randomly from three of the most prestigious journals, of which 97 reported significant results. However, only 36% of the replication studies did. Furthermore, the magnitude of the effect of the aftershocks was half that observed in the original studies. These results fueled the debate on the conclusions of the low rate of replicability in psychology(Protzko & Schooler, 2020), showing that the replicability rate in social psychology is only 25% and in cognitive psychology 50% (Open Science Collaboration, 2015). On the other hand, although clinical and school psychology have not been included very frequently in studies and discussions on replicability and are even considered “isolated”, it is unlikely that they are immune to this crisis.(Tackett et al., 2017). Even in view of the heterogeneity of the educational systems in the different countries, it is highly probable that the replicability rate is even lower.
The results of the replicability studies have not been encouraging, however, the authors themselves point out the need to carry out these studies in other contexts in order to determine the influence that could exist due to cultural differences.
Types of replication
In previous paragraphs it was indicated that replicability is the cornerstone of science (Open Science Collaboration, 2012); However, this does not coincide with the number of replications observed in the literature, and although in recent years researchers have shown more and more concern to know if the findings published in the literature are actually replicable(Świątkowski & Dompnier, 2017), the researchers themselves do not conduct replication studies. This situation opens up a series of questions: What are the reasons why, despite knowing its importance, a considerable number of these studies are not reflected in the scientific literature? and an even more important point, how can it be possible to promote the development of more replication studies?
It is important to know that a replication study is a procedure used to verify the veracity and accuracy of reported results. In a review of the literature it can be found that several types of replication were identified. In psychology, they are classified into direct replicas and conceptual replicas (Schmidt, 2009). When researchers repeat all relevant aspects of an original study and try to be as similar as possible to this one, it is defined as direct replication, which must be distinguished from conceptual replications, which are those that seek to test the hypothesis of previous studies using a different research design(Makel et al., 2012). Direct replication is rare(Patil et al., 2016) and although conceptual replicas can be found occasionally, they often lack a systematic approach (Schmidt, 2009). This has led to the conclusion that there is a replicability crisis in psychology(Anvari and Lakens, 2018), which has prompted a process to review the scientific evidence to generate the opportunity to reconsider some current practices that could be part of the origin of this crisis.
What prevents the replication of studies?
Some of these causes have been mentioned by various authors, many of whom have agreed that there are individual factors(Blanco et al., 2018), decision-making or degrees of freedom of the researcher(Gelman & Loken, 2014) as well as contextual factors, among which are included the pressure of some journals for the publication of positive studies, as well as the competitiveness and need for novelty in research and the lack of acceptance and adequate incentives for replicability studies (Blanco et al., 2017; John et al., 2012; Patil et al., 2016; Świątkowski & Dompnier, 2017).
Starting with individual factors, some studies connect this crisis with the serious flaws of the null hypothesis significance tests (NHST; Cumming, 2014; Savalei & Dunn, 2015), identifying as an underlying cause the dependence that would still have on these significance tests(Collins & Tabak, 2014), to the imperative of achieving statistical significance and the adoption of important search behaviors from various fields, selectively highlighting positive results(Brunner & Schimmack, 2016). For example, one of the things that happens in this regard is that researchers can become victims of confirmation bias and focus only on positive hypothesis confirmations, since by analyzing the data, researchers could report results that confirm their hypotheses while ignoring others who don't(Bakker et al., 2012; Bakker and Wicherts, 2011). These practices are related to the behavior and decision of the researchers themselves; However; Is it just a publication bias at the individual academic level? The refusal by the researchers to send null results for publication in addition to being able to signify conflict with the theoretical models, represent for them the belief that they will not be published(Ferguson and Heene, 2012); which would relate this individual factor to a publication bias, and the rejection of null results.(Świątkowski and Dompnier, 2017; Wagenmakers et al., 2011; Wetzels et al., 2011).
Likewise, it is also possible to observe that in some cases the hypotheses “emerge” as the data are explored instead of being formulated a priori, analyzing the data as necessary until the positive result is found to be published. This practice is known as p-hacking (multiple undisclosed tests without adjustment;Wicherts et al., 2016) that generate an excess of reports of significant results (Stevens, 2017). These practices are disclosed in the scientific literature as questionable research practices (PIC; John et al., 2012; Protzko & Schooler, 2020) that weaken the credibility and reproducibility of research results are very common today (Agnoli et al., 2017) and it brings with it the possibility of losing objectivity, maintaining theories in the face of inadequate tests, and therefore distorting the real scientific evidence.
Therefore, the statistical power would then be recognized as one of the reasons why replication studies fail to find the original effect, coupled with the fact that several studies have discovered direct and indirect evidence of the use of PIC among psychologists, such as biases selective in the information of methods and results, information of exploratory analyzes as confirmatory and generalized errors of portability of statistical results, demonstrating that the prevalence of PIC in different countries that include up to ten of these practices reaches more than a quarter of the all those surveyed in the studies, and those who have used a PIC more than 80%(Agnoli et al., 2017; Fiedler & Schwarz, 2016; John et al., 2012). For this reason, a possibility of generalization arises. However, it would be necessary for more studies to be carried out to determine the extent of these practices, especially in Latin American countries.
In addition to the individual factors, contextual factors are added that currently continue to respond to the lack of replications in psychology, but above all to the absence of replication studies, and the presence of the PICs already mentioned. Is there currently pressure to publish positive and novel results? Several authors over the years have affirmed this and it is that the systematic publication of studies that depend on their result, and where the published articles only include confirmatory results of the hypotheses presented, represent 90% of the volume of publication in psychology(Blanco et al., 2018; Fanelli, 2010). Statistically significant positive results are published more easily than statistically non-significant negative results, this problem is known as the "file drawer"(Fanelli, 2012; Ferguson and Heene, 2012)and it is related to making decisions based on the p-value, one of the limitations of the NHST procedure mentioned above. A significant p-value leads to the rejection of the null hypothesis, however, a non-significant p-value does not lead to the acceptance of the null hypothesis, which makes it inconclusive and difficult to interpret results.(Cohen, 1994); this being a reason why reviewers may prefer positive results and for them more conclusive.
If there is a pressure to publish positive results and a publication bias against negative results; researchers will be able to focus their interest more on obtaining “successful” results (Giner-Sorolla, 2012), although this involves the use of PIC (John et al., 2012; Schimmack, 2012; Simmons et al., 2011), and set aside negative results; With all this, it would not be surprising if the publication of null results decreased over time (Fanelli, 2012).
In addition, not only the significant results, but those that are considered novel are the ones that are more easily published. (Klein et al., 2014). This publication bias represents the direct distortion of scientific knowledge, all results regardless of their additions are relevant to science, provided they have been produced by solid methods(Fanelli, 2010). Is it important to publish negative results? Some authors mention that they represent 50% of knowledge, therefore their level of importance would be equivalent to that of positive results (Culebras, 2016); and the fact is that its disclosure would represent an increase in the knowledge of other researchers, by making it easier for them to know the negative results of research in the field of interest; it would also provide an optimization of expenses in human and material resources since in many cases it would mean that researchers can make modifications in their studies if what has already been discovered were disseminated (Tárraga and Rodríguez, 2016); Furthermore, if negative results are systematically omitted from the scientific literature, it is distorted and prevents its clarity for future studies or procedures such as analytical tests.(Świątkowski and Dompnier, 2017). Added to this, if there is an emphasis on novelty, a cumulative science construction is prevented; if journals today focus their acceptance criteria on studies with novel information, the importance of replications is neglected (Makel et al., 2012) despite the fact that the research community would collectively benefit if researchers made replicas (Koole and Lakens, 2012), the current situation of what can be published introduces biases that prevent researchers from taking the risk of taking this path.
To exemplify this situation, the 45 Latin American psychology journals included in the Scimago Journal & Country Rank for the year 2019 were reviewed and it was found that a considerable percentage (around 40%) discourages contributions in the form of replies, including as part of their evaluation criteria the theoretical and empirical novelty and originality in the studies that aspire to be published. Likewise, around 55% of these journals do not include information in their editorial evaluation criteria about replications, thus granting minimal chances of acceptance. Finally, only a small percentage leaves open the possibility of receiving replicate studies.(van Dijk et al., 2014).
Therefore, if these journals prioritize novelty, researchers will maximize their number of publications by concentrating on finding novel results.(Higginson & Munafò, 2016) Leaving out replication jobs, since they could risk immediate rejection if they submit these types of studies, so they are unlikely to be motivated to ever run them.
Why is this a problem?
The replica is a mechanism that allows to verify empirical experiences and knowledge exploration that enrich the different approaches to a problem (Ordoñez-Morales, 2014). If there is a crisis of replicability in psychology, not only is a process of adaptation of original designs limited, but also the new learning spaces that these can generate, as well as critical analysis, rethinking of problems and results that in greater Sometimes they are presented as the last word that other researchers, psychologists and students must accept unconditionally.
For this reason, the lack of replicability represents not only one but several problems in the different areas of psychology, the absence of this criterion obstructs the verification of the evidence. In clinical psychology, should a therapy or intervention that is not empirically supported be applied? The answer could be no, however, if you only take into account the positive results of an investigation that concludes the validity of a procedure or method, and thereby ensure its full validity, you are going against the advancement of psychology.
In previous paragraphs it was mentioned that clinical psychology and related areas have been little participants in the replicability discussions, representing a loss of opportunity for clinical scientists and the field in general, and although possible causes of this isolation have been attributed, has emphasized the concern that exists for this, given that currently the critical research evidence would not have a greater rigor, for example, the criteria of empirically supported therapies (ERT) in psychological interventions for specific disorders, only require two positive results to give this support, regardless of the number of negative results that may be obtained; in the context of negative results, these positive results may be attributable to chance(Tackett et al., 2017). So, if this information reaches other professionals as endorsed, they will choose to use it in their various interventions without a verification of procedures, methods or theories. However, clinical scientists are likely to have greater difficulties in implementing changes related to the points mentioned above, clinical research would benefit from the implementation of recommendations to improve replicability in the field by allowing their progress with empirically supported evaluation and intervention.
Likewise, in the field of psychometry it is generally accepted that a single instrumental study concludes that the instrument studied has sufficient metric qualities, without considering that obtaining validity evidence is a continuous process and various studies are required to consolidate the findings (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014).), Since it is possible that, for example, the internal structure found obeys to a greater extent the characteristics of the sample, but does not represent the construct. In addition, the use of outdated or inadequate procedures is also common (see Lloret, Ferreres, Hernández, & Tomás, 2014), so additional studies would serve to contrast and refute these findings (see Dominguez-Lara & Navarro-Loli, 2018).
In many cases, the results of research can fall into the points mentioned above, and be overestimated in magnitude, which implies the need for replicability studies that allow greater rigor to the research evidence.
What alternatives are there?
Some authors responding to the factors that contribute to this so-called replicability crisis describe methods and recommendations for this to change. Among these proposals is to implement statistical approaches that emphasize the estimation of parameters instead of the NHST(Asendorpf et al., 2013). In this sense, Bayesian statistics are presented as an alternative to remedy the bias against publications of non-significant results.(Świątkowski & Dompnier, 2017). Currently, more journals are added to require the approach of statistical power and sample size, allowing the use of new approaches such as Bayesian statistics and meta-analysis.
Consideration should be given to restructuring the incentive to replicate publications, since if the replicas remain unknown to the scientific community, researchers will not be motivated to carry out this type of study. Therefore, there must be outlets so that they can publish, either with separate space for replications in scientific journals or open access to the data as online material together with the original studies. (Asendorpf et al., 2013; Koole and Lakens, 2012). Likewise, a meta-analytic view of scientific knowledge could be considered(Cumming, 2014), considering that the value of a publication is many times determined by the amount of citations it can get (van Dijk et al., 2014), since quoting an original article could make a cocitation of the replication file; this could mean that at the same time as replicas get more citations, the original studies also benefit in the same way (Koole and Lakens, 2012).
Finally, the development of editorial policies that generate access to pre-registration and open data should be encouraged; which includes pre-registering hypotheses, designing, analyzing and making the collected data open access(Gelman & Loken, 2014)for example; archiving data, scripts, analysis and a description of them in a public way, would allow full access to other researchers and, therefore, facilitate replication. In addition, reproducible publishing workflows should be considered, allowing for process monitoring, and providing researchers with a space to store IRB materials, data, analysis scripts, etc. (Nosek et al. 2012); such as the Open Science Framework, which provides research with a hypothesis pre-registration platform and collection and analysis using anonymous data available in public mode after the study, which allows the materials to be available to other researchers (Tackett et al., 2017). If the journals implemented these spaces, it would prevent the initial research hypotheses of the studies from being modified at their convenience, the ICPs among researchers would be reduced, and the rate of false positives in the scientific literature would be lower. It should be considered that the registry can be the most accurate, transparent and reliable; however, the evidence shows how far we are from this ideal, and continues to provide studies that cannot be reproduced or evaluated; if the registration of a study is not adequate, despite this being very good it becomes unusable (Zúñiga, 2019). As a possible solution to this problem, it is important that authors and editors rely on tools that guide the writing and publication of scientific papers, as indicated by the EQUATOR network (Enhancing the Quality and Transparency of health Research), which will allow improvement the reliability and value of the research literature. The EQUATOR network is an international initiative established to promote high-quality reports on health research studies (Simera et al., 2010)and provides tools for authors including reporting guidance resources; guidelines for planning, writing, ethical considerations, sharing and publishing data. It also contains tools for peer reviewers evaluating research manuscripts, as well as for editors considering implementing policies that help improve the accuracy and integrity of reports; and value a content that guarantees the result of a transparent, reproducible and quality investigation.
Some journals have taken steps to improve the quality of research they publish (Ioannidis et al., 2010), thus contemplating the possibility of publishing replicas or null results; for example Adavances in Methods and Practices in Psychological Science has incorporated a section destined to compile replication studies; The new type of article, Registerd Replication Reports, aims to strengthen the foundation of psychological science by encouraging the publication of replications based on a shared and vetted protocol.
For research in science in general replicability is a prerequisite for obtaining valid conclusions, so it is necessary to incentivize these studies in psychology, and although there are barriers and challenges to implement them, and even more so in the field of scientific publications, these changes will be part of an improvement in psychological science as it would improve its quality and reputation as confidence for current publications increases. It is important to discover, but equally so is replicating, and in this way psychology could be regarded as a more rigorous and solid science.
Sharon Centeno-Leyva: Conceptualization, Research, Original Writing-draft.
Sergio Dominguez-Lara: Conceptualization, Writing-review and editing.
This study was self-funded.
CONFLICT OF INTERESTS
The authors indicate that there are no conflicts of interest with the writing of the manuscript.
Does not apply.
This study has been peer-reviewed in a double-blind manner.
DECLARATION OF DATA AVAILABILITY
Does not apply.
The authors are responsible for all statements made in this article. Neither Interactions nor the Peruvian Institute of Psychological Orientation are responsible for the statements made in this document.
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017). Questionable research practices among italian research psychologists. PLOS ONE, 12(3). https://doi.org/10.1371/journal.pone.0172792
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Anvari, F., & Lakens, D. (2018). The replicability crisis and public trust in psychological science. Comprehensive Results in Social Psychology, 3(3), 266–286. https://doi.org/10.1080/23743603.2019.1684822
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., Perugini, M., Roberts, B. W., Schmitt, M., van Aken, M. A. G., Weber, H., & Wicherts, J. M. (2013). Recommendations for Increasing Replicability in Psychology. European Journal of Personality, 27(2), 108–119. https://doi.org/10.1002/per.1919
Association for Psychological Science. (2013). Leading Psychological Science Journal Launches Initiative on Research Replication. Disponible en línea en: https://www.psychologicalscience.org/news/releases/initiative-on-research-replication.html (Consultado el 26 de junio de 2020).
Ato, M., López, J., & Benavente, A. (2013). Un sistema de clasificación de los diseños de investigación en psicología. Anales de Psicología, 29(3), 1038–1059. http://dx.doi.org/10.6018/analesps.29.3.178511
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060.
Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666–678. https://doi.org/10.3758/s13428-011-0089-5.
Blanco, F., Perales López, J. C., & Vadillo, M. A. (2018). Pot la psicologia rescatar-se a si mateixa?. Incentius, biaix i replicabilitat. Anuari de Psicologia de La Societat Valenciana de Psicologia, 18(2), 231–252. https://doi.org/10.7203/anuari.psicologia.18.2.231.
Brunner, J., & Schimmack, U. (2016). How replicable is psychology? A comparison of four methods of estimating replicability on the basis of test statistics in original studies. http://www.utstat.utoronto.ca/~brunner/zcurve2016/HowReplicable.pdf.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037/0003-066X.49.12.997.
Collins, F. S., & Tabak, L. A. (2014). Policy: NIH plans to enhance reproducibility. Nature, 505(7485), 612.
Culebras, J. M. (2016). Resultados negativos, cincuenta por ciento del conocimiento. Journal of Negative & No Positive Results, 1(1), 1–2. https://doi.org/10.19230/jonnpr.2016.1.1.926.
Cumming, G. (2014). The New Statistics. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966.
Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6(621). https://doi.org/10.3389/fpsyg.2015.00621.
Fanelli, D. (2010). Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data. PLoS ONE, 5(4), e10271. https://doi.org/10.1371/journal.pone.0010271.
Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. https://doi.org/10.1007/s11192-011-0494-7.
Ferguson, C. J., & Heene, M. (2012). A Vast Graveyard of Undead Theories. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059.
Fiedler, K., & Schwarz, N. (2016). Questionable Research Practices Revisited. Social Psychological and Personality Science, 7(1), 45–52. https://doi.org/10.1177/1948550615612150.
Gelman, A., & Loken, E. (2014). The Statistical Crisis in Science. American Scientist, 102(6), 460. https://doi.org/10.1511/2014.460.
Giner-Sorolla, R. (2012). Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science. Perspectives on Psychological Science, 7(6), 562–571. https://doi.org/10.1177/1745691612457576.
Higginson, A. D., & Munafò, M. R. (2016). Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. PLOS Biology, 14(11), e2000995. https://doi.org/10.1371/journal.pbio.2000995.
Ioannidis, John P.A., Tatsioni, A., & Karassa, F. B. (2010). A vision for the European Journal of Clinical Investigation: note from the new editors. European Journal of Clinical Investigation, 40(1), 1–3. https://doi.org/10.1111/j.1365-2362.2009.02229.x.
Ioannidis, John P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953.
Klein, R. A., Ratliff, K. A., Vianello, M., Adams Jr, R. B., Bahník, S., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Zeynep Cemalcilar, C. C. B., Chandler, J., Cheong, W., David, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., … Nosek, B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000.
Koole, S. L., & Lakens, D. (2012). Rewarding Replications. Perspectives on Psychological Science, 7(6), 608–614. https://doi.org/10.1177/1745691612462586.
Lloret, S., Ferreres, A., Hernández, A., & Tomás, I. (2014). El análisis factorial exploratorio de los ítems: una guía práctica, revisada y actualizada. Anales de de Psicología, 30(3): 1151-1169. http://dx.doi.org/10.6018/analesps.30.3.199361.
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research. Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688.
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. https://doi.org/10.1037/a0039400.
Open Science Collaboration. (2012). An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science. Perspectives on Psychological Science, 7(6), 657–660. https://doi.org/10.1177/1745691612462588.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716.
Ordoñez Morales, O. (2014). Replicar para comprender: prácticas investigativas para promover el razonamiento científico en estudiantes de psicología. Pensamiento Psicológico, 12(2). https://doi.org/10.11144/Javerianacali.PPSI12-2.rcpi.
Pashler, H., & Wagenmakers, E. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science. Perspectives on Psychological Science, 7(6), 528–530. https://doi.org/10.1177/1745691612465253.
Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539–544. https://doi.org/10.1177/1745691616646366.
Protzko, J., & Schooler, J. W. (2020). No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications. PeerJ, 8, e8014. https://doi.org/10.7717/peerj.8014.
Savalei, V., & Dunn, E. (2015). Is the call to abandon p-values the red herring of the replicability crisis? Frontiers in Psychology, 6, 245. https://doi.org/10.3389/fpsyg.2015.00245.
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551–566. https://doi.org/10.1037/a0029487.
Schmidt, S. (2009). Shall we really Do It Again? The Powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100.
Simera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010). Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Medicine, 8(1), 24. https://doi.org/10.1186/1741-7015-8-24.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
Stevens, J. R. (2017). Replicability and Reproducibility in Comparative Psychology. Frontiers in Psychology, 8, 862. https://doi.org/10.3389/fpsyg.2017.00862.
Świątkowski, W., & Dompnier, B. (2017). Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future. International Review of Social Psychology, 30(1), 111. https://doi.org/10.5334/irsp.66.
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. Perspectives on Psychological Science, 12(5), 742–756. https://doi.org/10.1177/1745691617690042.
Tárraga López, P. J., & Rodríguez Montes, J. A. (2016). ¿Se deben publicar los resultados negativos o no positivos? Journal of Negative & No Positive Results, 1(2), 43–44. https://doi.org/10.19230/jonnpr.2016.1.2.928.
van Dijk, D., Manor, O., & Carey, L. B. (2014). Publication metrics and success on the academic job market. Current Biology, 24(11), R516–R517. https://doi.org/10.1016/j.cub.2014.04.039.
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790.
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical Evidence in Experimental Psychology. Perspectives on Psychological Science, 6(3), 291–298. https://doi.org/10.1177/1745691611406923.
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832.
Zúñiga Rosales, Y. (2019). Red EQUATOR: el uso de guías de reporte para garantizar una publicación de calidad. Revista Cubana de Genética Comunitaria, 11(1), 4–6. http://revgenetica.sld.cu/index.php/gen/article/view/23/28.