Received date: Jan 24, 2016 Accepted date: Feb 25, 2016 Published date: Feb 27, 2016
Citation: Wikoff DS, Britt JK (2016) The Role of Systematic Review in the Practice of Toxicology and Risk Assessment–An Appreciation for the Primary Tool in Evidence-Based Practice. Toxicol open access 2:110. doi:10.4172/2476-2067.1000110
Copyright: © 2016 Wikoff DS, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at Toxicology: Open Access
Use of systematic reviews (SRs) as a tool to facilitate evidence-based toxicology (EBT) assessments is increasing, though the field has yet to develop an appreciation of the rigor required to appropriately utilize this tool. Toxicologists should recognize the weight of the term, understanding that a systematic review involves far more than conducting elements of a review systematically. Key aspects that appear to be currently underappreciated include development and publication of a protocol, the level of documentation involved in the conduct of a SR, and the overall level of effort required to maintain standards of SR. As many regulatory agencies and health organizations integrate systematic review into their procedures, it is clear that there is a need to develop best-science practices in EBT, as the methods developed for evidence-based medicine (EBM) do not always provide the best platform for evaluation of toxicological data. Such efforts are particularly needed for evidence integration, methods which allow for integration of multiple types of data, as well as application of the SR in both qualitative and quantitative hazard or risk assessments. Nonetheless, use of systematic review is advancing the field of toxicology, providing objectivity and transparency in evidence-based assessments.
The use of systematic reviews (SRs), which has long-been used in the fields of medicine and other scientific disciplines, as a tool in the field of toxicology is gaining significant interest [1-4]. This tool clearly aids in modernization of evidence-based decision making, though the field as a whole has yet to develop an appreciation of the rigor required to adequately utilize the systematic review as the primary tool in evidence-based toxicology (EBT). By definition, systematic review is a method for answering specific research questions – it uses a predefined, multistep process to identify, select, critically assess, and synthesize evidence from scientific studies to reach a conclusion [5,6]. Many systematic review frameworks exist within the field of evidencebased medicine (EBM) (e.g., IOM, AHRQ, GRADE, PRISMA); however, fewer frameworks and guidance tailored to EBT are available. Though not a comprehensive review, in this commentary, the role of systematic review in toxicology is highlighted, and in doing so, should provide an appreciation for the rigor and resources required to appropriately utilize this tool, as well as the need for continued development of best-science practices in EBT.
Efforts to integrate systematic review in the field of toxicology are substantiated by decades of use of systematic review as a tool in evidence-based medicine (EBM), as highlighted by the existence and operations of large organizations devoted to the systematic assessment of healthcare interventions, such as Cochrane and the Agency for Healthcare Research and Quality (AHRQ). Guzelian et al. were early adopters of the evidence-based vision, issuing a framework for evidence-based toxicology (EBT) in 2005 that was specifically focused on the determination of causation. Since that time, many efforts in the field involving regulatory, academic, non-profit, and private entities have furthered the integration of evidence-based practices into toxicology. For example, in 2011, the National Research Council (NRC) recommended that the U.S. Environmental Protection Agency (USEPA) utilize a consistent, transparent, and systematic approach for the identification, evaluation, and integration of data for assessing hazards to human health ; these recommendations were further delineated in 2014 . As a result of such, the Integrated Risk Information System (IRIS) Program is currently in the process of developing and implementing a systematic review process. The USEPA has also issued a draft handbook on the conduct of SRs in the IRIS program and is due to release an updated version imminently.
Other key efforts are highlighted by those from the National Toxicology Program’s Office of Health Assessment and Translation (OHAT); in 2012, OHAT began developing an approach for the implementation of SR methodology to carry out literature-based evaluations to reach conclusions about potential health hazards. In 2014, the group published an approach, followed by the issuance of a handbook and RoB tool in early 2015 . The U.S. Food and Drug Administration (FDA)’s  Center for Food Safety and Applied Nutrition (CFSAN), which has already been advocating the use of evidence-based review methods [11-13] recently (June, 2015) held a colloquium with the Society of Toxicology regarding SR; topics were based on integration of SR in human health assessments and included problem formulation and scoping, identification and selection of evidence based, harmonizing dose-response, and use of mechanistic data. State and other health organizations in the U.S. are also integrating SR; for example, in November of 2014, the Texas Commission on Environmental Quality (TCEQ) issued a position paper on Recommendations for Systematic Review and Evidence Integration. The International Agency on for Research on Cancer (IARC) is also utilizing elements of systematic review, as highlighted by a recent publication by Smith et al. in which the authors provide ten key characteristics of carcinogens as a basis for organizing data on the mechanisms of carcinogens. Other independent organizational efforts have made significant efforts to provide direction and examples of the integration of SR into toxicology, highlighted perhaps by those from, the Navigation Guide, a University of California, San Francisco Program on Reproductive Health and the Environment, as well as the Evidence Based Toxicology Collaboration , based out of the Johns Hopkins Bloomberg School of Public Health.
Most systematic review frameworks have common components including: problem formulation and protocol development, identification and evaluation of individual studies, assessment and integration of the body of evidence (qualitatively or quantitatively), and reporting of the SR. Problem formulation and protocol development is a common exercise in the field of EBM, though the rigor and resources needed to conduct this phase appear to be particularly underappreciated in the field of EBT. The early decisions made during protocol development have significant impact on the scope and form of the systematic review, thus underscoring the critical nature of a well-developed approach. The most obvious exercise in problem formulation involves development of objectives and/or hypothesis; in SR, this exercise is carried out via the development of PECO (population, exposure, comparator, and outcome) statements and associated rationale. Not as obvious, however, is the critical nature of well-formulated questions, as they have a significant impact on other components of the review – including the literature search strategy, data extraction, synthesis, and presentation of findings. In practice, formulation of the topic via PECO questions (or statements) is an iterative practice, which is best informed by a multidisciplinary team and considerations for the downstream implementation of the entire SR.
Development of PECO statements is an example of an aspect clearly differentiating SR methods in EBT relative to EBM. In EBM, PICO (population, intervention, comparator, and outcome) statements are utilized. The key difference, exposure versus intervention, is often significantly more difficult to define and standardize in the practice of EBT. Unlike the field of medicine, exposures, particularly to humans, in the field of toxicology are accidental in nature, or occur as part of industrial practices and/or low-level environmental exposures. As such, assessment of exposure in humans will often be complicated relative to that obtained from randomized control trials available in the field of medicine. Other study types (e.g., cohort, case-control, cross sectional) will often be the only source of information in humans. In contrast, assessment of exposure in animal studies is more straightforward, thus often leading toxicologists to a preference for this data type, despite availability of data characterizing potential hazards in humans – a topic which exemplifies the need for a priori approaches for both the critical assessment of individual studies as well as approaches for integrating the data across data streams. And, lastly, assessment of mixtures presents somewhat unique challenges in the practice of EBT; the definition of the mixtures and the definition of exposure to a given mixture can clearly have a significant approach on a SR.
Problem formulation involves additional aspects often not considered in a traditional narrative review. Included in these early exercises should be identification of team members, facilitators, sponsors, etc. as well as specific roles for each person. As established both in SR guidance, and in practice, integration of a multidisciplinary team that includes subject matter experts, as well as experts in systematic review, librarians, and potentially other experts (e.g., epidemiologists, physicians, industrial hygienists) as appropriate to the review. Conflict of interest (COI) statements are also a key component of initiating a review, though implementing standard processes for obtaining and managing COI information does not yet appear to be consistently practiced in the field toxicology. The conduct of pilot endeavors throughout the process should also not be undervalued. Such pilot exercises include: initial literature searches, pilot evaluation of screening criteria (i.e., inclusion and exclusion criteria), processes, and software, as well as pilot application of grading frameworks and/or criteria to the studies of interest. These exercises are crucial to informing the scope and implementation of the protocol and have a direct impact on consistency, efficiency, and transparency, particularly when users adhere to a strict documentation policy.
The best practices for the assessment of individual studies and subsequent integration of evidence are perhaps the topics of greatest research and debate currently in the field of EBT. There are differing opinions regarding the evaluation of the “quality” of individual studies with or without using checklists, scores, or grades versus assessment using more of a qualitative spectrum or continuum. One of the primary issues in developing best practices for individual study assessments is to first clearly define what is meant by study quality in a given SR, given that the parameters of interest to study quality may be dependent on the particular PECO statement of interest. Significant focus has been on assessment of risk of bias (RoB) (i.e., measure of the design and conduct of the study to determine credibility of the link between exposure and outcome; OHAT 2015), and, specifically, frameworks to evaluate RoB in parallel for multiple evidence streams (i.e., human, animal, mechanistic). However, risk of bias itself is not defined consistently, nor is it the only aspect of evaluating study “quality” that is important. Other “quality” aspects of individual studies, such as indirectness (i.e., applicability) and imprecision, are important considerations in determining overall quality and relevance. An appreciation for the rigor and efforts associated with evaluating individual study quality, including RoB, can be emphasized simply by the existence of more than 100 tools for evaluating such. And as a result, appreciation for the forethought and considerations regarding selection of such a framework, or in many cases, frameworks (depending on the scope and intentions of the SR), during problem formulation, cannot be underemphasized.
Similar issues exist with respect to assessment of the body of evidence, and of particular interest in toxicology, integration of multiple types of data, and application of the SR in both qualitative and quantitative hazard or risk assessments. This particular juncture appears to be of greatest need in terms of developing best practices. And foremost, first establishing that the application of the SR has a significant impact on the conduct of such. Unlike the use of SR in clinical medicine to evaluate interventions, SRs are used in toxicology to assess a broader range of outcomes and applications. For example, the outcome may be as broad as characterizing the potential for hazardous or adverse effects, thus requiring accommodation for multiple endpoints (e.g., hepatotoxicity, cardio toxicity, reproductive toxicity) within a single SR. Accordantly, the approach taken, as well as the depth of the assessment, would likely therefore be influenced by the volume of data available.
In other cases, the scope could be very narrow (e.g., specific birth defect observed following exposure to a compound during pregnancy), and the objective could risk-based and include the development of a health-based toxicity value, rather than qualitatively characterizing potential hazard. In such a case, the PECO and subsequent approach would likely be structured differently, with focus on candidate dataset selection and approaches for assessing the data qualitatively. For the later, methods may not involve standard quantitative approaches utilized in SRs (i.e., Meta analyses), but rather approaches for conducting dose-response modelling, etc. The application of SR with a risk context also raises additional challenges, ranging from reliance on a specific evidence stream (e.g., animal data from a high-dose carcinogenicity study with controlled exposures versus environmental epidemiological data) to considerations for kinetics and dynamics as well as environmental or consumer exposures (i.e., dose/exposure relevance) – all of which have a significant role in traditional risk assessment. It is thus notable that the NTP’s OHAT Handbook for Conducting Systematic review indicates that considerations for ADME and exposure should be made in developing overall hazard conclusions; however, guidance on integration of these parameters is not due to be available until 2016/2017 .
Many of these topics have been addressed in a series workshops held by the USEPA IRIS Program . At the recent USEPA workshop on Advancing Systematic Review held in December, 2015, it was evident that the there is significant interest in the application of systematic review in the discipline of toxicology, but that the field remains in its infancy with respect to determining best practices. During the workshop, various case studies were presented, demonstrating the unique nature of each systematic review, as well as the unique nature of an EBT SR versus an EBM SR.
The second day of the workshop focused on assessment and integration of mechanistic data in SR. As an evidence stream that is generally unique to EBT, fewer frameworks and guidance are available. Several presentations at the workshop were focused on a recent publication by Smith et al. , in which the authors provided ten key characteristics of carcinogens as a basis for organizing data on the mechanisms of carcinogens. Dr. Guyton, an author on the paper, presented applications of these data and discussed use of the characteristics within IARC evaluations. However, guidance on how these characteristics can be applied beyond organization of data is not yet available. For example, Smith et al. does not provide guidance on how to integrate null findings, how quality/validity and relevance are considered, or how the number of characteristics with positive/ negative influences the body of evidence. And, importantly, there is not yet a clear vision on how these characteristics can be applied relative to current practices in the assessment of mode of action for carcinogens, or how they could be used to evaluate high throughput data. There is also not yet a consensus that mechanistic data should be considered a separate stream – rather, should such data be considered contextual. Such a demonstration provides an excellent example of progress in the field of EBT, but also demonstrates the progress yet to be made.
And lastly, perhaps, an area that deserves certain appreciation is the amount of time and resources required conduct a systematic review. With respect to the amount of time, some of the exercises that differentiate the systematic review from a standard narrative review include: development and publication of a protocol, documentation of the literature search (including documentation of all records that were included/excluded), and a critical evaluation of each study using an approach determined a priori. Estimates of time needed to complete problem formulation are highly variable, but are not measured in minutes or hours; pilot screening has been estimates at 1->5 minutes per hit, full screening at 1-2 minutes/hit (plus time for conflict/group review), >2 hours/outcome while piloting individual study assessment, and ~1.5 to 3 hours/outcome (with outliers in both directions) for the bulk of individual study assessments [9,15]. Much of the time estimates are dependent on factors such as the number of collaborators, experience of team with SR processes, number of databases (and associated software compliance), establishment of internal processes and procedures, documentation, grading approaches, number of endpoints and outcomes, as well as overall complexity of the topic under investigation.
Typically, many of the exercises are also carried out by two evidence analysts, and the overall project informed by a multidisciplinary team (including a librarian). Though software programs are available to help facilitate various tasks within a systematic review, the resources and time required to conduct a SR relative to a standard narrative review are significantly greater. As such, there is also a need to balance rigor with efficiency, recognizing that not all SRs will achieve a similar level of detail or comprehensiveness. Key to achieving the balance is selecting tasks with most value added, and, most importantly, providing transparency to the decisions via a priori documentation and rationale.
The use of systematic review is advancing the field of toxicology, providing objectivity and transparency in our practice. The limited number of EBT SR publications relative to EBM highlight the infancy of the integration of this tool in toxicology. As we go forward, we must not haphazardly use the term systematic review, as it clearly bears weight – too often, already, the term is misused, referring only to elements of an exercise that were conducted systematically. We must also continue to move toward determining best practices, and in doing so, develop a greater appreciation for the tool that allows us conduct evidence-based toxicological assessments.
Dr. Wikoff and Britt report no conflicts of interest; either author or their employer received external funding in developing this commentary.