John Longshore1*, Sherene Banawan2, Heather Amidon2, Heather Todd2, Jeffrey Fu3, Mari Christensen3, Julie Tsai3, Grant Hillman3, Shannon Walter3, Felice Shieh3, Edward Lipford1, Guili Zhang3, Abha Sharma3 and John F Palma3
Received Date: December 19, 2014; Accepted Date: January 16, 2015; Published Date: January 20, 2015
Citation: Longshore J, Banawan S, Amidon H, Todd H, Fu J, et al. (2015) Comparison of Molecular Testing Methods for Detecting BRAF V600 Mutations in Melanoma Specimens with Challenging Attributes. J Mol Biomark Diagn 6:215. doi:10.4172/2155-9929.1000215
Copyright: © 2015 Longshore J, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at Journal of Molecular Biomarkers & Diagnosis
Introduction: To ensure the appropriate assignment of vemurafenib to patients with unresectable or metastatic melanoma, accurate detection of activating BRAF mutations is now a clinical imperative. However, the performance of commercially available test kits on challenging samples is unknown.
Methods: 126 formalin-fixed, paraffin-embedded melanoma samples were selected for challenging attributes, such as small sample size, that might affect test kit performance. The Qiagen BRAF RGQ PCR version 2 (RGQv2) test kit, intended for research use only, and the FDA-approved companion diagnostic cobas 4800 BRAF V600 Mutation Test were compared for their ability to detect the V600E mutation in challenging samples, using a single 5 μm unstained slide.
Results: of the 126 specimens, three samples were invalid by the RGQv2 test, three other samples were invalid by the cobas test, and an additional two samples were found invalid by both tests. For the 118 samples that yielded valid results with both tests, concordant results were observed for 105 (89.0% [95% CI, 82.1%-93.5%]) samples. Of the 12 discordant samples with sufficient material for further testing, next generation sequencing confirmed the cobas test result for 6 (50.0%) and confirmed the RGQv2 test result for the other 6 (50.0%) samples. Five (4.0%) of the RGQv2 test results yielded multiple positive mutation calls and two results had a sample control assay PCR cycle threshold (CT) >33, indicating insufficient amounts of DNA template, but gave accurate mutation calls. Workflow analysis showed that the total time to result was 5.65 hours for the cobas test to process 24 samples and 7.84 hours for RGQv2 assay to process 7 samples.
Conclusions: The two commercially available, PCR-based methods demonstrated similar abilities to detect BRAF V600 mutations in challenging melanoma samples. However, the total time-to-result, assay hands-on time, and diagnostic interpretation were more efficient and rapid with the cobas test.
Melanoma; BRAF; Cobas; Diagnostic; PCR; Next generation sequencing; Biomarker
FDA: United States Food and Drug Administration; FFPE: Formalin-Fixed, Paraffin-Embedded; RGQv2: Qiagen BRAF RGQ PCR version 2; NGS: Next Generation Sequencing; PCR: Polymerase Chain Reaction; CNB: Core Needle Biopsy; PPA: Positive Percent Agreement; NPA: Negative Percent Agreement; OPA: Overall Percent Agreement; FP: False Positive; FN: False Negative
Treatment options for malignant melanoma have undergone a true revolution with the introduction of targeted therapies. With the concomitant changes in diagnostic testing, patients with advanced or metastatic melanoma are now routinely screened for mutations in BRAF to identify patients who are most likely to respond to targeted therapies, such as vemurafenib and dabrafenib. Mutations in the BRAF gene are a common event in the development of malignant melanoma, accounting for 50% to 60% of all cases [1,2]. The majority of BRAF mutations occur at codon 600 in exon 15 of the BRAF gene, leading to constitutive activation of the cognate protein and downstream signaling through the MAPK pathway . The most frequent BRAF mutation leads to the substitution of glutamic acid for valine (V600E) and accounts for as many as 90% of BRAF mutations, although other activating mutations are known (e.g., V600K and V600R) .
Recognition of the important role played by BRAF kinase mutations in melanomas led to the development and subsequent approval of vemurafenib, a potent and selective inhibitor of mutated BRAF kinase, for the treatment of patients with unresectable or metastatic melanoma with the BRAF V600E mutation. Clinical trials with vemurafenib for metastatic melanoma showed improved overall and progression-free survival in patients with previously untreated melanoma whose tumors tested positive for this genetic change [4,5]. The development of vemurafenib represented a groundbreaking advance in the management of malignant melanoma patients with V600E-mutated tumors. Until recently, advanced melanoma was associated with a dismal prognosis; with few effective treatment options available, overall survival (OS) was measured in months [5,6]. Vemurafenib demonstrated potent activity against melanoma cell lines that carried the V600E mutation, but not against cell lines with wild-type BRAF . In a landmark phase 3 trial of patients with treatment-naive metastatic melanoma harboring the V600E mutation, vemurafenib significantly improved PFS and OS relative to the control arm . Based on these outcomes, vemurafenib received approval from the FDA for the treatment of patients with unresectable or metastatic melanoma with BRAF V600E mutation as detected by an FDA-approved test. In the clinical trials of vemurafenib, a positive result for the V600 mutation with the cobas® 4800 BRAF V600 Mutation Test was a key eligibility requirement [4,5]. Other commercially available tests are available for the evaluation of BRAF mutations for research rather than diagnostic purposes.
Given the improved melanoma patient outcomes with targeted therapies, it is imperative that test methods are highly sensitive to ensure that eligible patients receive the appropriate treatment. However, assay performance may be hampered if specimens submitted for analysis have challenging attributes, such as a low proportion of mutant cells, small specimen size, poor specimen quality, or high melanin content, which can interfere with PCR. BRAF tests with low sensitivity may yield false negatives or invalid results in those situations, which would result in patients being denied potentially life-prolonging therapy.
This study compared the analytical performance of the FDA-approved cobas® 4800 BRAF V600E Mutation Test versus the Qiagen BRAF RGQ PCR kit version 2, for research use only, for the detection of BRAF mutations in melanoma specimens with challenging attributes. These two tests were compared because they are the most commonly used PCR-based kits and, at the time of the study, only the cobas kit had approval for BRAF diagnostic testing. Since this study was completed, a new test, the THxID™ BRAF Kit (Hazelwood, MO), has been FDA approved in the US for advanced melanoma patients with BRAF V600E and V600K mutations to be treated with trametenib and dabrafenib.
Mutation testing methods
The cobas® 4800 BRAF V600 Mutation Test kit (“cobas test”; Roche Molecular Systems, Inc., Branchburg, NJ, USA) is an FDA-approved and CE-marked real-time PCR (RT-PCR), TaqMan-based in vitro diagnostic (IVD) assay designed to detect the presence of the BRAF V600E (1799T>A) mutation in FFPE melanoma specimens. Full descriptions of the assay and workflow have been described [8,9]. Although designed to detect the V600E mutation, the cobas test has some cross-reactivity with the V600K (GT>AA), V600D (TG>AC), and V600”E2” (GTG>GAA) mutations.
The Qiagen BRAF RGQ PCR kit version 2 (“RGQv2 test”; Qiagen Inc., Valencia, CA, USA) is an RT-PCR assay for research use only. It uses Scorpions® (bi-functional molecules in which a PCR primer is covalently linked to a probe) and Amplification Refractory Mutation System technologies to detect the V600E (c.1799 T>A), V600E2 (c.1799_1800TG>AA), V600K (c.1798_1799GT>AA), V600R (c.1798_1799GT>AG), and V600D (c.1799_1800TG>AT) mutations against a background of wild-type genomic DNA in FFPE tissue specimens . For control and experimental samples, the BRAF RGQ PCR Handbook for the version 2 assay specifies a CT of 20.95–33.00 as an acceptable range indicating the presence of sufficient DNA for optimal kit performance; a CT >33.00 indicates that mutations present at low levels may not be detected and results should be interpreted with caution; and a CT >35.00–45.00 indicates that only a few amplifiable copies of DNA are present and mutations are only likely to be reported if most copies are mutated . The mutation status of a sample is established by calculating the difference between the CT values (ΔCT) from the sample and a control sample as follows:
ΔCT = [Mutation CT] – [Control CT]
A ΔCT of ≤7.0 indicates that the sample is positive for the V600E mutation.
Next generation sequencing (NGS) was performed according to the manufacturer’s instructions on the Ion Torrent PGM (Life Technologies, Thermo Fisher Scientific, Carlsbad, CA) using a validated protocol for BRAF mutation detection with a limit of detection of 1% for V600E mutations . This method is a 5- to 7-day process that involves the generation of amplicons which are subject to pooling, ligation, emulsion PCR amplification, and NGS .
The study was conducted using a panel of 146 FFPE melanoma tissue specimens, were collected from a single site, from which 126 specimens with challenging attributes were selected (Figure 1). The challenging attributes were: (a) tissue sample <60 mm2; (b) CNB samples; (c) >50% necrosis; (d) >10% pigmentation (Figure 2); or (e) metastatic sites (Figure 3). Some of these attributes are known to compromise results acquired with PCR-based methods and can result in higher sample failure rates [12-14]. For example, decalcification steps for bone metastatic samples can lead to gross degradation of the target DNA. Also, no samples were excluded due to a low percentage of tumor cells. For the 126 samples, tumor content was at least 50% in 87 samples, was between 10% and 50% in 32 samples, and was 5% or less in 7 samples. This study was double blinded. Five 5-μm tissue curls were sectioned from each of the 126 panel specimens and blinded. All sections from each specimen were mounted on a slide; one was stained with hematoxylin and eosin, and all were coded. This section was assessed by a certified pathologist to confirm the diagnosis of melanoma and to assess tumor content, degree of pigmentation, and extent of necrosis according to laboratory protocol. Included samples could have multiple attributes; thus there were 42 samples with 1 attribute, 45 samples with 2 attributes, 25 samples with 3 attributes, and 3 samples with 4 attributes to yield 219 observable attributes.
DNA from one slide-mounted 5-µm section of each of the 126 clinical specimens was isolated using the cobas DNA Sample Preparation Kit and tested with the cobas test (Figure 4) . DNA from another 5-µm section was isolated as recommended by the manufacturer using the QIAamp® DSP DNA FFPE Tissue Kit and tested on the RGQv2 test (Figure 5) . Prior to PCR setup for the RGQv2 test, the amount of DNA to be used in the final PCR run was measured by a recommended sample assessment procedure based on CT values from the package insert. Although the cobas test requires macrodissection for samples with less than 50% tumor content, the RGQv2 test does not require macrodissection. In order to allow a fair comparison of outcomes between the 2 different methods, no macrodissection was performed on any samples prior to DNA extraction. This choice could potentially impart a bias favoring the RGQv2 kit. Therefore, to minimize assay performance bias, no samples were macrodissected for either test. The remaining two glass mounted slides were kept for additional discordance analysis, if required, and future studies.
Figure 4: cobas® 4800 BRAF V600 Mutation Test workflow. The assay is based on two major processes: (1) manual isolation and preparation of DNA from the specimen; (2) PCR amplification and detection. Macrodissection is normally required when specimens have <50% tumor but was not performed for this study for any samples. H&E = hematoxylin and eosin; PCR = polymerase chain reaction
Agreement analysis was assessed as positive percent agreement (PPA), negative percent agreement (NPA), and overall percent agreement (OPA) between the cobas test and the RGQv2 test for detecting BRAF V600 mutations. Specimens that gave discordant results between the two methods or invalid results for one of the methods were further subjected to NGS as a third mutation detection method, using existing DNA eluates. False positive (FP) and false negative (FN) rates were calculated for both methods using the following formulae and the cobas test as the reference method:
FP = FP/(FP + TN) and FN = FN/(TP + FN)
where TN and TP are true negatives and positives, respectively.
Specimens used in this study were included in a proposal and presented to the Carolinas HealthCare System Institutional Review Board (IRB File # 03-12-16EX). After review, this study met the criteria for exempt status. The IRB waived the need for consent since the testing was performed on de-identified samples.
Distribution of challenging attributes in melanoma tissue specimens
Of 146 original melanoma tissue samples, 126 specimens were selected for challenging attributes. Because specimens could have more than one challenging attribute, the 126 specimens represented a total of 219 challenging attributes, including tissue size <60 mm2 (n=82), CNB (n=24), >50% necrosis (n=20), >10% pigmentation (n=53), and metastatic sites (n=40).
Mutation test failure rate
Based on the quality assessment step using CT values with the RGQv2 test, DNA from 15 (11.9%) of the melanoma samples were classified as “interpret with caution” because the CT values were greater than 33 but generated a result due to manual interpretation. The remaining samples were within the acceptable range (CT = 20.95-33.00) for sample assessment control reactions. Of the prior 15 specimens, 13 had sufficient DNA yields for use with the cobas test (i.e., at least 5 ng/µL). The cobas test reported 5/126 (4%) invalid specimens, which were subjected to NGS. Of the 5 specimens called invalid by the cobas test, two specimens were invalid by RGQv2 and one specimen was also invalid by NGS and exceeded the CT >33 cutoff for a good quality sample. The RGQv2 test also reported invalid results from 5 specimens; two specimens were also invalid by the cobas test and three were unique to the RGQv2 test.
Out of 126 specimens, 8 samples were invalid for at least one of the test methods, resulting in 118 samples for analysis. Of the 8 invalid samples, 3 were invalid by the RGQv2 test only, 3 were invalid by the cobas test only, 2 were invalid for both methods. Using the RGQv2 test as the reference method, the initial agreement analysis showed a cobas/RGQv2 PPA of 49 /58 (84.5% [95% CI, 73.1%–91.6%]), NPA of 56 /60 (93.3% [95% CI, 84.1%–97.4%]), and OPA of 105/118 total samples (89.0% [95% CI, 82.1%–93.45%]) (Table 1). The cobas test had a FP rate of 6.7% and a FN rate of 15.5%.
|Positive agreement||84.48% [95% CI = 73.07% – 91.62%]|
|Negative agreement||93.33% [95% CI = 84.07% – 97.38%]|
|Overall agreement||88.98% [95% CI = 82.06% – 93.45%]|
|CI=confidence interval; MD=mutation detected; MND=mutation not detected|
Table 1: Correlation between cobas and RGQv2 results.
Thirteen specimens were discordant between the two test methods, and one of these samples did not have adequate DNA for NGS resolution analysis. For the remaining 12 specimens, NGS showed concordance with 6 (50%) of the cobas test results (4 wild-type and 2 mutated by the cobas test) and 6 (50%) of the RGQv2 test results (5 wild-type and mutated by the cobas test; Table 3). This resulted in 57 samples that were V600 (mutated) and 60 samples that were V600 (wild-type). Therefore, the final agreement values between the two test methods (cobas/RGQv2) were PPA 52/57 (91.2% [95% CI, 81.1%-96.2%]), NPA 59/60 (98.3% [95% CI, 91.1%-99.7%]), and OPA 111/117 (94.9% [95% CI, 89.3%-97.6%]; Table 2). The cobas test FP and FN rates were reduced to 1.7% and 8.8%, respectively. Twelve (75%) of 16 non-V600E mutations were detected with the cobas test. Additionally, five specimens tested with the RGQv2 test reported coincident multiple mutations, all of which included V600K plus either V600E or V600E2, and one specimen reported both of these mutations in conjunction with a V600D mutation.
Table 2: Test results from 13 discordant specimens.
|RGQv2 + NGS (Reference)|
|Positive agreement||91.23% [95% CI = 81.06%-96.19%]|
|Negative agreement||98.33% [95% CI = 91.14%-99.71%]|
|Overall agreement||94.87% [95% CI = 89.26%-97.63%]|
|CI=confidence interval; MD=mutation detected; MND=mutation not detected|
Table 3: Correlation between cobas and RGQv2 results after NGS sequencing of discordant samples.
Time to result
An estimation of the total time to result was derived from a single run of each instrument loaded with the maximum number of samples. This analysis resulted in a total time-to-result of 7.84 hours for RGQv2 to process 7 samples (Table 4) and 5.65 hours for cobas to process 24 samples. For the PCR reaction itself, the total reaction times are 2.67 hours for RGQv2 versus 1.75 for cobas, which is relevant for laboratories that use their own DNA extraction and quality assessment methods.
|Sample isolation||175 min||234 min|
|Total DNA isolation time||2.92 hours||3.90 hours|
|Sample assessment PCR setup||10 min||N/A|
|Sample assessment PCR run||115 min||N/A|
|Analysis/export data||10 min||N/A|
|Sample assessment total time||2.25 hours||N/A|
|PCR setup||25 min||10 min|
|PCR run||115 min||90 min|
|Export data||10 min||N/A|
|Data analysis||10 min||5 min|
|PCR total time||2.67 hours||1.75 hours|
|Total time||7.84 hours
Table 4: Workflow analysis for RGQv2 versus cobas. Workflow analysis is based on 7 samples for RGQv2 and 24 samples for cobas.
When this study was performed, the cobas test was the only CE-IVD and FDA-approved test for the identification of patients with tumors harboring the V600E mutation; however, other assays using differing technologies are commercially available for research use only (RUO) [16,17]. Recently, the bioMérieux THxID BRAF test was approved by the FDA for use as a companion diagnostic for the detection of V600E and V600K mutations when making treatment decisions regarding dabrafenib and trametinib for patients with metastatic melanoma . The Qiagen BRAF RGQ PCR version 2 kit is an RUO, CE-IVD approved test and it is commercially available for the detection of the clinically relevant V600E, V600E2, and V600K BRAF mutations, as well as other V600 mutations whose clinical relevance is unknown. In addition to comparing the cobas and RGQv2 test kits, we assessed sample processing turnaround times to allow consideration of whether the tests can achieve acceptable timescales for clinical decision-making. The analytical performance of the cobas test has also demonstrated cross-reactivity to BRAF V600D plasmid at ≥10% mutation, BRAF V600K plasmid at ≥35% mutation, and BRAF V600E2 plasmid at ≥65% mutation .
Although the V600E mutation is the most common at this codon, accounting for approximately 80% of valine substitutions, other mutations have been observed. V600K is clearly the next most common mutation and occurs in approximately 10% to 20% of melanoma samples [19,20]. Cross-reactivity against the V600K mutation has been documented for the cobas mutation test kit. In a recent study, the kit detected 75.0% (8 of 12) of BRAF V600K mutations confirmed by NGS . For detection of very rare or unknown mutations, methods such as Sanger sequencing may be useful in samples not requiring the high analytical sensitivity of a PCR-based assay.
In the authors’ experience, the majority of samples that receive BRAF testing has at least one challenging attribute. Both the cobas and the RGQv2 methods produced a high rate of valid results from 126 samples with challenging attributes. Eight samples were invalid by the cobas and/or the RGQv2 tests. In addition, 13 samples were discordant between the two tests, and one of these samples was invalid for NGS. After factoring in results from NGS, which was used to resolve discordant samples, the tests yielded an OPA of [111 of 117] 94.9%.
The cobas test yielded 5 false negatives based on the mutation calls that showed agreement with both RGQv2 and NGS; another sample reported the V600E mutation which was wild-type by RGQv2 and NGS . Although the RGQv2 instructions do not require macrodissection, if these specimens had been macrodissected prior to DNA extraction for the cobas test, the lack of sensitivity may have been resolved for these samples. The RGQv2 test yielded three false positives, including two V600E/E2 and one V600K, and it yielded three false negatives which were either V600E (two samples) or V600K (one sample) based on NGS.
Although the sample sizes were too small to allow statistical testing, the samples that provided the discordant calls between the cobas test and the RGQv2 test tended to be smaller samples (<60 mm2) with >10% pigmentation. Although approval of the cobas test by the FDA was specific to V600E mutations, the assay demonstrated cross-reactivity to V600K; these mutations are present in >5% of reported melanoma cases and therefore V600K results are reported this study . There were a total of 12 V600K mutations as determined by sequencing and the cobas test and RGQv2 test were able to identify 10 and 11 specimens, respectively.
Minimizing the presence of challenging tumor sample attributes can diminish the risk for invalid results. Nonetheless, in this study, challenging attributes had a nominal impact on assay performance for both the cobas and RGQv2 tests. Moreover, the cobas test demonstrated equivalent assay performance to the RGQv2 test despite the fact that 39 samples with less than 50% tumor content were not macrodissected. Two samples that were invalid by both assays were derived from metastatic bone samples.
Providing timely results for BRAF V600 mutation status is critical to patient management. Depending on the laboratory workflow for the case load of BRAF testing and other molecular tests, time to result becomes a differentiator for laboratory costs. In addition, invalid test results have important implications for patients, as the need to repeat tests, and potentially to re-biopsy patients, can lead to significant delays in effective patient treatment. This study yielded 10 separate samples for which an equal proportion of the results of RGQv2 or cobas testing did not agree with the NGS results. Although samples with insufficient DNA can often be manually evaluated, the process can contribute to a longer time to obtain results and introduces subjective interpretation of results. This issue is underscored by the longer time-to-result derived for a single instrument run of the cobas test versus the RGQv2 test (5.65 hours for 24 samples versus 7.84 hours for 7 samples, respectively). The faster time-to-result for a greater number of samples using the cobas test kit is particularly relevant for research and large clinical studies.
In summary, we have presented a comparison of two commercially available methods for the detection of V600E mutations in the BRAF gene from specimens of malignant melanoma with challenging specimen attributes. The two assays were found to perform similarly in their ability to detect V600E mutations in melanoma samples and neither assay was adversely impacted by the challenging attributes. Overall, the cobas test displayed a higher degree of mutation detection accuracy than the RGQv2 test when including the detection of V600K mutations. In addition, the total time to result, assay hands-on time, and diagnostic interpretation were more efficient with the cobas BRAF test. Current studies are underway to evaluate the performance of the cobas test relative to additional available tests.
This study was funded by Roche Molecular Systems. Reagents for this study were provided by Roche Molecular Systems and Qiagen. Co-authors Jeffrey Fu, Mari Christensen, Julie Tsai, Grant Hillman, Shannon Walter and Felice Shieh are/were employed by the funder of this study, Roche Molecular Systems. The cobas® 4800 BRAF V600 Mutation Test is a Roche Molecular Systems product. We have the following patents relating to material pertinent to this article: (US Patent Application 10/873,057, entitled “B-RAF Polynucleotides”, and 10/873,057, entitled “Genes”, and related foreign applications in-licensed from the Wellcome Trust; and US Patent Application 13/015,374, entitled “Diagnostic Test for Susceptibility to B-RAF Kinase Inhibitors”, owned by Roche Molecular Systems). This does not alter our adherence to all the BMC Cancer policies on sharing data and materials.
The authors would like to acknowledge Dr. H. Jeffrey Lawrence for assistance with this study, Dr. Maggie Merchant for medical writing assistance on behalf of Roche Molecular Systems, Inc., and Qiagen for use of the RotorGene instrument and DSP DNA isolation supplies.
Make the best use of Scientific Research and information from our 700 + peer reviewed, Open Access Journals