Assessing Clinical Trial Technology: Evaluating the TIME Study
Received Date: Mar 21, 2018 / Accepted Date: Mar 29, 2018 / Published Date: Mar 31, 2018
Purpose: Electronic data capture has quickly become the preferred means of capturing and storing clinical study data. Cost and efficiency savings have been documented, yet it is unclear how acceptable this technology is to participants. An evaluation survey has been created to assess participant attitudes to a clinical study website.
Methods: The TIME study is a clinical study that uses an online methodology to compare morning dosing of hypertensive medication with evening dosing. An evaluation questionnaire was developed and sent to participants to assess their views of the online methodology. The final questionnaire was organised into themes: functionality, personal contact, and trust. Negative and positive phrasing was used along with a Likert-type scale. The survey underwent 4 iterations before the content was finalised.
Results: 149 responses were received from 263 invitations. The mean scores for the three themes in the final 14 item questionnaire were as follows: functionality, 3.99; personal contact, 3.89; and trust, 4.03; suggesting an overall positive perception of the study methodology. Concerns regarding use of data and privacy were present in the responses but the overwhelming majority of responders chose to take part due to a sense of altruism and recognising the need for clinical research.
Conclusion: This questionnaire was specifically created to evaluate the TIME study website. Additional improvements to the questionnaire are necessary for more general usage. The feedback provided from participants reveals areas of the study website that require further development, and has reaffirmed known concerns participants have about the use of their data.
Keywords: Study questionnaire; Trial technology; Electronic data capture
User acceptance is a prerequisite to reaping the potential benefits of any new technology. In health care, patient satisfaction is often used as an indicator of the service performance and reflects patient’s values and expectations . When there is a match between the care expected and received, patients are satisfied . In other words, satisfaction is influenced not only by experiences but also by prior expectation .
The Treatment in Morning vs. Evening (TIME)  study is a clinical trial evaluating morning dosing of antihypertensive medication versus evening dosing to assess whether one provides greater cardiovascular protection. The study data and all follow ups are captured entirely online, using internet technology. The online methodology used in the TIME study is novel and it is unclear how acceptable this use of technology will be to study participants. In order to maintain or improve participation rates and minimise waste in clinical research it will be necessary to evaluate any new methods or innovations .
There have been several patient satisfaction surveys developed to evaluate telemedicine, or information technology in healthcare, including: Telemedicine Perception Questionnaire (TMPQ), Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Satisfaction Survey (TSQ). These questionnaires have all been designed specifically for use in telemedicine [6-9], and none were deemed suitable for use in assessing the TIME study. Problems with the aforementioned questionnaires include lack of measures of reliability or validity and limited evidence of practical application. Patient selection criteria were not always clearly specified so it is possible sampling bias may have be an issue [10,11]. It was identified by the authors that a suitable questionnaire was required that could be used to gather feedback for online-only clinical studies using a similar methodology as the TIME study.
The research objective was to create an online questionnaire capable of capturing clinical study participant feedback to evaluate the novel methodology used in the TIME study. Using similar methods to those reported in the development of telemedicine surveys, we devised and tested a questionnaire for the evaluation of specific components of the TIME study website.
A broad search of previous published literature on questionnaire development for similar applications was performed. The outcomes of this review were used to inform the questionnaire development process below.
Questionnaire ease of use
Care was taken to ensure that the questionnaire was not burdensome to the respondents, increasing the likelihood of complete and accurate responses . Potential questions were compiled and reviewed for inclusion by TIME study staff. Questions were then prioritised for inclusion if they appeared relevant to the trial technology and then edited to a reduced set of items. Correspondence was personalised and questions were kept short, interesting and user friendly. The survey also asked for reasons for not taking part. All of these are features that have been found to have a positive impact on response rates .
A number of reports have documented the extent of poor health literacy and its impacts on health status [14-16]; therefore, materials were designed with careful attention to readability. To ensure the questionnaire was suitable for a wide potential audience of trial participants, we aimed to adopt the recommended readability level between US 6th to 8th grade (11 to 14 years of age) English [17,18], measured using the Flesch-Kincaid reading ease scale .
Some questions were phrased positively and some negatively to minimise the “halo” effect, where responders may have an overall attitude towards the system and respond to all items consistently without reading the individual statements carefully . All questions were worded to avoid generalities or ambiguity and no emotional or persuasive terms were used. Questions were reviewed by study research staff familiar with the trial data collection process to ensure that they captured the participant experience. A Likert-type scale (from strongly disagree to strongly agree) was used for survey responses . Exceptions to this were a statement reflecting the previous participation in clinical research (question 1: “I have taken part in other clinical studies”), where a Yes, No or Not Sure options was required. Additionally, exceptions were made for two questions which were free text and allowed responders to give additional feedback as to why they decided to take part, or not take part, in the study. Prior to the questionnaire being sent to participants, all theme headings were removed and the order of the questions was randomised (all participants received the same questionnaire once randomised). No compensation or incentives were offered for completing the survey.
TIME online methodology
TIME study is conducted online, so there is a perception that participants are familiar with web-based technology. An online questionnaire was considered an appropriate and efficient method of gathering feedback. A web based questionnaire was also deemed the most suitable as it would not require additional software to be installed. The questionnaire was developed using Microsoft C# .NET and responses were stored in the TIME study SQL Server database. An encrypted Uniform Resource Locator (URL), unique for each individual, was emailed to TIME study participants who had been randomised and to any participants who had not been randomised but had consented to being contacted. This encrypted URL took the individual directly to the questionnaire when clicked.
To validate the questionnaire we tested face validity, content, and construct validity . Face validity was addressed by receiving feedback from study personnel and academic staff. The members of staff included the principal investigator, the trial physician, the senior software developer, the trial statistician and the study administrative assistant. Where differences of opinion occurred, the majority opinion was taken. Content validity was achieved by ensuring the questionnaire covered all relevant themes, including reported advantages and disadvantages of online clinical study methodology identified through the literature review. Construct validity was investigated by examining the results of the testing procedure and comparing these with published findings. The reliability of the questionnaire was tested using internal consistency. Internal consistency is used to assess the consistency of results across items within a questionnaire. The index of internal consistency used was Cronbach’s Alpha , which is the average of all possible split-half estimates. Test-retest reliability was not measured at this point, as it requires that the same subjects are exposed a second time to the questionnaire. This will, however, be assessed at the end of TIME to evaluate the consistency of the questionnaire from one time point to another. Participants will be re-sent the same questionnaire to measure consistency of answers between the trial start and finish.
The initial questionnaire was created during December 2015. Principal component analysis was used through iterations of questionnaire development to identify the extent that questions were relatable to intended themes and that the questions were appropriately grouped. The authors wished to identify how easy participants found the study website to use, what participants views were about the study being conducted entirely online with no nurse or clinician contact, and what participants thought of their personal data being used in this manner. These 3 themes were named: Functionality, Personal Contact, and Trust. Cronbach’s Alpha statistic was calculated to check the internal consistency of the questions within each theme and ensure questions were correctly associated to the theme. Cronbach Alpha scores above 0.7 are preferable , but for an exploratory study, a value above 0.6 is considered tolerable . Questions that were answered inconsistently, or that did not relate to the intended theme, were considered redundant and were reworded or removed. This iterative process went through four rounds of development before the final questionnaire was felt to be performing as required. Each round involved the questionnaire being responded to by between 121 and 297 newly enrolled patients from January 2015 to July 2016.
Three questions were not analysed in the questionnaire development (question 1, question 16, and question 17). These were either free text or required a Yes/No/Not known response, and so were not suitable for data analysis. These three questions were included to collect additional information as to why participants had decided to take part or not in the TIME study, and whether this had been their first experience of participating in a clinical study.
The final version of the 14 question TIME evaluation survey (Figure 1) was emailed to 263 participants, 149 (57%) of whom responded, between July 2016 and January 2017. The mean age of respondents was 64 (median 67) and 72 were male (48%). 103 (69%) participants indicated that they had never taken part in a clinical study before, 41 (27%) indicated that they had, and 6 (4%) participants were unsure or did not know.
The Cronbach Alpha of the themes for the final questionnaire were: functionality 0.764, personal contact 0.690, and trust 0.668. Due to the Cronbach Alpha scores being below 0.7 for personal contact and trust a correlation matrix (Table 1) was calculated to detect redundant items, and to give confidence that questions pertaining to specific themes were sufficiently related to one another. There were no redundant questions in the final version of the questionnaire. The readability for the survey achieved 82.3 (age 11 to 12) using the Flesch-Kincaid reading ease scale (using https://readable.io/text/). The frequency distribution of the questionnaire responses support an assumption of validity (frequency distribution shown in Table 2). The possible total scores ranged from 14 to 70 for the questionnaire (the greater the score, the more positive the overall impression of the study website). The mean scores for the three themes were: functionality, 3.99; personal contact, 3.89; and trust, 4.03; suggesting an overall positive perception of the study methodology. Table 3 shows the frequency distribution of the total score as the percentage of responses that fell within the questionnaire range. 99% of the questionnaire responses had a total score above the median score of 41, which indicates an overall positive impression of the study website and the themes contained within the questionnaire . Table 4 lists the common themes yielded by analysis of the free text responses.
|Item no.||Question||Strongly disagree||Disagree||Neither agree or disagree||Agree||Strongly agree|
|1||I would have preferred to have someone explain the TIME study to me on the phone||29(19%)||54(36%)||59(40%)||7(5%)||0(0%)|
|2||I trust the TIME study to protect my personal information||5(3%)||0(0%)||11(7%)||81(55%)||52(35%)|
|3||I am concerned my TIME study information could be shared without my permission||35(24%)||64(43%)||30(20%)||18(12%)||2(1%)|
|4||I would have preferred to meet with someone from the TIME study||35(24%)||46(31%)||63(42%)||5(3%)||0(0%)|
|5||I am happy to share my personal information with the TIME research team||1(<1%)||1(<1%)||14(9%)||92(62%)||41(28%)|
|6||The TIME study website instructions were clear||2(1%)||4(3%)||8(5%)||91(61%)||44(30%)|
|7||The British Heart Foundation/British Hypertension Society/University of Dundee logos reassured me about the TIME study website||2(1%)||3(2%)||42(28%)||79(53%)||23(16%)|
|8||I trust the TIME study to keep my participation in the study private||2(1%)||2(1%)||10(7%)||94(63%)||41(28%)|
|9||The TIME website did not always work||37(25%)||70(47%)||32(22%)||5(3%)||5(3%)|
|10||I like being able to take part in TIME without the need to visit a nurse or GP||3(2%)||3(2%)||29(19%)||82(55%)||32(22%)|
|11||Signing up to the TIME study was easy||4(3%)||8(5%)||6(4%)||89(60%)||42(28%)|
|12||It was not obvious how to use the TIME study website||35(24%)||72(48%)||24(16%)||15(10%)||3(2%)|
|13||I like that the TIME study is online||1(<1%)||1(<1%)||20(13%)||83(56%)||44(30%)|
|14||The layout of the TIME study website was confusing||37(25%)||76(52%)||29(19%)||5(3%)||2(1%)|
Table 1: Correlation Matrix of Survey Questions.
Table 2: Questionnaire items and frequency distribution (n and %) of responses (n=149).
|Range of Score||No. of Respondents (%)|
Table 3: Distribution of Total Scores (n=149).
The Functionality theme (Figure 2) was associated with whether patients went on to be randomised (t-test statistics=4.8, d.f.=147, p=<0.0001), such that lower scores were associated with a reduced likelihood of randomisation. Participants who were more comfortable with the lack of personal contact (Figure 3) offered by the TIME study were also more likely to be randomised (t-statistic=2.51, d.f.=147, p=0.0132). Concerns relating to privacy and protection of patient data (Figure 4) were not statistically associated with being randomised (tstatistic= 0.64, d.f.=147, p=0.5254).
The results from this questionnaire indicate that participant satisfaction was positive. Patients are concerned about privacy and confidentiality but the overwhelming majority of responders to the survey (Table 4) indicated altruistic attitudes to research specifically or took part for “the greater good” in general. Lack of face-to-face contact and a reduced sense of intimacy have been suggested as a disadvantage; however, in general, participants were comfortable and enthusiastic about the study being conducted entirely online. The concern of participants about privacy and confidentiality of their medical data, use and reliability of the required technology, and the generally positive attitude towards home telecare are in line with those reported in the published literature [6,9,21,25-29] and, therefore, support the construct validity of the questionnaire. Test-retest reliability will be conducted on the questionnaire at the end of the TIME study.
|Common Theme||No of respondents|
|Interested due to having hypertension||10|
|Due to being asked by GP or Biobank||13|
|Altruistic reasoning/helping others||53|
|Interested in research or improving research||55|
|Easy study to take part in||4|
|Family history of Heart Disease||1|
Table 4: Free Text Responses for Taking Part (n=149).
One weakness of any survey, whether self-administered, conducted by telephone or face-to-face, is that the instrument itself may promote change. Respondents can easily fall into role selection when answering questionnaires (i.e. respondents are likely to anticipate what the interviewer expects of them in an interpersonal situation and act accordingly) since no one is present to observe the role they are taking and challenge it. Specifically for self-administered surveys, a problem may arise if respondents misunderstand the questions or wish for clarification of some items . It is not known whether these factors played a part in the responses received.
An issue highlighted by the literature as affecting survey responses is perceived anonymity and privacy. The anonymity of subjects is imperative, since individual’s responses to survey items may be affected by their fear of identification. It is important to ensure anonymity and to assure participants that their responses will only be presented in aggregate form without identifying them. It is not known whether fear of identification would have played a part in the responses we received, or if responders even considered this possibility.
The developed questionnaire has several other limitations. This study was unable to specifically contact individuals who had not completed consent or had asked not to be contacted when they did consent. Individuals with concerns over privacy or who have anxiety using technology are unlikely to have signed up to the TIME study and therefore could not complete the questionnaire. Data on why individuals did not consent or decided not to take part would have been invaluable and likely added more insight into the more meaningful changes needed to increase participation in the study. Given the small sample size, it is unknown how representative the questionnaire is of the general TIME study population. Improvements are necessary to render the questionnaire re-usable and generalizable to other research studies that also use information technology.
The questionnaire underwent several iterations in attempts to improve questions and make them more relatable to the themes the investigators wished to evaluate. This took longer than anticipated and ultimately reduced the data collected in the final questionnaire as study recruitment was drawing to an end. The prolonged development also prevented further improvements aimed at increasing the internal consistency of the trust and personal contact themes. On-going development of the questionnaire will be necessary to strengthen its validity.
Electronic data capture systems have been proven to improve data quality over time. Their digital nature, and online availability, can provide fast and easy insight into outcomes such as adherence to treatment protocols and patient satisfaction . The developed questionnaire is an example of how web-based technologies can be used to evaluate clinical study technology. The questionnaire successfully identified the views of TIME study participants around the themes chosen by the investigators.
Participation rates in clinical studies have failed to improve over the last decade despite concerted efforts by researchers. Evaluation and feedback on existing studies will be essential to ensure that participants are not dissuaded from future research and that the technologies used meet with their expectations. The results of evaluation questionnaires such as described above can be used to improve trial procedures and electronic data capture tools. On-going development and assessment of participant satisfaction questionnaires will be necessary to improve the design and implementation of future technology-based clinical research applications.
Ethical approval and consent to participate
Ethical approval has been obtained from the Tayside Committee on Medical Research Ethics. MREC reference: 11/AL/0309. UKCRN ID: 17071. TIME is registered as ISRCTN: 18157641. The trial is performed in line with Good Clinical Practice guidelines and International Society of Pharmacoepidemiology (ISPE) Good Pharmacoepidemiology Practice Guidance. All participants have consented to taking part in the TIME study.
Consent for publication
All listed authors fulfil the requirements for authorship and agree to submission of the manuscript in its current form.
Availability of supporting data
Anonymised data from the study can be made available to bona fide researchers on application.
There are no conflicts of interest.
The TIME study has been funded by a grant from the British Heart Foundation, Greater London House, 180 Hampstead Road, London, NW1 7AW
David Rorie conceived the idea. The initial draft of the manuscript was created by David Rorie. Thomas MacDonald, Robert Flynn and Amy Rogers reviewed and edited the manuscript. All listed authors fulfil the requirements for authorship and agree to submission of the manuscript in its current form.
Medicines Monitoring Unit, University of Dundee.
- Donabedian A (1988) The quality of care. How can it be assessed? JAMA 260: 1743–1748.
- Greeneich DS, Long CO, Miller BK (1992) Patient satisfaction update: research applied to practice. Appl Nurs Res 5: 43–48.
- Hall JA, Dornan MC (1988) Meta-analysis of satisfaction with medical care: description of research domain and analysis of overall satisfaction levels. Soc Sci Med 27: 637–644.
- Rorie DA, Rogers A, Mackenzie IS, Ford I, Webb DJ, et al. (2016) Methods of a large prospective, randomised, open-label, blinded end-point study comparing morning versus evening dosing in hypertensive patients: the Treatment In Morning versus Evening (TIME) study. BMJ Open 6: e010313.
- Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE (2012) Knowledge translation of research findings. Implement Sci [Internet]. BioMed Central 7: 50.
- Bakken S, Grullon-Figueroa L, Izquierdo R, Lee NJ, Morin P, et al. (2006) Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc 13: 660-667.
- Yip MP, Chang AM, Chan J, MacKenzie AE (2003) Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare 9: 46–50.
- Britton BP (2003) First home telehealth clinical guidelines developed by the American Telemedicine Association. Home Healthc Nurse 21: 703–706.
- Finkelstein SM, MacMahon K, Lindgren BR, Robiner WN, Lindquist R, et al. (2012) Development of a remote monitoring satisfaction survey and its use in a clinical trial with lung transplant recipients. J Telemed Telecare 18: 42–46.
- Doolittle GC, Yaezel A, Otto F, Clemens C (1998) Hospice care using home-based telemedicine systems. J Telemed Telecare 4: 58-59.
- Currell R, Urquhart C, Wainwright P, Lewis R (2000) Telemedicine vs. face to face patient care: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2000: CD002098.
- Demiris G (2006) Principles of survey development for telemedicine applications. J Telemed Telecare 12: 111–115.
- Edwards P (2002) Increasing response rates to postal questionnaires: systematic review. BMJ 324: 1183.
- Ratzan SC, Parker RM, Selden CR, Zorn M (2000) National Library of Medicine Current Bibliographies in Medicine: Health Literacy. Bethesda, MD Natl Institutes Heal 2000.
- Nielsen-Bohlman L, Panzer AM, Kindig DA (2004) Health Literacy Washington, D.C.: National Academies Press; 2004.
- Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association (1999) Health literacy: report of the Council on Scientific Affairs. JAMA 281: 552–557.
- Stossel LM, Segar N, Gliatto P, Fallar R, Karani R (2012) Readability of patient education materials available at the point of care. J Gen Intern Med 27: 1165–1170.
- Badarudeen S, Sabharwal S (2010) Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res 468: 2572–2580.
- Kincaid JP, Braby R, Mears JE (1988) Electronic authoring and delivery of technical information. J Instr Dev 11: 8–13.
- Friedman, Charles P, Wyatt J (2006) Evaluation Methods in Biomedical Informatics. Computers and Medicine 2nd ed. XVII: 388.
- Demiris G, Speedie S, Finkelstein S (2000) A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare 6: 278–284.
- Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2: 53–55.
- Aron A, Aron E (1999) Statistics for psychology. 2nd ed. Upper Saddle River, NJ: Prentice Hall.
- Hair J., Black WC, Babin BJ, Anderson RE, Tatham RL (2006) Multivariate data analysis. Prentice Hall Pearson Education.
- Chang HH, Chang CS (2008) An assessment of technology-based service encounters & network security on the e-health care systems of medical centers in Taiwan. BMC Health Serv Res 8: 87.
- Heinzelmann PJ, Williams CM, Lugn NE, Kvedar JC (2005) Clinical outcomes associated with telemedicine/telehealth. Telemed J E Health 11: 329–347.
- Williams TL, Esmail A, May CR, Griffiths CE, Shaw NT, et al. (2001) Patient satisfaction with teledermatology is related to perceived quality of life. Br J Dermatol 145: 911–917.
- Mair F, Whitten P (2000) Systematic review of studies of patient satisfaction with telemedicine. BMJ 320: 1517–1520.
- Aoki N, Dunn K, Johnson-Throop KA, Turley JP (2003) Outcomes and methods in telemedicine evaluation. Telemed J E Health 9: 393–401.
- Wildeman MA, Zandbergen J, Vincent A, Herdini C, Middeldorp JM, et al. (2001) Can an online clinical data management service help in improving data collection and data quality in a developing country setting? Trials 12: 190.
Citation: Rorie DA, MacDonald TM, Rogers A, Flynn RWV (2018) Assessing Clinical Trial Technology: Evaluating the TIME Study. J Clin Trials 8: 342. DOI: 10.4172/2167-0870.1000342
Copyright: © 2018 Rorie DA, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Select your language of interest to view the total content in your interested language
Share This Article
- Total views: 1101
- [From(publication date): 0-2018 - Jan 22, 2019]
- Breakdown by view type
- HTML page views: 1059
- PDF downloads: 42