Utilizing the Delphi Survey Approach: A Review

The Delphi survey is a method of gathering the collective opinion of a group of experts on a particular topic. It is based on the premise that ‘pooled intelligence’ enhances individual judgment and captures the collective opinion of a group of experts [1]. It provides an opportunity for experts (panelists) to communicate their opinions and knowledge anonymously about a complex problem or a topic of interest, to see how their evaluation of the issue aligns with others, and to change their opinion, if desired, after reviewing and reconsideration of the collective findings of the group’s ideas [2]. Dalkey and Helmer in 1963:458, suggest that it can gather “the most reliable consensus of opinion of a group of experts”. It was first utilized by the Greeks. The paper presents the: historical and philosophical perspectives, key characteristics, advantages and disadvantages of the Delphi survey.

Nursing adopted the Delphi approach in the 1970s, and since it has been applied within the health care field [8]. It has been applied widely for exploring research problems in nursing education, and practice [9]. For example: Facione in 1990 applied it to gain consensus on the definition of critical thinking [10]; while Ganga-Limando in 2001 used it to construct a conceptual framework for nursing education in francophone Africa and Kennedy in 2004 applied the Delphi survey to devise a model of exemplary midwifery practice [11]. Within health care, Austin-Lane, Girasek and Barbour in 2004 built a Conceptual Framework of Influences on State Tobacco Control using a modified Delphi survey.

Philosophical Orientation
Scientific knowledge using the positivistic paradigm is generated through an application of logical reasoning and objective principles by various means that include: observation; measuring; and quantifying with a view to making generalizations [12]. The philosophical underpinnings that guide the Delphi survey are closely aligned to positivism, mainly because of the intention and objectives of the study which are primarily to build consensus and require the use of quantifiable methods [13]. Although applying the Delphi approach does generate qualitative data due to the use of open ended questions, the narrative data is analyzed for the major and most recurring themes and therefore aligned to positivistic philosophy.
Whilst the Delphi survey is consistent with the principles of positivism because of the use of quantitative methods, numerical data and statistical analysis, its philosophical assumptions also have a base in Hegelian dialectic. This approach utilizes the principles of building consensus through the generation of thesis, antithesis and synthesis [14] (see Figure 1). In this regard, thesis is an idea generation stage (ideas generation in round 1); antithesis takes place when there is a conflict of opinion (evaluation ideas) while synthesis occurs when consensus is achieved (re-evaluation of ideas) [15]. A "thesis" is opposed to an "anti-thesis," which the thesis generally provokes. They confront each other, correct one another mutually or destroy each other but combine and finally engender a synthesis [16]. This is a continuous process, so that synthesis does not necessarily mean an end but can give rise to another thesis. Linstone and Turoff in 2002 suggest that whilst the Hegelian dialectic does not automatically lead to a new agreement, when this does happen, the resulting synthesis is likely to be strong consensus [17].  [14]).

Research Process
The Delphi survey is a systematic process which aims to: gather information on a specific issue, involves a group of experts that reach consensus through iterative rounds with the use of questionnaires, whose opinions are anonymous, expert panelists do not meet physically and maybe geographically dispersed ( Figure 2). Skulmoski et al. in 2007 suggest that the Delphi survey is well suited for application when knowledge about a phenomenon is incomplete. Skulmoski et al. in 2007 state that such a survey can be utilized well when the goal is to enhance understanding of problems, opportunities and solutions. The method must be selected carefully being cognizant of its advantages and disadvantages [18].

Advantages
This approach offers several advantages, which makes it an important research methodology for health and nursing research. It utilizes experts in the field and brings together the collective wisdom of expert panelists in a cost effective manner [19]. According to Cook in 1994, it facilitates group communication and sharing of information among experts panelists, anonymously, and paradoxically also allows independent thinking. It allows the expert panellists to focus on key issues within the questionnaire, which in turn prevents them getting side tracked. Content validity is assured by means of involving expert panelists and iterative rounds [20]. It provides anonymity and confidentiality to the expert panelists, which in turn prevents dominance by influential individuals and avoids group pressure and group think [21]. The Delphi survey can incorporate participants: with relevant expertise; from geographically diverse locations; and large panel size [7]. Furthermore, the rationale for such a survey can stem from the ability to utilize a sample that is geographically dispersed and where members cannot and do not meet face to face therefore avoids dominance by certain high profile candidates might have a confounding effect on the study. The aspect of anonymity may encourage openness to express honest opinions, and multiple iterative rounds gives the participants an opportunity to re-evaluate their ideas leading to increased content validity and finally, the questionnaire is self-reported and self-administered [22].

Disadvantages
The Delphi survey poses some important disadvantages as well. Such surveys can be time consuming due to their iterative nature, and the expert panelists may lose interest in the research study overtime [23]. Another disadvantage cited in the literature is related to a clear definition of consensus. Literature suggests ranges between 51% to 70% percentage agreements represents consensus [24]. Furthermore, there are no clear guidelines suggesting definitions of experts, panel size and sampling techniques [25]. Another difficulty with this approach is that the attrition rates are high and increase with the number of rounds [26]. Together with lack of clear guidance on panel size, this could be problematic.
The nature of Delphi survey is iterative and overtime and this may result in the panelist changing their minds during the course of the study about issues, and this might hinder consensus building. The passage of time maybe a problem as be a factor that is true now may be different within the next few months along the time when data is collected. Linked to the selection of experts is the quality of their input (which is self-reported): this is critical to the outcome of the Delphi survey, as is true for many other survey research methodologies [9].

Key Characteristics
The key characteristics of a Delphi survey are: expert panel, iteration of rounds and controlled feedback, statistical summaries of group response, anonymity and consensus building [8,27,28] ( Figure  2). The Delphi survey is applied by means of a series of questionnaires that are completed anonymously by individuals on the expert committee. It is a process of group communication without the group ever meeting face to face. The responses from each set of questionnaires are analyzed, summarized and then sent back to the participants until a large extent of consensus is reached on the area of interest [29]. Both qualitative and quantitative data can be generated through a Delphi survey [30]. It is a flexible approach and can be modified, whilst maintaining the main principles, to achieve the purpose of the research [31,32].

Expert Panel
Hasson et al. in 2000 suggest that, "Studies employing the Delphi survey make use of individuals who have knowledge of the topic being investigated". The inclusion of a panel of experts is based on the rationale that the pool intelligence of a group of experts is better than one expert when exact knowledge on a topic is not available [33]. The selection of the right panel of experts is the most critical factor in the success of a Delphi survey as this depends on their collective expertise [34].
Studies applying the Delphi survey usually use non-random, purposive samples. The sample selected when employing such a survey is referred to as the "panel of experts" [23]. Purposive sampling refers to the sample being selected purposely and depends on the researcher's judgment, in line with the aim of the study, regarding whom he/she judges to be typical of the population and is particularly knowledgeable about the issues being studied [29,24].Whilst, some authors are liberal in their definitions of experts and suggest that these are people possessing the relevant knowledge and experience of a particular topic, within the context of a specific study [3]. Needham and de Loë in 1990 suggest that expertise lies along a continuum which includes experts with subjective expertise, mandated expertise and objective expertise. Donohoe and Needham in 2008 further suggest that experts be identified considering their proximity to the issue under investigation. Subjective expertise -possessing knowledge by being affected by the issue under study; Mandated expertiseknowledge and experience related to the job description and role requirement; Objective expertise -knowledge gained due to academic position, education and research.

Panel Size
This refers to the number of expert panelists to be included in the study (Polit and Beck 2008). There are no clear guidelines suggesting the numbers to be included in studies applying the Delphi survey because the sample is purposively selected and it depends on the problem being investigated. Some studies have used 15 participants whilst others have used 60 [35]. Needham and de Loë in 1990 suggest a sample size of a minimum of 10 (a smaller size does not generate enough ideas) and a maximum of 50 participants (a larger sample results in cost inefficiencies related to time, product and the iteration process). Whilst, DeVilliers, De Villiers and Kent in 2005 define sample size depending on whether it is homogenous or heterogeneous and suggest the following numbers: if they are from the same discipline (15 -30) or from differing ones (5-10) per professional group. Delphi survey studies do not call for a representativeness of the sample in terms of statistical purposes; therefore, sample size principles differ from those in other surveys [34].

Iterative Rounds
Delphi surveys are conducted over a series of iterative rounds, and expert panelists are expected to complete a series of questionnaires until consensus is reached [24]. Whilst there are no strict guidelines on the right number of rounds to be undertaken, generally the number of rounds shown in literature was between two and four [21]. Brockhoff ([Sa]) investigated the performance of Delphi groups in relation to the number of rounds and concluded that it was not reasonable to extend the number of rounds beyond the third one [7]. The purpose of the rounds is to: round 1 (thesis stage) to generate ideas; round 2 to review and evaluate ideas (antithesis stage) against the group summaries and round 3 re-evaluate ideas and to arrive at consensus (synthesis stage) ( Figure 2).

Data collection Instruments
The data collection instruments are designed to solicit information about a topic of interest from expert panelists by the use of written questions [22]. The data solicited is self-reported by the panelist, meaning that it represents their knowledge, perception or experience and is self-administered: they complete the questionnaire themselves [12]. The questions can be closed ended or open ended [23].

Questionnaire Development (Qualitative)
The purpose of the first round is to generate ideas regarding the issue of interest. Literature review can inform the development of the content and concepts of the questions for the first questionnaire [24]. The data from the participants' input and literature reviews informs the development of subsequent question questionnaires. The first questionnaire can be partly structured and partly semi-structured. The first questionnaire is partly structured to solicit demographic data and contact information. This questionnaire is largely open-ended, to solicit information and ideas from the experts own knowledge and experience [22].

Questionnaire Development (Quantitative)
The aim of the second round and subsequent rounds is to evaluate and re-evaluate ideas and develop consensus among the expert panelists. The concepts identified in the first questionnaire inform the formulation and development of the second questionnaire, usually with close ended questions. The questions use a Likert scale that aims at gathering information about their level of agreement or disagreement (strongly agrees; agree; undecided; disagree; strongly disagree). Such a scale is designed to determine the opinion of the participant on an issue under investigation [12]. Polit and Beck (2008:51) suggest that researchers who are concerned about their research process may undertake a pilot study which is a small-scale version of the major study. They add that if questionnaires are used, then they should be pretested for length, clarity and overall adequacy [24].

Data Collection Process
A database with the current information pertaining to the expert panelists needs to be created, to help the researcher determine whom to include in the study, with attention to accomplishing the objectives of the study. Furthermore, considering the nature of Delphi survey that requires reiterative rounds and therefore mandates a database of the participants' contacts.
The data collection process can be time consuming as the rounds are iterative and can take up to six months, therefore needs to be factored in the research timelines. The expert panellists can be contacted before the start of the study to gauge their interest and ability to participate in the study, as it does require time and effort. Questionnaires can be posted or emailed. Follow up of expert panellists is important does impact the number of individuals who will finally be part of the study [36].
The process of data collection for the second and third rounds is identical to the first round. Data analysis is done in-between rounds ( Table 1). Following the first round the questionnaires are analyzed as per qualitative data analysis protocol. Then the second questionnaire is developed based on the strong themes and ideas that are identified. The third questionnaire is modified depending on the cumulative responses to the second questionnaire. Areas where there was no consensus in the 2nd round can be omitted from the third round.
The second questionnaire is sent to the participants who respond to the first questionnaire. Some studies are known to include the entire sample (initial sample); even those that did not respond. In other studies, only those who participated in the previous round are included in the subsequent round [6]. The rationale being that those who do not return questionnaires may have not responded because they were not sufficiently interested or thought they have nothing to contribute towards the study.

Data Analysis Qualitative Data Analysis
This process can be done by utilizing qualitative analysis computer packages or done manually. The unstructured data from the open ended questions in questionnaire 1 is transcribed into Word documents, verbatim. The documents are analyzed to identify relationships and patterns; similarities and differences are identified. Words and phrases can be grouped by cutting and pasting the Word document into clusters of similar ideas and concepts and highlighting in different colours. This helps in grouping similar concepts together and identifying the most commonly occurring concepts. The analysis of this phase can be undertaken independently by the researcher and an assistant; the notes were compared to validate the concepts that occurred. The concepts that most commonly occur are then developed into close ended questions and developed into the 2nd questionnaire.

Quantitative Data Analysis
The computer package SPSS (Statistical Package for Social Scientists) can be utilized. Ascertaining the groups' collective opinion requires the use of descriptive statistics, in consultation with a statistician [29]. Descriptive statistics are used mainly because the questionnaires (second and third) are designed to collect nominal and ordinal data. The nominal data examines the percentages in terms of the percentage of agreement, while the ordinal data examines the measures of central tendency and includes: the means and level of dispersion such as the standard deviation [24]. The statistical analyses conducted are designed to measure the level of agreement related to the concepts in the questionnaire. The statistical tests usually relevant are Percentage agreement (PA), mean (M) and standard deviation (SD).

Validity and Reliability Validity
It is the extent to which a method measures what it is intended to measure [37,38]. Validity includes face validity and content validity.

Face Validity
Refers to whether the instrument looks as though it is measuring the appropriate concepts [24]. It is a subjective judgment that the instrument measures what it intends to measure in terms of the relevance and presentation of the questionnaire [39]. It includes the questionnaire being: readable; exhibiting clarity of content and language; being unambiguous; and clear [40].

Content Validity
Refers to the judgments of a panel of experts about the extent to which the content of the questionnaire appears logically to examine and comprehensively include the characteristics of the domain being explored. The Delphi approach is particularly strong in achieving content validity [19]. Content validity can be improved by: developing data collection instrument that is informed by: published literature; pretesting the tools; carefully analyzing the data from the first questionnaire; and including expert panelist who are knowledgeable on the topic being studied. Furthermore Rounds 2 and 3 confirm the validity of the content and concepts by giving the panel of experts an opportunity to review these. This suggests that the final results are high in content validity [22].

Reliability
This refers to the ability of the instrument to yield similar results when repeating the same study using similar conditions, producing the same or similar results consistently [22,37,21]. Keeney, Hasson and McKenna in 2011 suggest that the Delphi survey enhances reliability because: in the decision making process the members of the expert panel do not meet face to face, which eliminates group bias or group thinking; panel size and iterative rounds increases the reliability.

Ethical Principles Applied to the Delphi Approach Approval and Ethical Clearance
To ensure that the study maintained high ethical standards, the proposed protocol is usually submitted for ethical clearance, to the relevant research and ethics boards.

Right to Autonomy and Informed Consent
This is safeguarded by explaining the benefits, rights and risks involved in the research study in writing and securing consent by the return of the questionnaire. A covering letter that explains the purpose of the study at every round of the study is attached. Additionally, the researcher's details are made available to the participants so that they have the opportunity to contact the researcher if they had any questions. They are required to sign the cover page of the questionnaire and return it to the researcher, though returning the questionnaire itself indicates implied as agreement to partake in the study and implied consent. These measures support the ethical principles of respect and the right of self-determination and of obtaining an informed consent [12].

Right of Self-Determination
This is respected by: participants choose what information they share with the researcher; and are at liberty to withdraw from the study at any time without penalty. However, once the aggregate group summaries are developed withdrawal from the study becomes complex as the data is no longer individual but in aggregate group summaries.

Anonymity and Confidentiality
This refers to concealing the identity of the participants in all documents resulting from the research. Anonymity is a key feature of the Delphi survey, which serves four fundamental purposes: it assures the expert panelists' rights; prevents group think; prevents dominance by influential or high profile individuals; and encourages independent decision-making [21]. The Delphi survey is a group communication process whereby the participants may never meet each other or know who has participated in the process [28]. Furthermore, there is the guarantee of anonymity of the participants' individual responses and these are never known to one another [9]. Though this may potentially lead to lack of accountability for the response, this is a possibility not unique to a Delphi survey and is true for other self-report surveys as well [41]. Anonymity can be achieved at various levels. Participants may be completely unknown to each other; furthermore there should be no potential for other panelists to ascribe a response to any individual and the researcher should not reveal their identities. In highly specialized areas the panelists may deduce who the others are; though is that their judgments and opinions are anonymous, and this has been termed quasi-anonymity [9,8]. The nature of the Delphi survey requires that the researcher follow up the expert panelist, which prevents total anonymity from that aspect as well, rendering it as quasi-anonymous [23]. Confidentiality is maintained by delinking the data from the person and utilizing codes numbers for returned questionnaires.

Conclusion
The Delphi technique is a research approach that is underutilized in nursing. However, for certain types of research it offers great potential and usefulness as an alternative approach. This is true particularly for areas where there is scant research and the opinions of experts can provide valuable insight into the problems.
It has been utilized in nursing since the 70s in the western world and later in health care field generally. It is a systematic approach and like all other approaches has advantages and disadvantages. It facilitates group communication and sharing of information among a group of expert to build consensus. Major disadvantages are related to lack of clear guidelines related to definitions of experts, panel size and sampling techniques. The key characteristics of a Delphi survey are: expert panel, iteration of rounds and controlled feedback, statistical summaries of group response, anonymity and consensus building.
The Delphi approach can be a useful method if it is systematically applied with attention the principles of the approach and universal principles of sound ethical research.