View source for Delphi Method (expert for identification)


The Delphi method is a technique used, primarily, to make quantitative assessments basing the anonymity as one of the main features compared with other measure methods. Normally is referred as a ‘quasi-anonymity’ because the respondents know one another, but their opinions and judgments remain in complete anonymity always. The technique consists in the structuring of a process of communication through experts to solve complex problems by series of questionnaires per rounds. The name was coined by the philosopher Kaplan, in honor of the Oracle of Delphi, at the beginning of the Cold War to obtain the most reliable consensus of opinion. This is characterized as a method for structuring a group communication process, making it more effective under the premise of a group of judgments is more valid than individual ones for dealing complex problems.

Organizations and companies use it when they want to tackle significant decision-making that will set the future guidelines. The results of this analysis have direct implications for the development or revision of a lot of aspects inside the companies. Furthermore, there are relevant applications of this method as forecasting, issue identification, prioritization or concept and framework development that makes it unique and a very powerful tool. The method accuracy can be evaluated by comparing it with other methods of measure with direct interaction, structured and unstructured. After the application of this method, other supplementary analysis to take into account may be the distributional estimates, the individual skills learning and different forms of feedback, as ask the subjects to write their reasons for the answers. Finally, the ethical considerations that have this method are delicate because the participants do not meet face to face, so they can react to ideas unbiased by the participants' identities. This article explains the Delphi methodology and their principles aspects and applications.


== Introduction ==

The Delphi method is a technique used to make quantitative assessments to solve complex problems by series of questionnaires made to experts in the fields of study. Companies consider it, mainly, when they have to tackle important decision-making that will set the future orientations. The results obtained in the analysis have direct implications for the evolution of their procedures, policies or programs (Loo 2002).

=== History === 

The first experiment using the method was implemented for the improvement in the scores of horse races. The name was coined by the philosopher Kaplan, in honour of the Oracle of Delphi, at the beginning of the Cold War to obtain the most reliable consensus of opinion (Loo 2002). All this work was secret because of its warlike character. Then, in the early 1960s, some information about the method commenced being published in the literature (Woudenberg 1991). The Delphi method has been used by many different disciplines during the years trying to get the maximum accuracy in every study field and continues these days being a powerful tool in development and improvement.

=== Objective of the method ===

The main objective of this technique is to obtain the most reliable predicted future based on the consensus of a group of experts in the study field (Okoli and Pawlowski 2004).

=== Techniques ===

Delphi method exists in two different forms. Commonly, is used the paper-and-pencil version (called ‘Delphi Exercise’). In this situation, a small team design and send a questionnaire which is sent and answered to the respondent group. After this questionnaire is returned to the developers and they summarize the results. Based on them, is been developed a new questionnaire for the respondents and successively. The respondents have one opportunity, at least, to reevaluate their answers. Moreover, there is another one called ‘Delphi Conference’ that is increasing in popularity over time. This technique replaces the team by a computer which has been programmed to compute the results of the group of study.

Comparing both techniques, there are advantages and disadvantages e.g. the ‘Delphi conference’ eliminates the delay caused in the summarization of each round. However, to apply it is required that the communication characteristics be perfectly defined to avoid possible compilation errors or misunderstanding (Linstone and Turoff 1976).

=== Main Characteristics ===

The key characteristics of the Delphi method can be defined  as seven different (McKenna 1994):
* The anonymity has to be guaranteed for the participants.
* The usage of a group of experts for obtaining data.
* The usage of two or more rounds between making a summary of the results of the previous round communicated to and evaluated by panel members.
* The participants or respondents can meet in face-to-face discussion.
* The Systematic emergence of a concurrence of judgment or opinion.
* The usage of sequential interviews and/or questionnaires.
* The usage of frequency distributions to identify agreement patterns.

Additionally, Robert Loo (Loo 2002) express his point of view of the characteristics as five main points that can be summarized in the seven presented before and the 
necessity of at least three or four iterations of questionnaires and feedbacks.

== Methodology ==


[[File:Theorical framework.jpg|500px|thumb|'''Figure 1.'''  Theoretical framework of Delphi qualitative in research (Figure adapted from (Mirzaaghabeik and Vosoughifar 2016) for the article)]]
The methodology of this technique can be divided into 3 simple points (Okoli and Pawlowski 2004):

* '''Selection of the Delphi methodology''': Make the schedule and organize which technique is going to be used as the number of rounds and participants is going to be. Delphi is very desirable because do not require that the experts meet personally.
* '''Procedure for selecting experts''': Being unsure of the identification of the relevant experts and invite them to participate in the study. This task is especially critical for the further development of the method because the group of decision-making requires qualified experts that have a deep understanding of the issues of study. The procedure consists of five steps:
** Prepare a Knowledge Resource Nomination Worksheet.
** Populating the Knowledge Resource Nomination Worksheet with names.
** First-round contacts - Nominations for additional experts.
** Ranking experts by qualification.
** Invite the experts to the study.

* '''Data collection and analysis method''': The mechanism for administering the questionnaires can be by any communication platform (e.g. Email). For administrate them, are involved three steps: 
** Brainstorming the important factors.
** List the most important ones.
** Rank the list to let the participants know.

Furthermore, in Figure 1 can be seen the simple theoretical structure of the complete process from the application requirements until the group response (data collection) (Mirzaaghabeik and Vosoughifar 2016).

=== Number of participants ===

The sample size is a practical consideration when the research is faced. For the number of participants, there are not fast or hard rules, but it has to be considered a number of factors (Skulmoski, Hartman, and Krahn 2007):

* '''Heterogeneity of the sample''': when is a homogeneous group, a smaller sample of between ten to fifteen is sufficient to obtain significant results. However, if disparate groups are involved, the size of the sample increases until hundreds of participants. Heterogeneous samples can increase the difficulty and the complexity of collecting data, conducting the analysis and verify the results.
* '''Decision quality''': There is a reduction in group error as size increases.
* '''Internal or external verification''': As the sample increases, the results obtained will be more convincingly can be to verify.

=== Number of rounds ===

The number of rounds is variable and depends mainly on the research purpose. In overall, with two or three iterations is sufficient for obtaining a good result, but depends directly on the homogeneity of the sample. If the work sample is heterogeneous, there will be required three or more rounds and, on the other hand, if the sample is homogeneous, fewer than three rounds may be enough to reach the consensus (Skulmoski, Hartman, and Krahn 2007).

== Applications ==

Delphi has been approved rapidly with hundreds of studies worldwide. Furthermore, the applications of this method have grown in a lot of different ways e.g. project management or social phenomena as human values and attitudes or quality of life (Sackman 1974). Okoli et al.  listed examples of the method applications dividing them into two main applications: Forecasting and Issue (identification/prioritization) and Concept/ Framework Development. Table 1 reflects the method applications in information systems research. 

In the research process, the applications of the Delphi method are very diverse. It can stand out among them: the specification of research questions, the selection of variables of interest/generation of propositions, the preliminary identification of causal relationships, the identification of the research topic or the definition of constructs and creation of a common language for discourse (Okoli and Pawlowski 2004).

{| class="wikitable"
|+ style="text-align: left;" | Table 1. Delphi method applications in information system research (Table adapted from (Okoli and Pawlowski 2004) for the article).
|-
! scope="col" | Application !! scope="col" | Example studies
|-
! scope="row" |  Forecasting and issue 
Identification / Prioritization 
| Identify the most critical issues in the coming 3-5 years

Forecast changing in the international business environment over the next decade and the changes impact 

Identify critical issues in IS in the coming 5 years 

Identify a prioritized list of international data communications activities vital to multinational corporations in managing information exchanges for control and implementation of global business  strategies

Forecast of the role of the systems in the 21st century 

|-
! scope="row" | Concept / Framework 
Development 
| Develop a framework of the main areas of the IS field 

Develop a descriptive framework of elemental knowledge manipulation activities 

Develop a capability-based typology of information technologies within the financial services industries 

Develop a conceptual taxonomy of organizational design actions-mechanisms to enhance technology users’ propensity to innovate in information technology 

Develop a ranked list of common risk factors for software projects as a foundation for theory building about IS project risk management 

|}

== Accuracy of the method ==

The accuracy of the Delphi method can be measured by the comparison between different judgment methods. In most studies, no statistical comparison between methods is made. So as is shown in Table 2, and based on bibliography (Woudenberg 1991), the judgment methods can be put on a scale of accuracy so that the less accurate is at the left and the most accurate at the right part.

{| class="wikitable"
|+ style="text-align: left;" | Table 2. Judgement methods in order of increasing accuracy (Table adapted from (Woudenberg 1991)).
|-
! colspan="1" style="col" |  
! colspan="3" style="col" | Traditional methods
! colspan="2" style="col" | Newly Methods

|-
! scope="row" | Method 
| Random individual || Staticized group || Unstructured, direct interaction || Structured, direct interaction || Delphi method 
|-
! scope="row" | Accuracy
|  Lower ||   ||   ||   || Higher
|}

As discussed above, when it is used the Delphi’s method, is normal to expect higher accuracy as compared to unstructured, direct interaction or the staticized group.

== Supplementary analysis ==

In addition to this technique, there are substudies that can be done to deepen in the search of information on the study that is being carried out, Norman Dalkey (Dalkey 1969), made a selection of the most interesting results:

* '''Distributional Estimates''': Is a valid hypothesis with respect to opinion is that participants have in their minds a fuzzy probability distribution over the quantity and when are requested to develop a single estimate, they select a measure of central tendency for this distribution.
* '''Learning''': Even the experiments are being done by experts; one consideration here is whether the estimation task is a skill that can be learned. Experiments made suggest that with the pass of the rounds, the learning of the participants get increased. Furthermore, there is no evidence that provides the existence of group knowledge. 
* '''Other forms of feedback''': Include other types of feedback in addition to the of previous-round answers, makes more efficient and accurate the questionnaires. This could be asking the participants who are in the extremes to know their reasons about the answers. 
* '''Sexual differences''': In average, the female participants are less accurate than men in their responses and they are more likely to change their answers than male participants. 
* '''Differences due to major''': are expected that participants whose major subject is one of the fields of study, would produce more accurate answers than another participant expert in a different matter.
* '''Comparison with simple iteration''': Without feedback, there is no improvement or in some cases, there is degradation in the participants’ answers.
* '''Time''': There is a fairly acute limit on the number of factors and the amount of processing that can be dealt with profitability.
* '''Self-evaluation''': Generally is used a scale from one to five for rating. The main hypothesis is that a subgroup is more accurate than the total group, but this hypothesis cannot be confirmed in every case, so it’s necessary to make a deep analysis to confirm it in every case of study.

== Ethical considerations ==

In Delphi method, is not necessary that the participants meet with each other face and accordingly. They can behave and present to some ideas impartially, that is to say, the participants have the freedom to react unbiased by the pressures and the identities of the rest. Moreover, the anonymity is one of the main relevant points of this method, making it a characteristic from other consensus and judgments methods.

To maintain the rigour of the technique, Sumsion (Sumsion 1998) suggested a 70% of response rate. To achieve it, the researcher has to know the identity of the participants and non-respondent must be pursued. Additionally, the complete anonymity presents some term problems. Because of that, is referred as a ‘quasi-anonymity’ and the respondents know one another, but their opinions and judgments remain in complete anonymity always.
Furthermore, based on the experimental results (Dalkey 1969) and comparing the face-to-face groups with the anonymous groups, the median response of the questionnaire is more accurate in the anonymous, even though the final decision is that there is not statistically significant between them.

== Annotated bibliography ==

Dalkey, Norman. 1969. “An Experimental Study of Group Opinion.” Futures 1(5):408–26. Retrieved (http://linkinghub.elsevier.com/retrieve/pii/S001632876980025X).
*The book was used to get the information about the comparison face to face and anonymous interaction in the pages 21-24 and the supplementary analyses in pages 50-72. 
Linstone, Harold A. and Murray Turoff. 1976. “The Delphi Method: Techniques and Applications.” Technometrics 18(3):363. Retrieved (http://www.millennium-project.org/FRMv3_0/04-Delphi.pdf).
*The book is a very complete source of information on the subject being discussed. It was used to develop the techniques of the Delphi method that are described in pages 5-7.
Loo, Robert. 2002. “The Delphi Method: A Powerful Tool for Strategic Management.” Policing: An International Journal of Police Strategies & Management 25(4):762–69. Retrieved (http://www.emeraldinsight.com/doi/10.1108/13639510210450677).
*The journal article gives an overview of the Delphi method focused on the strategic management. Introduction, History and Characteristics were checked in pages 1-2.
McKenna, Hugh P. 1994. “The Delphi Technique: A Worthwhile Research Approach for Nursing?” Journal of Advanced Nursing 19(6):1221–25. Retrieved (http://doi.wiley.com/10.1111/j.1365-2648.1994.tb01207.x).
*The journal article was reviewed for amplifying the knowledge in the characteristics of the Delphi method in page 1222.
Mirzaaghabeik, Hossein and Hamid Reza Vosoughifar. 2016. “Comparison between Quality and Quantity Seismic Damage Index for LSF Systems.” Engineering Science and Technology, an International Journal 19(1):497–510. Retrieved (http://dx.doi.org/10.1016/j.jestch.2015.08.013).
*The journal article was used to adapt the Figure 4 Theoretical framework of Delphi for the article (Figure 1) in page 500.
Okoli, Chitu and Suzanne D. Pawlowski. 2004. “The Delphi Method as a Research Tool: An Example, Design Considerations and Applications.” Information & Management 42(1):15–29. Retrieved (http://linkinghub.elsevier.com/retrieve/pii/S0378720603001794).
*The journal article gives a very good overview of the method on page 2. It was used to adapt the Table 1 for the article from the Applications of the Delphi method in page 3 & 15 and ‘Methodology’ in pages 6-14.
Sackman, H. 1974. Delphi Assessment: Expert Opinion, Forecasting and Group Process. Retrieved (https://www.researchgate.net/publication/235209130_Delphi_Assessment_Expert_Opinion_Forecasting_and_Group_Process).
*The book is divided in three big parts; a deep analysis of the delphi method, the method versus social science standards and the Delphi evaluation. The book was used in the first and third chapter to amplify the knowledge about the applications in page 3 and annonimity and accountability in pages 62-65, respectively.
Skulmoski, Gregory J., Francis T. Hartman, and Jennifer Krahn. 2007. “The Delphi Method for Graduate Research” edited by D. Heymann. Journal of Information Technology Education 6(1):1–21. Retrieved (https://www.researchgate.net/publication/220590716_The_Delphi_Method_for_Graduate_Research).
*The journal article from the ‘Journal of Information and Technology’ was checked for the ‘Number of participants’ on page 10 and the ‘Number of rounds’ in page 11 to complete the information of this article.
Sumsion, Thelma. 1998. “The Delphi Technique: An Adaptive Research Tool.” British Journal of Occupational Therapy 61(4):153–56. Retrieved (http://journals.sagepub.com/doi/10.1177/030802269806100403).
*The journal article gives an overview of the Method, including the number of participants, rounds and suggested uses in management. The article was used to obtain the data of the response suggested rate for each round to maintain the rigour of the technique on page 154.
Woudenberg, Fred. 1991. “An Evaluation of Delphi.” Technological Forecasting and Social Change 40(2):131–50. Retrieved (http://linkinghub.elsevier.com/retrieve/pii/004016259190002W).
*The journal article Explains the accuracy of the method and the characteristics attached to it. For the article, it was used to check the History on page 132, the accuracy of the method and the comparison of other judgement methods in pages 132 & 134 and the Characteristics of Delphi in page 133.