Wideband Delphi - Revision history

S220027: /* Introduction */

2022-03-27T21:11:31Z

Introduction

← Older revision		Revision as of 21:11, 27 March 2022
Line 15:		Line 15:

	The original Wideband Delphi steps were<ref name="Boehm" ></ref>:		The original Wideband Delphi steps were<ref name="Boehm" ></ref>:
	1.The Coordinator provides each expert with a specification and an estimate sheet.		1.The Coordinator provides each expert with a specification and an estimate sheet.
	2.The coordinator convenes group meetings where experts discuss estimation issues with the coordinator and with each other.		2.The coordinator convenes group meetings where experts discuss estimation issues with the coordinator and with each other.
	3.Experts fill out forms anonymously.		3.Experts fill out forms anonymously.
	4.The coordinator prepares and distributes summaries of estimates		4.The coordinator prepares and distributes summaries of estimates
	5.The coordinator convenes group meetings with a particular focus on getting the experts to discuss points where their estimates vary widely		5.The coordinator convenes group meetings with a particular focus on getting the experts to discuss points where their estimates vary widely
	6.The expert filled out the form anonymously again and repeated steps 4 through 6 as needed.		6.The expert filled out the form anonymously again and repeated steps 4 through 6 as needed.
	It was later improved by Neil Potter and Mary Sakry of The Process Group to better embed The project management framework and came out the modern version.<ref name="Stell" >A. Stellman and J. Greene (2005), Applied software Project Management. 1st Edition. O’Reilly Media Inc.</ref>		It was later improved by Neil Potter and Mary Sakry of The Process Group to better embed The project management framework and came out the modern version.<ref name="Stell" >A. Stellman and J. Greene (2005), Applied software Project Management. 1st Edition. O’Reilly Media Inc.</ref>

S220027: /* Introduction */

2022-03-27T21:11:19Z

Introduction

← Older revision		Revision as of 21:11, 27 March 2022
Line 14:		Line 14:
	Wideband Delphi was first proposed by Barry Boehm and John A. Farquhar in the 1970s and promoted by Boehm in his book Software Engineering Economics (1981)<ref name="Farq" > Farquhar JA (1970) A preliminary inquiry into the software estimation process. The Rand Corporation, Santa Monica, CA, Technical Report RM-6271-PR</ref><ref name="Boehm" >Boehm BW (1981) Software engineering economics. Prentice-Hall, Englewood Cliffs, NJ</ref>.Unlike the traditional Delphi method, it encourages discussion among participants. When Boehm(1981) applied it to cost estimation, he found that facilitating discussion and wider channels of communication could produce more accurate results and contribute to experience sharing and team building. Compared with the Rand Delphi method of narrowband communication, this improved method is called the wideband Delphi method. Recent research in software project evaluation shows that evaluations that benefit from group discussions tend to be more accurate. <ref name="MIT"/>		Wideband Delphi was first proposed by Barry Boehm and John A. Farquhar in the 1970s and promoted by Boehm in his book Software Engineering Economics (1981)<ref name="Farq" > Farquhar JA (1970) A preliminary inquiry into the software estimation process. The Rand Corporation, Santa Monica, CA, Technical Report RM-6271-PR</ref><ref name="Boehm" >Boehm BW (1981) Software engineering economics. Prentice-Hall, Englewood Cliffs, NJ</ref>.Unlike the traditional Delphi method, it encourages discussion among participants. When Boehm(1981) applied it to cost estimation, he found that facilitating discussion and wider channels of communication could produce more accurate results and contribute to experience sharing and team building. Compared with the Rand Delphi method of narrowband communication, this improved method is called the wideband Delphi method. Recent research in software project evaluation shows that evaluations that benefit from group discussions tend to be more accurate. <ref name="MIT"/>

	The original Wideband Delphi steps were<ref name="Boehm" ></ref>:		The original Wideband Delphi steps were<ref name="Boehm" ></ref>:
	1.The Coordinator provides each expert with a specification and an estimate sheet.		1.The Coordinator provides each expert with a specification and an estimate sheet.
	2.The coordinator convenes group meetings where experts discuss estimation issues with the coordinator and with each other.		2.The coordinator convenes group meetings where experts discuss estimation issues with the coordinator and with each other.

S220027: /* Robustness */

2022-03-27T21:05:42Z

Robustness

← Older revision		Revision as of 21:05, 27 March 2022
Line 145:		Line 145:

	=== Robustness ===		=== Robustness ===
	- Not easy to ~~promise~~.		- Not easy to guarantee.
	Expert judgment can be biased by irrelevant and misleading information. For example, a client's request for a project budget may cause an estimator (willing to meet the client) to favor that budget. But Wideband Delphi is designed to gather collective assessments from experts, and structured group consensus procedures help reduce bias without the risk that some voice or influential individual will distort the results.		Expert judgment can be biased by irrelevant and misleading information. For example, a client's request for a project budget may cause an estimator (willing to meet the client) to favor that budget. But Wideband Delphi is designed to gather collective assessments from experts, and structured group consensus procedures help reduce bias without the risk that some voice or influential individual will distort the results.

S220027: /* Robustness */

2022-03-27T21:05:11Z

Robustness

← Older revision		Revision as of 21:05, 27 March 2022
Line 145:		Line 145:

	=== Robustness ===		=== Robustness ===
	- Not easy.		- Not easy to promise.
	Expert judgment can be biased by irrelevant and misleading information. For example, a client's request for a project budget may cause an estimator (willing to meet the client) to favor that budget. But Wideband Delphi is designed to gather collective assessments from experts, and structured group consensus procedures help reduce bias without the risk that some voice or influential individual will distort the results.		Expert judgment can be biased by irrelevant and misleading information. For example, a client's request for a project budget may cause an estimator (willing to meet the client) to favor that budget. But Wideband Delphi is designed to gather collective assessments from experts, and structured group consensus procedures help reduce bias without the risk that some voice or influential individual will distort the results.

S220027: /* Introduction */

2022-03-27T20:59:14Z

Introduction

← Older revision		Revision as of 20:59, 27 March 2022
Line 9:		Line 9:

	=== Introduction ===		=== Introduction ===
	Delphi method is a feedback anonymous correspondence method in essence. It originated from the prediction project conducted by Rand Corporation at the request of the US military during the Cold War. It assumes that the judgment of a group is more effective than that of an individual (usually an expert). The general process is to collect, summarize, and make statistics on the problems to be predicted after obtaining the opinions of experts, and then feedback anonymously to the experts. Opinions are solicited again, and then centralized again, and then feedback again until unanimous opinions are reached.		Delphi method is a feedback anonymous correspondence method in essence. It originated from the prediction project conducted by Rand Corporation at the request of the US military during the Cold War. The classic Delphi method is a method in which companies form a specialized forecasting organization, which includes several experts and corporate forecasting organizers, and follow a prescribed procedure to solicit experts' opinions or judgments about future markets back to back and then make forecasts. It assumes that the judgment of a group is more effective than that of an individual (usually an expert). The general process is to collect, summarize, and make statistics on the problems to be predicted after obtaining the opinions of experts, and then feedback anonymously to the experts. Opinions are solicited again, and then centralized again, and then feedback again until unanimous opinions are reached.

	Wideband Delphi is a variation of the Delphi estimating method where subject matter experts complete multiple rounds of producing estimates individually, with a project team discussion after each round, until a consensus is achieved. For Wideband Delphi, those who created the highest and lowest estimates explain their rationale, following which everyone reestimates. The process repeats until convergence is achieved. <ref name="PMBOK">A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English) — 2021, pp178. xxvi, 67, 274 Seiten</ref>		Wideband Delphi is a variation of the Delphi estimating method where subject matter experts complete multiple rounds of producing estimates individually, with a project team discussion after each round, until a consensus is achieved. For Wideband Delphi, those who created the highest and lowest estimates explain their rationale, following which everyone reestimates. The process repeats until convergence is achieved. <ref name="PMBOK">A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English) — 2021, pp178. xxvi, 67, 274 Seiten</ref>

S220027: /* Abstract */

2022-03-27T20:58:58Z

Abstract

← Older revision		Revision as of 20:58, 27 March 2022
Line 2:		Line 2:

	== Abstract ==		== Abstract ==
	The Wideband Delphi estimation method is a technique for estimating project workload based on participant consensus, mainly used in Management of Software development Projects. It is based on the Delphi method, famously developed by the RAND Corporation in the 1950s-1960s as a forecasting tool. The classic Delphi method is a method in which companies form a specialized forecasting organization, which includes several experts and corporate forecasting organizers, and follow a prescribed procedure to solicit experts' opinions or judgments about future markets back to back and then make forecasts. In the 1970s, Barry Boehm and John A. Farquhar proposed a wideband variant of the Delphi method. The term "wideband" is used because the wideband Delphi technique involves more interaction and more communication between participants than the Delphi method. It has since been adapted in many industries to estimate a wide range of tasks, from statistical data collection results to sales and marketing forecasts.<ref name="BASI" >Basili VR, Caldiera G, Rombach HD (1994b) The goal question metric approach.</ref> <ref name="SS" >In: Marciniak JJ (ed) Encyclopedia of software engineering, vol 1, 2nd edn. Wiley, New York, pp 528–532 </ref> The main advantages of Wideband Delphi are simplicity of implementation, low cost, and no need for historical data.		The Wideband Delphi estimation method is a technique for estimating project workload based on participant consensus, mainly used in Management of Software development Projects. It is based on the Delphi method, famously developed by the RAND Corporation in the 1950s-1960s as a forecasting tool. In the 1970s, Barry Boehm and John A. Farquhar proposed a wideband variant of the Delphi method. The term "wideband" is used because the wideband Delphi technique involves more interaction and more communication between participants than the Delphi method. It has since been adapted in many industries to estimate a wide range of tasks, from statistical data collection results to sales and marketing forecasts.<ref name="BASI" >Basili VR, Caldiera G, Rombach HD (1994b) The goal question metric approach.</ref> <ref name="SS" >In: Marciniak JJ (ed) Encyclopedia of software engineering, vol 1, 2nd edn. Wiley, New York, pp 528–532 </ref> The main advantages of Wideband Delphi are simplicity of implementation, low cost, and no need for historical data.
	This paper will describe the main steps of Wideband Delphi and analyze its application in different areas, such as software development projects, presenting the role and advantages it plays in project management. This paper will also give practical examples with diagrams to help understand the process of its use.<ref name="MIT" >Valerdi R. 10.4. 2 convergence of expert opinion via the Wideband delphi method: An application in cost estimation models[C]//Incose International Symposium. 2011, 21(1): 1246-1259.</ref>		This paper will describe the main steps of Wideband Delphi and analyze its application in different areas, such as software development projects, presenting the role and advantages it plays in project management. This paper will also give practical examples with diagrams to help understand the process of its use.<ref name="MIT" >Valerdi R. 10.4. 2 convergence of expert opinion via the Wideband delphi method: An application in cost estimation models[C]//Incose International Symposium. 2011, 21(1): 1246-1259.</ref>
	In addition, the paper will discuss the strength and weakness within the existing framework and how it can be integrated with SCRUM(a framework for developing, delivering, and sustaining products in a complex environment<ref name="SCRUM" >Schwaber, Ken; Sutherland, Jeff (November 2017), The Scrum Guide: The Definitive Guide to Scrum: The Rules of the Game, retrieved May 13, 2020</ref>.） in Planning Poker.		In addition, the paper will discuss the strength and weakness within the existing framework and how it can be integrated with SCRUM(a framework for developing, delivering, and sustaining products in a complex environment<ref name="SCRUM" >Schwaber, Ken; Sutherland, Jeff (November 2017), The Scrum Guide: The Definitive Guide to Scrum: The Rules of the Game, retrieved May 13, 2020</ref>.） in Planning Poker.

S220027: /* Annotated bibliography */

2022-03-27T20:57:14Z

Annotated bibliography

← Older revision		Revision as of 20:57, 27 March 2022
Line 192:		Line 192:

	Annotation：This paper not only reviews the origin and concept of Delphi method and Wideband Delphi method, but also uses cost estimation model to verify the effect of Wideband Delphi method in Expert Support.		Annotation：This paper not only reviews the origin and concept of Delphi method and Wideband Delphi method, but also uses cost estimation model to verify the effect of Wideband Delphi method in Expert Support.


	'''4. A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English), pp170-181. '''		'''4. A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English), pp170-181. '''

S220027: /* Annotated bibliography */

2022-03-27T20:57:05Z

Annotated bibliography

← Older revision		Revision as of 20:57, 27 March 2022
Line 193:		Line 193:
	Annotation：This paper not only reviews the origin and concept of Delphi method and Wideband Delphi method, but also uses cost estimation model to verify the effect of Wideband Delphi method in Expert Support.		Annotation：This paper not only reviews the origin and concept of Delphi method and Wideband Delphi method, but also uses cost estimation model to verify the effect of Wideband Delphi method in Expert Support.

	4. ''' A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English), pp170-181. '''		'''4. A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (English), pp170-181. '''

	Annotation: This section of the book lists various commonly used estimation methods and meeting patterns, including Wideband Delphi. This gives us a clearer overview of Wideband Delphi's role and domain in the project estimation environment.		Annotation: This section of the book lists various commonly used estimation methods and meeting patterns, including Wideband Delphi. This gives us a clearer overview of Wideband Delphi's role and domain in the project estimation environment.

S220027: /* Advantage and disadvantage */

2022-03-27T20:56:47Z

Advantage and disadvantage

← Older revision		Revision as of 20:56, 27 March 2022
Line 139:		Line 139:
	The merits and demerits of this method can be evaluated from the following aspects. <ref name="CSP"/> <ref name="MIT"/> Then we can generally understand why Wideband Delphi is currently more likely to be used in areas where the process paradigm is more fixed but the content is more complex, such as software development.		The merits and demerits of this method can be evaluated from the following aspects. <ref name="CSP"/> <ref name="MIT"/> Then we can generally understand why Wideband Delphi is currently more likely to be used in areas where the process paradigm is more fixed but the content is more complex, such as software development.
	=== Involvement of Experts ===		=== Involvement of Experts ===
	- Strong, but it's hard to achieve optimum. Like other group estimation methods, Wideband Delphi requires extensive participation from multiple experts, who should represent a variety of expertise and perspectives on the software being developed. The experts remain anonymous throughout the process, so they are free to express their opinions and criticisms, carefully coordinated by the moderator. It asks experts to give reasons for their estimates and then debate those reasons; The validity of the results and the expertise of the participants depends largely on who attended the workshop. If the topic is too specialized, the barrier to expert participation can be high. Valerdi holds a view that a major problem has also been the tendency for experts to over-simplify particular issues, and treat them as isolated events<ref name="MIT"/>. This is particularly the case in forecasting, where experts tend to think in terms of linear sequential events rather than applying a systems view that involves complex chains and associations. Design of survey instruments can help mitigate these effects. A similar question could be presented several times with slightly different wording to measure the consistency of ~~experts’opinions~~.		- Strong, but it's hard to achieve optimum. Like other group estimation methods, Wideband Delphi requires extensive participation from multiple experts, who should represent a variety of expertise and perspectives on the software being developed. The experts remain anonymous throughout the process, so they are free to express their opinions and criticisms, carefully coordinated by the moderator. It asks experts to give reasons for their estimates and then debate those reasons; The validity of the results and the expertise of the participants depends largely on who attended the workshop. If the topic is too specialized, the barrier to expert participation can be high. Valerdi holds a view that a major problem has also been the tendency for experts to over-simplify particular issues, and treat them as isolated events<ref name="MIT"/>. This is particularly the case in forecasting, where experts tend to think in terms of linear sequential events rather than applying a systems view that involves complex chains and associations. Design of survey instruments can help mitigate these effects. A similar question could be presented several times with slightly different wording to measure the consistency of experts’ opinions.

	=== Data Requirement===		=== Data Requirement===
	- Less information required and it's acceptable. The access method is simple but time-consuming. Wideband Delphi does not require any historical project data. But data can be difficult to quantify and time-consuming to obtain when large numbers of samples are needed.If the procedure is complex and has many levels, data reprocessing of expert evaluation results is required after each feedback.		- Less information required and it's acceptable. The access method is simple but time-consuming. Wideband Delphi does not require any historical project data. But data can be difficult to quantify and time-consuming to obtain when large numbers of samples are needed. If the procedure is complex and has many levels, data reprocessing of expert evaluation results is required after each feedback.

	=== Robustness ===		=== Robustness ===
Line 149:		Line 149:

	=== Flexibility ===		=== Flexibility ===
	- ~~Hign~~. Input from Wideband Delphi does not require any specific information and does not require any adjustments when applied to different situations and environments.		- High. Input from Wideband Delphi does not require any specific information and does not require any adjustments when applied to different situations and environments.

	=== Complexity ===		=== Complexity ===
	- It is brief and easy to understand. Wideband Delphi is intuitive and easy to use.The steps are clear and the participants are guided quickly under the framework.		- It is brief and easy to understand. Wideband Delphi is intuitive and easy to use. The steps are clear and the participants are guided quickly under the framework.

	=== Support level ===		=== Support level ===
Line 159:		Line 159:
	=== Reusability ===		=== Reusability ===
	- Poor.		- Poor.
	Wideband Delphi does not provide an estimation model that can be reused in similar projects. Each time an estimate is required, the ~~estimate~~ procedure must be completely reapplied.		Wideband Delphi does not provide an estimation model that can be reused in similar projects. Each time an estimate is required, the estimation procedure must be completely reapplied.

	=== Ability to predict ===		=== Ability to predict ===
	- Depends on the expert selection and question setting, but sometimes hard to meet expectations.		- Depends on the expert selection and question setting, but sometimes hard to meet expectations.
	Human experts can consider (but implicitly) anomalous and unique project characteristics and their interactions. For example, experts can handle differences between past project experience and the new technology, architecture, or application involved in a new project. However, the human experts involved in the estimation process can vary widely in their individual estimates. Although the group estimation process should compensate for the influence of individual experts, group estimation can still be unstable. At the same time, estimates based on human judgment can be biased by a number of factors, such as the personal expertise, experience and preferences of human experts, as well as the influence of other experts involved in the estimation process and irrelevant and misleading information. For example, human estimators tend to underestimate tasks that are considered easy and overestimate tasks that are considered difficult.		Human experts can consider (but implicitly) anomalous and unique project characteristics and their interactions. For example, experts can handle differences between the past project experience and the new technology, architecture, or application involved in a new project. However, the human experts involved in the estimation process can vary widely in their individual estimates. Although the group estimation process should compensate for the influence of individual experts, group estimation can still be unstable. At the same time, estimates based on human judgment can be biased by a number of factors, such as the personal expertise, experience and preferences of human experts, as well as the influence of other experts involved in the estimation process and irrelevant and misleading information. For example, human estimators tend to underestimate tasks that are considered easy and overestimate tasks that are considered difficult.

	=== Dealing with uncertainty ===		=== Dealing with uncertainty ===
	- Normal.		- Normal.
	Wideband Delphi can handle uncertain information (that is, the expertise of human experts). It can also provide information about the uncertainty of estimates~~, for~~ example, experts provide their estimates in the form of three values: minimum, maximum, and most likely.		Wideband Delphi can handle uncertain information (that is, the expertise of human experts). It can also provide information about the uncertainty of estimates. For example, experts provide their estimates in the form of three values: minimum, maximum, and most likely.
	However, this method does not provide any system technology to deal with uncertainty; This requires the use of additional analytical techniques, such as operations on simple probability distributions.		However, this method does not provide any system technology to deal with uncertainty; This requires the use of additional analytical techniques, such as operations on simple probability distributions.

	=== Range Estimation ===		=== Range Estimation ===
	- Limited and relatively small.		- Limited and relatively small.
	The applicability of Wideband Delphi to estimating different types of activities is limited by the expertise of experts in the relevant domain. In practice, estimation is limited to project activities in which the participating human estimators have ~~experiences~~.		The applicability of Wideband Delphi to estimating different types of activities is limited by the expertise of experts in the relevant domain. In practice, estimation is limited to project activities in which the participating human estimators have experience.

	=== Usability and Experience Acquisition ===		=== Usability and Experience Acquisition ===

S220027: /* Application */

2022-03-27T20:53:26Z

Application

← Older revision		Revision as of 20:53, 27 March 2022
Line 115:		Line 115:
	Planning Poker is a consensus-based estimation technique used to estimate the amount of work or relative size described by users in Scrum.<ref name="Poker" >Cohn, Mike (November 2005). "Agile Estimating and Planning". Mountain Goat Software. https://athena.ecs.csus.edu/~buckley/CSc131_files/Planning%20Poker.pdf </ref>		Planning Poker is a consensus-based estimation technique used to estimate the amount of work or relative size described by users in Scrum.<ref name="Poker" >Cohn, Mike (November 2005). "Agile Estimating and Planning". Mountain Goat Software. https://athena.ecs.csus.edu/~buckley/CSc131_files/Planning%20Poker.pdf </ref>
	Planning Poker combines three estimation techniques: Wideband Delphi technique, analog estimation, and WBS estimation.		Planning Poker combines three estimation techniques: Wideband Delphi technique, analog estimation, and WBS estimation.
	In planning poker estimation technique, the estimation of user stories is obtained by playing planning poker. The entire Scrum team is involved and it produces quick but reliable estimates. Like Wideband Delphi, it iterates to improve the accuracy of estimates， uses a non-anonymous meeting approach, also to estimate working hours.		In the planning poker estimation technique, the estimation of user stories is obtained by playing planning poker. The entire Scrum team is involved and it produces quick but reliable estimates. Like Wideband Delphi, it iterates to improve the accuracy of estimates， uses a non-anonymous meeting approach, and also to estimate working hours.

	Plan poker is played with a pack of cards. When using the Fibonacci sequence, the card has the numbers - 1, 2, 3, 5, 8, 13, 21, 34, etc. These numbers represent "story points." Each estimator has a deck of cards. When one team member holds up the card, the number on the card should be large enough for all team members to see.		Plan poker is played with a pack of cards. When using the Fibonacci sequence, the card has the numbers - 1, 2, 3, 5, 8, 13, 21, 34, etc. These numbers represent "story points." Each estimator has a deck of cards. When one team member holds up the card, the number on the card should be large enough for all team members to see.
	One of the team members was chosen as the host. The moderator reads the description of the user story for which it is being estimated. If the assessors have any questions, the product owner will answer them.<ref name="Poker2">Kjetil Moløkken-Østvold, Nils Christian Haugen, Hans Christian Benestad, Using planning poker for combining expert estimates in software projects,Journal of Systems and Software,Volume 81, Issue 12,2008,Pages 2106-2117,ISSN 0164-1212,https://doi.org/10.1016/j.jss.2008.03.058.</ref>		One of the team members was chosen as the host. The moderator reads the description of the user story for which it is being estimated. If the assessors have any questions, the product owner will answer them.<ref name="Poker2">Kjetil Moløkken-Østvold, Nils Christian Haugen, Hans Christian Benestad, Using planning poker for combining expert estimates in software projects,Journal of Systems and Software,Volume 81, Issue 12,2008,Pages 2106-2117,ISSN 0164-1212,https://doi.org/10.1016/j.jss.2008.03.058.</ref>

	Each estimator privately selects a card that represents his or her estimate. Cards are not displayed until all estimators have made their choices. At that point, all cards are flipped and held up at the same time so that all team members can see each estimate.		Each estimator privately selects a card that represents his or her estimate. Cards are not displayed until all the estimators have made their choices. At that point, all cards are flipped and held up at the same time so that all team members can see each estimate.
	Estimates are likely to change in the first round. The overestimators and underestimators explained the reasons for their estimates. It should be noted that all discussions are for understanding only and not personal. The moderator must ensure the same.		Estimates are likely to change in the first round. The overestimators and underestimators explained the reasons for their estimates. It should be noted that all discussions are for understanding only and not personal. The moderator must ensure the same.
	The team ~~can~~ discuss the story and their estimates for a few more minutes.		The team could discuss the story and their estimates for a few more minutes.
	Facilitators can take notes during discussions, which will help develop specific stories. After the discussion, each estimator reestimated by selecting a card again. The cards are kept secret again until everyone has estimated them and then handed in at the same time.		Facilitators can take notes during discussions, which will help develop specific stories. After the discussion, each estimator reestimated by selecting a card again. The cards are kept secret again until everyone has estimated them, and then handed in at the same time.
	This process is repeated until the estimate converges to one that can be used for the story. The estimated number of rounds may vary from user story to user story.		This process is repeated until the estimate converges to one that can be used for the story. The estimated number of rounds may vary from user story to user story.

	Planning poker now ~~becomes~~ a popular method of ~~for~~ relative estimation of complex empirical work, such as engineering or software development.<ref name="agile" >Agile Stationery. Why is Planning Poker a popular estimation technique in agile and scrum teams? https://agilestationery.com/pages/planning-poker</ref> It encourages team members to think independently by asking them to reveal their score simultaneously with their peers. This avoids the cognitive bias known as anchoring<ref name="Wiki">Anchoring (cognitive bias). https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)</ref> and exposes situations where team members hold different information or perspectives on the work.		Planning poker has now become a popular method of relative estimation of complex empirical work, such as engineering or software development.<ref name="agile" >Agile Stationery. Why is Planning Poker a popular estimation technique in agile and scrum teams? https://agilestationery.com/pages/planning-poker</ref> It encourages team members to think independently by asking them to reveal their score simultaneously with their peers. This avoids the cognitive bias known as anchoring<ref name="Wiki">Anchoring (cognitive bias). https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)</ref> and exposes situations where team members hold different information or perspectives on the work.
	Influential Lean Management theory categorises estimation as waste because it does not directly deliver value. Agile practitioners therefore try to reduce the time spent producing estimates, which have a poor history of inaccuracy anyway.		Influential Lean Management theory categorises estimation as waste because it does not directly deliver value. Agile practitioners therefore try to reduce the time spent producing estimates, which have a poor history of inaccuracy anyway.
	Typically, conversations during the game will be focused on the perspectives that differ the most. This means that details that will not ~~effect~~ the estimate are less likely to be discussed and the conversation is much more efficient.		Typically, conversations during the game will be focused on the perspectives that differ the most. This means that details that will not affect the estimate are less likely to be discussed and the conversation is much more efficient.
	Another reason this poker game process makes planning cheaper is that it takes relative complexity as the input, rather than a complex design document that may not be accurate. There is simply too much detail to be discovered down the road, and without an implementation in-progress much of that detail is not discoverable.		Another reason this poker game process makes planning cheaper is that it takes relative complexity as the input, rather than a complex design document that may not be accurate. There is simply too much detail to be discovered down the road, and without an implementation in-progress much of that detail is not discoverable.
	Relative complexity however, is a tractable question. Only a broad outline of the solution is needed to determine this, and that outline can be produced in a fraction of the time, dramatically reducing the effort needed to produce an estimate.		Relative complexity, however, is a tractable question. Only a broad outline of the solution is needed to determine this, and that outline can be produced in a fraction of the time, dramatically reducing the effort needed to produce an estimate.
	----		----
	Obviously, Wideband Delphi can be used in other areas as ~~well，but~~ there are few suitable cases to be found. An interesting case is that Indonesian scholars discussed the key requirements to be considered in the national cyber defense framework to prevent foreign surveillance after PRISM, and proposed effective mitigation control measures to safeguard and protect the national interests of the country. <ref name="PRIISM" >Y. Nugraha, I. Brown and A. S. Sastrosubroto, "An Adaptive Wideband Delphi Method to Study State Cyber-Defence Requirements," in IEEE Transactions on Emerging Topics in Computing, vol. 4, no. 1, pp. 47-59, Jan.-March 2016, doi: 10.1109/TETC.2015.2389661.</ref>There Wideband Delphi method was used to determine the analysis framework in conjunction with governance and legal remedies. They identified 25 mitigation control measures to address policy makers' priorities, which were divided into five deep defence elements.		Obviously, Wideband Delphi can be used in other areas as well， but there are few suitable cases to be found. An interesting case is that Indonesian scholars discussed the key requirements to be considered in the national cyber defense framework to prevent foreign surveillance after PRISM, and proposed effective mitigation control measures to safeguard and protect the national interests of the country. <ref name="PRIISM" >Y. Nugraha, I. Brown and A. S. Sastrosubroto, "An Adaptive Wideband Delphi Method to Study State Cyber-Defence Requirements," in IEEE Transactions on Emerging Topics in Computing, vol. 4, no. 1, pp. 47-59, Jan.-March 2016, doi: 10.1109/TETC.2015.2389661.</ref>There Wideband Delphi method was used to determine the analysis framework in conjunction with governance and legal remedies. They identified 25 mitigation control measures to address policy makers' priorities, which were divided into five deep defence elements.
	The difficulty in applying this method lies with the experts.In general, Wideband Delphi can be used in important situations where a large number of experts are involved and a full exchange of views is required.		The difficulty in applying this method lies with the experts. In general, Wideband Delphi can be used in important situations where a large number of experts are involved and a full exchange of views is required.
	Of course, the characteristic of expert participation will also become the limitation of its application. First, this approach is not appropriate when one (especially the moderator) wants to build relationships and continuous communication among the participants, or when one wants to make decisions directly. Secondly, if the internal consistency and reliability of expert judgment are poor, the actual meaning cannot be inferred from the results. Lack of reliability can also result from inconsistent understanding of concepts by experts.		Of course, the characteristic of expert participation will also become the limitation of its application. First, this approach is not appropriate when one (especially the moderator) wants to build relationships and continuous communication among the participants, or when one wants to make decisions directly. Secondly, if the internal consistency and reliability of expert judgment are poor, the actual meaning cannot be inferred from the results. Lack of reliability can also result from inconsistent understanding of concepts by experts.