We continue what has become a week focused on the conversations which happen around a simulation with a discussion of how to collect multi-point estimates.
It doesn't take much searching to turn up 100's of blog posts, presentations and journal articles on the best way to create three-point estimates. However, many times when creating a simulation, asking experts is the only option available.
Uncertainty vs. Risk
"If our partner doesn't come through, which is the worst case, it will cost us $1.5 million."
The range of the estimate should be about the uncertainty of a situation, not the individual risks. Particularly with worst-case scenarios, people have a tendency to start lumping in "what-if" cases. As the manager, don't dismiss those concerns, but capture them as risks rather than including in the range of the estimate.
Remember - there can be risks that improve the outcome - be sure to capture those as well.
"Best case, we can get it done in an hour, worst case, 10 years."
Particularly for people new to estimating, there is a tendency to go crazy with the extremes. On the flip side, people can appear overly certain about a particular outcome. Both cases require you to help guide the conversation - some key questions can help -
- Can you imagine how this could cost twice/half what you think?
- What are the conditions that could lead to the extreme scenario you're describing?
- What additional information could help reduce your uncertainty?
The worst thing a manager can do is collect estimates then turn around and commit a team to hitting the best-case. A lot of times, the unwillingness of team members to enthusiastically engage with the estimation process stems from being burned in the past.
To get good estimates, you need to prove you'll use them appropriately, making the experts partners in the estimation process.
We've only scratched the surface here, providing a few ideas to keep in mind. Estimating is a skill and as a manager it is important to help experts through the process and help build their estimation ability.
Photo by Victor rawpixel on Unsplash
While the native tools in DartCannon provide a powerful view into your simulations, if you want to perform your own analysis or create a custom chart, you'll want to use the Excel Export feature.
Excel Import/Export is a pro feature which requires a paid subscription. Sign Up Now!
Exporting an Excel file is as easy as clicking 'Export Excel' from the Simulation menu.
The resulting XLSX will contain the following worksheets:
- Items (for budgets) or Tasks (for plans)
- Simulation Results
- Result Ranges & Probabilities
- Critical Paths (only for plans)
Creating a Custom Chart
Lets use this to create a version of the normal results chart, only with a different color scheme. We'll use the 'Holiday Presents' simulation created in the Getting Started guide.
First, we'll go to the 'Simulation Results' worksheet which contains the 'raw' output of the simulation. There are three columns here -
- Raw Value- this is the value
- Simulation Count - the histogram, or number of times this particular value was simulated
- Cumulative Percent - the running percentage.
There is far more resolution in this export than displayed online where data is further binned depending on the display. Here, the data is at the maximum resolution.
This makes it easy to create a new chart. We select our three columns, then go to 'Insert' and select a 'Combo Chart' which will put our results on separate axes.
Before we look at the chart, we need to dive in and remove the first row from the datasets - go to 'Select Data' under 'Design' in 'Chart Tools':
Then edit each of the series to set the label to the second cell and the range starting at row 3:
The default chart isn't bad, but we can clean it up a lot:
While not an Excel tutorial, there is one tip that can be hard to find. If you want your bars to fill the space completely without gaps, you want to change the following settings:
- Set 'Series Overlap' to '100%'
- Set 'Gap Width' to '0%'
By making that change, tweaking colors, adding labels and a callout for the most likely outcome, we end up with the following:
Not too shabby and far more personalized than the default chart.
We've only scratched the surface of what is possible once you have the raw data in hand - hopefully it inspires you.
It is important to note that the export is "static" - changing the estimates in the excel will not change the results. Because the output is based on the last simulation run on DartCannon, it is suggested you increase the number of runs to the maximum before exporting for best results.
For more information on the Excel Import/Export feature, check out the help page in our documentation.
Every week we collect articles from around the web related to probability and estimating in a business context.
Happy Friday! Here's a rundown of this week in DartCannon
What We Wrote
- Accepting an Estimate - DartCannon is about creating simulations of complex business problems, but once the simulation is done the work of getting buy in starts. We shared some pointers of how to approach those conversations and potential pitfalls to look out for.
- The Curse of Parallel Tasks - It is so easy for our intuition to lead us astray as problems get complex. Here we show how you can be fooled even in the simple case of tasks with equal estimates performed in parallel.
What We've Been Reading
- If you only know one thing about Risk Management... (J. Goodpasture, Musings on project management) John is always good at bringing complex ideas back to fundamentals. Evidentially coupling of projects is on everyone's mind, as we wrote about parallel tasks, John wrote about the importance of schedule slack where it is unavoidable.
- Real-world Monte Carlo analysis (Cook, M. S., Paper presented at Project Management Institute Annual Seminars & Symposium, Nashville, TN) It likely comes as no surprise that we're slightly geeky around here, so no one should be shocked that we read this conference paper from 2001. Great if you're looking for a more rigorous review of the application of monte-carlo analysis, the simulation method underlying DartCannon.
DartCannon Change Spotlight
We are constantly continuing to improve DartCannon and while the majority of changes are too small to call out, we want to highlight larger improvements.
Excel Import Enhancements
In addition to overall performance improvements to XLSX import and export, we've added some smarts to the import process. Now, DartCannon will automatically attempt to identify which sheet contains your items and risks AND try to identify what each column represents.
Excel import/export is a Pro feature, so you need to subscribe to take advantage.
An end-of-week quote:
A goal without a plan is just a wish - Antoine de Saint-Exupéry
Hope everyone has a great weekend!
You've gone through the work of collecting credible estimates, building and finally running your simulation. You've got a range of possible outcomes, now what?
The Goal: Acceptance
At this point, you want to make sure that stakeholders accept the project or budget as simulated and everyone is aligned to the possible outcomes. It may mean setting a target, either a dollar figure for a budget or a date for a project, but that isn't necessary.
This means having some important conversations about Risk and Assumptions.
Talking About Risk
Generally, it is impossible or impractical to aim for 100% chance of success - the levels for the worst case in all but the simplest of simulations far exceeds what most organizations would accept. Not planning for the worst case means tolerating some amount of risk.
Generally there are a variety of approaches to dealing with risk - transferring (or insuring), mitigating (reducing or eliminating the effect), avoiding (reducing the likelihood) or accepting (hoping it doesn't happen). With simulations, there is often another tool - additional research.
We've written before about the difference types of risk - knowable (epistemic) and unknowable (aleatoric). If some of the uncertainty comes from a lack of knowledge, you could decide to take the time or spend the money to reduce uncertainty, reducing the 'knowable' risk.
Even once all levers have been pulled, you're sure to end up accepting a certain amount of risk - whether that is 5% or 50% is up to your organization's risk tolerance. It is important that, as a manager, you don't unilaterally take on that risk.
Conversations About Assumptions
A nice simulation gives the impression that everything is known, but of course there are always assumptions that should be evaluated:
- Good Estimates - If the individual item estimates are bad, the end result, no matter how pretty, won't be accurate.
- Independence - An underlying assumption of DartCannon is that all items are independent, while in many situations all items will trend high or low.
- Completeness - Are any tasks or line items missing? Are there unknowns that could turn bad?
While the goal is to get alignment around risk tolerance, there are a number of directions that the conversation can go off the rails.
- Focus on extremes - The extremes are typically so infinitesimally likely, that spending time on the worst and best cases is unhelpful.
- Discuss individual estimates - Trying to tweak parameters to get a desirable outcome is unlikely to change the overall picture and is second-guessing the work done creating the simulation.
- Placing burden of risk on the manager - There is a lot to unpack here and we're going to write more on this in the future, but DartCannon simulations are about uncertainty which, without resources, cannot be influenced.
Having productive conversations about risk tolerance is largely why DartCannon exists - don't let your work end when your simulation is complete.
Photo by Jakob Owens on Unsplash
Without thinking too hard, what is the most likely overall duration of 3 tasks, performed in parallel, each with an estimated duration from 1 to 3 days with a most likely time of 2 days?
I'll share the answer in a moment, but this is easily simulated in DartCannon. Starting with an empty new project, we'll quickly add 3 tasks:
And look at the result:
There is less than a 12.8% chance of being less than 2 days.
Why This Happens
When tasks are performed in parallel, the project is always waiting on the longest running task to complete. The more tasks running in parallel, the more likely that at least one of them is running long.
Perhaps you knew that parallel tasks add risk, but I'd bet that you probably wouldn't have guessed just how extreme the effect would be.
For this simple case, it is relatively easy to make an exact calculation of the
likelihood the entire set finishes in under the half-way point. If each task
has a 50% chance, the chance for 3 is
.5 x .5 x .5 or 12.5% - exactly what
our simulation showed. Generalizing this for a few tasks gives the following:
|Number||Chance < 50%|
This is fairly extreme and highlights the problem with relying on intuition. In this trivial example we can easily do the math to see the problem and with some experience, would know what to look for. The real problem comes in more complicated projects where risks may be hidden in a more complex network - these are the times where simulations are essential to understand potential outcomes and risks.
Tips & Takeaways
- Always simulate your projects - Don't just rely on intuition - it is too easy to overlook a gotchya or corner case that adds a lot of risk.
- Avoid choke points where many tasks converge - Funnelling work into points that depend on everything that comes before may seem "clean", but it actually results in increased risk where you have to wait on the longest task.
- Parallelizing is still a timesaver - If you can run multiple tasks in parallel, you still should! Just be aware of how it will really effect the scehdule.
It is a common misconception that adding more tasks is 'free' from a schedule perspective as long as they are perfomed in parallel. What we've shown here directly is that the benefit may not be a great as you'd think depending on how the project is structured.
Photo by Victor Sánchez Berruezo on Unsplash