Application of Quantitative Risk Management Practices to

Preparing to load PDF file. please wait...

0 of 0
100%
Application of Quantitative Risk Management Practices to

Transcript Of Application of Quantitative Risk Management Practices to

Preliminary Investigation
Caltrans Division of Research, Innovation and System Information
Application of Quantitative Risk Management Practices to Project Development: A Survey of State Practice and Related Research
Requested by James Monroe, Caltrans Division of Project Management
July 22, 2015
The Caltrans Division of Research, Innovation and System Information (DRISI) receives and evaluates numerous research problem statements for funding every year. DRISI conducts Preliminary Investigations on these problem statements to better scope and prioritize the proposed research in light of existing credible work on the topics nationally and internationally. Online and print sources for Preliminary Investigations include the National Cooperative Highway Research Program (NCHRP) and other Transportation Research Board (TRB) programs, the American Association of State Highway and Transportation Officials (AASHTO), the research and practices of other transportation agencies, and related academic and industry research. The views and conclusions in cited works, while generally peer reviewed or published by authoritative sources, may not be accepted without qualification by all experts in the field.
Table of Contents
Executive Summary ................................................................................................................. 2 Background .......................................................................................................................... 2 Summary of Findings ........................................................................................................... 2 Gaps in Findings .................................................................................................................. 5 Next Steps ........................................................................................................................... 5
Detailed Findings ..................................................................................................................... 6 Consultation with State DOTs .............................................................................................. 6 Other State DOT Quantitative Risk Management Practices.................................................25 Domestic Research and Resources ....................................................................................28 International Research and Resources ...............................................................................31
Contacts ..................................................................................................................................33

Executive Summary
Background
Currently, Caltrans’ Capital Outlay Support (COS) program employs quantitative risk management practices to contain costs associated with the construction of transportation projects. COS management is interested in learning about a different application of quantitative risk management practices, one that assesses risk earlier in the life cycle of a transportation project—before the project reaches the construction phase. These practices would assess and quantify the risks encountered during project development, when project teams produce plans and specifications, develop cost estimates, complete environmental evaluations and advertise to the construction community.
Caltrans would like to learn from the experiences of other transportation agencies applying this type of risk management. To assist Caltrans in this information-gathering effort, CTC & Associates examined domestic and international published and in-process research that addresses the application of quantitative risk management practices to transportation project development. To supplement the literature review, CTC contacted members of the AASHTO Standing Committee on Planning to inquire about the member agencies’ quantitative risk management practices during project development.
Summary of Findings
We sought information about the use of quantitative risk management practices applied during project development using a literature search and contacts made to representatives of state departments of transportation (DOTs) who indicated that these practices were employed by their agencies. We begin below with results of the state DOT queries.
Consultation with State DOTs
We sent an email query to members of the AASHTO Standing Committee on Planning seeking to identify state agencies applying quantitative risk management during the project development process. Responses to that query prompted us to follow up with representatives from six state DOTs—Minnesota, Nevada, New York, Utah, Virginia and Washington—to gather more information. A seventh state—Vermont—is developing quantitative risk management practices for use during project development but had nothing to share at this time.
The following describes some common themes identified in our conversations with these six agencies and a review of documents describing the agencies’ practices.
Background
Washington State DOT was an early adopter of quantitative risk management during project development. WSDOT began development of a formal risk assessment process—Cost Estimate Validation Process (CEVP)—in 2002 and initiated its use on a set of projects in 2003. Nevada DOT began conducting its version of quantitative risk assessment in the summer of 2008. Since 2009, New York State DOT has adopted quantitative risk management to a lesser degree, applying the practices on a handful of projects. Utah DOT first investigated the use of WSDOT’s CEVP model on a problematic project more than 10 years ago. Four years ago, UDOT revisited its risk management program, employing a full-time staff member to oversee the agency’s risk management activities. Minnesota DOT began applying quantitative risk management practices to a large program of projects in 2013. In Virginia, traditional and design-

Produced by CTC & Associates LLC

2

build projects are subject to qualitative assessment, while quantitative risk practices, in varying degrees, are applied to public-private partnership projects.
The Process
The quantitative risk management processes employed by five of the six states we spoke to— Minnesota, Nevada, New York, Utah and Washington—are similar. The process begins with a workshop to bring together a range of interested parties—the project manager and team, internal and external subject matter experts, cost estimate and construction experts, and other specialists as needed—to identify and quantify risks early in the project development process. These workshops can be led by consultants or by trained in-house staff. A quantitative tool is used to assess the risk data gathered during the workshop. All states conduct some form of follow-up after completion of the initial workshop and generation of risk-related documents to track risks throughout the phases of project development.
The Quantitative Tool
States have a range of qualitative and quantitative tools to address risk management during project development and most often use project cost as a general guideline in determining which tool to use. Quantitative risk management processes that employ a consultant’s probabilistic modeling tool are typically applied to large projects of $100 million or more or to projects of lower cost but significant complexity. Instead of using project cost to determine when to apply quantitative practices, MnDOT applies a set of complexity definitions.
Typically, these tools apply a Monte Carlo simulation technique to determine the impact of risks identified by workshop participants by running simulations that identify a range of possible outcomes for multiple scenarios. Some states maintain their own Excel-based quantitative tools that are developed in-house or by a consultant for use with lower-cost projects. These tools may also apply Monte Carlo simulation techniques. Tool output includes risk registers that quantify and rank risks, tornado diagrams, expected values and decision trees.
Risk Management Program Impacts
While some states note the challenges of quantifying the benefits of risk management, NDOT cites significant cost savings resulting from quantitative assessment of a major program of projects in Las Vegas. MnDOT has found that cost estimates developed with the aid of quantitative risk assessment are let close to or under the budget initially developed. For UDOT, the benefits of risk management also lie in the communication that occurs during the risk workshop, bringing people together to bridge gaps in the agency. WSDOT offers anecdotal evidence that quantitative risk management results in better cost estimates and the establishment of risk reserves, and better equips project managers to address issues that arise in the field.
Challenges
Below are some of the challenges identified by interviewees:
• Finding time to conduct the workshop and integrate findings into a procurement document can be challenging.
• Striking the right balance of quantifying and managing risk with the most effective use of limited resources can be difficult.
• Holding project managers accountable for costs over the life cycle of a project’s development can represent a culture change.

Produced by CTC & Associates LLC

3

• Earning staff confidence and demonstrating competence in the process can take time.
• Risk registers can become too cumbersome when they include 50 to 100 items. Not everything has to be assessed using a quantitative model.
• The ranking of risks generated by some software programs can be misleading, and the project team may fail to pay adequate attention to high-impact, low-probability risks.
Recommendations for Success
Interviewees offered these and other recommendations to agencies implementing a quantitative risk management program during project development:
• Ensure executive and management support.
• Be patient and start small.
• Employ an effective facilitator to lead the risk workshops.
• Avoid shortchanging the philosophical assessment of risk management in favor of a sole focus on tools.
Federal Highway Administration Analysis Required for “Major Projects”
Our discussion with NDOT brought to light a legislatively mandated requirement for Federal Highway Administration (FHWA) to examine financial plans submitted by state DOTs undertaking a “Major Project” as defined by the rule (projects with a total cost exceeding $500 million or with high visibility). FHWA applies a risk-based probabilistic assessment, similar to the assessments conducted by NDOT and other states interviewed for this Preliminary Investigation, to ensure that the state-provided cost estimates are “reasonable and supportable.”
Other State DOT Quantitative Risk Management Practices
To supplement the information obtained through interviews, we gathered information from agency web sites and other sources about the quantitative risk management practices used by other state DOTs during project development. We highlight the practices of three state DOTs:
• Florida DOT. Levels of risk analysis include a consultant-led independent risk analysis workshop for complex projects or projects with total costs greater than $500 million.
• Montana DOT. Project cost determines the level of risk analysis during estimating and design project management. Rarely does the agency expect to apply the Cost Risk Assessment workshop specified for complex or major projects.
• Ohio DOT. The agency’s new cost estimate review process is similar to FHWA’s examination of cost estimates associated with FHWA-classified Major Projects.
Domestic Research and Resources
In this section we provide a sampling of publications from FHWA, NCHRP and Strategic Highway Research Program 2 (SHRP 2) that offer guidance on quantitative risk analyses. We also provide additional information about the quantitative analysis FHWA conducts on projects classified as Major Projects, and details of an NCHRP project in process, scheduled to conclude in October 2015, that is expected to identify tools and techniques state DOTs can use to manage risk across the agency.

Produced by CTC & Associates LLC

4

International Research and Resources
In this section we highlight conference papers that describe a new quantitative model used in connection with Australian road projects, and a standard for cost estimation for projects that require funding from the Australian government. Also included in this section are an international scan report that identifies structures for successful risk management and a practice standard for project risk management issued by an international project management organization. The final citation in this section revisits the CEVP model developed for WSDOT in a discussion of how the model has been applied to large transportation projects in Canada.
Gaps in Findings
The summaries of consultations with six state DOTs are not intended to represent an exhaustive review of the activities among state DOTs in the area of quantitative risk management. It also appears that agency practices are evolving, with these summaries representing a snapshot in time.
We received a relatively low level of response to our initial inquiry, and other state DOTs not identified in this report may support robust risk management programs that conduct a quantitative assessment of risk during the phases of project development.
Next Steps
Moving forward, Caltrans could consider:
• Contacting any of the states identified in this Preliminary Investigation as using quantitative risk management practices during project development to gather more information about specific areas of interest to Caltrans. These contacts might address:
o The staffing needs and training requirements to support quantitative risk assessment practices during project development (Florida, Minnesota, Nevada, New York, Utah and Washington).
o The role of workshops in gathering the information needed for risk analysis (Florida, Minnesota, Nevada, Ohio, Utah and Washington).
o The development of agency tools (NDOT’s Cost Estimating Wizard, UDOT’s Excel-based risk model for Monte Carlo analysis and WSDOT’s risk-based estimate self-modeling tool).
o The role played by consultants in the risk management process (Florida, Minnesota, Nevada, Ohio, Utah and Washington).
• Contacting Vermont Agency of Transportation to learn more about its ongoing activities to initiate a quantitative risk management program that examines risks during the phases of project development.
• Consulting with NYSDOT later this year to inquire about the impact of the relatively new Office of Project Management in formalizing risk management practices across the agency.
• Contacting MnDOT in early 2016 to learn more about the guidance documents now in development that will provide further direction on what is needed to conduct a quantitative risk assessment and when it is most appropriate to conduct this type of analysis.

Produced by CTC & Associates LLC

5

Detailed Findings
Consultation with State DOTs
We distributed an email to members of the AASHTO Standing Committee on Planning to seek information from agencies applying quantitative risk management practices to the project development process. We received responses from 11 state DOTs that can be categorized as follows:
• Not currently applying quantitative risk management practices during project development—Arkansas, Connecticut, Missouri and North Dakota.
• Currently developing quantitative risk management practices for use during project development—Vermont.
• Using a risk evaluation matrix—Virginia. • Currently applying quantitative risk management practices during project development—
Minnesota, Nevada, New York, Utah and Washington.
Below we summarize our conversations with representatives from six state DOTs—Minnesota, Nevada, New York, Utah, Virginia and Washington—about the agencies’ use of quantitative risk management practices during project development.
The summaries below are organized in the following topic areas: • Background. • The Process. • The Quantitative Tool. • Implementing the Process. • Risk Management Program Impacts. • Challenges. • Recommendations for Success. • What’s Next. • Related Resources.
Not all summaries include all topic areas.
Minnesota
Contact: Christopher Roy, State Design Engineer, Minnesota Department of Transportation, 651-366-3182, [email protected]
Background Christopher Roy notes that MnDOT has long been conducting some form of risk management, but the agency’s practices have evolved over time. During the period 2007 to 2009, a cost estimating and cost management initiative within the agency led MnDOT to consider a more

Produced by CTC & Associates LLC

6

comprehensive approach to risk management; see Related Resources for a technical manual that describes the agency’s risk management processes.
While MnDOT had demonstrated its proficiency in estimating costs, it was determined that more work could be done on controlling the scope of a project and measuring risk, which pointed to a new focus on risk management. As Roy indicates, the agency was expending significant efforts on developing a risk register—with a project team spending a day or more developing the register—while less time was typically devoted to monetizing risk, examining the basis for the probabilities and costs associated with each item in the register.
The Corridors of Commerce program, created by the Minnesota Legislature in 2013 to fund the construction, reconstruction and improvement of trunk highways, presented an opportunity for MnDOT to apply its quantitative risk management practices to a set of 11 projects.
The Process
All MnDOT projects, regardless of project size and complexity, require some form of qualitative or quantitative risk assessment. Unlike other agencies, MnDOT does not use project cost to determine which projects will be subject to a quantitative risk assessment. Instead, the agency applies a set of complexity definitions developed by Pennsylvania DOT that appear in NCHRP Report 574: Guidance for Cost Estimation and Management for Highway Projects During Planning, Programming, and Preconstruction (see page 29 of this Preliminary Investigation for a citation for this publication). Projects identified as “major projects” under this criteria are subject to a quantitative risk analysis, referred to in MnDOT documents as a Type III risk analysis. Complex smaller projects may also be subject to more than simply a qualitative assessment using a risk register.
A Type III risk analysis for a major project requires a consultant-led risk analysis workshop to identify project risks. The analysis begins with the project’s schedule and cost estimate. Workshop participants provide a “three-point estimate” that reflects the optimistic (low), most likely and pessimistic (high) values for the activity or cost element. A quantitative Monte Carlo simulation conducts a simultaneous evaluation of the impact of all risks identified and quantified by the workshop participants. For other projects not deemed to require a consultant-led workshop, MnDOT uses an internal Monte Carlo tool to quantify project risks.
The Quantitative Tool
MnDOT’s consultant-led risk workshops have employed the @RISK tool by Palisade, which is used in conjunction with Excel to apply Monte Carlo simulations by using the project team’s riskrelated qualitative assessments to calculate and track multiple scenarios, and identify the probabilities and risks associated with alternatives. Risk workshop products include a risk register and risk management plan. Risk management plans that result from risk workshops are updated frequently as circumstances require.
For medium-size and larger projects not subject to a risk workshop, MnDOT uses Deltek Acumen Risk. This Monte Carlo risk analysis tool produces a risk register that allows the agency to examine alternatives to identify the most effective risk response plans for both cost and schedule. MnDOT first used this tool to consider schedule risks during construction. The tool can be used at the project level or on specific project segments.

Produced by CTC & Associates LLC

7

Risk Management Program Impacts
At various times, MnDOT has received an influx of unexpected funding from sources such as the state-funded Corridors of Commerce program and the American Recovery and Reinvestment Act of 2009. Roy notes that the large programs of projects developed to apply these funds can present challenges to planners. In some cases, not every project identified in the initial stages of programming can be built as proposed when cost estimates may be subject to significant fluctuation. With the application of quantitative risk management practices to its first five projects for the Corridors of Commerce program, Roy notes that only one of the five projects was let over its estimated cost, while the other four projects were let 20 percent under the initial cost estimate. Roy notes that, ideally, projects will be let on target with the initial estimate, and not above or below.
Challenges
Moving to a formalized quantitative risk assessment program can represent a culture shift for project managers who will be held accountable for the budget on a project that could be four years away from letting. However, Roy notes that the quantitative tools allow project managers to make informed decisions on how to hit a project cost target as the project life cycle unfolds.
What’s Next
MnDOT is developing white papers that will provide additional guidance on what is needed in connection with quantitative risk management and when it is most appropriate to employ the quantitative practices for MnDOT projects. Roy expects these guidance documents to be available by early 2016.
Related Resources
Cost Estimation and Cost Management, Technical Reference Manual, Minnesota Department of Transportation, 2008. http://dotapp7.dot.state.mn.us/edms/download?docId=670233 The manual highlights the following issues to consider:
• Project complexity. Project complexity is the key driver for determining the type of risk analysis. Project size is not necessarily a determinant of project complexity. Small projects can be complex and require a more rigorous analysis.
• Use of consultants. Employ external consultants for Type III risk analyses. Retain the consultant who conducts the initial risk analysis for updates, whether periodic or as required by project circumstances.
• Resulting risk management plan. Develop risk management plans and update them frequently. The level of detail in the risk analysis plan corresponds to the level of risk management.
The following sections of the report provide more in-depth discussion of issues related to quantitative risk practices:
• Risk register (see page 428 of the report; page 434 of PDF).
• Estimate ranges—Monte Carlo analysis (see page 437 of the report; page 443 of the PDF).
• Risk workshops (see page 440 of the report; page 446 of the PDF).

Produced by CTC & Associates LLC

8

The Project Risk Management Process, Minnesota Department of Transportation, undated. http://www.dot.state.mn.us/pm/documents/guidance/The%20Project%20Risk%20Managem ent%20Process%20SHORT%20(2).docx This draft document describes MnDOT’s risk management process and provides links to other risk management topics, including quantitative risk management.
Quantitative Risk Analysis—Level 3, Minnesota Department of Transportation, undated. http://www.dot.state.mn.us/pm/documents/guidance/quantitative-risk-analysis.docx This document describes the quantitative tool used to conduct the risk analysis and how a risk register is used to enter data that will be used in the quantitative analysis.
@RISK6, Palisade Corporation, 2015. http://www.palisade.com/risk/ From the web site:
@RISK (pronounced “at risk”) performs risk analysis using Monte Carlo simulation to show you many possible outcomes in your spreadsheet model—and tells you how likely they are to occur. It mathematically and objectively computes and tracks many different possible future scenarios, then tells you the probabilities and risks associated with each different one. This means you can judge which risks to take and which ones to avoid, allowing for the best decision making under uncertainty.
@RISK also helps you plan the best risk management strategies through the integration of RISKOptimizer, which combines Monte Carlo simulation with the latest solving technology to optimize any spreadsheet with uncertain values. Using genetic algorithms or OptQuest, along with @RISK functions, RISKOptimizer can determine the best allocation of resources, the optimal asset allocation, the most efficient schedule, and much more.
Deltek Acumen Risk, Deltek, Inc., 2015. http://www.deltek.com/products/ppm/risk/acumen-risk From the web site:
Acumen Risk is a Monte Carlo risk analysis tool combining true cost and schedule risk analysis against a native project plan together with identified risk events from a project risk register.

Produced by CTC & Associates LLC

9

Nevada
Contact: Amir M. Soltani, Project Management Chief, Nevada Department of Transportation, 775-888-7321, [email protected]
Background In the summer of 2008, NDOT began conducting Cost Risk Assessments (CRAs) to quantify the risks associated with major capacity projects (over $100 million) and innovative delivery projects such as design-build.
Today, most of the projects NDOT subjects to a comprehensive quantitative risk analysis are large projects. Amir Soltani estimates that the agency conducts about eight CRAs a year. The agency conducts more limited risk assessments on specific elements of a project that have been deemed to present the potential for significant risk. For example, the agency has completed a CRA on the contractual aspects of certain design-build projects.
NDOT limits its application of quantitative risk management to major capacity projects. The NDOT Cost Estimating Wizard, a cost-estimating tool used for lower-cost projects since 2007, is addressed later in this summary.
The Process A facilitated CRA workshop is required on projects with total project costs over $100 million and highly suggested for projects between $10 million and $100 million. The workshop facilitator is trained in risk assessment and analysis, and in the application of a probabilistic tool that analyzes risks with potential impact on the project’s scope, budget and schedule. Participants in an NDOT CRA include the consultant representatives leading the session, members of the project and cost-risk teams, and internal and external subject matter experts.
The Quantitative Tool The risk management tool used in NDOT’s CRAs was developed by HDR, Inc., an international engineering firm. (Soltani notes that the HDR tool is similar to the tool used in CEVP to support WSDOT’s risk management program; see page 20 of this Preliminary Investigation for information about the WSDOT process.) The tool generates:
• Probabilistic modeling. The user enters the probability that each risk will occur, along with estimates of cost and schedule impact (low, high and most likely). The model evaluates up to 20 risks.
• Risk registers. These simple spreadsheets allow the agency to track the risks identified in the CRA.
The CRA reports included in Related Resources provide examples of the quantitative tool’s output.
NDOT has an internal version of this tool, developed in 2008, that does not include updates or enhancements reflected in the HDR tool currently used in CRA workshops. NDOT often uses its internal tool to more closely manage specific elements of a project, such as right of way.

Produced by CTC & Associates LLC

10
ProjectsRisk Management PracticesRisk ManagementProject DevelopmentRisk