A Framework for Systems Engineering Development of Complex Systems

In developing complex systems, evaluating potential schedule and cost risks is essential. With the Incremental Commitment Model (ICM), schedule tasks can be evaluated against manufacturing and technology risk drivers. In this article, these risk drivers are analyzed using a comprehensive approach with emphasis being placed on quantitative risk analysis through Monte Carlo simulation. Through modeling the behavior of a hypothetical project schedule for a notional spacecraft system, the authors show how the ICM framework is implemented in complex system development. The result is a repeatable, inherent, risk-driven commitment process that can stabilize and synchronize systems engineering and acquisition processes.

The level of complexity needed to develop spacecraft systems and other emerging technologies require programs to develop risk management and risk planning techniques that can potentially identify schedule and cost risks as early as possible during the acquisition life cycle. According to the Government Accountability Office (GAO), studies have shown that there has been an increase in schedule and cost overruns that involve complex systems and emerging technologies. This is often the case when projects exceed scheduled activity durations, resulting in frequent budget overruns. There are a plethora of risks that factor into inaccurate schedule estimates, including the elusive emerging requirements to the lack of process understanding.

Unfortunately, it is common to observe how requirements established during the earlier stages of an acquisition life cycle are changed to accommodate customer requests, thus impacting schedule and delivery costs. These impacts invariably affect scheduled activities from the design through the development and production of the complex system. To improve implementation and the understanding of the life-cycle processes for the complex system, it is essential that the development of a work breakdown structure (WBS), or an architecture and its interfaces within the appropriate hierarchical levels of decomposition, be accurately structured.

The structure of this architecture coincides with the development and implementation of activities that are required for the design, development, and production stages of the life cycle. As a result, the developed schedule activities influence the cost of delivery. Since scheduled activities impact the cost to develop complex systems, it can be shown that there is an inherent relationship between the complex systems architecture and the process activities required for schedule and cost estimating. As a result, project schedule and cost estimation play an important role in driving key acquisition life-cycle decisions.

Developing and delivering complex systems requires the management of complex risks such as uncertainty usage, schedule uncertainties, uncertainties associated with technology maturity, manufacturing maturity, technical design, and technical complexity [1]. Acquisition life-cycle decisions can be potentially flawed if the systems engineering development model isn’t appropriately matched to the complex system being developed. To address the challenge of selecting a candidate systems engineering development model, the Committee on Human-System Design Support for Changing Technology recommended implementation of the ICM as a reasonably robust framework for the “progressive reduction of risk through the full life cycle of system development, to produce a cost-effective system that meets the needs of all the stakeholders” [1]. The ICM integrates key strengths or attributes of other models into an integrated framework while introducing risk decision anchor points throughout the life cycle [2]. To complement implementation of this model, our research provides a hypothetical example that incorporates maturity risk drivers—technology readiness levels (TRL) and manufacturing readiness levels (MRL)—within a notional ICM framework as an approach to assess schedule and cost risks during the development of a complex system.

The method chosen to evaluate schedule and cost risk drivers for this research is known as Monte Carlo simulation. This method is used to model probabilistic behaviors of activities throughout the acquisition life cycle. The term Monte Carlo has been used interchangeably with probabilistic simulation since it is a technique used to randomly select numbers from a probability distribution or sampling. For our work, Monte Carlo simulation modeled the behaviors of schedule and cost uncertainties while providing traceability and consequence between the risk drivers and activities within the acquisition life cycle. Specifically, the simulation modeled the behaviors of tasks and activities derived from the spacecraft systems (the WBS) after mapping various risk drivers to those tasks [3]. This allows for exploration of the likelihood and consequences that these risk drivers have scheduled task activities throughout the acquisition life cycle.

The ICM Framework

The ICM was developed to ensure the flexibility of implementing one or more frameworks throughout each stage of an acquisition life cycle. The model was built upon five key principles that are critical for system development: 1) customer satisfaction; 2) incremental growth of system definition and stakeholder commitment; 3) iterative system definition and development; 4) concurrent system definition and development; and 5) management and project risk [2]. These principles are also proven strengths of other models such as the Waterfall, iterative, Rational Unified Process (RUP), and spiral development frameworks.

The ICM is unique in that it merges these key principles into one framework [2] to provide a process model that is robust with a central focus to progressively reduce potential risks throughout the entire life cycle, culminating in the development of complex systems that are both schedule- and cost-effective. The ICM framework also provides the decision-maker with the flexibility to recognize potential risks that coincide with system’s maturity and complexity of scope. To accommodate the ensuing consequences of potential risk drivers, the ICM framework implements a risk management principle that associates risk-driven tasks and activities for each stage of the acquisition life cycle.

The construct of the ICM framework is comprised of two major stages. Stage I, the Incremental Definition, entails the initial design stages of the system where the conceptual definition and feasibility studies are conducted for a better understanding of the system and stakeholder commitment. Stage II, Incremental Development and Operations, is where the increments of operational capability are developed and integrated into schedules that correlate to the development and evolution of the complex system. The activities within each stage are risk-driven to account for process agility and rigor to ensure that the system objectives are met throughout the systems development life cycle [2]. The concepts that the ICM framework is built upon include:

 

• Early verification and validation concepts of the V-model.
• Concurrency concepts of the concurrent engineering model.
• Concepts from Agile and Lean models.
• Risk-driven concepts of the spiral model.
• Phases and anchor points of the RUP.
• Systems of systems acquisition concepts of the spiral model.

Synergistic structuring of one or more process models within the ICM framework provides the tailoring flexibility to accommodate the varying maturity characteristics of any complex system; Barry Boehm and Jo Ann Lane provide a more detailed discussion regarding the ICM framework in [2]. An illustration of an integrated DoD/ICM life-cycle framework is provided in Figure 1. This view aligns milestones A, B, and C, representing the designated commitment point of key stages. A more detailed discussion of anchor points can be found in both [1] and [4].

Figure 1: ICM and DoD Milestone Traceability
(Click on image above to show full-size version in pop-up window.)

Our work focused on how the ICM framework is implemented in the development of a complex system by modeling the behavior of a hypothetical project schedule of a notional spacecraft system [3]. The modeling technique, Monte Carlo simulation, will be used to help the decision-maker evaluate schedule durations and cost estimates that are impacted by risk drivers. It will also aid the decision-maker with establishing preliminary risk management assessments.

Spacecraft System WBS

The WBS shown in Figure 2 illustrates how the notional spacecraft system is defined, developed, and maintained throughout the acquisition life cycle [4]. The hierarchical breakdown of the spacecraft is used to understand the products contained within each level of decomposition. The hierarchical levels that comprise the products provide the work structure necessary to develop tasks and activities throughout the acquisition life cycle. The products include hardware, software, documents, and processes. However, the WBS in this case is only focused on the hardware of the spacecraft at a third level of decomposition. Each level of decomposition was used to identify all of the subsystems and components that influence the preceding level and are aggregated to the top of the hierarchy through functional relationships.

Figure 2: WBS of Notional Spacecraft System
(Click on image above to show full-size version in pop-up window.)

Spacecraft System Project Schedule

The spacecraft system’s project schedule was used to provide key activity dates and durations that are associated with the products of the WBS. The WBS sets the foundation of all scheduled activities, thus impacting cost estimates. The duration of the scheduled activities was driven by many factors such as the technical complexity of work to be performed, manufacturing availability of components to be developed, and the technical maturity of components to be assembled. These factors were risk drivers that had an effect on the uncertainties of the project schedule.

The project schedule illustrates two key elements: the influence that risk drivers have on schedule tasks, and the influence that tasks and activities have on each other through precedence relationships. For example, let’s say that Task 1 must end before Task 2 can begin or Task 3 cannot begin until Tasks 1 and 2 have ended, respectively [5]. Because of the shared interrelationships of the schedule’s tasks, it is inevitable that any overrun in scheduled activities will most likely impact the duration of other tasks and activities of the project schedule—thus increasing the likelihood of cost overrun. Table 1 illustrates a sample project schedule for the development of the spacecraft system implementing the ICM framework.

Table 1: Sample Spacecraft Project Schedule With Notional ICM Framework
(Click on image above to show full-size version in pop-up window.)

Risk Management

To evaluate the proposed ICM framework effectively, it is important to organize a risk management approach that ensures the identification and quantification of risks and uncertainties that may impact a project’s schedule and cost [6]. Because of the increasing complexity of the development of spacecraft systems, it is likely that a project’s schedule tasks are ultimately interrelated and associated with cost [7]. In the context of system development programs, schedule and cost risks may determine whether or not the program will complete the systems development on schedule and on budget. If the program successfully meets or exceeds the schedule and budget expectations of the customer, it will likely be due to the effective implementation of risk management processes [8].

The ICM model implements risk management anchor points throughout each stage of the life cycle in order to improve the possibility of success for the development of the complex system [2]. The project schedule of the notional spacecraft system illustrated similar risk anchor points and was modeled with the appropriate stage tasks and activities against associated risks. Cost and schedule risks co-exist because of inherent uncertainties regarding the time and costs required to complete tasks of a project’s schedule [9]. To understand the uncertainties of schedule and cost risks, a risk mitigation strategy must be implemented to minimize the impact of these risks.

The risk mitigation strategy to be implemented with our research includes the following steps:

 

1. Risk Identification. Evaluation of a risk’s probability of occurrence and the impacts or consequence of risks against schedule and costs.
2. Risk Assessment. Quantification of the information acquired from risk identification to assess project schedule, cost, and technical risks.
3. Risk Analysis. Quantification of risk data in terms of probability of occurrence and the eventual consequence(s) if a risk does occur.
4. Risk Mitigation. Determination of actions to be implemented to reduce schedule and cost risks.

These steps are supported by empirical data that show how cost estimates are often linked to activity durations via schedule risk results. This is often seen when schedule risk analysis results are used as input to cost risk analysis and is primarily implemented to identify the uncertainty in activity durations in order to assess cost risks [3].

Risk Identification

The risk identification process begins by evaluating key risks (independent variables) and their respective uncertain impacts, throughout the acquisition life cycle, on the project’s schedule and cost variables. The risk identification process is implemented by a team of experts who evaluate project tasks and activities against the categorized risks that have varying probabilities of occurrences within each stage of the life cycle. It should also be noted that the risk identification process can be implemented with empirical data for the complex system being developed.

Our research identified maturity risks (TRL and MRL) as technical risks to be evaluated against project events. A brief description for TRL and MRL ratings are provided in Table 2 [10]. The risks were evaluated independently against tasks and activities that would impact subsystems and components of the spacecraft’s WBS. This approach allows a quick and simple explanation of how these risk drivers impact the final schedule and cost results of the project schedule.

Table 2: TRL and MRL Rating Descriptions and Relationships

The maturity risk drivers (TRL and MRL) are independent variables that carry a correlating metric weighting system that establishes the readiness to implement based upon factors such as the stability, technical complexity, and maturity of a spacecraft’s systems and subsystems. Had the research been more detailed, the results of the weighting metric system would be used to establish an entrance and exit criteria for identifying the most appropriate framework or framework combinations to be implemented throughout the life cycle. However, that step was bypassed for this study; the given ICM framework was considered baseline for a nominal matured spacecraft system. The hypothetical impact for both TRL and MRL risk drivers are shown in Figure 3 with a brief explanation. The example provided for that figure summarizes the ratings for each risk driver in a pre- and post-mitigated format. However, the focus of this research is to demonstrate how a system or subsystem may have different maturity ratings throughout the WBS and illustrate the impacts on tasks and activity durations.

Figure 3: Risk Identification of MRL-1
(Click on image above to show full-size version in pop-up window.)

Figure 3 illustrates the process used to identify risks that affect the project schedule (generated using [5]). Each risk is evaluated against a task where they’re rated based upon the probability of occurrence and their severity effects on schedule and cost. This example demonstrates a pre-and-post mitigation evaluation—quantified after running the Monte Carlo simulation—to address the schedule’s behavioral uncertainties. The first step in the Monte Carlo simulation development required a qualitative risk assessment of the defined risk drivers. The qualitative risk assessment is performed with the use of the WBS and the identification of an expert familiar with the tasks and activities of the development and production phases. The input from the expert will be used to establish the qualitative risk assessment with the following important steps:

 

1. Develop a working list of risk drivers that pose a threat to the project schedule, cost, or development performance.
2. Develop a risk ranking guide that establishes the probability of risk occurrence. This study uses very low (VL), low (L), medium (M), high (H), very high (VH) probabilities of occurrence for activities and tasks of the WBS affecting schedule and cost.
3. Identify the impacts or consequences of the risk drivers by evaluating the probability of occurrence and the magnitude of impact on the schedule, cost, and development performance. Establish qualitative descriptions to identify the assessed risk drivers with the use of a risk matrix.
4. Establish a final risk score for each risk driver after completing steps 1-3, followed by populating the risk matrix to illustrate the magnitude of all identified risks. Figure 3 illustrates how steps 1-4 were implemented and ranked for MRL-1.

Risk Assessment

Risk assessment is the process of classifying risks into categories characterized by their frequency of occurrence and the severity of their consequences. The risk assessment can be performed through either qualitative or quantitative evaluations as well as through a comprehensive evaluation combining both assessment types. Qualitative risk assessment is considered to be the process of prioritizing risks based upon the risk’s probability of occurrence ranging from unlikely to most likely. The second aspect of the qualitative risk assessment is when the risks are prioritized based upon the risk’s severity of consequence. Quantitative risk assessment is considered to be the process of prioritizing risks using statistical techniques to estimate the project’s numerical outcome (schedule/cost behaviors) based upon identified project risks through the use of probability distributions. Monte Carlo simulation is commonly used to model project behavior [11].

For our research, the quantitative risk assessment was based upon the project schedule that was developed for the spacecraft system. The goal was to understand the levels of uncertainty inherent within tasks and activities of a project’s schedule. These uncertainties (task durations) were evaluated by probability distributions via three-point estimates and are acquired either through empirical data or expert judgment. Duration values associated with each task or activity risk are represented by subjective bounds least likely, most likely, and optimistic, and are analyzed after the Monte Carlo simulation is run.

Monte Carlo simulation is designed to iterate the project schedule’s tasks and activities multiple times by randomly selecting task or activity duration values for each iteration from the probability distribution type chosen. The outcome results of the simulation were then used to provide the possible end dates of all tasks and activities based upon the respective associated risk drivers for the spacecraft’s project schedule.

Risk Analysis and Results

Risk analysis was conducted using Monte Carlo simulation to illustrate how the ICM framework can be implemented for the development of a spacecraft system [3]. The benefit of using this technique is that it generates schedule and cost estimates for uncertain input values through the use of probability distributions. It does this by randomly generating values and iteratively modeling the behavior of tasks and activities of a project schedule.

The methodology was used to demonstrate how TRL and MRL risk drivers can be mitigated within the ICM framework, and to understand and assess whether the project schedule will meet the required completion date without budget overruns. In order to properly model the behavior of the project schedule, the ICM framework was baselined with the DoD life-cycle stages [3]. Within each stage, planned tasks and activities were developed for a spacecraft’s development and traced against the TRL and MRL risk drivers through risk identification. An example of the mapping process, illustrating the relationship of the risk drivers and the life-cycle stages, is provided in Figure 4.

Figure 4: TRL and MRL Maturity Trace to DoD Acquisition Life Cycle

Once the risk drivers were mapped to the appropriate activities, the uncertainty of the risks were pre- and post-mitigated through qualitative analysis with the options of using expert opinion and empirical data. Although our work was hypothetical, the data collection method included both expert opinion and empirical data. The probability distribution chosen to be simulated throughout the life cycle is frequently used to model expert opinion or empirical data. The distribution used was adapted so that the expert can provide three-point estimates that represent pessimistic, most likely, and optimistic inputs. The estimates correspond directly to the time estimates used as input variables for the Monte Carlo simulation. Therefore, the triangular distribution was chosen because it is the most commonly used distribution for modeling expert opinion and empirical data. The triangular distribution is also used when there is very little information known about the parameters outside the approximate estimate of its pessimistic, most likely, and optimistic variables. In addition, the uniform distribution was not chosen because it is known to be a very poor modeler of expert opinion and empirical data (since all of the values within its range have equal probability density). Thus, the density falls sharply to zero at the pessimistic and optimistic endpoint estimates.

The example (provided in Figure 3) identifies MRL-1 as the risk driver imposed upon the concept studies task of the Exploration and Valuation stages of the life cycle. The MRL-1 risk driver was then categorized and identified with a medium impact and low probability of occurrence. An illustration of the risk rating is provided in the risk matrix of Figure 3, in addition to detailing pre- and post-mitigation definitions for that risk driver. The risk identification and mitigation process is a critical step in the event that contingency scenarios need to be formulated to successfully complete the project.

Now that all steps have been taken to develop the spacecraft’s project model, the simulation is run and the results are evaluated to be realistic or unrealistic. After running the pre-mitigated project schedule, the statistical completion dates are provided in the project schedule results without the imposed maturity risks shown in Figure 5. The results of the spacecraft’s project schedule were then used to establish the baseline model for the ICM acquisition life cycle. The next step, as outlined in the Risk Identification section, requires factoring the risk drivers into the project schedule’s tasks and activities, followed by running another simulation to determine the final schedule outcome. The modeled behavior of the risk-driven schedule is provided in the project schedule with the imposed maturity risks of Figure 5.

Figure 5: Spacecraft Schedule and Cost Results of Non-Risk Imposed vs. Risk Imposed Simulations
(Click on image above to show full-size version in pop-up window.)

Figure 5 also illustrates the spacecraft’s cost result for non-risk- and risk-imposed simulations. The non-risk-imposed probability distributions for both schedule and cost outcome aided in the evaluation of the uncertainties of the maturity risk drivers that are incorporated into the risk-imposed results. Thus, it is clear that the risk-imposed simulation results have increased in cost when compared to the pre-mitigated results.

Risk Mitigation and Conclusions

The objective of this study was to evaluate the implementation of the ICM framework within a DoD acquisition life cycle while imposing TRL and MRL risk drivers. This was accomplished by the implementation of qualitative and quantitative statistical techniques and the use of Monte Carlo simulation to predict the probability of meeting the program’s projected schedule and cost estimates. The project schedule activities developed for this study correlated to activities that comprised a notional ICM framework. As a result, the notional ICM framework established the baseline project schedule to be evaluated against technology and manufacturing maturity throughout the system development life cycle.

The evaluation was successfully performed using a step-by-step risk management process that was quantified through simulation. The triangular probability distribution was chosen to be used throughout the Monte Carlo simulation. This distribution type was chosen because three-point estimates (pessimistic, most likely, and optimistic) were used to represent workflow or activity durations of the life cycle. It should be noted that although life-cycle-critical paths were not identified and discussed in this study, the Monte Carlo simulation generated random variables to predict activity durations from each critical path probability distribution; this was ultimately used to develop the overall probability distribution of the system’s project schedule. Because of the project schedule’s simulation, insight was given to the decision-maker that revealed the consequences of the imposed maturity risks against the predicted schedule and cost estimates.

The project results contain qualitative (expert opinion) and quantitative (empirical) data. It is assumed that the degree of subjectivity associated with an expert’s input is consistent with the expected data of the project schedule. However, if the subjective inputs are inaccurate, the results of the simulation can be very sensitive and reflect inaccurate schedule and cost estimates. Therefore, since the risk scores were developed by experts with some degree of subjectivity, it is important to consider evaluating the credibility of those experts in order to quantify their input. This evaluation can be performed with a technique called the classical method [12]. Although not used within our study, it is a credible approach to consider when validating subjective inputs.

The results of the Monte Carlo simulation demonstrated that the technology and manufacturing maturity risks influence the overall schedule and cost to develop the complex system; this is explicitly shown in the final schedule and cost results of Figure 5. The pre-mitigated status of the project schedule represents a simulation run where the maturity risk drivers are not applied to the activities of the project schedule. The post-mitigated status of the project schedule represents a simulation run where the maturity risk drivers are applied to the activities of the project schedule and a risk-mitigation strategy is developed but not implemented. This is consistent with the schedule slip and the cost increase illustrated in Figure 5. However, a simulation run that implements the risk-mitigated strategy should show improvement in the final project schedule and cost metrics. It should also be noted that the simulated results were used to illustrate two important elements:

 

1. There may be schedule and cost consequences when applying maturity risk drivers to the project schedule of the acquisition life cycle.
2. The analysis can provide a level of confidence in meeting the projected schedule and cost estimates of the overall project.

As a result of the risk management process developed for this study, decision-makers have a significant alternative mitigation strategy that can be implemented in order to minimize potential schedule and cost overruns.

For future research, this study establishes a framework for evaluating the impacts of maturity risks against schedule and cost and produces results that quantify a hypothetical baseline ICM framework. It also establishes the risk assessment approach to quantitatively evaluate and compare the metrics of other life-cycle models, further identifying the strengths and tailorability of the ICM framework.

Dr. Karl L. Brunson, George Washington University
Dr. Jeffrey Beach, George Washington University
Thomas A. Mazzuchi, George Washington University
Dr. Shahram Sarkani, George Washington University