June 2020 – Systems Engineering and T&E Synchronization
As systems increase in complexity and adopt agile development strategies, testing and evaluation (T&E) provides a meaningful feedback loop to the systems engineering process. Systems engineering provides the process and tools to build the right effective products in the best way. A meaningful feedback loop requires improved synchronization between systems engineering and test and evaluation.
This month’s articles provide summaries of high-level initiatives and technical case studies to achieve System Engineering and T&E synchronization.
This month’s Guest Editorial comes from Major General Michael Brewer and Colonel Matthew Magness from the Air Force Operational Test and Evaluation Center (AFOTEC). It describes how AFOTEC is refining their processes to achieve agile acquisition goals. The Adaptive Relevant Testing (ART) initiative outlines six key principles: 1) early operational test involvement, 2) the ability to tailor to the situation, 3) continuous and cumulative feedback, 4) streamlining process and products, 5) integrated and combined testing, and 6) adaptive testing based on discovery. They adopt a philosophy of testing as a service (TaaS) as a practical application of these principles and highlight the value of this approach.
In our Inside the Beltway, J. Michael Barton, Ph.D. interviews Dr. Steven J. Hutchison, Director for Test and Evaluation in the Department of Homeland Security. Dr. Hutchinson discusses his career in test and evaluation; how his role in T&E has changed over time; how the T&E profession has changed over time; the importance of putting the evaluation goals before the test plan; “shift left”; and importantly for this issue how T&E is a life cycle activity. I enjoyed his perspective immensely, especially his closing remarks on the importance of having an excellent team.
In the T&E Handbook article, Mark A. London, Ph.D. discusses sample size calculations for continuous data. This is the complement to his previous tutorial on sample size for binary data. This primer on sample size calculations provides useful tools for determining optimal test sizes that maximize test resources at allowable risk levels.
Our first technical article, “Integrating Test into a Secure Systems Engineering Process” by William D. Bryant, Ph.D. and Lt. Col. R. Lane Odom, highlights the importance of designing in cyber resilience via Systems Security Engineering (SSE). The authors note how current practices often wait until too late in the system development to start cyber testing activities and fail to incorporate feedback into the system design. They summarize methods and tools for providing cyber assessments to include risk assessments and how testing can serve as a feedback loop.
Terry Murphy and Patrick Kastner provide an informative article on, “Incorporating Test and Evaluation into Acquisition Contracts Improving Acquisition Outcomes” in our second technical article. The article provides an exhaustive list of T&E questions that need to be answered and incorporated into contracts, because as the authors note, “If it is not written into the contract, it will not happen.” This article is a great resource for testers before they review contracting documents.
In our third article, “Model-based Systems Engineering to Support Test-driven Development,” 2d Lt Katherine E. Cheney and John M. Colombi, Ph.D. discuss approaches for testing autonomous systems. They develop a 16-step process to accelerate testing for autonomous agents. They apply their process to the testing of a multi-rotor small Unmanned Aircraft System conducting a wide area search mission, starting with the system architecture and mission and concluding with the appropriate system/mission analysis.
In “Under-body Blast Verification and Validation Methodology,” written by Andrew W. Drysdale, et al. from Combat Capabilities Development Command provides an overview of their methodology and tools. The authors highlight that this under-body blast methodology is complicated by the modeling process which involves finite-element modeling, multiple sources of code (both commercial and government), varying granularity, and a variety of input-data sources. They note that simultaneously full system testing is limited and also subject to sources of variability.
Our fifth technical article written by Ian Levitt, et al. from the Federal Aviation Administration (FAA) is titled, “Research, Development, Test & Evaluation (RDT&E) in the Trajectory Based Operations (TBO) Integrated Test Environment (TITE).” The authors discuss the need for testing that can be conducted holistically (e.g., end-to-end, system-of-systems) and the integration of T&E activities throughout the acquisition life cycle. The article provides an overview of their goals and how public/private partnership enabling industry-differentiating live, virtual, constructive environment will enable both test and evaluation objectives.
In our final article, “Exploring Agile Acquisition Principles and Practices for FAA Complex System Delivery,” Mindy Dowling, et al. from the FAA explore the concepts of agile acquisition for the FAA in the context of the complex and interconnected systems. They highlight the importance of safety in the FAA mission and ponder how agile principles align with the FAA’s values of safety and security. They provide two examples and note the benefits of agile values. They conclude that the incorporation of agile processes into the FAA’s acquisition strategy is an ongoing discussion, but that the goal of speed will not replace the commitment to T&E best practices.
Enjoy the issue!
Laura Freeman, Ph.D.