December 2020 – T&E for Cybersecurity and Readiness
This month our feature column is by Michèle Flournoy, Avril Haines, and Gabrielle Chefitz. The column provides an executive summary of a recent report the three authored entitled, “Building Trust through Testing.” The summary highlights how new technologies such as machine learning (ML) and artificial intelligence (AI) require new testing approaches. The full report is available at: https://cset.georgetown.edu/wp-content/uploads/Building-Trust-Through-Testing.pdf. They make 11 recommendations providing a useful framework to pave the way for future Test and Evaluation methods for ML/AI.
In our next feature, J. Michael Barton, Ph.D. continues his outstanding series of interviewing experts from across the Test and Evaluation (T&E) community. In this issue, Ed Kraft, Ph.D., reflects on his career in T&E starting in wind tunnels and progressing into senior technical advisor roles. He advises T&E practitioners to continue to learn and innovate.
In our first technical article, Col Martha Monroe and Jeff Olinger from AFOTEC discuss Mission-Based Risk Assessment Process for Cyber (MRAP-C) – a “systematic risk analysis to characterize system failure modes and identify potential cyber vulnerabilities.” AFOTEC Commander, Major General Sears, has directed all Acquisition Category (ACAT) I programs start using MRAP-C, noting, “MRAP-C is the most revolutionary cyber test process to affect the Air Force in the last decade and we are working with our partners across the acquisition and test communities to implement it as soon as possible.” The end of the article links to resources for ITEA readers looking to learn more about MRAP-C.
In the next technical article, Robin Deiulio and Kevin Alix from the Test Resource Management Center (TRMC) highlight the need and apply agile best practices to accelerated DoD cybersecurity. The article summarizes efforts to streamline processes, provide certified software and enterprise services in the TMRC that will enable rapid and agile development environments across the RDT&E enterprise.
In his article on Test Strategy and Architecture, Gary Honea provides his thoughts on how a test strategy and architecture sets programs up for success. He highlights the challenge that test and evaluation can be overlooked in the effort to deliver capabilities fast with improved capabilities. He advocates for a single accountable authority, a “Test Architect,” to develop an integrated test strategy and architecture for a program. This article describes the elements of a test strategy and architecture and applies them to the development cycle of an example program.
In our fourth technical article, Lt Col Logan O. Mailloux, Ph.D., Capt Martin “Trae” Span, and Robert F. Mills, Ph.D. tackle the challenge of “Defining Testable Systems Security Requirements for an Autonomous Space Vehicle.” They highlight the challenge of defining security requirements for complex systems with cyber dependencies. Using a tailored System-Theoretic Process Analysis for Security (STPA-Sec), they illustrate the requirements generation process on a notional autonomous space vehicle.
Our final article from Joshua Walker and John M. Borky, Ph.D. discusses the value of Systems Modeling Language (SysML) to enhance the test planning and documentation process. They argue such an approach is needed for systems that are growing in complexity due to their software code base. They highlight the potential loss of information and redundancies in the test process without a structure to link test products and information together. The article provides a clear walkthrough of how using model-based systems engineering supported with SysML can provide clear test planning and documentation.
I would also like to remind you that this issue of the Journal is our second issue where we are testing digital delivery. The Publications Committee proposed to the Board of Directors a two-issue trial (September 2020 and December 2020) for digital copies of the Journal. We are also conducting a survey of the readership to understand the implications of a digital Journal.1 The survey is available here: https://forms.office.com/Pages/ ResponsePage.aspx?id=hGiVYK0Q-kCGPU8yweOjev7HDZKp2WNCvh4LYQYVzMxUNDRGWUVOMVM1OUhaTklERzRRR1FVREMwUy4u.
I would appreciate your feedback as we move forward into new directions with the Journal.
Enjoy the Issue!
1 The Journal has subscription-based members as well. Physical subscriptions to the Journal will still receive hard copies during this prototyping phase and into the future.
September 2020 – Innovations in Modeling and Simulation (M&S) Use for T&E
Before we dive into the issue, I would like to first announce that we will be using this issue of the Journal to prototype digital delivery – I could not think of a better issue than an M&S issue to prototype a digital version of the Journal. As Pete discussed in his President’s column, ITEA is faced with the same challenges that many other organizations are right now due to Covid-19. While that is a motivating factor, we are also excited about lots of other opportunities that a digital journal will provide. The Publications Committee has proposed to the Board of Directors a two issue trial (September 2020 and December 2020) for digital copies of the Journal.1 Some benefits we are working to implement include digital copies of articles for all authors with the built in copyright agreement to promote sharing on personal media sites; highlighting special articles for open access, and reducing our environmental impact by reducing the number of plastic bags mailed out to each of you. Of course as good testers, during this prototype test we will be collecting data and working with you all to understand the potential benefits and shortfalls of this approach.
Innovations in modeling and simulation that make it credible for test and evaluation are occurring all of the time. Current initiatives in digital engineering across industry and government agencies will continue to propel modeling and simulation capabilities for complex systems and environments forward. These capabilities provided needed tools for test and evaluation allowing us to depict places and time where real systems and/or humans cannot go. They have the ability to better inform live testing, shape the bounds of testing, augment data, and allow us to answer questions that are unfeasible to answer via live test and evaluation. This issue dedicated to innovations in modeling and simulations provides perspectives and examples of the advances enabling these transformative changes to test and evaluation.
In our Inside the Beltway column this month, we have another great article in our interview series. J. Michael Barton, Ph.D. provides a compelling interview with Dr. C. David Brown. Dave takes us through his career in the Army, his introduction to test and evaluation, how testing has changed, how it is still the same, the importance of certification and education for the T&E workforce, and the importance of data. It is a great review of the importance of test and evaluation and how we can continue to strive to advance the test community.
In this month’s Historical Perspectives column, Tom Cash shares his experience in developing test plans for the Distributed Interactive Simulation (DIS) capability. He highlights the confluence of events that lead to DIS being a credible testing and training tool, a great perspective given the wide use of modeling and simulation tools used today in testing and training.
In our first technical article, “A Live, Virtual, and Constructive Based Approach to Building Progressive Confidence in Air Manned-Unmanned Team (MMUT) Systems,” the authors outline a staged process for developing T&E programs that progresses from purely synthetic environments to full integration of Live, Virtual, and Constructive. Derek Cook, Mark Jennings, and Adrian Britton, Ph.D. leverage their collective experience, knowledge of United Kingdom Air Ranges, and work at QinetiQ to illustrate the key challenges and enablers of such a program. Their MMUT example provides a logical sequence for others to learn from.
In our second technical article, a team for the Scientific Test and Analysis Techniques (STAT) Center of Excellence provides a case study, “Design of Experiments (DOE) in Characterizing Hypersonic Flow on a Wind Tunnel Model.” The article describes the STAT process, provides fundamentals of DOE, and highlights the strengths of the designs in supporting numerous technical objectives. The detailed analysis uses graphics to show the results of the design and how not only did the design approach provide a defensible strategy for testing experimental technologies, but also provided a basis for validation of computational fluid dynamic models. They highlight that this approach has the potential to reduce future test points needed in both the computational fluid dynamic (CFD) model and wind tunnel tests.
Next, Edward M. Kraft, Ph.D. paints a bold new vision for how test and evaluation can be transformed and is a vital element of the digital engineering revolution in his article “Digital Engineering and the Future Transformation of Test & Evaluation.” His analysis highlights the alignment of the current DoD Digital Engineering Strategy with the key T&E transformation steps. He highlights a philosophy of thinking about T&E as an authoritative source of validation information for digital engineering. Finally, he describes how our processes might change to support such a revolution, calling for a Digital TEMP Test and Evaluation Master Plan to enable these processes.
In the next article a team spanning North Carolina Agricultural and Technical State University and Georgia Tech Research Institute showcases, “A Software Tool for Evaluating Unmanned Autonomous Systems.” The Data-Drive Intelligent Prediction Tool (DIPT) was developed for testing a multi-platform Unmanned Aerial Vehicle. Leveraging Learning Classifier Systems and Fuzzy Logic, the team developed a tool to infer operating states and perceptions of UAV using externally available information and states. They highlight not only the data-driven tools, but the process they put in place to achieve success.
In our fifth technical article, Christopher Paust, Tyler Chan, and Laurie Beth Smith show the “Impact of Swarming Surface Vessels and Aerial Scoring on DoD Test Ranges.” The article provides great background on systems against swarming surface threats: SWARM Command & Control (C2) and SWARM Scoring. A key is that SWARM C2 enables one operator to control multiple High Speed Maneuverable Surface Targets (HSMSTs) simultaneously. The article not only describes the capability, but also the verification and validation process the team conducted to ensure the SWARM C2 is adequate for testing. Key benefits they highlight are reduced costs, lower manpower, and the ability to provide complex scenarios not previously possible in test.
In our final technical article, Cedrick Reid from KBR writes about “Live Testing of LASER Systems – Safe Execution.” Cedrick describes the increasing presence of High Energy LASERs (HEL) in systems, and the need to protect humans from these energy sources. The article showcases a proven process for ensuring safety and coordinating the safety requirements with T&E objectives.
Enjoy the issue!
1 The Journal has subscription based members as well. Physical subscriptions to the Journal will still receive hard copies during this prototyping phase and into the future.