2020 ITEA Journal Issues Abstracts

ITEA Journal of Test and Evaluation

ONLINE Issues (ITEA Members Only)


December 2020 – T&E for Cybersecurity and Readiness

This month our feature column is by Michèle Flournoy, Avril Haines, and Gabrielle Chefitz. The column provides an executive summary of a recent report the three authored entitled, “Building Trust through Testing.” The summary highlights how new technologies such as machine learning (ML) and artificial intelligence (AI) require new testing approaches. The full report is available at: https://cset.georgetown.edu/wp-content/uploads/Building-Trust-Through-Testing.pdf. They make 11 recommendations providing a useful framework to pave the way for future Test and Evaluation methods for ML/AI.

In our next feature, J. Michael Barton, Ph.D. continues his outstanding series of interviewing experts from across the Test and Evaluation (T&E) community. In this issue, Ed Kraft, Ph.D., reflects on his career in T&E starting in wind tunnels and progressing into senior technical advisor roles. He advises T&E practitioners to continue to learn and innovate.

In our first technical article, Col Martha Monroe and Jeff Olinger from AFOTEC discuss Mission-Based Risk Assessment Process for Cyber (MRAP-C) – a “systematic risk analysis to characterize system failure modes and identify potential cyber vulnerabilities.” AFOTEC Commander, Major General Sears, has directed all Acquisition Category (ACAT) I programs start using MRAP-C, noting, “MRAP-C is the most revolutionary cyber test process to affect the Air Force in the last decade and we are working with our partners across the acquisition and test communities to implement it as soon as possible.” The end of the article links to resources for ITEA readers looking to learn more about MRAP-C.

In the next technical article, Robin Deiulio and Kevin Alix from the Test Resource Management Center (TRMC) highlight the need and apply agile best practices to accelerated DoD cybersecurity. The article summarizes efforts to streamline processes, provide certified software and enterprise services in the TMRC that will enable rapid and agile development environments across the RDT&E enterprise.

In his article on Test Strategy and Architecture, Gary Honea provides his thoughts on how a test strategy and architecture sets programs up for success. He highlights the challenge that test and evaluation can be overlooked in the effort to deliver capabilities fast with improved capabilities. He advocates for a single accountable authority, a “Test Architect,” to develop an integrated test strategy and architecture for a program. This article describes the elements of a test strategy and architecture and applies them to the development cycle of an example program.

In our fourth technical article, Lt Col Logan O. Mailloux, Ph.D., Capt Martin “Trae” Span, and Robert F. Mills, Ph.D. tackle the challenge of “Defining Testable Systems Security Requirements for an Autonomous Space Vehicle.” They highlight the challenge of defining security requirements for complex systems with cyber dependencies. Using a tailored System-Theoretic Process Analysis for Security (STPA-Sec), they illustrate the requirements generation process on a notional autonomous space vehicle.

Our final article from Joshua Walker and John M. Borky, Ph.D. discusses the value of Systems Modeling Language (SysML) to enhance the test planning and documentation process. They argue such an approach is needed for systems that are growing in complexity due to their software code base. They highlight the potential loss of information and redundancies in the test process without a structure to link test products and information together. The article provides a clear walkthrough of how using model-based systems engineering supported with SysML can provide clear test planning and documentation.

I would also like to remind you that this issue of the Journal is our second issue where we are testing digital delivery. The Publications Committee proposed to the Board of Directors a two-issue trial (September 2020 and December 2020) for digital copies of the Journal. We are also conducting a survey of the readership to understand the implications of a digital Journal.1 The survey is available here: https://forms.office.com/Pages/ ResponsePage.aspx?id=hGiVYK0Q-kCGPU8yweOjev7HDZKp2WNCvh4LYQYVzMxUNDRGWUVOMVM1OUhaTklERzRRR1FVREMwUy4u.

I would appreciate your feedback as we move forward into new directions with the Journal.

Enjoy the Issue!

Endnote
1  The Journal has subscription-based members as well. Physical subscriptions to the Journal will still receive hard copies during this prototyping phase and into the future.


September 2020 – Innovations in Modeling and Simulation (M&S) Use for T&E

Before we dive into the issue, I would like to first announce that we will be using this issue of the Journal to prototype digital delivery – I could not think of a better issue than an M&S issue to prototype a digital version of the Journal. As Pete discussed in his President’s column, ITEA is faced with the same challenges that many other organizations are right now due to Covid-19. While that is a motivating factor, we are also excited about lots of other opportunities that a digital journal will provide. The Publications Committee has proposed to the Board of Directors a two issue trial (September 2020 and December 2020) for digital copies of the Journal.1 Some benefits we are working to implement include digital copies of articles for all authors with the built in copyright agreement to promote sharing on personal media sites; highlighting special articles for open access, and reducing our environmental impact by reducing the number of plastic bags mailed out to each of you. Of course as good testers, during this prototype test we will be collecting data and working with you all to understand the potential benefits and shortfalls of this approach.

Innovations in modeling and simulation that make it credible for test and evaluation are occurring all of the time. Current initiatives in digital engineering across industry and government agencies will continue to propel modeling and simulation capabilities for complex systems and environments forward. These capabilities provided needed tools for test and evaluation allowing us to depict places and time where real systems and/or humans cannot go. They have the ability to better inform live testing, shape the bounds of testing, augment data, and allow us to answer questions that are unfeasible to answer via live test and evaluation. This issue dedicated to innovations in modeling and simulations provides perspectives and examples of the advances enabling these transformative changes to test and evaluation.

In our Inside the Beltway column this month, we have another great article in our interview series. J. Michael Barton, Ph.D. provides a compelling interview with Dr. C. David Brown. Dave takes us through his career in the Army, his introduction to test and evaluation, how testing has changed, how it is still the same, the importance of certification and education for the T&E ­workforce, and the importance of data. It is a great review of the importance of test and evaluation and how we can continue to strive to advance the test community.

In this month’s Historical Perspectives column, Tom Cash shares his experience in developing test plans for the Distributed Interactive Simulation (DIS) capability. He highlights the confluence of events that lead to DIS being a credible testing and training tool, a great perspective given the wide use of modeling and simulation tools used today in testing and training.

In our first technical article, “A Live, Virtual, and Constructive Based Approach to Building Progressive Confidence in Air Manned-Unmanned Team (MMUT) Systems,” the authors outline a staged process for developing T&E programs that progresses from purely synthetic environments to full integration of Live, Virtual, and Constructive. Derek Cook, Mark Jennings, and Adrian Britton, Ph.D. leverage their collective experience, knowledge of United Kingdom Air Ranges, and work at QinetiQ to illustrate the key challenges and enablers of such a program. Their MMUT example provides a logical sequence for others to learn from.

In our second technical article, a team for the Scientific Test and Analysis Techniques (STAT) Center of Excellence provides a case study, “Design of Experiments (DOE) in Characterizing Hypersonic Flow on a Wind Tunnel Model.” The article describes the STAT process, provides fundamentals of DOE, and highlights the strengths of the designs in supporting numerous technical objectives. The detailed analysis uses graphics to show the results of the design and how not only did the design approach provide a defensible strategy for testing experimental technologies, but also provided a basis for validation of computational fluid dynamic models. They highlight that this approach has the potential to reduce future test points needed in both the computational fluid dynamic (CFD) model and wind tunnel tests.

Next, Edward M. Kraft, Ph.D. paints a bold new vision for how test and evaluation can be transformed and is a vital element of the digital engineering revolution in his article “Digital Engineering and the Future Transformation of Test & Evaluation.” His analysis highlights the alignment of the current DoD Digital Engineering Strategy with the key T&E transformation steps. He highlights a philosophy of thinking about T&E as an authoritative source of validation information for digital engineering. Finally, he describes how our processes might change to support such a revolution, calling for a Digital TEMP Test and Evaluation Master Plan to enable these processes.

In the next article a team spanning North Carolina Agricultural and Technical State University and Georgia Tech Research Institute showcases, “A Software Tool for Evaluating Unmanned Autonomous Systems.” The Data-Drive Intelligent Prediction Tool (DIPT) was developed for testing a multi-platform Unmanned Aerial Vehicle. Leveraging Learning Classifier Systems and Fuzzy Logic, the team developed a tool to infer operating states and perceptions of UAV using externally available information and states. They highlight not only the data-driven tools, but the process they put in place to achieve success.

In our fifth technical article, Christopher Paust, Tyler Chan, and Laurie Beth Smith show the “Impact of Swarming Surface Vessels and Aerial Scoring on DoD Test Ranges.” The article provides great background on systems against swarming surface threats: SWARM ­Command & Control (C2) and SWARM Scoring. A key is that SWARM C2 enables one operator to control multiple High Speed Maneuverable Surface Targets (HSMSTs) simultaneously. The article not only describes the capability, but also the verification and validation process the team conducted to ensure the SWARM C2 is adequate for testing. Key benefits they highlight are reduced costs, lower manpower, and the ability to provide complex scenarios not previously possible in test.

In our final technical article, Cedrick Reid from KBR writes about “Live Testing of LASER Systems – Safe Execution.” Cedrick describes the increasing presence of High Energy LASERs (HEL) in systems, and the need to protect humans from these energy sources. The article showcases a proven process for ensuring safety and coordinating the safety requirements with T&E objectives.

Enjoy the issue!

Endnote
1  The Journal has subscription based members as well. Physical subscriptions to the Journal will still receive hard copies during this prototyping phase and into the future.


June 2020 – Systems Engineering and T&E Synchronization

As systems increase in complexity and adopt agile development strategies, testing and evaluation (T&E) provides a meaningful feedback loop to the systems engineering process. Systems engineering provides the process and tools to build the right effective products in the best way. A meaningful feedback loop requires improved synchronization between systems engineering and test and evaluation.

This month’s articles provide summaries of high-level initiatives and technical case studies to achieve System Engineering and T&E synchronization.

This month’s Guest Editorial comes from Major General Michael Brewer and Colonel Matthew Magness from the Air Force Operational Test and Evaluation Center (AFOTEC). It describes how AFOTEC is refining their processes to achieve agile acquisition goals. The Adaptive Relevant Testing (ART) initiative outlines six key principles: 1) early operational test involvement, 2) the ability to tailor to the situation, 3) continuous and cumulative feedback, 4) streamlining process and products, 5) integrated and combined testing, and 6) adaptive testing based on discovery. They adopt a philosophy of testing as a service (TaaS) as a practical application of these principles and highlight the value of this approach.

In our Inside the Beltway, J. Michael Barton, Ph.D. interviews Dr. Steven J. Hutchison, Director for Test and Evaluation in the Department of Homeland Security. Dr. Hutchinson discusses his career in test and evaluation; how his role in T&E has changed over time; how the T&E profession has changed over time; the importance of putting the evaluation goals before the test plan; “shift left”; and importantly for this issue how T&E is a life cycle activity. I enjoyed his perspective immensely, especially his closing remarks on the importance of having an excellent team.

In the T&E Handbook article, Mark A. London, Ph.D. discusses sample size calculations for continuous data. This is the complement to his previous tutorial on sample size for binary data. This primer on sample size calculations provides useful tools for determining optimal test sizes that maximize test resources at allowable risk levels.

Our first technical article, “Integrating Test into a Secure Systems Engineering Process” by William D. Bryant, Ph.D. and Lt. Col. R. Lane Odom, highlights the importance of designing in cyber resilience via Systems Security Engineering (SSE). The authors note how current practices often wait until too late in the system development to start cyber testing activities and fail to incorporate feedback into the system design. They summarize methods and tools for providing cyber assessments to include risk assessments and how testing can serve as a feedback loop.

Terry Murphy and Patrick Kastner provide an informative article on, “Incorporating Test and Evaluation into Acquisition Contracts Improving Acquisition Outcomes” in our second technical article. The article provides an exhaustive list of T&E questions that need to be answered and incorporated into contracts, because as the authors note, “If it is not written into the contract, it will not happen.” This article is a great resource for testers before they review contracting documents.

In our third article, “Model-based Systems Engineering to Support Test-driven Development,” 2d Lt ­Katherine E. Cheney and John M. Colombi, Ph.D. discuss approaches for testing autonomous systems. They develop a 16-step process to accelerate testing for autonomous agents. They apply their process to the testing of a multi-rotor small Unmanned Aircraft System conducting a wide area search mission, starting with the system architecture and mission and concluding with the appropriate system/mission analysis.

In “Under-body Blast Verification and Validation Methodology,” written by Andrew W. Drysdale, et al. from Combat Capabilities Development Command provides an overview of their methodology and tools. The authors highlight that this under-body blast methodology is complicated by the modeling process which involves finite-element modeling, multiple sources of code (both commercial and government), varying granularity, and a variety of input-data sources. They note that simultaneously full system testing is limited and also subject to sources of variability.

Our fifth technical article written by Ian Levitt, et al. from the Federal Aviation Administration (FAA) is titled, “Research, Development, Test & Evaluation (RDT&E) in the Trajectory Based Operations (TBO) Integrated Test Environment (TITE).” The authors discuss the need for testing that can be conducted holistically (e.g., end-to-end, system-of-systems) and the integration of T&E activities throughout the acquisition life cycle. The article provides an overview of their goals and how public/private partnership enabling industry-differentiating live, virtual, constructive environment will enable both test and evaluation objectives.

In our final article, “Exploring Agile Acquisition Principles and Practices for FAA Complex System Delivery,” Mindy Dowling, et al. from the FAA explore the concepts of agile acquisition for the FAA in the context of the complex and interconnected systems. They ­highlight the importance of safety in the FAA mission and ponder how agile principles align with the FAA’s values of safety and security. They provide two examples and note the benefits of agile values. They conclude that the incorporation of agile processes into the FAA’s acquisition strategy is an ongoing discussion, but that the goal of speed will not replace the commitment to T&E best practices.

Enjoy the issue!
Laura Freeman, Ph.D.
Senior Editor


March 2020 – The Right Mix of T&E Infrastructure

To accomplish our jobs as Test and Evaluation (T&E) professionals, we need the right mix of infrastructure to support our mission. As our systems evolve and become increasingly complex, the “right mix” of infrastructure must also evolve to keep pace. This issue contains a variety of articles that reflect the diverse infrastructure needs of the T&E community.

In this issue, we are experimenting with a few new types of articles. We have included an historical perspectives article and two short technical articles.

In our historical perspectives article, Christian Gelzer reflects on the longest flight in the X-15. In the first technical short article, Joseph Crossin discusses developing the next generation of radars that will leverage emerging technologies such as 5G wireless networks. In the next technical short article, Guenever Aldrich and Tomasz Wojtaszek discuss the need for collaboration to achieve spectrum sharing. I enjoyed reading these contributions and welcome your feedback on these new article ideas. Our technical articles reflect a wide range of T&E infrastructure needs.

Brian Mork, Col USAF (Ret), Ph.D. from the Air Force Research Lab authored an article on a “Three Dimensional Cyber Risk Model.” He highlights that we need to move beyond two-dimensional risk assessments when an intelligent adversary is involved. He shows how thinking in three dimensions can provide better decision making for design engineers, program managers, and red teams that devise cyber combat theories.

In the next article, Huat Ng and Wayne Devereux from KBR highlight the importance of Live Virtual and Constructive (LVC) environments in test design. Noting the recent and substantial advances in models and simulations made possible by advances in information technology and visualization, they highlight that now is an ideal time to incorporate more models and simulations in test design. They highlight that these methods can model more complex environments than available on operational test ranges and inform conditions for operational testing. Finally, they provide a decision flow chart for leveraging LVC in test design. An electricaloptical/infrared sensor case study reinforces their ideas.

MAJ Kurt Klingensmith and Jon Alt, Ph.D. from the Research and Analysis Center in Monterey contributed an article on “A Systems Architecting and Engineering Approach to Creating Enterprise Data Strategy.” This article highlights how data curation is a critical element of test infrastructure. They provide overall architecture and examples of functional decomposition for various stages of data management to guide the reader to an enterprise data strategy. Themes they highlight that should resonate with ITEA readers are that data is an enterprise asset, leveraging it requires an educated workforce, and that data strategies address people, processes, and technologies.

In the article, “Homeland Security Test & Evaluation Infrastructure: It’s Time to Add to the Mix,” Wayne Dumais, Deputy Director for the Office of Test and Evaluation at the Department of Homeland Security (DHS), highlights the broad range of DHS missions and the corresponding infrastructure needs. He advocates for a consortium approach to developing T&E infrastructure for homeland security missions including the DoD, academia, and industry. One of the many interesting challenges Wayne highlights is the construction of tunnels for T&E purposes. He ponders how one constructs these tunnels so that they meet safety standards, but also reflect the operational mission.

In the article, “Directed Energy Wind Tunnels,” the authors reinforce the need for leveraging modeling and simulation as part of an integrated test program. This article reinforces the themes in the article on LVC for testing. It shows the advantage of using a combination of computational fluid dynamic simulations, wind tunnel testing, and flight test. Their work is motivated by the important role that directed energy will play in future US Air Force aircraft. They highlight that these results should provide a rational for incorporating models, simulations, and ground test into integrated test and evaluation strategies for future programs of record.

In the final article, Bryan S. Kelly, Lt. Col. USAF, highlights the need for automated regression testing for aircraft implementing agile software development. An issue that many of us face is the growing complexity of software on all platforms. Kelly highlights the importance of software quality through historical examples.

He provides a case study for the F-16 Center Display Unit. He advocates for broad Air Force adoption of automated regression testing noting that the current manual test methods are not responsive to the Air Force’s adoption of agile software development.

Enjoy the issue!
Laura Freeman, Ph.D.
Senior Editor