2020 ITEA Journal Issue Abstracts

2020 ITEA Journal Issue Abstracts

ONLINE Issues (ITEA Members Only)


June 2020 – Systems Engineering and T&E Synchronization

As systems increase in complexity and adopt agile development strategies, testing and evaluation (T&E) provides a meaningful feedback loop to the systems engineering process. Systems engineering provides the process and tools to build the right effective products in the best way. A meaningful feedback loop requires improved synchronization between systems engineering and test and evaluation.

This month’s articles provide summaries of high-level initiatives and technical case studies to achieve System Engineering and T&E synchronization.

This month’s Guest Editorial comes from Major General Michael Brewer and Colonel Matthew Magness from the Air Force Operational Test and Evaluation Center (AFOTEC). It describes how AFOTEC is refining their processes to achieve agile acquisition goals. The Adaptive Relevant Testing (ART) initiative outlines six key principles: 1) early operational test involvement, 2) the ability to tailor to the situation, 3) continuous and cumulative feedback, 4) streamlining process and products, 5) integrated and combined testing, and 6) adaptive testing based on discovery. They adopt a philosophy of testing as a service (TaaS) as a practical application of these principles and highlight the value of this approach.

In our Inside the Beltway, J. Michael Barton, Ph.D. interviews Dr. Steven J. Hutchison, Director for Test and Evaluation in the Department of Homeland Security. Dr. Hutchinson discusses his career in test and evaluation; how his role in T&E has changed over time; how the T&E profession has changed over time; the importance of putting the evaluation goals before the test plan; “shift left”; and importantly for this issue how T&E is a life cycle activity. I enjoyed his perspective immensely, especially his closing remarks on the importance of having an excellent team.

In the T&E Handbook article, Mark A. London, Ph.D. discusses sample size calculations for continuous data. This is the complement to his previous tutorial on sample size for binary data. This primer on sample size calculations provides useful tools for determining optimal test sizes that maximize test resources at allowable risk levels.

Our first technical article, “Integrating Test into a Secure Systems Engineering Process” by William D. Bryant, Ph.D. and Lt. Col. R. Lane Odom, highlights the importance of designing in cyber resilience via Systems Security Engineering (SSE). The authors note how current practices often wait until too late in the system development to start cyber testing activities and fail to incorporate feedback into the system design. They summarize methods and tools for providing cyber assessments to include risk assessments and how testing can serve as a feedback loop.

Terry Murphy and Patrick Kastner provide an informative article on, “Incorporating Test and Evaluation into Acquisition Contracts Improving Acquisition Outcomes” in our second technical article. The article provides an exhaustive list of T&E questions that need to be answered and incorporated into contracts, because as the authors note, “If it is not written into the contract, it will not happen.” This article is a great resource for testers before they review contracting documents.

In our third article, “Model-based Systems Engineering to Support Test-driven Development,” 2d Lt ­Katherine E. Cheney and John M. Colombi, Ph.D. discuss approaches for testing autonomous systems. They develop a 16-step process to accelerate testing for autonomous agents. They apply their process to the testing of a multi-rotor small Unmanned Aircraft System conducting a wide area search mission, starting with the system architecture and mission and concluding with the appropriate system/mission analysis.

In “Under-body Blast Verification and Validation Methodology,” written by Andrew W. Drysdale, et al. from Combat Capabilities Development Command provides an overview of their methodology and tools. The authors highlight that this under-body blast methodology is complicated by the modeling process which involves finite-element modeling, multiple sources of code (both commercial and government), varying granularity, and a variety of input-data sources. They note that simultaneously full system testing is limited and also subject to sources of variability.

Our fifth technical article written by Ian Levitt, et al. from the Federal Aviation Administration (FAA) is titled, “Research, Development, Test & Evaluation (RDT&E) in the Trajectory Based Operations (TBO) Integrated Test Environment (TITE).” The authors discuss the need for testing that can be conducted holistically (e.g., end-to-end, system-of-systems) and the integration of T&E activities throughout the acquisition life cycle. The article provides an overview of their goals and how public/private partnership enabling industry-differentiating live, virtual, constructive environment will enable both test and evaluation objectives.

In our final article, “Exploring Agile Acquisition Principles and Practices for FAA Complex System Delivery,” Mindy Dowling, et al. from the FAA explore the concepts of agile acquisition for the FAA in the context of the complex and interconnected systems. They ­highlight the importance of safety in the FAA mission and ponder how agile principles align with the FAA’s values of safety and security. They provide two examples and note the benefits of agile values. They conclude that the incorporation of agile processes into the FAA’s acquisition strategy is an ongoing discussion, but that the goal of speed will not replace the commitment to T&E best practices.

Enjoy the issue!
Laura Freeman, Ph.D.
Senior Editor


March 2020 – The Right Mix of T&E Infrastructure

To accomplish our jobs as Test and Evaluation (T&E) professionals, we need the right mix of infrastructure to support our mission. As our systems evolve and become increasingly complex, the “right mix” of infrastructure must also evolve to keep pace. This issue contains a variety of articles that reflect the diverse infrastructure needs of the T&E community.

In this issue, we are experimenting with a few new types of articles. We have included an historical perspectives article and two short technical articles.

In our historical perspectives article, Christian Gelzer reflects on the longest flight in the X-15. In the first technical short article, Joseph Crossin discusses developing the next generation of radars that will leverage emerging technologies such as 5G wireless networks. In the next technical short article, Guenever Aldrich and Tomasz Wojtaszek discuss the need for collaboration to achieve spectrum sharing. I enjoyed reading these contributions and welcome your feedback on these new article ideas. Our technical articles reflect a wide range of T&E infrastructure needs.

Brian Mork, Col USAF (Ret), Ph.D. from the Air Force Research Lab authored an article on a “Three Dimensional Cyber Risk Model.” He highlights that we need to move beyond two-dimensional risk assessments when an intelligent adversary is involved. He shows how thinking in three dimensions can provide better decision making for design engineers, program managers, and red teams that devise cyber combat theories.

In the next article, Huat Ng and Wayne Devereux from KBR highlight the importance of Live Virtual and Constructive (LVC) environments in test design. Noting the recent and substantial advances in models and simulations made possible by advances in information technology and visualization, they highlight that now is an ideal time to incorporate more models and simulations in test design. They highlight that these methods can model more complex environments than available on operational test ranges and inform conditions for operational testing. Finally, they provide a decision flow chart for leveraging LVC in test design. An electricaloptical/infrared sensor case study reinforces their ideas.

MAJ Kurt Klingensmith and Jon Alt, Ph.D. from the Research and Analysis Center in Monterey contributed an article on “A Systems Architecting and Engineering Approach to Creating Enterprise Data Strategy.” This article highlights how data curation is a critical element of test infrastructure. They provide overall architecture and examples of functional decomposition for various stages of data management to guide the reader to an enterprise data strategy. Themes they highlight that should resonate with ITEA readers are that data is an enterprise asset, leveraging it requires an educated workforce, and that data strategies address people, processes, and technologies.

In the article, “Homeland Security Test & Evaluation Infrastructure: It’s Time to Add to the Mix,” Wayne Dumais, Deputy Director for the Office of Test and Evaluation at the Department of Homeland Security (DHS), highlights the broad range of DHS missions and the corresponding infrastructure needs. He advocates for a consortium approach to developing T&E infrastructure for homeland security missions including the DoD, academia, and industry. One of the many interesting challenges Wayne highlights is the construction of tunnels for T&E purposes. He ponders how one constructs these tunnels so that they meet safety standards, but also reflect the operational mission.

In the article, “Directed Energy Wind Tunnels,” the authors reinforce the need for leveraging modeling and simulation as part of an integrated test program. This article reinforces the themes in the article on LVC for testing. It shows the advantage of using a combination of computational fluid dynamic simulations, wind tunnel testing, and flight test. Their work is motivated by the important role that directed energy will play in future US Air Force aircraft. They highlight that these results should provide a rational for incorporating models, simulations, and ground test into integrated test and evaluation strategies for future programs of record.

In the final article, Bryan S. Kelly, Lt. Col. USAF, highlights the need for automated regression testing for aircraft implementing agile software development. An issue that many of us face is the growing complexity of software on all platforms. Kelly highlights the importance of software quality through historical examples.

He provides a case study for the F-16 Center Display Unit. He advocates for broad Air Force adoption of automated regression testing noting that the current manual test methods are not responsive to the Air Force’s adoption of agile software development.

Enjoy the issue!
Laura Freeman, Ph.D.
Senior Editor