ITEA Journal Issue Abstracts

ITEA Journal Issue Abstracts

ONLINE Issues (ITEA Members Only)

Not a Member? Sample Articles Below:

Building Trust through Testing by Flournoy, Haines, and Chefitz

ITEA is going online! As part of our shift to an online Journal we will be making select articles available to the general public.  The December 2020 featured column is by Michèle Flournoy, Avril Haines, and Gabrielle Chetitz.  The column provides an executive summary of a recent report the three authored entitled, “Building Trust through Testing.”  The summary highlights how new technologies such as machine learning (ML) and artificial intelligence (AI) require new testing approaches.  They make 11 recommendations providing a useful framework to pave the way for future Test and Evaluation methods for ML/AI.

Interested in more great articles like this, join ITEA for only $95 or contribute articles to the Journal – all authors get a full copy of the issue in which their article appears.

March 2021 – New Initiatives in Developmental and Integrated T&E

This month’s issue focuses on new initiatives in developmental and integrated T&E.  In this month’s Interview with Expert’s column, J. Michael Barton, Ph.D. talks to Matthew Reynolds, a founding member of ITEA in 1980! Matthew Reynolds discusses how T&E has changed over the course of his career to emerge into a discipline distinct from systems engineering.  He highlights how ITEA shaped his career and is an excellent organization for workforce development and sharing of ideas, tools, and best practices.  He concludes noting it is truly a golden age for T&E!  This column is a great reflection on the rewards of a career in T&E.
We have a book review this month from Mark ­London, Ph.D. on Doing Bayesian Analysis, 2nd Edition by John R. Kruschke. Dr. London provides great insights into how the book might be useful to practitioners both by providing an accessible introduction to Bayesian analyses and providing numerous examples with code. 
Our first article comes from Arjuna (AJ) ­Pathmanathan, Director of the National Cyber Range Complex (NCRC).  AJ notes, “the NCRC provides Department of Defense (DoD) and other government and industry users with representative environments, a distributed network infrastructure, and other tools and services to support cybersecurity testing and evaluation, research and development, mission rehearsal, and workforce training events.” The article summarizes the origins of the NCRC and its evolution to the capability it provides today, which includes complex cybersecurity testing in real-world scenarios to include dynamic input from users. 
Our next article is a combined effort from Air Force Test Center (AFTC) and Air Force Operational Test and Evaluation Center (AFOTEC).  The article highlights the operational need to rethink the contractor/developmental/operational testing sequence to meet the need of modernization and new acquisition strategies designed to accelerate weapon systems development. Colonel Keith M. Roessig, Vice Commander of the AFTC, and Colonel Matthew T. Magness, Vice Commander of the AFOTEC illustrate the effectiveness of a Combined Test Force (CTF) under one chain of leadership and how it benefited the Ground Based Strategic Deterrent (GBSD) test program. They highlight that by “capitalizing on unity of command, the GBSD CTF will promote real-time combined test planning, execution, and reporting to meet developmental and operational test objectives.” This article provides a great example for potential future test force structures. 
In our next article, authors from the UKL Ministry of Defence, Defence and Science Technology Laboratory (Victoria va der Byl, Richard Cawthorne, Brian Gillett, and John O’Hara) describe their innovative approach for maximizing the impact of an industry day by capturing data.  Their goal is to facilitate the future of UK T&E capabilities. They designed a unique approach that leverages data captures coupled with the structured brainstorming technique, world café. The results allowed them to clearly capture in numerous charts the challenges, capabilities, and investment needs for the future of T&E.  Notably, they also captured non-technology challenges.  The article is a great illustration of how government organizations can bring clarity to the vast amount of input provided by industry days.
Tom Roltsch’s article on the sequential probability ratio test (SPRT) introduces a Bayesian approach for designing reliability tests.  The SPRT and the Bayesian SPRT are used to estimate the mean time between failures, 50th percentile (MTBF50), and stopping rules for testing given the MTBF50 is significantly above or below a threshold level. Tom provides an intuitive explanation of the test, the mathematical basis, and a great example illustrating the stopping rules.  These types of methods provide new ways of determining how much testing is enough in an integrated test paradigm.  
Our final article comes from a team from the Institute for Defense Analyses. Matthew Avery, Ph.D., Jim Simpson, Ph.D., and Heather Wojton, Ph.D. examine the question of “how much testing is enough?”  They compare and contrast current challenges and capabilities of the T&E community with the report that resulted from a joint Military Operations Research Society (MORS) and the International Test and Evaluation Association (ITEA) symposium in 1994. The paper serves as a great reference for publications in T&E, (citing 68 publications!), that tackle changes in statistical methods, modeling and simulation, pooling data, and costs and benefits.  The article is a great summary of what has changed and what challenges still remain.  The final section highlights new challenges in T&E to include software intensive systems, cybersecurity, ­artificial intelligence, and autonomy.  A larger analysis is available directly from IDA.
 I would like to thank all of this month’s authors for their excellent contributions and encourage the ITEA readership to contribute to our upcoming issues.  In June, we will cover “Training the Future T&E Workforce,” September should be an exciting issue where we will tackle “Testing Artificial Intelligence and Collaborative Autonomous Systems,” and December gives us the opportunity to show off our best practices and accomplishments with “Success Stories in T&E.”  Please consider contributing your work to one of 
these issues.
Enjoy the issue!