ITEA Journal Issue Abstracts

ITEA Journal Issue Abstracts

ONLINE Issues (ITEA Members Only)

Not a Member? Sample Articles Below:

Building Trust through Testing by Flournoy, Haines, and Chefitz

ITEA is going online! As part of our shift to an online Journal we will be making select articles available to the general public.  The December 2020 featured column is by Michèle Flournoy, Avril Haines, and Gabrielle Chetitz.  The column provides an executive summary of a recent report the three authored entitled, “Building Trust through Testing.”  The summary highlights how new technologies such as machine learning (ML) and artificial intelligence (AI) require new testing approaches.  They make 11 recommendations providing a useful framework to pave the way for future Test and Evaluation methods for ML/AI.

Interested in more great articles like this, join ITEA for only $95 or contribute articles to the Journal – all authors get a full copy of the issue in which their article appears.

https://www.itea.org/wp-content/uploads/2021/02/ITEA_Journal_Dec20-Building-Trust-through-Testing.pdf


June 2021 – Training the Future T&E Workforce

The ITEA Journal of Test and Evaluation - Vol 42 No 2, June 2021 Cover

Before we launch into The Issue at a Glance, I would like to take the time to thank our associate editors and publication team at The ITEA Journal.  Associate editors provide recommendations for articles, conduct interviews, and keep ITEA’s content relevant to the Test and Evaluation (T&E) profession.  We are currently looking for new associate editors. If you are interested in joining our team, or would like to learn more about what is involved, please let me know at journal@itea.org.  

This month in our Conversations with Experts, J. Michael Barton, Ph.D., talks with Pete Christensen.  Pete reflects on a career in the DHS and DoD.  His advice to us all is, “Stick with It and Volunteer for ­Everything.”

This issue, I am especially excited about our Workforce of the Future column.  Since this is a workforce issue, we are launching this new column, but it will remain a fixture for future issues. Our goal is to capture more papers from students (our future workforce) in The ITEA Journal.  These papers could come from internship experiences, research projects, or experiential learning projects.  The only requirement is that a student should be the lead author on papers in this column.  This issue’s column features articles written by interns from the High Performance Computing Modernization Program (HPCMP) Internship Program.  This program supports undergraduate and graduate student summer internship opportunities at DoD laboratories and test centers where students learn high performance computing skills.  Virginia To provides a great introduction to this column. In “Future with Computing,” Virginia provides an overview of the DoD HPCMP Internship Program.  The three articles that follow all come from last year’s summer internship program and include:
• James Hughes, Selena Hamilton, and Virginia To, “Immersive Environments for Visual Analytics”
• Aiden Kenny, Mariya Occorso, and Vincent Perry, “Visual Analytic Application Toolset for Large-Scale Data”
• Sage Leone, Jaime Infantolino, Adam Childs, and Cleon Anderson, “Topic Modeling and Visualization for Test Incident Report Data”

In our first technical article, “How to Train Your Space Tester: Big Picture Challenges Facing Space Test Training,” Michael Nayak, Ph.D., Christina Straight, Evelyn Kent, and Jarred Langhals discuss four of the big challenges that need to be addressed for a possible future “Space Test Pilot School,” and their impact on US Space Force operators.

In our second technical article, William Rowell, Ph.D., Steven Oimoen, Ph.D., Leonard Truett, Ph.D., and Timothy Shelton discuss, “Applying Scientific Test and Analysis Techniques in the Agile/DevSecOps T&E Environment.” The article provides guidance and illustrative examples.  

In technical article three, Vincent Perry, Mariya Occorso, Aiden Kenny, and Dale Shires discuss “Integrating Analysis and Visualization for Heterogeneous T&E Data.”  Their work is motivated by the convergence of autonomous systems and big data. 

In “Test in the 21st Century: Enabling a Digital Workforce to Support Tomorrow’s T&E Needs,” Alexander Hillman and Rebeka Melber discuss the need for a data savvy workforce in this new digital era.  They highlight the need for leadership support in developing, recruiting, and educating the evolving T&E workforce. 

In our final article for this workforce issue, Shokoufeh Mirzaei, Ph.D., Erich Brownlow, James Brownlow, Ph.D., and Robert Poulson, Ph.D. discuss new methods for modeling tracking errors in “Stochastics Differential Equation Modeling of Target Tracking Errors.”  They showcase how stochastic differential equations offer a solution to modeling systems whose output is autocorrelated and also converges to a specific value over time.


March 2021 – New Initiatives in Developmental and Integrated T&E

This month’s issue focuses on new initiatives in developmental and integrated T&E.  In this month’s Interview with Expert’s column, J. Michael Barton, Ph.D. talks to Matthew Reynolds, a founding member of ITEA in 1980! Matthew Reynolds discusses how T&E has changed over the course of his career to emerge into a discipline distinct from systems engineering.  He highlights how ITEA shaped his career and is an excellent organization for workforce development and sharing of ideas, tools, and best practices.  He concludes noting it is truly a golden age for T&E!  This column is a great reflection on the rewards of a career in T&E.
 
We have a book review this month from Mark ­London, Ph.D. on Doing Bayesian Analysis, 2nd Edition by John R. Kruschke. Dr. London provides great insights into how the book might be useful to practitioners both by providing an accessible introduction to Bayesian analyses and providing numerous examples with code. 
 
Our first article comes from Arjuna (AJ) ­Pathmanathan, Director of the National Cyber Range Complex (NCRC).  AJ notes, “the NCRC provides Department of Defense (DoD) and other government and industry users with representative environments, a distributed network infrastructure, and other tools and services to support cybersecurity testing and evaluation, research and development, mission rehearsal, and workforce training events.” The article summarizes the origins of the NCRC and its evolution to the capability it provides today, which includes complex cybersecurity testing in real-world scenarios to include dynamic input from users. 
 
Our next article is a combined effort from Air Force Test Center (AFTC) and Air Force Operational Test and Evaluation Center (AFOTEC).  The article highlights the operational need to rethink the contractor/developmental/operational testing sequence to meet the need of modernization and new acquisition strategies designed to accelerate weapon systems development. Colonel Keith M. Roessig, Vice Commander of the AFTC, and Colonel Matthew T. Magness, Vice Commander of the AFOTEC illustrate the effectiveness of a Combined Test Force (CTF) under one chain of leadership and how it benefited the Ground Based Strategic Deterrent (GBSD) test program. They highlight that by “capitalizing on unity of command, the GBSD CTF will promote real-time combined test planning, execution, and reporting to meet developmental and operational test objectives.” This article provides a great example for potential future test force structures. 
 
In our next article, authors from the UKL Ministry of Defence, Defence and Science Technology Laboratory (Victoria va der Byl, Richard Cawthorne, Brian Gillett, and John O’Hara) describe their innovative approach for maximizing the impact of an industry day by capturing data.  Their goal is to facilitate the future of UK T&E capabilities. They designed a unique approach that leverages data captures coupled with the structured brainstorming technique, world café. The results allowed them to clearly capture in numerous charts the challenges, capabilities, and investment needs for the future of T&E.  Notably, they also captured non-technology challenges.  The article is a great illustration of how government organizations can bring clarity to the vast amount of input provided by industry days.
 
Tom Roltsch’s article on the sequential probability ratio test (SPRT) introduces a Bayesian approach for designing reliability tests.  The SPRT and the Bayesian SPRT are used to estimate the mean time between failures, 50th percentile (MTBF50), and stopping rules for testing given the MTBF50 is significantly above or below a threshold level. Tom provides an intuitive explanation of the test, the mathematical basis, and a great example illustrating the stopping rules.  These types of methods provide new ways of determining how much testing is enough in an integrated test paradigm.  
 
Our final article comes from a team from the Institute for Defense Analyses. Matthew Avery, Ph.D., Jim Simpson, Ph.D., and Heather Wojton, Ph.D. examine the question of “how much testing is enough?”  They compare and contrast current challenges and capabilities of the T&E community with the report that resulted from a joint Military Operations Research Society (MORS) and the International Test and Evaluation Association (ITEA) symposium in 1994. The paper serves as a great reference for publications in T&E, (citing 68 publications!), that tackle changes in statistical methods, modeling and simulation, pooling data, and costs and benefits.  The article is a great summary of what has changed and what challenges still remain.  The final section highlights new challenges in T&E to include software intensive systems, cybersecurity, ­artificial intelligence, and autonomy.  A larger analysis is available directly from IDA.
 
 I would like to thank all of this month’s authors for their excellent contributions and encourage the ITEA readership to contribute to our upcoming issues.  In June, we will cover “Training the Future T&E Workforce,” September should be an exciting issue where we will tackle “Testing Artificial Intelligence and Collaborative Autonomous Systems,” and December gives us the opportunity to show off our best practices and accomplishments with “Success Stories in T&E.”  Please consider contributing your work to one of 
these issues.
 
Enjoy the issue!