December 2018 – Advanced Instrumentation and Information Systems Technology for T&E

The theme for this issue is “Advanced Instrumentation and Information Systems Technology for Test and Evaluation (T&E),” and the issue includes the President’s Corner, two Remembrances, a Guest Editorial, an Inside the Beltway, the feature “Capturing the Knowledge Base of Test and Evaluation,” the first article that will be part of the ITEA Test and Evaluation Handbook, and seven technical articles.

President’s Corner, written by William Keegan, President of the Board of Directors for ITEA, discusses the successes for ITEA during 2018, upcoming events, news about ITEA, and key themes, features, and articles in this issue of the The ITEA Journal. He ends his feature with best wishes for 2019.

For this issue, with great sadness, we honor two great supporters of ITEA. Stu Burley was a Life Member of ITEA, founder and lifeblood of the ITEA Mid-Pacific Chapter, and a continuous supporter of holding ITEA events in Kauai. When our 2019 Symposium is held in Kauai, we can again honor him. Austin Vick literally opened his home to ITEA event attendees. He was a founding member of the Roadrunner and White Sands Chapters, supporting ITEA Workshops in Las Cruces and El Paso. He designed and built custom homes,
and, for Workshops in Las Cruces, he opened his home for us to host receptions. Both of these great men built a legacy of volunteering at educational institutions in their communities, mentoring future scientists and engineers, and sponsoring scholarships. From their large extended families, the students these two great men mentored, and the significant voluntary assistance they gave to growing ITEA, Stu Burley and Austin Vick are remembered and dearly missed.

Our Guest Editorial, written by Bernard “Chip” Ferguson, is “The Test Resource Management Center Vision for Cyber Test and Evaluation in the Future,” and he discusses the mandate and goal to build offensive and defensive cyber operations capabilities. With these capabilities comes the need to also build the test infrastructure for development and operational T&E to meet the demands of successful cyber operations.

For the Inside the Beltway feature “A Groundswell for Test and Evaluation,” Laura Freeman, PhD predicts a new era for T&E. She states that it is easier than ever before to publish and share tools that facilitate T&E analyses and that advances in data science and statistics are greatly increasing the inclusions of subject matter expertise in test planning and data extraction. These improvements will help decision makers manage risk and evaluate systems and capabilities.

The next two features are the first steps to develop an ITEA Test and Evaluation Handbook. The feature “Capturing the Knowledge Base of Test and Evaluation” is provided as an overview of the motivation for developing the handbook, the goal of including draft sections of the handbook in each issue of The ITEA Journal, and the draft topics expected for the handbook. This feature is a call for help: what topics are missing, who might volunteer to draft one of the topics or a portion of the topics, and what topics should be modified or otherwise expanded? The second feature related to the handbook is “Ethical Issues in Test and Evaluation” written by Matt Reynolds. The topic of this article is listed under subheading of “Reporting” near the end of the outline of the handbook. Readers can suggest edits/additions to this article or to the draft outline of the handbook at CTEP-handbook@itea.org.

Our first of seven technical articles, “A New Way to Justify Test and Evaluation Infrastructure Investments,” written by Pull Piscopo and Patricia Bronson, Ph.D., summarizes the Institute for Defense Analyses methodology that was used to successfully justify and secure full funding for a proposed infrastructure investment augmentation. The authors propose areas where the methodology, initially used for defense-related hypersonic systems development, could be reused for other future T&E infrastructure needs.

The second technical article in this issue, “The Reliability Metric, Mean Time Between Service Interruption, and Reliability, Availability, and Maintainability Testing of Information Technology Systems” written by Tom Roltsch, states that acquiring Information Technology (IT) systems that have high reliability and maintainability will result in two benefits: highly available IT services and low maintenance costs. The author provides options for the models used to get good estimates of failure rates and repair times.

Our next article, “Use-Inspired Hardware and Software Convergence for Immersive Visualization and Analytics” by J. Michael Barton, PhD et al., proposes a goal of a use-inspired, user centric, and data flow oriented visual analytics capability. The team’s research explores new techniques, algorithms, and technologies that are usable, sustainable, and applicable over the life cycle of the project/system. The article provides use case results and future goals.

In our fourth technical article, “Design of Experiments for Operational Cybersecurity Test and Evaluation,” written by Alex Hoover, he summarizes how the Design of Experiments (DOE) process can be applied to T&E of operational cybersecurity systems. He states that even if cybersecurity does have domain-unique concerns, DOE tools are readily applicable. He explains how leveraging standard DOE can reduce the cyber test space.

For the fifth article in this issue, “The United States Coast Guard Icebreaker Program Shifting Left to Ensure Success,” Steve Hutchison, PhD et al., present a paper highlighting international cooperation between Canada and the United States to thoroughly examine designs for the next generation of heavy polar icebreakers. The modeling and research produced remarkable results to help manage costs and risks, shift acquisition schedules left, influence future icebreaker hull designs, and ensure needed operational capability in the Polar Regions for both nations.

In the sixth article, George Hollenbeck and Javier Villafañe in “Model-Based Test Architecture: An Abstraction to Improve Reality” explain the concept of Test Architecture (TA) and discuss how it is defined and used. One advantage of a TA is the ability to manage linked information in a searchable tool; this, in turn, allows the test architect to create structured test models for all levels of decomposition of the system. These methods align with model-based systems engineering, and the test architecture model supports internal and external process reviews.

The last technical article, “Progress in Migration of Flight Test Analysis Routines to Python,” authored by John C. Bretz, highlights how flight test can be analyzed in the Python open-source software suite. The paper summarizes the evaluation and summary of findings for comparing Python use to use of previous traditional flight test software analysis choices. His paper summarizes several ITEA Workshop presentations that were very well attended, and it describes Python modules and commands that can be helpful to preforming flight test data processing, analysis, and presentation of results.

I hope you enjoy this fourth issue of 2018 for The ITEA Journal of Test and Evaluation. By the time you receive issue 39-4 in December, the March 2019 issue 40-1 is being finalized. The theme in March will be “Statistical Methods in T&E.” For the next issue (the second issue of 2019), 40-2, the deadline for submissions is just after March 1, 2019, and the theme will be “Aligning Modernization of DoD Test Ranges with National Defense Strategy.” We have posted all themes and descriptions for the remainder of 2020-2021 on the ITEA website. Please provide feedback on the choice of themes, and please write early and often.

September 2018 – Test and Evaluation of Hypersonic Systems

The theme for this issue is “Test and Evaluation of Hypersonic Systems,” and the issue includes the President’s Corner, the feature “Testing Hypersonic Systems 101,” “Walt Hollis Remembered,” and seven technical articles.

President’s Corner, written by William Keegan, President of the Board of Directors for ITEA, discusses upcoming events, news about ITEA, and key themes, features, and articles in this issue of the The ITEA Journal.

The editorial “Testing Hypersonic Systems 101” is provided as an overview of the motivations to develop hypersonic systems and the requirements for test and evaluation research and development necessary to field safe and effective hypersonic systems for travel, cargo, and military purposes.

Our first of seven technical articles, “Navigating the Testing of Hypersonics in the 21st Century,” written by Elizabeth Hill and Greg DeVogel, discusses the gaps in developmental and operational testing capabilities required for hypersonic systems. The authors propose areas where testing hypersonic systems may require changes to processes or additional test assets.

The second technical article in this issue, “Interface Management in Systems Engineering” written by George Gardner, Ph.D., describes the activity of interface management—particularly in the practice of systems engineering. He compares the naming conventions and the interface management tools used across some federal agencies to ensure smooth integration, improve interoperability, and provide easier and cheaper technical enhancement. This is a reprint to correct errors in a previous printing of this article.

Our next article, “Defining Resiliency” by Sarah Pramanik, Ph.D., proposes a framework for defining resiliency in a way that facilitates how a system is designed, implemented, and tested. She suggests that the definition of resiliency is a necessary first step, and she proposes a resilient system is able to recover to an acceptable level of performance once it has been under duress. The author also discusses examples, such as the story of the A-10 aircraft’s resiliency to combat damage.

In our fourth technical article, “Modernizing Test and Evaluation with High Performance Computing and Advanced Networks,” is written by J. Michael Barton, Ph.D., et al., and the authors summarize progress to date for the High Performance Computing Modernization Program. The authors look at the future of computing, high performance computing, and advanced networking.

In the fifth technical article, “One War Training System,” Matt Morrisset discusses the emerging need for centralized data storage at the tactical training ranges. He discusses the types of information being recorded during a mission and how a centralized data store makes sense.

For the sixth article in this issue, “Independent Verification and Validation (IV&V),” David Thrasher and Jessica Pippard present tools that can be created to assist in performing IV&V. They suggest tools and methods that assist in examining the source code and that help run the executables. These tools assist the development team to improve the final product.

In the seventh and last technical article, Jeff Kalibjian authored “Overcoming Data Security Challenges in Testing Environments” to highlight the fact that data stored (at rest) is often more at risk than data in motion (in transmission). The author explains that once the data are post processed they may be more at risk. He offers some alternative data security solutions.

I hope you enjoy this third issue of 2018 for The ITEA Journal of Test and Evaluation. By the time you receive issue 39-3 in September, the December 2018 issue 39-4 is being finalized. That theme in December will be “Advanced Instrumentation and Information Systems Technology for T&E.” For the next issue (the first issue of 2019), 40-1, the deadline for submissions is just after December 1, 2018, and the theme will be “Statistical Methods in T&E.” We have posted all themes and descriptions for the remainder of 2019 – 2021 on the ITEA website. Please provide feedback on the choice of themes, and please write early and often.

June 2018 – Unmanned and Autonomous Vehicle Testing

The theme for this issue is “Unmanned and Autonomous Vehicle Testing,” and the issue includes two Guest Editorials, an Inside the Beltway feature, a Cultivating the T&E Workforce feature, the President’s Corner, and six technical articles.

The first of our two Guest Editorials is written by Major General Matthew Molloy. His topic is “T&E in a Time of Risk and Change – Answering the Call: Prototyping, Experimentation, and Acquisition at the Pace of Need.” He states that expediting the fielding of critical warfighting capabilities is essential. We need to find a way to unbridle our processes to allow for serendipity, experimenting, falling forward, pivoting with technology, prototyping, and delivering a center-mass product with clear growth potential.

Our second Guest Editorial, written by Wilson Felder, Ph.D., is “Systems Engineering and Test and Evaluation: a Professional Society Collaboration.” This feature provides new information concerning an initiative to create a roadmap for the growth and development of systems engineering. Several professional societies, including ITEA, have been invited to collaborate in this initiative with the International Council on Systems Engineering (INCOSE). Readers are asked to provide feedback on four possible mechanisms to achieve the desired goals.

Our Inside the Beltway article is titled, “A Conversation with The Honorable Bob Behler, Director, Operational Test and Evaluation (DOT&E).” Director Behler was interviewed by Eileen Bjorkman, Ph.D., a member of the Publications Committee of The ITEA Journal of Test and Evaluation. Director Behler discussed his background and path to becoming DOT&E, and he discussed his desire to bring attention to the growth in the dependence on software and the need to secure it. Director Behler also explained his focus areas and priorities.

The President’s Corner, written by William Keegan, President of the Board of Directors for ITEA, discusses upcoming events, news about ITEA, and key themes, features, and articles in this issue of The ITEA Journal. As a Cultivating the T&E Workforce feature, we’ve included a short “2018 Update on Simulation: Helping Develop the Future Workforce” as an initiative to motivate high school students toward the sciences and engineering. The new national program for certification in Modeling and Simulation (M&S) has increased the number of certifications in Florida and now has been adopted by Virginia. To date, 280 high school students and teachers have earned the M&S Certification, and many of these students are employed in the M&S industry.

Our first of six technical articles, “The Future of Autonomous Ground and Surface Systems Testing,” written by Camille Robbins and Melissa Steffen, Ph.D., discusses the new challenges that require fundamental changes in testing for autonomous systems. Testing sensors and decision-making logic in all possible operational environments are likely not possible. Range and instrumentation improvements are the beginning to improve the ability to more fully test autonomous systems.

The second technical article in this issue, “Overcoming the Challenges in Test and Evaluation of Autonomous Robotic Systems” written by Darryl Ahner, Ph.D., et al., describe eight autonomous system test challenges faced by the test and evaluation community. These challenges span from test adequacy to post-acceptance testing. The authors conclude with five challenges that require test research funding.

For our next article, “A Bioinformatics Approach to Finding Candidate Organism Strains for True Positive Detection Testing of Biological Warfare Agent Detectors,” by Felicia Ellis, et al., proposes a method to identify and use candidate biological agent strains that are less likely to be detected, thus providing a more rigorous test of the detectors. The authors point out that this goal is meant to add realism to the testing of the detectors and to make the controlled lab testing environment more realistic compared to what field detection would encounter if it were feasible. The search for candidates continues.

In our fourth technical article, “Applying the Gaps Model of the Service Industry: Five Ways that Prevent Test & Evaluation from Meeting Customer Expectations,” written by Allan Alfafara, CTEP, Test and Evaluation (T&E) is compared to being a service organization. The author uses the Gaps Model of Service quality to illustrate how T&E as a service might fail to satisfy customers. He presents solutions that might be needed to help recover the good standing of T&E and prevent future unwanted gaps in the services provided by T&E.

In the fifth technical article, “Estimating Operational Resilience in a Cyber Contested Environment,” Paul Johnson discusses a methodology to highlight the complicated problem of estimating operational resilience based on the portfolio of cybersecurity. The author presents a method for estimating operational resilience by using data from a variety of tests and by using a variety of inputs.

In the sixth and last technical article, Angus McLean, Ph.D., et al., authored “LVC-Enabled Testbed for Autonomous System Testing” to highlight how the test requirements for autonomous systems differ from piloted aircraft and to propose a Live, Virtual, Constructive (LVC) environment solution. The authors explain how the LVC environment can be improved using real data, and they discuss the advantages of the LVC approach.

I hope you enjoy this second issue of 2018 for The ITEA Journal of Test and Evaluation. By the time you receive issue 39-2 in June, the September 2018 issue 39-3 is being finalized. That theme in September will be “Test and Evaluation of Hypersonic Systems.” For the next issue (the fourth issue of 2018), 39-4, the deadline for submissions is just after September 1, 2018, and the theme will be “Advanced Instrumentation and Information Systems Technology for T&E.” We have posted all themes and descriptions for the remainder of 2018 – 2021 on the ITEA website. Please provide feedback on the choice of themes, and please write early and often.

March 2018 – Testing Using Facilities Around the World

The theme for this issue is “Testing Using Facilities Around the World,” and the issue includes two Guest Editorials, an Inside the Beltway feature, the President’s Corner, and five technical articles.

Our first Guest Editorial, written by Keith Joiner, PhD, is “International Test and Evaluation (T&E): Looking Towards the United States Not Outward from the United States.” This feature provides a unique description of international T&E looking from a perspective external to the United States. The author also  provides information on the areas where Australia has parts to play in international T&E. He foresees the need for allies of the United States to find ways to resynchronize T&E capabilities with the United States.

Our second Guest Editorial, written by Fred Moorefield, is “Modernizing DoD’s Spectrum Technology Test and Evaluation Capabilities to Meet Growing Military Requirements.” In this feature, he states that achieving electromagnetic spectrum (EMS) superiority provides important tactical, operational, and strategic advantages. This goal of EMS superiority will be paramount  in future military operations. He concludes that new EMS technology development will require new T&E capabilities to understand impacts and interoperability capabilities in the electromagnetic environment.

Our Inside the Beltway article is titled “Reciprocally Using International Test Facilities Under the Director, Operational Test and Evaluation (DOT&E) Test and Evaluation Program (TEP).” Gloria Deane and Mitchell Dossett state the two fundamental considerations of the TEP: that adequate testing is necessary to ensure that systems work as expected for warfighters and that  T&E cooperation with allies improves interoperability in coalition operations. The authors explain the unique TEP Reciprocal Use of Test Facilities project agreement, and they state that TEP agreements have contributed to improved interoperability in the coalition warfare.

President’s Corner, written by William Keegan, President of the Board of Directors for ITEA, discusses upcoming events, news  about ITEA, and key themes, features, and articles in The Journal.

Our first of five technical articles, “National Advanced Spectrum and Communications Test Network (NASCTN): A Framework for Leveraging T&E Facilities,” written by Keith Hartley, et al., discusses the need for a collaborative and flexible spectrum-sharing network. This network is needed because the growth of broadband wireless services continues worldwide, and the crowded  spectrum environment drives regulators to look increasingly at spectrum sharing. The authors explain that NASCTN was designed  and chartered as a cooperative agreement to quickly unite expertise, equipment, facilities, and resources to support spectrum-sharing test measurements. Expanding wireless services across government and commercial users will require increased  spectrum access and technologies, and NASCTN is a model for expansion and replication to meet future needs.

The second technical article in this issue, “Next Generation Intelligent Test Network Environments” written by J. Michael Barton, Ph.D., and Venkat Dasari, Ph.D., describes many developments that are leading to a seemingly overwhelming flood of data to be evaluated for tests. For instance, the growth of networked systems of systems, an array of advanced instrumentation  providing data at enormous aggregate rates, and testers needing near-real-time feedback about the state of tests as they occur, in combination, may require intelligent test networks to manage the bandwidth and sequence the computations to smart instrumentation. The authors state that current technologies are available to field intelligent instrumentation and programmable networks to process the growing flow of data from tests.

For our next article, “Estimating the Effect of Avionics Complexity on Safety Assurance” by Sarah Sheard, et al., the authors  review the research to date from the Federal Aviation Administration’s Complexity and Safety Project. They state that, even with the problems associated with complexity, we must find ways to regain the ability to predict and control systems. The team  created and tested a formula for complexity that can be applied early in system development, and they applied the formula to  three varied test cases.

Our fourth technical article, “Services-Based Testing of Autonomy (SBTA)” is a peer-reviewed article, and Christopher Eaton, et al. suggest a method to test newly incorporated services, enabling a consistent test and evaluation (T&E) verification and validation (V&V) capability, designated TEVV. The authors state that the Services-Based Testing of Autonomy will provide a cost effective, scalable, and efficient means to test autonomy services, and TEVV will be a key asset as autonomous systems have significant mission growth.

In the fifth and last technical article, “Measuring the Influences of Mission Command,” Robert Pokorny, et al., discuss a means of developing metrics for mission command so that the United States Army can measure the effects of new systems or processes on mission command. This research leveraged a scoring system called Performance Assessment in Complex Environments (PACE) that has been successfully used on other projects in varied domains. Since the effectiveness of mission command enables  mission execution, the developed metrics may be used to evaluate the effectiveness of new technology and may also reduce the operational risk of introduction of new, disruptive processes or technologies. Software that can unobtrusively measure mission  command effectiveness could provide a key missing metric in operational mission testing.

I hope you enjoy this first issue of 2018 for The ITEA Journal of Test and Evaluation. By the time you receive issue 39-1 in  March, the June 2018 issue 39-2 is being finalized. That theme will be “Unmanned and Autonomous Vehicle Testing.” For the next issue (thethird issue of 2018), 39-3, the deadline for submissions is just after June 1, 2018, and the theme will be “Test and Evaluation of Hypersonic Systems.” We have posted all themes and descriptions for the remainder of 2018 – 2021 on the ITEA  website. Please  provide feedback on the choice of themes, and please write early and often.