Upcoming Issue Themes for The ITEA Journal

Upcoming Issue Themes for The ITEA Journal

The ITEA Journal of Test and Evaluation offers a forum for sharing knowledge and ideas crucial to our changing T&E workforce. Authors can submit articles for submission with standard review and editing, or for the more lengthy ITEA peer review or referee processes. Please consider writing an article, share this document with coworkers, and provide feedback on the themes.

Please submit your contributions today to journal@itea.org , attention: Publications Chair.

Thank you in advance for your assistance,
Steve “Flash” Gordon, PhD
ITEA Publications Committee Chair 
steve.gordon@gtri.gatech.edu


Authors can submit articles for standard review and editing or for the more lengthy ITEA peer review or referee processes. Please consider writing an article, share this document with coworkers, and provide feedback. Articles of general interest to ITEA members and The ITEA Journal of Test and Evaluation readers are always welcome, and authoring these articles is a great way to contribute to our industry.

Submission

The ITEA Journal offers a forum for sharing knowledge and ideas crucial to our changing T&E workforce. Please submit your contributions today to journal@itea.org, attention: Publications Chair. Manuscript guidelines are found at www.itea.org under Share and Publications tabs.  Dr. Steve Gordon (steve.gordon@gtri.gatech.edu)

Articles for The ITEA Journal include specialty features, each 2-3 pages long, and key technical articles:

  • Book Reports for T&E or related topics such as systems engineering, scientific principles, and acquisition.
  • Featured Capability describes unique, innovative capabilities; demonstrates how they support T&E.
  • Historical Perspectives recall how T&E was performed in the past or a significant test or achievement, often based on personal participation in the “old days” of T&E.
  • Tech Notes discusses innovative technology that has potential payoff in T&E applications or could have an impact on how T&E is conducted in the future.
  • Cultivating the T&E Workforce addresses the future of T&E by looking for ways to encourage students to pursue and remain in Science, Technology, Engineering, and Mathematics courses and majors, and for innovations in professional education for the T&E workforce.
  • T&E Range Challenges Related to Fauna, Flora, and Civilian Activities addresses range encroachment. Papers will be accepted for this topic during any issue in order to judge interest. 
  • Scientific Methods in T&E solicits articles on innovations in statistical/mathematical methods for T&E.
  • Peer-Reviewed Articles. Authors may submit articles designated to be peer-reviewed. The process usually takes 30 extra days.  These peer-reviewed articles will be highlighted in The ITEA Journal

Upcoming Themes for The ITEA Journal of Test and Evaluation


Test and Evaluation of Hypersonic Systems (Issue 39-3, September 2018).  Two technological challenges of testing hypersonic systems are to (1) advance hypersonic technology itself, staying ahead of rivals, and (2)  develop countermeasures for use in combat situations when an adversary employs such technology against our systems.   By the very nature of these hypersonic systems, testing and evaluation will have very significant challenges – some yet to be adequately described.   These challenges include having the range space, instrumentation, and data capture systems to conduct the testing live or having the trusted simulations to conduct the testing in a live-virtual-constructive environment.  Secondly, testing the countermeasures on both sides of engagements may be a challenge.  It should be noted that hypersonic speed provides potential improvements in operational flexibility for accomplishing the mission (such as short notice urgent space launch or engaging time-critical targets), diminishing the effectiveness of current detection and countermeasures systems, and potential reductions in operational cost to accomplish the mission.  Because of hypersonics, we will need testing improvements in aerothermodynamics, materials science, hypersonic navigation, guidance and control systems, endo-atmospheric and exo-atmospheric flight dynamics, instrumentation including telemetry, and extremely large data flows.
(Manuscript deadline: June 1, 2018)
           

Advanced Instrumentation and Information Systems Technology for T&E (Issue 39-4, December 2018) Testing requires data and analysis of the data from instrumentation and information systems.  In order to improve these systems, science and technology (S&T) advancements are sought in these areas: Time-Space Position Information (TSPI), Advanced Sensors, Advanced Energy and Power Systems, Non-Intrusive Instrumentation, Range Environmental Encroachment, and Human Performance Measurement and Assessment.  For this theme, articles related to these S&T areas are requested, and articles describing advancements in real-time data analytics, automated analysis, and automated test reporting are also welcomed.  Advanced instrumentation and information systems could produce an overwhelming amount of data from tests, causing a sense of drowning in data — absent advances in data analytics.
(Manuscript deadline: September 1, 2018)

Statistical Methods in T&E (Issue 40-1, March 2019).  Mathematical and statistical methods have traditionally been used in testing.  Some new approaches in using statistical methods provide new tools to allow testers to estimate how much testing is enough.  Too little testing and too much testing waste money.  Application of statistical methods coupled with disciplined up-front analyses, may help increase the scientific underpinning of tests.  Up-front analyses include following accepted processes to determine outputs, determining how to measure the outputs accurately, identifying what super-set of inputs may affect the outputs, determining the critical few inputs and how to measure the inputs accurately, setting statistical confidence, evaluating the design for statistical power, and making trade-offs.  How are designation of standard operating procedures and conducting measurement system analyses related to reducing unnecessary noise and maintaining statistical power?  Would upfront analyses also include verification and validation of requirements and verification, validation, and accreditation of modeling and simulation to support the test?  What are the consequences of unnecessary noise in the systems that support testing?  How can we increase statistical power of tests without increasing testing?
(Manuscript deadline: December 1, 2018)

T&E Range Challenges Related to Flora, Fauna, and Civilian Activities (Issue 40-2, June 2019).  Unless properly handled or accommodated, environmental encroachment can cause limitations to testing and training; these limitations can include sonar energy employment restrictions, communications degradations, range radar interference, reduced land and sea maneuver space, and increased cost.  The science and technology (S&T) research in this area includes advanced marine mammal monitoring, wind turbine mitigation to range radars, endangered species monitoring, monitoring of electromagnetic interference (EMI) from systems like high voltage power lines, and solar power tower effects.  Are there lessons learned or success stories in preventing incompatible development near test and training ranges?  Papers related to these topics, to developing alternative ranges, and to range encroachment issues in general are requested.  Also, papers that discuss ways to complement range testing with gaming or simulation are welcomed.  T&E challenges related to fauna, flora, and civilian activities will continue to impact test and training ranges, but the limitations can be moderated.  Are there range initiatives that seek to protect flora and fauna and the environment?  Best practices in this area would be of interest to the readers. 
(Manuscript deadline: March 1, 2019)

Drowning in Data: How to Gain Timely Information and Knowledge from Data (Issue 40-3, September 2019).  Digital technology and accelerating improvements to digital technology provide us with the ability to acquire, create, and store data at unprecedented rates and volumes.  Literature searches that would have taken days or weeks in the age of library card catalogs now can be accomplished in seconds over the internet.  Data, or more generally, information, has become big business, in addition to being the business of T&E.  Yet, technological solutions come with their own problems: the immense flow of data from tests has not been met by a commensurate growth in the ability to exploit the data to gain information and knowledge.  This theme examines the issues and the potential solutions for the need to extract meaning and value from the mountains of data.  Articles are invited on such topics as data acquisition, storage, archiving, access, validation, exploitation, and visualization; data as a service; cloud computing; service-oriented architecture; metadata syntax and semantics; instrumentation; accelerating the process of acquiring data to making a decision; data and sensor fusion; data preservation; distributed and nonrelational databases; and related topics. The theme seeks insights, lessons learned, and success stories of gaining information and knowledge in a timely manner from test data.
(Manuscript deadline: June 1, 2019)

The Right Mix of T&E Infrastructure (Issue 40-4, December 2019). Our T&E infrastructure is regularly evaluated for downsizing, improvement, or changes in ownership.  The right footprint of T&E infrastructure depends on the tests in the pipeline and future systems in design.  How do we know what and how much is necessary?  How do we find the right sources of information (who knows?), and how can we search for facilities available nationally and internationally?  Can we share government, industry, and university facilities within and across country boundaries?  Would this type of sharing cause conflict of interest issues? Will overlap of contractor testing, developmental testing, and operational testing reduce or increase the demand on test infrastructure?  Can a shift to earlier developmental testing in representative operational environments and a push for integrated testing reduce the load on test infrastructure?
(Manuscript Deadline: September 1, 2019)

Accelerating T&E to the Speed of Need (Issue 41-1, March 2020). The speed of need is the speed required to transition the system tested from definition of a user need to the initial operation of the capability in the time operationally necessary.  The demands of war have shortened the timeline on requirements for military systems.  Rapid acquisition and rapid fielding initiatives arising from urgent operational needs have created an entire industry in the defense community.  The pace at which transportation security and border protection measures need to be deployed is similarly increased.  We see information technology (IT), especially software and cyber threats and defenses, change with a frequency of months, not years.  Test and evaluation (T&E) must be responsive to the acquisition timelines.  The Federal Aviation Administration, Border Patrol, law enforcement, and many other organizations are adapting to the changing speed of need.  This issue takes a candid look at agile software development processes, cyberspace T&E, defense IT acquisition reform, rapid acquisition and fielding, reconfigurable test capability, testing on demand, reuse, and other ideas for streamlining the T&E process in support of accelerating deployment of new products, services, and capabilities.
(Manuscript Deadline: December 1, 2019)

Systems Engineering and T&E Synchronization (Issue 41-2, June 2020).  These activities are part of an integrated, solid T&E process.  Systems engineering provides the process and tools to build the right effective products in the best way.  Reliability strives to develop a system that is available and suitable for the intended use and resilient to disruption.  Lifecycle support looks at maintainability and supportability with long-term ownership costs in mind.  Testing makes sure these requirements and others are satisfied by the designed and produced system.  Blending these initiatives into an integrated T&E program could help us field the right system for the user.  Invited papers could include discussions of success stories, lessons learned, drawbacks, benefits, good intentions gone awry, and alternative views.
(Manuscript Deadline: March 1, 2020)

Innovations in Modeling and Simulation (M&S) Use for T&E (Issue 41-3, September 2020). M&S allows depiction of places and times where the users of the M&S cannot otherwise go.  These battlespaces and times could be in the past, future, or into areas where enemy defenses or other physical barriers prevent immediate travel.  In light of this powerful benefit of M&S, there are at least two situations where M&S gives the most benefit in T&E.  The first is the reduction of testing cost.  Rather than running flight tests or destroying weapons systems, we simulate the activity in the laboratory to determine if the system satisfies the developmental documentation and performance and if the system meets operational effectiveness and suitability.  We can then verify performance with much fewer flight tests — saving money by running fewer high-cost live tests.  The second is the ability to use M&S to generate operational edge conditions that we cannot create in the live operational environment.  We can simulate operational conditions that would be expensive or impossible to create in a physical operational test. Examples could be testing aircraft in lethal environments or after the departure from safe flight or load testing an air traffic control system with more tracks than would be possible in the live operational environment to prove performance resilience requirements.
(Manuscript Deadline: June 1, 2020)

T&E for Cyber Security and Readiness (Issue 41-4, December 2020).  Key information passed through network connections improves the speed and lethality of combat operations; yet, use of networks opens doors to vulnerabilities.  Network connections for home computers, smart phones, social media, and entertainment add enjoyment; yet, ease of use often equates to increased ease of misuse and scamming.  Systems that support the military, our finances, our health records, and our other personal information must pass risk management, information assurance/security, net readiness, and cyber readiness tests. Yet, these tests, when passed, do not provide 100% assurance of protection.  Systems and the networks that connect them are subject to attacks from many sources; however, the goal of the attack is nearly always to take something valuable.  Money, personal information, trust, freedom, military information and plans, or intellectual property are often taken with very minimal effort and cost.  How much testing is required to provide an acceptable level of risk and enduring protection to expected attacks?
(Manuscript deadline: September 1, 2020)

New Initiatives in Developmental and Integrated T&E (Issue 42-1, March 2021).   These initiatives include, but are not limited to, mission engineering and improved interoperability testing across mission support levels, implementing the Developmental Evaluation Framework (DEF), designation of Chief Developmental Testers, improving reliability T&E, and improving cybersecurity testing.   Systems need to be tested in relevant mission areas and relevant levels of warfare so that the systems perform as intended in the Service, Joint, and Coalition warfare missions in which the system may be employed.  Part of that requirement includes the interoperability to effectively and safely operate in those types of operations, levels of warfare, and battlespaces.  What are the benefits and lessons learned of implementing the DEF to improve DT&E planning and streamline some T&E documentation?  In designating Chief Developmental Testers, how has that initiative improved DT and OT results?  Has the emphasis on designing for reliability planning and reliability growth and improved estimation of reliability resulted in realized improved reliability and risk reduction? How have tabletop exercises and consistent cybersecurity T&E assistance helped improve cybersecurity DT&E?
(Manuscript deadline: December 1, 2020)

Training the Future T&E Workforce (Issue 42-2, June 2021).  Test and evaluation over the next decade will need a workforce of professionals from many academic disciplines.  The academic majors will certainly include science, technology, engineering, and math (STEM); yet, management, communications, psychology, and other types of majors also may be needed for the T&E profession.  We will need a steady supply of the right academic majors from our technical schools, colleges, and universities, and we will need initial training for the incoming workforce to be ready to become T&E professionals.  The need for an inflow of new talent suffers from a constrained supply and competes with many demands for the same disciplines from industry, academia, and other parts of the government.  Increasing the throughput of the right new talent would help considerably.  And, innovative ways to attract the new workforce, provide recurring training to the existing workforce, and fund career enhancement will help T&E retain the workforce needed.
(Manuscript deadline March 1, 2021)

Testing Artificial Intelligence and Collaborative Autonomous Systems (Issue 42-3, September 2021).  For these systems, we must test and train as we fight.  Unmanned and autonomous vehicles fly reconnaissance, target location, and combat sorties.  They work for us.  They fly in combat and public airspaces.  They travel on the ground, in the air and space, on the water, and under water on a variety of missions.  Unmanned and autonomous vehicles may be fully controlled by humans, semi-automated, or fully autonomous individually or in swarms or dissimilar teams.  How can we test the collaborative software that allows these systems to complete many missions, including operating in autonomous swarms.  How have we conducted T&E of these systems in the past?  Are there best practices or lessons learned?  Are there any guidelines for how to test the semi-automated and autonomous behaviors in representative operational environments? How can we test these systems and their “brains” in degraded environments in order to determine operational robustness to combat conditions and to cyber degradations?
(Manuscript deadline: June 1, 2021) 

Success Stories in T&E (Issue 42-4, December 2021).  Are there improved testing regimens under which systems have been evaluated, resulting in lives saved, costs reduced, and/or battles won?   For this issue, we will mostly focus on the high-profile T&E experiences where the testing helped influence product changes that made the product better in terms of cost, safety, or effectiveness.  Feedback from tests at or before the design phase can be implemented with minimal cost to the program and can help improve the cost, suitability, and effectiveness of the system.  Product or system modifications as a result of feedback from tests after the development phase of acquisition may cost more, but the test results later in a program are often essential to building the right product.  This issue is also seeking examples of tests where, for some reason, systems were fielded even though they were lacking in effectiveness, suitability, or resiliency.  So, we would also welcome “Less than Successful Stories in T&E” in this issue as well.  Examples of how things have gone wrong help us understand the importance of making sure things go right.
(Manuscript deadline: September 1, 2021)

Previously-Used Themes

T&E at the Speed of Need (Issue 33-1, March 2012).  The speed of need is the time between definition of a user need and initial operation of the capability.  The demands of war have shortened the timeline on requirements for military systems.  Rapid acquisition and rapid fielding initiatives arising from urgent operational needs have created an entire industry in the defense community.  Commensurately hastened is the pace at which transportation security and border protection measures need to be deployed.  We see information technology (IT), especially software, change with a frequency of months not years.  Test and evaluation (T&E) must be responsive to the acquisition timelines.  The Federal Aviation Administration, Border Patrol, law enforcement, and many other organizations have adapted to the changing speed of need.  This issue takes a candid look at agile software development processes, cyberspace T&E, defense IT acquisition reform, rapid acquisition and fielding, reconfigurable test capability, testing on demand, reuse, and other ideas for streamlining the T&E process in support of accelerating deployment of new products, services, and capabilities.

Drowning in Data, Thirsty for Information (Issue 33-2, June 2012).   Digital technology and Moore’s Law provide us with the ability to acquire, create, and store data at unprecedented rates and volumes.  Literature searches that would have taken days or weeks in the age of library card catalogs can be accomplished in seconds over the internet.  Data, or more generally, information, has become big business, in addition to being the business of T&E.  Yet, technological solutions also come with their own problems and the morass of data has not led to a commensurate growth of knowledge or ability to exploit the data.  This issue examines the plethora of data growing exponentially and the ever critical demand to extract meaning and value from the data.  Articles are invited on such topics as data acquisition, storage, archiving, access, validation, exploitation, and visualization; data as a service; cloud computing; service-oriented architecture; metadata syntax and semantics; instrumentation; accelerating the process from acquiring data to implementing a decision; data and sensor fusion; data preservation; distributed and nonrelational databases, and related topics. 

Strategic Partnering: We are Doing More without More (Issue 33-3, September 2012).   Shrinking budgets and accelerating technology development put ever-increasing pressure on product development and T&E organizations: shorter schedules, fewer personnel, and less access to facilities.  Extreme environments and complex systems add additional demands.  To provide the requisite test capabilities, the T&E community must be agile and responsive, as well as innovative.  The automobile and aircraft industries have long spread liability and gained benefit from seeking standard parts production from independent manufacturers or specific systems development delegated to risk sharing partners.  Strategic partnering takes such forms as outsourcing, reuse, and collaboration; common test and training infrastructure; integrating developmental and operational testing; shared facilities among government, industry, and academia.  Cooperation comes with its own issues: some loss of control, policy or statutory impediments, protection of proprietary rights, and conflicting goals of leadership.  The issue addresses all forms of partnering, allowing testers to do more without having more resources to accomplish their task.

Cultivating the T&E Workforce (Issue 33-4, December 2012).   Test and evaluation are individually professions — not academic disciplines, and, as such, we cannot merely recruit more test and evaluation professionals as needed.  We recruit engineers, physicists, computer scientists, mathematicians, chemists, and other degreed professionals; and we train them in test and evaluation.  As technology changes and systems and instrumentation become more complex, T&E professionals need to continue formal education as well as improve T&E expertise.  In addition, we need to consistently attract young people to the disciplines of science, technology, engineering, and mathematics.  Cultivating the T&E workforce requires asking the question “what should the T&E professional’s background consist of today?”  The internet gives us nearly immediate access to the four W’s – who, what, when, and where – but it cannot provide why and how, which require human reasoning.  Appealing to younger professionals means communicating with them in their preferred mode, such as the social networking medium and recognizing that T&E will evolve as IT drives us in new directions and as younger leaders assume their roles in legacy organizations.  We need to prepare the future workforce for T&E, and prepare T&E for them.  This issue solicits ideas on improving the current workforce and growing the next generation: the use and benefit of certification, such as T&E, modeling and simulation, project management; Science-Technology-Engineering-Mathematics (STEM) initiatives; educating leadership; the role of the Service academies in DoD; internships; and many more topics.                          

T&E Challenges and Issues (Issue 34-1, March 2013).  Challenges are the rule not the exception in test and evaluation (T&E), and innovation is a necessity not a choice.  Budget and schedule constraints are a fixture, but the daunting task facing us is addressing the technological hurdles.  New system complexity increases as technology allows designers to do more in a smaller volume.  Test technology must evolve faster than those systems to be tested, and must produce more sensitive instruments than the systems to be measured.  The number of ranges available and facilities on those ranges continues to diminish, and encroachment on land and spectrum accelerates.  Autonomous and cognitive systems present all new test requirements, and cyberspace testing pushes us beyond thinking outside the box — there is no box.  T&E challenges range from characterizing the required test environment, providing production representative articles for operational testing, and producing adequate threat systems through assimilating emerging technology, distributed testing, and shortage of critical skills, to loss of test facilities, hypersonics, and integrating new with legacy systems.

The Changing Face of Developmental Test and Evaluation (Issue 34-2, June 2013).  Within the Department of Defense, big changes are taking place in developmental test and evaluation. The Office of the Deputy Assistant Secretary of Defense (DASD), Developmental Test and Evaluation (DT&E), was established effective June 23, 2009, “to serve as the principal advisor to the Office of the Secretary of Defense and the Under Secretary of Defense for Acquisition, Technology and Logistics in matters relating to DT&E in defense acquisition programs.” The Air Force has restructured its developmental test centers into the Air Force Test Center. The Army Test and Evaluation Command sought efficiencies by reorganizing and incorporating the functions of its Developmental Test Command into higher headquarters. The US Marine Corps has established a developmental testing organization. After three years of change this issue takes a look at subsequent developments; the impact of the changes on the services, their contractors, and industry; integrated testing, system of systems testing, joint and developmental evaluation, new organizational conflict of interest policy, and related ideas.                                                                  

Truth in Data (Issue 34-3, September 2013). The test and evaluation community is moving to a more rigorous and scientific basis. Vocal support has come from the Office of the Director of Operational Test and Evaluation in the Department of Defense, championing rigor and objectivity in T&E, and the Air Force Flight Test Center has an initiative in statistically defensible T&E. Many organizations have embraced design of experiments for test planning and evaluation; however, there are two caveats: statistics is more than just design of experiments; and statistics can be used to prove anything, even the truth. A scientifically based T&E process is naturally rooted in systems engineering and takes an end-to-end view to testing. Modeling and simulation and distributed testing are common tools, which introduce the requirements for verification, validation, accreditation and certification. Looking beyond T&E, there are commonalities with the operations research/systems analysis community, which uses many of the same fundamental approaches to evaluate mission performance against desired operational outcomes, but during real-world operations. Truth in Data seeks insights into, lessons learned and success stories from scientific rigor in T&E and the quest for more efficient and effective testing. It also investigates common ground with the operations research/systems analysis community.

T&E in the Global Marketplace (Issue 34-4, December 2013). The economy is global with a degree of connectedness which has never existed. Ownership of companies is fluid, moving overseas only to return, resulting in sister organizations distributed around the world. Product development is often spread among many partners to distribute cost and risk, share revenue, and to cultivate markets. Software companies have development partners around the world allowing work to progress 24 hours a day. The interrelationships also present problems, not least of which are multiple time zones, culture, language, units of measure, security and protecting intellectual property. T&E in the Global Marketplace looks outside the confines of the Department of Defense to testing in other government agencies, in industry and universities, and recognizes the “I” in ITEA.

How Much Testing is Enough? (Issue 35-1, March 2014).  How do we right-size tests? Could we test until funding, time remaining, and systems delivered by the program office are depleted?  Perhaps we can use scientific methods to decide when to test more or when to stop testing.  What is often called the frequentist methods of setting statistical confidence and estimating statistical power can provide initial guidelines for sizing of test events.  Bayesian methods may also help us answer these questions. However, other scientific tools may also be needed.  Right-sizing instrumentation suites, gathering no more and no less than the data needed, and analyzing the data as it streams in could help decision-makers know when to stop testing. 

Training the Future T&E Workforce (Issue 35-2, June 2014).  Test and evaluation over the next decade will need a workforce of professionals from many academic disciplines.  The academic majors will certainly include science, technology, engineering, and math (STEM); yet, management, communications, psychology, and other types of majors also may be needed for the T&E profession.  We will need a steady supply of the right academic majors from our technical schools, colleges, and universities, and we will need initial training for the incoming workforce to be ready to become T&E professionals.  The need for an inflow of new talent suffers from a constrained supply and competes with many demands for the same disciplines from industry, academia, and other parts of the government.  Increasing the throughput of the right new talent would help considerably.  And, innovative ways to attract the new workforce, provide recurring training to the existing workforce, and fund career enhancement will help T&E retain the workforce needed.  

Modeling and Simulation Use in T&E (Issue 35-3, September 2014).  Is the full power of modeling and simulation (M&S) used today in T&E?  M&S allows us to go to places that we cannot otherwise go or to use systems that we otherwise cannot use.  Where should we make changes to use less or more M&S in better or different ways?  How can we make sure the M&S is good enough for the purpose intended—that is, are there lessons to be learned about verification, validation, and accreditation of M&S or software in general that we should consider?   How can we make sure the databases are good enough—that is, are there lessons learned about verification, validation, and certification of databases that we should consider?  Have we developed ways to combine live test data with M&S data to provide a more complete model of system performance across a broader operational envelop?  Can use of M&S help Developmental Test determine if the system is ready for low rate initial production? Articles are invited in the areas of technology development, policy, leveraging training venues for T&E, success stories of using M&S, and lessons learned from using M&S.

Unattended Vehicle Testing (Issue 35-4, December 2014).  For these systems, we must test and train as we fight.  Unattended vehicles fly reconnaissance, target location, and combat sorties.  They fly in combat and public airspaces.  They travel on the ground, on the water, and under water on a variety of missions.  Unattended vehicles may be fully controlled by humans, semi-automated, or fully autonomous individually or in dissimilar teams.   How have we conducted T&E of these systems in the past?  Are there best practices or lessons learned?  Are there any guidelines for how to test the semi-automated and autonomous behaviors in representative operational environments? How can we test these systems in degraded environments in order to determine operational robustness to combat conditions and to cyber degradations? 

The Right Mix of T&E Infrastructure (Issue 36-1, March 2015).  Our T&E infrastructure is regularly evaluated for downsizing, improvement, or changes in ownership.  The right footprint of T&E infrastructure depends on tests in the pipeline and future systems in design.  How do we know what and how much is necessary?  Can we share government, industry, and academia facilities within and across country boundaries?  Would this type of sharing cause conflict of interest issues? Will overlap of contractor testing, developmental testing, and operational testing reduce the demand on test infrastructure?  Can a shift earlier for developmental testing in representative operational environments and for integrated testing reduce the load on test infrastructure?  Will sharing (or dual use) of training and test venues help in use of limited infrastructure? 

Test Methodology Rigor (Issue 36-2, June 2015).  The T&E workforce ideally uses the scientific method to plan, conduct, and analyze tests.  Test rigor includes appropriate use of the scientific method, and test rigor is more than statistics and design of experiments.  Test rigor includes following accepted processes to determine outputs that must be measured, how to measure the outputs accurately, what super-set of inputs may affect outputs, how to determine the critical few inputs, how to measure the inputs accurately, how to set the statistical confidence desired and plan the test design for statistical power, and how to make the best trade-offs.  Test methodology rigor would include designation of standard operating procedures and measurement system analysis to reduce noise and maintain statistical power.   Rigor in testing would also include verification and validation of requirements and of modeling and simulation and databases to support tests.

T&E of Information Assurance, Information Security, and Cyber Readiness (Issue 36-3, September 2015).  Key information passed through network connections improves the speed and lethality of combat operations; yet, use of the networks opens the door to vulnerabilities.  Network connections for home computers, smart phones, social media, and entertainment add enjoyment; yet, ease of use often equates to increased ease of misuse and scamming.  Systems that support the military, our finances, our health records, and our other personal information must pass information assurance, information security, net readiness, and cyber readiness tests. Yet, these tests, when passed, do not provide 100% assurance of protection.  Systems and the networks that connect them are subject to all sorts of attacks from all sorts of sources; however, the goal of the attack is nearly always to take something valuable.  Money, personal information, trust, freedom, military information and plans, or intellectual property are often taken with very minimal effort and cost.  How much testing is required to provide an acceptable level of protection to expected attacks?    

Blending Systems Engineering, Reliability, Lifecycle Support, and Testing (Issue 36-4, December 2015).  These activities are part of an integrated, solid T&E process.  Systems engineering provides the process and tools to build the right, effective product in the best way.  Reliability strives to develop a system that is available and suitable for intended use and resilient to disruption.  Lifecycle support looks at maintainability and supportability with long-term ownership costs in mind.  Testing makes sure these requirements and others are satisfied by the system as it is designed and built.  Blending these initiatives and their development targets into an integrated T&E program could help us field the right system for the user.  Invited papers could include discussions of success stories, lessons learned, drawbacks, benefits, good intentions gone awry, and alternative views.  

Leveraging Training and Experimentation Infrastructure and Events for T&E (Issue 37-1, March 2016).  Missions, scenarios, and supporting Live-Virtual-Constructive environments for testing, training, and experimentation have different focus areas. Yet, because of fiscal realities and OPTEMPO concerns, T&E should consider leveraging the future warfighting environments of experimentation and the current operational environments of training, especially since both experimentation and training include warfighting staffs, command and control centers, and joint and coalition warfighters?  T&E can add a good measure of data collection and disciplined analysis to experimentation and training events.  Articles that examine this cross-use of test-training or test-experimentation environments, plans for such sharing, or articles on pros and cons of such sharing are encouraged. 

Inter-Agency and International T&E (Issue 37-2, June 2016).  Improved transportation and communications systems worldwide have impacted our leisure and business activities. The improved connectivity and shared needs across countries, coalitions, and operations have also increased the desire for collaborative testing internationally and inter-agency testing within countries.  Shared T&E environments and scenarios across homeland security, law enforcement, and defense will become commonplace.  Likewise, T&E related to airspace control will include the Federal Aviation Authority, the National Air and Space Agency, homeland security, and defense.  Sharing the international T&E infrastructure and expertise will likely benefit all countries involved.  Articles about T&E of systems intended to operate worldwide and/or in support of missions that cross the boundaries of homeland security, law enforcement, and national defense will be of interest. 

Changes to Developmental and Integrated Testing (Issue 37-3, September 2016).  Smooth handoffs are likely to save time and money and improve quality, suitability, and effectiveness.  This issue seeks articles that cover current and future plans for changes to developmental testing and articles on how developmental and operational/acceptance testing are being integrated to allow smoother handoffs that improve testing, save time and funding, and/or improve the product.  This topic seeks articles beyond the Department of Defense; so, articles from industry, other federal agencies, and international partners will be welcomed.  The theme of this issue could easily be reworded as “How can early testing assist in the transition to most efficiently facilitate full independent testing?’ or “How does final independent testing collaborate with early developmental testing to ensure a smooth transition and effective and efficient testing?”.

Statistical Methods in T&E (Issue 37-4, December 2016).  Mathematical and statistical methods have traditionally been used in testing; yet, some new approaches in statistical methods allow testers to estimate how much testing is enough.  Too little testing and too much testing waste time and money.  Application of statistical methods, coupled with disciplined up-front analyses, may help increase the scientific unpinning of tests.  The up-front tasks include following accepted processes to determine outputs, determining how to measure the outputs accurately, identifying what super-set of inputs may affect the outputs, determining the critical few inputs, setting statistical confidence, evaluating the design for statistical power, and making trade-offs.  Designation of standard operating procedures and conducting measurement system analyses help reduce unnecessary noise and maintain statistical power.  Upfront tasks also include verification and validation of requirements and verification, validation, and accreditation of modeling and simulation to support the test.
Blending Systems Engineering, Lifecycle Support, Reliability, and Testing (Issue 38-1, March 2017). These activities are part of an integrated, solid T&E process.  Systems engineering provides the process and tools to build the right, effective product in the best way.  Reliability strives to develop a system that is available and suitable for intended use and resilient to disruption.  Lifecycle support looks at maintainability and supportability with long-term ownership costs in mind.  Testing makes sure these requirements and others are satisfied by the system as it is designed and built.  Blending these initiatives and their development targets into an integrated T&E program could help us field the right system for the user.  Invited papers could include discussions of success stories, lessons learned, drawbacks, benefits, good intentions gone awry, and alternative views. 

Training the Future T&E Workforce (Issue 38-2, June 2017).  Test and evaluation over the next decade will need a workforce of professionals from many academic disciplines.  The academic majors will certainly include science, technology, engineering, and math (STEM); yet, management, communications, psychology, and other types of majors also may be needed for the T&E profession.  We will need a steady supply of the right academic majors from our technical schools, colleges, and universities, and we will need initial training for the incoming workforce to be ready to become T&E professionals.  The need for an inflow of new talent suffers from a constrained supply and competes with many demands for the same disciplines from industry, academia, and other parts of the government.  Increasing the throughput of the right new talent would help considerably.  And, innovative ways to attract the new workforce, provide recurring training to the existing workforce, and fund career enhancement will help T&E retain the workforce needed.              
T&E of Cyber Security and Readiness (Issue 38-3, September 2017).  Key information passed through network connections improves the speed and lethality of combat operations; yet, use of networks opens the door to vulnerabilities.  Network connections for home computers, smart phones, social media, and entertainment add enjoyment; yet, ease of use often equates to increased ease of misuse and scamming.  Systems that support the military, our finances, our health records, and our other personal information must pass information assurance, information security, net readiness, and cyber readiness tests. Yet, these tests, when passed, do not provide 100% assurance of protection.  Systems and the networks that connect them are subject to all sorts of attacks from all sorts of sources; however, the goal of the attack is nearly always to take something valuable.  Money, personal information, trust, freedom, military information and plans, or intellectual property are often taken with very minimal effort and cost.  How much testing is required to provide an acceptable level of protection to expected attacks?  How do we predict future possible attacks?              

T&E for Enhanced Security (Issue 38-4, December 2017).  This issue seeks articles about testing for enhanced security in the land, sea, air, space, and cyber domains.  The theme includes testing by homeland security and law enforcement of systems used to protect people and infrastructure, including water, power, natural gas/petroleum, food, pharmaceuticals, transportation, and communications processing and distribution systems.  The United States National Guard, the Department of Homeland Security, and State homeland security components have unique testing needs and experiences, and articles on these topics are encouraged.  How do the Federal Aviation Authority and the National Aeronautics and Space Administration collaborate on testing the systems that control and ensure the safety of the nation’s airspace?  How are robotic vehicles tested?  Articles from international partners are also encouraged in these listed areas and in the areas of national defense and homeland security.

Testing Using Facilities around the World (Issue 39-1, March 2018).  Where in the world are the valuable, unique T&E facilities and other resources that test planners should be aware of?  Our T&E infrastructure is regularly evaluated for downsizing, improvement, or changes in ownership.  The T&E ranges worldwide have challenges related to fauna, flora, and civilian activities.  Obviously, the right footprint of T&E infrastructure depends on the tests in the pipeline and future systems in design.  How do we know what and how much is necessary?  Can we share government, industry, and university facilities within and across country boundaries? And, if we do, what are the risks and rewards?
(Manuscript deadline: December 1, 2017)

Unmanned and Autonomous Vehicle Testing (Issue 39-2, June 2018).  For these systems, we must test and train as we intend to fight.  Unmanned and autonomous vehicles fly reconnaissance, target detection, and combat sorties.  They fly in combat and public airspaces.  They travel on the ground, in the air and space, and on and under the water on a variety of missions.  Unmanned and autonomous vehicles may be fully controlled by humans, semi-automated, or fully autonomous individually or in dissimilar teams.  They have been demonstrated to operate in autonomous swarms.  How have we conducted T&E of these systems in the past?  Are there best practices or lessons learned?  Are there any guidelines for how to test the behaviors in representative operational environments?  How can we test these systems in degraded environments in order to determine operational robustness to combat conditions and to cyber and other degradations?
(Manuscript deadline: March 1, 2018)