Validation in EW

According to the principles of System Engineering (SE) [5], the process models of the life cycle for a complex HW/SW system may be many and varied, but all resort to Test & Evaluation to generate the evidence base necessary for the completion of the development at component, unit or system level, and the acceptance of the result.

Test & Evaluation process is fundamental support tool for engineers and end users throughout the entire life cycle of an EW product, spanning from proof-of-concept to the end of its operative life.
An optimized design of the Test & Evaluation plan is therefore essential in order to ensure cost-effectiveness in the Verification (activity aimed at determining that the manufacturer has fulfilled all contractual requirements), Validation (activity aimed at determining the validity of the product in the targeted operational environment), and consequently the Acceptance of any EW product by the user.

There are three distinct types of T&E defined in US DoD regulation ([4]):

• Developmental Test and Evaluation (DT&E) includes any testing used to assist the development and maturation of products, product elements, or manufacturing or support processes.
• Operational Test and Evaluation (OT&E) is the field test, under realistic conditions, of any item (or key component) of the equipment for determining its effectiveness and suitability for use in combat by typical military users and the evaluation of the results of such tests.
• Live fire T&E (LFT&E) follows, and, though it offers the highest correspondence to the operational environment, it is limited by its costs.

Planning

The acquisition strategy of EW End Users, such as for instance the UK MoD and US DoD, explicitly includes the agreement of the Test & Evaluation approach with the contractor in a very early stage, and specific groups within their organizations have been appointed with T&E responsibilities and tasks.

Generally, the T&E process is well described in [1], as it consists of a sequence of actions – which are captured by the ITEAP (Integrated Test, Evaluation & Acceptance Plan) or TEMP (Test & Evaluation master plan, [7]) documentation:

• Definition of the T&E strategy in order to achieve final acceptance
• Determine the V&V requirements
• Define Test objectives, as well as measures of performance and test criteria
• Planning of the tests, including the necessary resources
• Execution of tests
• Evaluation of the results
• Iteration as per necessity

The results of this operational T&E “loop” will then determine the final acceptance of the system. The Integrated Test, Evaluation & Acceptance Plan (ITEAP) is generally due at the System Design Review of the program, and evolves throughout the design phase.

Measures

EW capability V&V (Verification and Validation) process is based on the definition of tests and measures by which the system performance or effectiveness will be evaluated. These are known as Measures of Performance (MOPs) and Measures of Effectiveness (MOEs).
The MOPs are generally more applicable to DT&E and are generally tied directly to contractual technical performance requirements, while MOEs apply to OT&E.

Considering diverse EW capabilities, some example of MOP/MOE that might be considered in the verification and validation phases are reported in Table 1 (modified from [1]). The list of MOP/MOE is not exhaustive and is provided as reference. It highlights the operational relevance of the measures to be considered in the operational T&E phase.

The determination of MOP/MOE leads to the need for the so-called T&E Capabilities, which are preferably developed by an independent entity. These need to be designed for use in several environments and support diverse EW V&V applications. The key attribute of a T&E capability is by its nature the degree of correctness, accuracy, completeness and repeatability of the information generated. The fulfilment of this “quality” indicator determines the fitness of the T&E capability for its intended use.

As the T&E capability often resorts to Modelling & Simulation techniques, the fitness is intrinsically connected to the credibility of the utilized simulation. Therefore, the Verification, Validation & Accreditation of the simulation [4] is also an important pre-requisite for the entire T&E process outcome.

T&E Capabilities

“A Test and Evaluation (T&E) capability is a combination of facilities, equipment, people, skills and methods, which enable the demonstration, measurement and analysis of the performance of a system and the assessment of the results” ([1]).

Firstly, it is worth highlighting that an EW function can be tested at unit, system or platform level. It is accordingly referred to as Unit Under Test (UUT), System under Test (SUT) or Platform under Test (PUT). A T&E solution might apply to one or more of these configurations.

The connection and interaction between the EW capability and the T&E solution can be implemented in different ways:

Radiation mode: the T&E asset and the System/Platform under Test (SUT/PUT) exchange RF signals on air. The EW system (integrated on the final platform or temporarily deployed on the field) is stimulated by the T&E asset, which generates and transmits E.M. signal scenarios designed in order to realistically simulate the operational conditions of interest for validating the EW function and calculating MOEs.

Direct Injection mode: the UUT/SUT/PUT exchanges signals (RF, digital data, commands …) with the T&E asset via cable. This is the typical configuration for laboratory testing, since radiation might not be permitted for safety reasons and the distance for the synthesis of the far field antenna beam might not be compatible with a laboratory facility.

Different configurations for direct injection exist:

• Stimulating the RF ports – the T&E asset generates and receives signals in the radiofrequency band of the UUT/SUT/PUT. These signals can be captured, routed and injected directly in the antenna ports, bypassing the antenna radiation element.
  This allows accurately testing EW sensors with multiple antennas, e.g. direction finders, since the synthetically generated test signals can be controlled in phase and amplitude in order to simulate the direction of arrival of the emitter radiation. Bypassing the antenna, it is then required to simulate its radiation pattern or transfer characteristic in the test signal itself.
• Interfacing the IF stage – in case of sensors that implement an up-conversion stage from an intermediate frequency, it might be convenient to interface the UUT/SUT at IF level as the electronics of the T&E asset might benefit from the power levels at IF and the availability of commercial items at lower frequencies.
  This configuration might be adequate for developmental test rigs (focused on unit testing) or for providing an integrated “built-in” T&E function within the tactical EW/ISR sensor.
• Exchange of Digital data – Moving the focus only on the digital section of the EW sensor and the constituent SWCIs, the test strategy might rely on the exchange of digital data only.
  This means that the test vector is:
  • Generated digitally as signal descriptor,
  • Converted into the input data format of the EW sensor itself, and
  • Injected into a digital communication interface of the EW sensor (e.g., MILBUS for exchanging data, commands, processing results).
• Direct stimulation of HMI – In specific applications, it might be enough to stimulate the Human to Machine Interface of the EW sensor, and consequently the operator reactions. This requires sequences of commands and predefined datasets as input of the SW front-end, which interprets and displays them as if it were real. In this way, only the interaction of the operator with the EW system is verified. This might be of interest for training applications, which focus on the interactions that might happen in the “virtual reality” seen by the trainee.
• Hood mode: In this configuration, due to the unavailability of accessible RF test interfaces, the test signals are transmitted by probes located in very near range to the SUT/PUT antennas, and the test setup is protected from other interferences by antenna hoods. This allows injecting signals in the SUT without dismounting or modifying any part of it. The drawback is the limited accuracy of the signals that are actually received by the SUT, which might allow only health check and GO/No-Go testing.

T&E Use Cases

Considering the operational use and the life cycle of EW sensors, which are designed for integration on board a specific platform (vehicle, ship, manned aircraft, UAV, missile, projectile), it is possible to identify a list of T&E use cases. The list is not exhaustive but well provides an insight into the variety of applications and goals for testing an EW function:

• Health Testing of Line Replaceable Unit (LRU)
• GO/No-Go testing on the flight line
• Development and Verification of EW Units
• EW sensor characterization
• EMI/EMC, Specialized Testing or Parametric Measurements
• Factory Acceptance of EW System
• Verification of Sensor-on-Platform Integration
• Site Acceptance of EW System
• Training of EW operators
• EW Library generation (also part of EW Operational Support)
• Mission Planning (also part of EW Operational Support)

MODELLING & SIMULATION

Modelling & Simulation represents one of the key capabilities for EW Test & Evaluation. Modelling & simulation is the enabler for moving from physical to virtual T&E ([2]), and attempts at best tradeoff between coverage of the “test space” and V&V costs.

However, in order for using Simulation in the T&E process, it is necessary to achieve enough confidence in the fidelity of the simulation and confirm of its credibility. For this reason, the simulation is expected to undergo a Verification, Validation, and Accreditation (VV&A) process.
The decision to use the simulation will depend on the simulation’s capabilities and correctness, the accuracy of its results, and its usability in the specified application. Specifically, the validation step will ensure that the fidelity of the simulation is adequate for the specific purpose, whereas the accreditation consists of the official certification that it is acceptable for use (credibility), [3].

Trends in T&E

The current acquisition processes of most MODs fit the procurement of hardware-oriented systems, which are potentially produced in series. From [6]: “(…) typical practice is to determine whether or not a design is adequate for its purpose before committing to advancing the production decision.
For programs that are dominated by manufacturing cost, this approach reduces the possibility that a costly reworking of a system might become necessary should a defect be identified only after fielding of units in the operational environment has begun”.

State-of-the-art EW sensors largely differ from this concept since:

• they are more and more SW-intensive ([8])
• they are federated or integrated in Systems of Systems ([8],[10])
• they are customized and conform to the platforms they will be installed on
• they span over the cyberspace ([9], [10])

This urges for a different approach, which can be effectively supported by “Integrated Testing”. This strategy aims at “testing the capability as it is intended to be used”, and favours an early start of testing and validation within the program.
The advantage is to involve as soon as possible the future user of the system, through the integration of the developmental testing and the operational testing steps, as well as the two testing teams.
This approach is often described as means to “include the voice of the end user”.

Final objective would be establishing “a collaborative model built upon shared data and reciprocity of test results that is ultimately an enabling process for delivering working capability”.
The T&E capability would then transform into a responsive, on-demand, ‘‘testing as a service’’ construct.

It is worth mentioning that the overlapping of the EW domains and the Cyberspace has an impact also on the T&E concept and methodology. The trend is to scale the T&E coverage from the RF and video-signals level to the upper layers of IT and command & control (C2). The threat to be simulated might be vehiculated by an RF signal, but have a Measurement of Effectiveness that can be seen only at higher communication protocol level (e.g., break in content field of the transmitted data package).

In addition, Artificial Intelligence emerged recently as technology enabler for future EW.
This urges for change in T&E as well, [11]: “the Defence Department needs to reform its existing testing and verification system—its methods, processes, infrastructure, and workforce—in order to help decision-makers and operators understand and manage the risks of developing, producing, operating, and sustaining AI-enabled systems. (…) for the next five to 10 years, the Defence Department will likely rely on supervised learning systems, and testing ML/DL (Machine Learning/Deep Learning) systems will likely require large sets of labelled, representative data”.