In computer science, fault injection is a testing technique for understanding how computing systems behave when stressed in unusual ways. This can be achieved using physical- or software-based means, or using a hybrid approach. Widely studied physical fault injections include the application of high voltages, extreme temperatures and electromagnetic pulses on electronic components, such as computer memory and central processing units. By exposing components to conditions beyond their intended operating limits, computing systems can be coerced into mis-executing instructions and corrupting critical data.
In software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths; in particular error handling code paths, that might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part of developing robust software.J. Voas, "Fault Injection for the Masses," Computer, vol. 30, pp. 129–130, 1997. Robustness testingKaksonen, Rauli. A Functional Method for Assessing Protocol Implementation Security. 2001. (also known as syntax testing, fuzzing or fuzz testing) is a type of fault injection commonly used to test for vulnerabilities in communication interfaces such as protocols, command line parameters, or APIs.
The propagation of a fault through to an observable failure follows a well-defined cycle. When executed, a fault may cause an error, which is an invalid state within a system boundary. An error may cause further errors within the system boundary, therefore each new error acts as a fault, or it may propagate to the system boundary and be observable. When error states are observed at the system boundary they are termed failures. This mechanism is termed the fault-error-failure cycleA. Avizienis, J.-C. Laprie, Brian Randell, and C. Landwehr, "Basic Concepts and Taxonomy of Dependable and Secure Computing," Dependable and Secure Computing, vol. 1, pp. 11–33, 2004. and is a key mechanism in dependability.
History The technique of fault injection dates back to the 1970sJ. V. Carreira, D. Costa, and S. J. G, "Fault Injection Spot-Checks Computer System Dependability," IEEE Spectrum, pp. 50–55, 1999. when it was first used to induce faults at a hardware level. This type of fault injection is called Hardware Implemented Fault Injection (HWIFI) and attempts to simulate hardware failures within a system. The first experiments in hardware fault involved nothing more than shorting connections on circuit boards and observing the effect on the system (bridging faults). It was used primarily as a test of the dependability of the hardware system. Later specialised hardware was developed to extend this technique, such as devices to bombard specific areas of a circuit board with heavy radiation. It was soon found that faults could be induced by software techniques and that aspects of this technique could be useful for assessing software systems. Collectively these techniques are known as Software Implemented Fault Injection (SWIFI).
Software implemented fault injection SWIFI techniques can be categorized into two types: compile-time injection and runtime injection.
Compile-time injection is an injection technique where source code is modified to inject simulated faults into a system. One method is called mutation testing which changes existing lines of code so that they contain faults. A simple example of this technique could be changing a = a + 1 to a = a – 1
Code mutation produces faults which are very similar to those unintentionally added by programmers.
A refinement of code mutation is Code Insertion Fault Injection which adds code, rather than modifying existing code. This is usually done through the use of perturbation functions which are simple functions which take an existing value and perturb it via some logic into another value, for example
int pFunc(int value) { return value + 20; }
int main(int argc, char * argv[]) { int a = pFunc(aFunction(atoi(argv))); if (a > 20) { /* do something */ } else { /* do something else */
In this case, pFunc is the perturbation function and it is applied to the return value of the function that has been called introducing a fault into the system.
Runtime Injection techniques use a software trigger to inject a fault into a running software system. Faults can be injected via a number of physical methods and triggers can be implemented in a number of ways, such as: Time Based triggers (When the timer reaches a specified time an interrupt is generated and the interrupt handler associated with the timer can inject the fault. ); Interrupt Based Triggers (Hardware exceptions and software trap mechanisms are used to generate an interrupt at a specific place in the system code or on a particular event within the system, for instance, access to a specific memory location).
Runtime injection techniques can use a number of different techniques to insert faults into a system via a trigger.
Corruption of memory space: This technique consists of corrupting RAM, processor registers, and I/O map. Syscall interposition techniques: This is concerned with the fault propagation from operating system kernel interfaces to executing systems software. This is done by intercepting operating system calls made by user-level software and injecting faults into them. Network Level fault injection: This technique is concerned with the corruption, loss or reordering of network packets at the network interface.
These techniques are often based around the debugging facilities provided by computer processor architectures.
Protocol software fault injection
Complex software systems, especially multi-vendor distributed systems based on open standards, perform input/output operations to exchange data via stateful, structured exchanges known as "protocols." One kind of fault injection that is particularly useful to test protocol implementations (a type of software code that has the unusual characteristic in that it cannot predict or control its input) is fuzzing. Fuzzing is an especially useful form of Black-box testing since the various invalid inputs that are submitted to the software system do not depend on, and are not created based on knowledge of, the details of the code running inside the system.
Hardware implemented fault injection The hardware fault injection method consists in real electrical signals injection into the DUT (devices under testing) in order to disturb it, supposedly well working, at hardware low level, and deceive the control - detection chain (if present) in order to see how and if the fault management strategy is implemented.
This technique is based on a nail bed, necessary to contact the electronics of the products, where a specific test pad has been left free (Design for testability of PCB), and properly inject disturbance signals, in order to see the product's control- reaction arm's reaction if present.
It's a technique often used to certify and validate reaction to fault in high reliability products, where safety is involved (Military, Medical, Autonomous vehicle).
The proper low level physical injection is very critical, not only because the different types of signals should have different types of disturbance (voltage, coupling impedance etc.), but also a very appropriate timing when the disturbance shall be injected, to avoid serious damage to the DUT generation's dangerous short circuit.
Any single components present in the DUT (from dipole type resistor, capacitor, diode) to tripolar transistors, CMOS, ECC) up to complex chip (Low voltage converter, power unit, CPU, RAM, ECC), is susceptible to a fault method (short, open, drift ECC) that can be studied during a FMEDA analysis using statistical method (MIL HBK) and determine the most dangerous with a Functional safety approach (ISO 26262 A), then real test shall be executed on any permutation there considered risky.
Because of the wide complexity a dedicated and complex tools is necessary.
Characteristics of fault injection Faults have three main parameters.
Type: What type of fault should be injected? For example stuck-to-value, delay, ignoring some functions, ignoring some parameters/variable, random faults, the bias fault, the noise, etc. The amplitude of each fault is also important. Time: When should be activated? For example the activation time of fault or the activation condition of fault. Location: Where should be in the system? For example fault in the link/connection between systems, faults within systems/subsystems/function, etc.
These parameters create the fault space realm. The fault space realm will increase exponentially by increasing system complexity. Therefore, the traditional fault injection method will not be applicable to use in the modern cyber-physical systems, because they will be so slow, and they will find a small number of faults (less fault coverage). Hence, the testers need an efficient algorithm to choose critical faults that have a higher impact on system behavior. Thus, the main research question is how to find critical faults in the fault space realm which have catastrophic effects on system behavior. Here are some methods that can aid fault injection to efficiently explore the fault space to reach higher fault coverage in less simulation time.
Sensitivity analysis: In this method, sensitivity analysis has been used to identify the most important signals that have a higher impact on the system's specification. By identifying those important signals or parameters, the fault injection tool will focus on those effective signals instead of focusing on all signals in the system. Reinforcement learning:Moradi, M., Oakes, B.J., Saraoglu, M., Morozov, A., Janschek, K. and Denil, J., 2020. EXPLORING FAULT PARAMETER SPACE USING REINFORCEMENT LEARNING-BASED FAULT INJECTION. In this method, the reinforcement learning algorithm has been used to efficiently explore the fault space and find critical faults.
Fault injection tools
Although these types of faults can be injected by hand the possibility of introducing an unintended fault is high, so tools exist to parse a program automatically and insert faults.
Research tools
A number of SWIFI Tools have been developed and a selection of these tools is given here. Six commonly used fault injection tools are Ferrari, FTAPE, Doctor, Orchestra, Xception and Grid-FIT.
MODIFI (MODel-Implemented Fault Injection) is a fault injection tool for robustness evaluation of Simulink behavior models. It supports fault modelling in XML for implementation of domain-specific fault models.Rickard Svenningsson, Jonny Vinter, Henrik Eriksson and Martin Torngren, "MODIFI: A MODel-Implemented Fault Injection Tool", Lecture Notes in Computer Science, 2010, Volume 6351/2010, 210-222. Ferrari (Fault and ERRor Automatic Real-time Injection) is based around software traps that inject errors into a system. The traps are activated by either a call to a specific memory location or a timeout. When a trap is called the handler injects a fault into the system. The faults can either be transient or permanent. Research conducted with Ferrari shows that error detection is dependent on the fault type and where the fault is inserted.G. A. Kanawati, N. A. Kanawati, and J. A. Abraham, "FERRARI: A Flexible Software-Based Fault and Error Injection System," IEEE Transactions on Computers, vol. 44, pp. 248, 1995.