Sensory substitution is a change of the characteristics of one sensory modality into stimuli of another sensory modality.
A sensory substitution system consists of three parts: a sensor, a coupling system, and a stimulator. The sensor records stimuli and gives them to a coupling system which interprets these signals and transmits them to a stimulator. In case the sensor obtains signals of a kind not originally available to the bearer it is a case of sensory augmentation. Sensory substitution concerns human perception and the plasticity of the human brain; and therefore, allows us to study these aspects of neuroscience more through neuroimaging.
Sensory substitution systems may help people by restoring their ability to perceive certain defective sensory modality by using sensory information from a functioning sensory modality.
History The idea of sensory substitution was introduced in the 1980s by Paul Bach-y-Rita as a means of using one sensory modality, mainly tactition, to gain environmental information to be used by another sensory modality, mainly vision. Thereafter, the entire field was discussed by Chaim-Meyer Scheff in "Experimental model for the study of changes in the organization of human sensory information processing through the design and testing of non-invasive prosthetic devices for sensory impaired people". The first sensory substitution system was developed by Bach-y-Rita et al. as a means of brain plasticity in congenitally blind individuals. After this historic invention, sensory substitution has been the basis of many studies investigating perceptive and cognitive neuroscience. Sensory substitution is often employed to investigate predictions of the embodied cognition framework. Within the theoretical framework specifically the concept of sensorimotor contingencies is investigated utilizing sensory substitution. Furthermore, sensory substitution has contributed to the study of brain function, human cognition and rehabilitation. More recently the philosophy and psychology of sensory substituon and augmentationMacpherson, Fiona (2018) (ed.) Sensory Substitution and Augmentation, Proceedings of the British Academy, Oxford: Oxford University Press, https://academic.oup.com/british-academy-scholarship-online/book/31069. in an edited volume of that name by Fiona Macpherson.
Physiology When a person becomes blind or deaf they generally do not lose the ability to hear or see; they simply lose their ability to transmit the sensory signals from the periphery (retina for visions and cochlea for hearing) to brain. Since the vision processing pathways are still intact, a person who has lost the ability to retrieve data from the retina can still see subjective images by using data gathered from other sensory modalities such as touch or audition.
In a regular visual system, the data collected by the retina is converted into an electrical stimulus in the optic nerve and relayed to the brain, which re-creates the image and perceives it. Because it is the brain that is responsible for the final perception, sensory substitution is possible. During sensory substitution an intact sensory modality relays information to the visual perception areas of the brain so that the person can perceive sight. With sensory substitution, information gained from one sensory modality can reach brain structures physiologically related to other sensory modalities. Touch-to-visual sensory substitution transfers information from touch receptors to the visual cortex for interpretation and perception. For example, through fMRI, one can determine which parts of the brain are activated during sensory perception. In blind persons, it is seen that while they are only receiving tactile information, their visual cortex is also activated as they perceive sight objects.Bach-y-Rita P. Brain Mechanisms in Sensory Substitution, Academic Press New York:1972. Touch-to-touch sensory substitution is also possible, wherein information from touch receptors of one region of the body can be used to perceive touch in another region. For example, in one experiment by Bach-y-Rita, touch perception was able to be restored in a patient who lost peripheral sensation due to leprosy.Bach-y-Rita P. Nonsynaptic Diffusion Neurotransmission and Late Brain Reorganization, Demos-Vermande, New York :1995.
Technological support In order to achieve sensory substitution and stimulate the brain without intact sensory organs to relay the information, machines can be used to do the signal transduction, rather than the sensory organs. This brain–machine interface collects external signals and transforms them into electrical signals for the brain to interpret. Generally, a camera or a microphone is used to collect visual or auditory stimuli that are used to replace lost sight and hearing, respectively. The visual or auditory data collected from the sensors is transformed into tactile stimuli that are then relayed to the brain for visual and auditory perception. Crucially, this transformation sustains the sensorimotor contingency inherent to the respective sensory modality. This and all types of sensory substitution are only possible due to neuroplasticity.
Brain plasticity Brain plasticity refers to the brain's ability to adapt to a changing environment, for instance to the absence or deterioration of a sense. It is conceivable that cortical remapping or reorganization in response to the loss of one sense may be an evolutionary mechanism that allows people to adapt and compensate by using other senses better. Brain imaging studies have shown that upon visual impairments and blindness (especially in the first 12–16 years of life) the visual cortices undergo a huge functional reorganization such that they are activated by other sensory modalities. Such cross-modal plasticity was also found through functional imaging of congenitally blind patients which showed a cross-modal recruitment of the occipital cortex during perceptual tasks such as Braille reading, tactile perception, tactual object recognition, sound localization, and sound discrimination. This may suggest that blind people can use their occipital lobe, generally used for vision, to perceive objects through the use of other sensory modalities. This cross modal plasticity may explain the often described tendency of blind people to show enhanced ability in the other senses.
Perception versus sensing While considering the physiological aspects of sensory substitution, it is essential to distinguish between sensing and perceiving. The general question posed by this differentiation is: Are blind people seeing or perceiving to see by putting together different sensory data? While sensation comes in one modality – visual, auditory, tactile etc. – perception due to sensory substitution is not one modality but a result of cross-modal interactions. It is therefore concluded that while sensory substitution for vision induces visual-like perception in sighted individuals, it induces auditory or tactile perception in blind individuals. In short, blind people perceive to see through touch and audition with sensory substitution. Through experiments with a Tactile-visual sensory substitution (TVSS) device developed by Bach-y-Rita subjects described the perceptual experience of the TVSS as particularly visual, such that objects were perceived as if located in the external space and not on the back or skin. Further studies using the TVSS showed that such perceptual changes were only possible when the participants could actively explore their environment with the TVSS. These results have been underpinned by many other studies testing different substitution systems with blind subjects such as vision-to-tactile substitution, vision-to-auditory substitution and vision-to-vestibular substitution Such results are also reported in sighted subjects, when blindfolded and deliver further support for the sensorimotor contingency theory.
Different applications Applications are not restricted to disabled persons, but also include artistic presentations, games, and augmented reality. Some examples are substitution of visual stimuli to audio or tactile, and of audio stimuli to tactile. Some of the most popular are probably Paul Bach-y-Rita's Tactile Vision Sensory Substitution (TVSS), developed with Carter Collins at Smith-Kettlewell Institute and Peter Meijer's Seeing with Sound approach (The vOICe). Technical developments, such as miniaturization and electrical stimulation help the advance of sensory substitution devices.
In sensory substitution systems, we generally have sensors that collect the data from the external environment. This data is then relayed to a coupling system that interprets and transduces the information and then replays it to a stimulator. This stimulator ultimately stimulates a functioning sensory modality. After training, people learn to use the information gained from this stimulation to experience a perception of the sensation they lack instead of the actually stimulated sensation. For example, a leprosy patient, whose perception of peripheral touch was restored, was equipped with a glove containing artificial contact sensors coupled to skin sensory receptors on the forehead (which was stimulated). After training and acclimation, the patient was able to experience data from the glove as if it was originating in the fingertips while ignoring the sensations in the forehead.
Tactile systems To understand tactile sensory substitution it is essential to understand some basic physiology of the tactile receptors of the skin. There are five basic types of tactile receptors: Pacinian corpuscle, Meissner's corpuscle, Ruffini endings, Merkel nerve endings, and free nerve endings. These receptors are mainly characterized by which type of stimuli best activates them, and by their rate of adaptation to sustained stimuli. Because of the rapid adaptation of some of these receptors to sustained stimuli, those receptors require rapidly changing tactile stimulation systems in order to be optimally activated. Among all these mechanoreceptors Pacinian corpuscle offers the highest sensitivity to high frequency vibration starting from a few tens of Hz to a few kHz with the help of its specialized mechanotransduction mechanism.
There have been two different types of stimulators: electrotactile or vibrotactile. Electrotactile stimulators use direct electrical stimulation of the nerve ending in the skin to initiate the action potentials; the sensation triggered, burn, itch, pain, pressure etc. depends on the stimulating voltage. Vibrotactile stimulators use pressure and the properties of the mechanoreceptors of the skin to initiate action potentials. There are advantages and disadvantages for both these stimulation systems. With the electrotactile stimulating systems a lot of factors affect the sensation triggered: stimulating voltage, current, waveform, electrode size, material, contact force, skin location, thickness and hydration. Electrotactile stimulation may involve the direct stimulation of the nerves (percutaneous), or through the skin (transcutaneous). Percutaneous application causes additional distress to the patient, and is a major disadvantage of this approach. Furthermore, stimulation of the skin without insertion leads to the need for high voltage stimulation because of the high impedance of the dry skin, unless the tongue is used as a receptor, which requires only about 3% as much voltage. See also Brainport This latter technique is undergoing clinical trials for various applications, and been approved for assistance to the blind in the UK. Alternatively, the roof of the mouth has been proposed as another area where low currents can be felt.
Electrostatic arrays are explored as human–computer interaction devices for touch screens. These are based on a phenomenon called electrovibration, which allows microamperre-level currents to be felt as roughness on a surface.