Visual indexing theory, also known as FINST theory, is a theory of early visual perception developed by Zenon Pylyshyn in the 1980s. It proposes a pre-attentive mechanism (a ‘FINST’) whose function is to individuate salient elements of a visual scene, and track their locations across space and time. Developed in response to what Pylyshyn viewed as limitations of prominent theories of visual perception at the time, visual indexing theory is supported by several lines of empirical evidence.
Overview
Fingers of instantiation
'FINST' abbreviates ‘FINgers of INSTantiation’. Pylyshyn describes visual indexing theory in terms of this analogy.Pylyshyn, Z.W. (1989). The role of location indexes in spatial perception: a sketch of the FINST spatial index model. Cognition, 32, 65–97. Imagine, he proposes, placing your fingers on five separate objects in a scene. As those objects move about, your fingers stay in respective contact with each of them, allowing you to continually track their whereabouts and positions relative to one another. While you may not be able to discern in this way any detailed information about the items themselves, the presence of your fingers provides a reference via which you can access such information at any time, without having to relocate the objects within the scene. Furthermore, the objects' continuity over time is inherently maintained — you know the object referenced by your pinky finger at time t is the same object as that referenced by your pinky at t−1, regardless of any spatial transformations it has undergone, because your finger has remained in continuous contact with it.
Visual indexing theory holds that the visual perceptual system works in an analogous way. FINSTs behave like the fingers in the above scenario, pointing to and tracking the location of various objects in visual space. Like fingers, FINSTs are: Plural. Multiple objects can be independently indexed and tracked by individual FINSTs simultaneously. Adhesive. As indexed objects move around in the visual scene, their FINSTs move with them. Opaque to the features of the objects they index. FINSTs reference objects according to their location only. No additional information about their referents is conveyed via the FINST mechanism itself.
Role in the visual perception process
Individuation
FINSTs operate pre-attentively — that is, before attention is drawn or directed to an object in the visual field. Their primary task is to individuate certain salient features in a scene, conceptually distinguishing these from other stimuli. Under visual indexing theory, FINSTing is a necessary precondition for higher level perceptual processing.
Pylyshyn suggests that what FINSTs operate upon in a direct sense is 'feature clusters' on the retina, though a precise set of criteria for FINST allocation has not been defined. "The question of how FINSTs are assigned in the first instance remains open, although it seems reasonable that they are assigned primarily in a stimulus-driven manner, perhaps by the activation of locally distinct properties of the stimulus-particularly by new features entering the visual field."
FINSTs are subject to resource constraints. Up to around five FINSTs can be allocated at any given time, and these provide the visual system information about the relative locations of FINSTed objects with respect to one another.
Tracking
Once an object has been individuated, its FINST then continues to index that particular feature cluster as it moves across the retina. "Thus distal features which are currently projected onto the retina can be indexed through the FINST mechanism in a way that is transparent to their retinal location." By continually tracking an objects' whereabouts as it moves about, FINSTs perform the additional function of maintaining the continuity of objects over time.
Attentional facilitation
Under visual indexing theory, an object cannot be attended to until it has first been indexed. Once it has been allocated a FINST, the index provides the visual system with rapid and preferential access to the object for further processing of features such as colour, texture and shape.
While in this sense FINSTs provide the means for higher-level processing to occur, FINSTs themselves are "opaque to the properties of the objects to which they refer." FINSTs do not directly convey any information about an indexed object, beyond its position at a given instant. "Thus, on initial contact, objects are not interpreted as belonging to a certain type or having certain properties; in other words, objects are initially detected without being conceptualised."Pylyshyn, Z.W. (2000). Situating vision in the world. Trends in Cognitive Sciences 4,(5), 197-207. Like the fingers described above, FINSTs' role in visual perception is purely an indexical one.
Theoretical context
Visual indexing theory was created partly in response to what Pylyshyn viewed as limitations of traditional theories of perception and cognition — in particular, the spotlight model of attention, and the descriptive view of visual representation.
Spotlight and zoom-lens models of attention
The traditional view of visual perception holds that attention is fundamental to visual processing. In terms of an analogy offered by Posner, Snyder and Davidson (1980): "Attention can be likened to a spotlight that enhances the efficiency of detection of events within its beam".Posner, M. I., Snyder, C. R. R. and Davidson, B. J. (1980). Attention and the Detection of Signals. Journal of Experimental Psychology: General, Vol. 109, No. 2, 160-174. This spotlight can be controlled volitionally, or drawn involuntarily to salient elements of a scene,Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3–25. but a key characteristic is that it can only be deployed to one location at a time. In 1986, Eriksen and St. James conducted a series of experiments which suggested that the spotlight of attention comes equipped with a zoom-lens. The zoom-lens allows the size of the area of attentional focus to be expanded (but due to a fixed limit on available attentional resources, only at the expense of processing efficiency).Eriksen, C. W. and St. James, J. D. (1986). Visual attention within and around the field of focal attention: A zoom lens model. Perception & Psychophysics, 40 (4), 225-240.
According to Pylyshyn, the spotlight/zoom-lens model cannot tell the complete story of visual perception. He argues that a pre-attentive mechanism is needed to individuate objects upon which a spotlight of attention could be directed in the first place. Furthermore, results of multiple object tracking studies (discussed below) are "incompatible with the proposal that items are accessed by moving around a single spotlight of attention." Visual indexing theory addresses these limitations.
Descriptive view of visual representation
According to the classical view of mental representation, we perceive objects according to the conceptual descriptions they fall under. It is these descriptions, and not the raw content of our visual perceptions, that allow us to construct meaningful representations of the world around us, and determine appropriate courses of action. In Pylyshyn's words, "it is not the bright spot in the sky that determines which way we set out when we are lost, but the fact that we see it (or represent it) as the North Star".Pylyshyn, Z. W. (2001). Visual indexes, preconceptual objects, and situated vision. Cognition 80, 127-158. The method by which we come to match a percept to its appropriate description has been the subject of ongoing investigation (for example the way in which parts of objects are combined to represent their whole),Hoffman, D. D. and Richards, W. A. (1984). Parts of recognition. Cognition 18, Issues 1–3, 65–96. but there is a general consensus that descriptions are fundamental in this way to visual perception.
Like the spotlight model of attention, Pylyshyn takes the descriptive model of visual representation to be incomplete. One issue is that the theory does not account for demonstrative, or indexical references. "For example, in the presence of a visual stimulus, we can think thoughts such as `that is red' where the term `that' refers to something we have picked out in our field of view without reference to what category it falls under or what properties it may have." Relatedly, the theory has problems accounting for how we are able to pick out a single token among several objects of the same type. For example, I may refer to a particular can of soup on a supermarket shelf sitting among a number of identical cans that answer to the same description. In both cases, a spatiotemporal reference is required in order to pick out the object within the scene, independently of any description that object may fall under. FINSTs, Pylyshyn suggests, provide just such a reference.
A deeper problem for this view, according to Pylyshyn, is that it cannot account for objects' continuity over time. "An individual remains the same individual when it moves about or when it changes any (or even all) of its visible properties." If we refer to objects solely in terms of their conceptual descriptions, it is not clear how the visual system maintains an object's identity when those descriptions change. "The visual system needs to be able to pick out a particular individual regardless of what properties the individual happens to have at any instant of time." Pylyshyn argues that FINSTs' detachment from the descriptions of the objects they reference overcomes this problem.
Experimental evidence
Three main types of experiments provide data that support visual indexing theory. Multiple tracking studies demonstrate that more than one object can be tracked within the visual field simultaneously, subitizing studies suggest the existence of a mechanism that allows small numbers of objects to be efficiently enumerated, and subset selection studies show that certain elements of a visual scene can be processed independently of other items. In all three cases, FINSTs provide an explanation of the phenomenon observed.Pylyshyn, Z.W. (1994). Some primitive mechanisms of spatial attention. Cognition 50, 363–384.
Multiple object tracking studies
Multiple object tracking describes the ability of human subjects to simultaneously track the movement of up to five target objects as they move across the visual field, usually in the presence of identical moving distractor objects of equal or greater number. The phenomenon was first demonstrated by Pylyshyn and Storm in 1988,Pylyshyn, Z.W. and Storm, R. (1988). Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spatial Vision. 3, 179–197. and their results have been widely replicated (see Pylyshyn, 2007 for a summary.Pylyshyn, Z.W. (2007). Multiple Object Tracking. Scholarpedia, 2(10):3326.)
Experimental setup
In a typical experiment, a number of identical objects (up to 10) are initially shown on a screen. Some subset of these objects (up to five) are then designated as targets — usually by flashing or changing colour momentarily — before returning to being indistinguishable from the non-target objects. All of the objects then proceed to move randomly around the screen for between 7 and 15 seconds. The subject's task it to identify, once the objects have stopped moving, which objects were the targets. Successful completion of the task thus requires subjects to continually track each of the target objects as they move, and ignore the distractors.
Results
Under such experimental conditions, it has been repeatedly found that subjects can track multiple moving objects simultaneously. In addition to consistently observing a high rate of successful target tracking, researchers have shown that subjects can: