Monday, October 27, 2008
Dorkbot CBR at Manuka CCAS
Thursday, October 02, 2008
Synesthesia and Cross-Modality in Contemporary Audiovisuals
Though written about a year ago, this essay has just been published in Senses and Society. It's related to the Synchresis project posted earlier but makes a more rigorous investigation of synaesthesia, as it is (so often) applied to fused or algorithmic audiovisuals. After a quick tour through the history of synaesthesia in the arts, it uses some nifty perceptual neuroscience to argue for an alternative model, of contemporary audiovisuals as cross-modal objects that reveal the space of relation between modalities - the map. It takes work by Andrew Gadow (below) and Robin Fox as case studies, but also touches on Oskar Fischinger, Robert Hodgin, Norman McLaren and others. The version here has plenty of pics and vids; for a more paper-based experience grab the pdf (and please use the print version for any citations). Oh and pardon the American spellings here - journal style.
In the age of ubiquitous digital media, synesthesia is everywhere. In human, neurological form, it is rare: for perhaps three in a hundred people, a stimulus in one sensory modality automatically induces a sensation in another. Auditory-to-visual synesthesia, or “colored hearing” is much rarer still. Yet now this phenomenon is realised, apparently, inside every digital music player, on VJ screens in every club, in robot lightshows. On these screens sound is transformed into visual pattern and form instantly and automatically; an exotic perceptual phenomenon becomes a technically mediated commonplace.
In fact digital synesthesia is a trope that occurs in the production and use of mainstream digital media, as well as the media arts. Computer users find audio visualisers built in to their music players; as this software shows, audiovisual relations can now be reduced to an algorithm, a formal procedure that interprets (sound) and emits (image) data; though in this case the results are mostly mundane psychedelia. In some recent music video audio visualisations are integrated into the narrative and performative conventions of the genre; in Justin Timberlake’s Lovestoned video (2007) the singer’s image is constructed from the flickering bands of a visualised audio spectrum. Here a technically guaranteed unity of sound and image is literally reinscribed on to the performing artist. In contemporary media arts practice the same techniques – computational analysis of sound driving generated visual elements – are widespread, and its aesthetics more diverse. In custom-coded audio visualisations artists such as Marius Watz (2005) and Robert Hodgin (2007) (below) construct visualisations tuned to specific soundtracks; the automatism of digital synesthesia animates specific, constructed worlds of form and image. The algorithm becomes an endlessly variable and dynamic intermediary between sound and image.
In a distinct but related approach, some media artists have opted instead to simplify or reduce that audiovisual relation, ofen bypassing computation altogether. In Carsten Nicolai’s Telefunken works (2000,2004) the stereo output of an audio CD player is connected to the audio and video inputs of a television screen; what is heard as synthetic tones and noisy drones, is seen on the screen as patterns of monochrome form and line. In an approach I will refer to here as transcoding, sound and image are linked through a direct transfer of signal, a simple cross-wiring.
Australian artist Robin Fox plugs audio from a custom-built digital synthesiser into an oscilloscope; in the resulting hybrid instrument, Fox explores a territory in which signal is simultaneously heard and seen; every sound is a form in motion, every form a sound (above). The connection, the cross-wiring of sound to image, literally manifests the sensory cross-over of synesthesia; more, the work itself seems to somehow induce synesthetic experience. The correspondence between sound and image is immediate, agile and intense; the audiovisual relation is completely consistent, somehow self-evident, yet continually surprising. There’s a feeling of something like revelation; one reviewer describes Fox’s Backscatter DVD (2005) as “mesmerizing” and “overwhelming,” and hints at a sense of “greater significance or higher purpose” (Baker Fish 2005). Andrew Gadow’s work approaches the same relationship from the other side; working with an old video synthesiser, he transfers its image signal directly into audio. The scan-line structure of the video signal becomes audible as modulations of a 50 Hz hum; flickering, disintegrating visual textures become abrasive but intricately detailed buzzsaw audio. Again the subjective experience can be powerful, a visceral sense of force or encounter; the audiovisual coupling is so close that it seems to disappear, distinct modalities fuse into raw sensation.
Synesthesia is widely used as an analogy around this work. The analogy provides a mapping that aligns subjective sensation with audiovisual signals; it maps perceptual or even neurological structures onto technical structures. The analogy also plays another role, foregrounding sensation in the reception of the artworks; proposing to operate, for the subject, at the level of direct sensation. Finally, synesthesia also connects this contemporary work with a historical artistic tradition. The new automatic or transcoded fusion of sound and image seems to mark the culmination of a practice spanning music, painting, film and electronic media and aspiring, as Jeremy Strick writes, to the ideal of synesthesia as “the unity of the senses, and, by extension, the arts.” (Strick 2005: 15) The 2005 Visual Music exhibition, curated by Strick, documents this tradition in detail, as well as making a bid for its continuation into the present:In digital media ... music and visual art are ... created out of the same stuff, bits of electronic information ... . [T]he aspiration to novel experience created by the compounding of sensation and association has never been more possible. (ibid.)
This paper’s main aim is to test this analogy, and the related historical drive that Strick suggests; to consider if, and how, such practice can be thought of as synesthetic, and examine structural parallels between synesthesia as a perceptual and neurological phenomenon, and the automatic or transcoded linking of audio and visual media? Following the tradition of artistic synesthesia that Strick invokes, the approach here is to provisionally ignore the glaring gap in this analogy, between subjective sensation and objective, technical artefact. Scientific work in perception and neuropsychology is drawn in for a more detailed account of synesthesia; but it also offers an alternative model for this practice, based on theories of cross-modal interactions in normal perception. Close correlations between sound and image are, after all, an everyday perceptual occurrence. From this perspective, tightly correlated audiovisuals direct us towards the abstract structures that are its generative materials: signal, as distinct from image or sound; and the map, the pattern of correlation between signals in different domains. Although artists such as Fox and Gadow use obsolete, analog technologies, their work is a sensory manifestation of these characteristically contemporary abstractions.
Synesthesia
Scientists studying synesthesia define it as occurring “when stimulation of one sensory modality automatically triggers perception in a second modality” (Harrison and Baron-Cohen 1996: 3). There are many documented forms of synesthesia; stimuli such as numbers, letters, words, days of the week, and musical tones may trigger perceived color and shape; taste, smell and pain can also trigger perceptions of shape. Estimates of the prevalence of synesthesia vary widely between 1 in 20 and 1 in 20 000; one recent study found a prevalence of around 3% and showed that calendar-color and letter- and number-color forms occur most often, while audiovisual synesthesia or “colored hearing” is comparatively rare (Simner et al 2006).
After long being debunked or treated as a curiosity, synesthesia has attracted increasing scientific attention, and validation, in recent years. Neuropsychologist Richard Cytowic undertook one of the first modern studies (Cytowic 1989). As well as a basic validation – finding that synesthesia is a real phenomenon with a neurological basis – Cytowic proposed a set of diagnostic and clinical features of the condition (Cytowic 1996: 23-31). He found that synesthesia is “involuntary but elicited,” an automatic perceptual experience that cannot be supressed or controlled. Synesthetic perceptions are “durable and generic,” meaning that an individual’s cross-sensory connections do not change over their lifetime, and that synesthetic perceptions are elementary and general, rather than “elaborated” – for example colors and simple shapes, rather than a detailed mental picture. Cytowic also points out that synesthetic perceptions are unusually memorable: some synesthetes use their triggered percepts as an index that aids their recall of the evoking stimulus. Moreover, Cytowic states, synesthesia is an emotional experience, “accompanied by a sense of certitude (the ‘this is it’ feeling)” that he links to William James’ description of religious ecstasy, and in particular the affect of noesis, “knowledge that is experienced directly, an illumination that is accompanied by a feeling of certitude.” It is in part this affective dimension that leads Cytowic to propose a linkage between synesthesia and the limbic brain – associated with emotion and a sense of “salience.”
More recent science has continued to investigate the neurology and psychology of synesthesia, using modern imaging techniques to show activation in the anatomy of the synesthetic brain, as well as behavioural experiments that seek out the parameters of synesthetic experience. Recent work has confirmed Cytowic’s finding that synesthesia is involuntary and perceptually real, though the results also suggest that there is significant variation between synesthetic individuals (see for example Hubbard and Ramachandran 2005). Discussion centers on where synesthesia occurs in the notional chain of perceptual processing; for a minority of synesthetes it seems to occur early in the chain, before cognitive processes such as attention; for the rest it seems to occur later; Ramachandran and Hubbard label these forms"lower” and “higher” (2001: 14). There is general consensus that synesthesia has a neural basis in the form of increased connectivity between normally separate neural regions or modules (contrary to Cytowic’s limbic model), though the connective mechanism and architecture is debated. Some, including Ramachandran and Hubbard, propose that this connectivity is a result of defective “pruning” of neural connections, suggesting that synesthetic cross-wiring is a normal early developmental stage. The notion of synesthesia as common, underlying or originary is supported by the correlations between synesthetic and “normal” cross-sensory associations; despite individual differences in color-tone mapping, non-synesthetes and those with “colored hearing” make similar mappings between pitch and lightness (Marks 1996: 72). Similarly Ramachandran and Hubbard (2001:19) cite the consistency of an (albeit simple) mapping between shape and sound to support the same point: asked to link two shapes, one round, one spiky, with two names, bouba and kiki, subjects overwhelmingly associate kiki with the angular form and bouba with the rounded one. The authors continue, in one of the more expansive examples of synesthesia science, to propose links between this “normal” synesthesia and the angular gyrus, an anatomical region associated with cross-modal association as well as numeracy, the neurology of metaphor, emotion, art and the evolution of language. In this formulation synesthesia is an extreme case that offers clues to the neurology of normal – and significant – human abilities to associate and synthesise disparate sensations and concepts.
Synesthesia in the Arts: Models and Maps
The history of synesthesia in the modern arts is well documented, and will not be recounted in detail here. More important for this argument is a sense of the major strains or variants of the concept, and their creative implications. This lineage has been traced to Romantic and Symbolist interests in the correspondence of the senses; poems of Rimbaud and Baudelaire correlated letters, colors, smells and sounds (see for example Cook 1998:25). In what Judith Zilczer (2005: 26) terms the mystical strain of artistic synesthesia, these sensory correspondences were held to refer to a higher, unitary reality, informed by Theosophy and Romantic philosophy. Kandinsky, reputedly a synesthete with “colored hearing,” is the best known of a group whose painterly abstraction was informed by musical and synesthetic analogies. In Kandinsky’s Concerning the Spiritual in Art (1977: 25) sensation, and especially color, can set the soul vibrating like a musical instrument: his aim was a form of absolute or “nonobjective” visual art, comparable to music. Cook (1998: 46) describes this model, informed by Goethe’s philosophy, as triangular. Both sound and color derive from the spiritual, or higher vibration, at the apex; thus sound and color have no inherent correspondence but “correspond to one another in so far as they embody the same ultimate meaning.” In a second wave of visual music between the wars, artists such as Paul Klee, Man Ray, Georgia O’Keefe and Arthur Dove adopted more concrete and structural models of correspondence, attempting to map harmony, counterpoint and rhythm into the abstract picture plane (see for example Zilczer 2005: 52-67). Synesthesia itself plays a shifting role in this context. Kandinsky and composer Alexander Scriabin seem to have experienced it, while many other artists were inspired by, or in some cases literally borrowed, synesthestic correspondences. Discussing the influence of Kandinsky’s note-color correpondences on Schoenberg, Cook (1998: 49) proposes a “cultural synesthesia” – where the idea of sensory correspondence can carry a cultural value independent of its actual experience.
In fact cultural synesthesia – evoked, suggested, implied or idealised synesthesia – dominates the visual music tradition; there are very few instances of actual, spontaneous, automatic audiovisual correspondences. In the work of Messiaen, Scriabin and perhaps Schoenberg (via Kandinsky), synesthetic experience formed the basis for a systematised set of pitch-color correspondences, though even these are not straightforward. The correspondences are different for each composer, as we would expect based on recent science. Moreover each is conditioned by what Cook (1998:46) argues is a mixture of subjective and cultural factors. Any correspondence between the continuous color spectrum and the discrete values of the Western twelve-tone scale is dubious – though these correspondences flourished in the early twentieth century in the “color organs” of Rimington and others (see for example Cook 1998:37 and Peel 2006). Later emblematic practitioners of the visual music tradition, John and James Whitney, used tightly composed but again ultimately arbitrary relations between sound and vision. If, as Strick argues, this creative tradition aspires towards synesthesia, when it comes to practically manifesting that sensory relation it founders on the problem of the map, the pattern of correspondences. Of all possible relations between sound and vision, what is it that makes one different, or preferable, to another? While recent science suggests some underlying perceptual commonalities, the devil, and the aesthetic, is in the detail.
Animator Oskar Fischinger provides a near-precedent for transcoded audiovisuals, and demonstrates one possible solution to the question of the map. Fischinger’s Ornament Sound experiments of 1932 (above) explored the double identity of the optical film soundtrack, printing regular visual patterns into the 3mm-wide sound strip at the edge of the frame, enabling them to be automatically rendered as sound. Fischinger (1932) emphasised the potential of this technique for composers: “control of every fine gradation and nuance is granted to the music-painting artist.” This form also promised a newfound “definitive” control over performance: “his creation, his work, can speak for itself directly through the film projector.” Fischinger also recognises the visual interest in recorded sound waves; and although he anticipated their use in conjunction with animation, he did not envisage the “sounding ornaments” as visual content in themselves. Nonetheless, Fischinger had found a space of audiovisual correspondence that was preexisting and “definitive,” yet seemingly had limitless creative potential. The later work of animator Norman McLaren developed Fischinger’s techniques, synchronising hand-drawn optical soundtracks with animation in Dots (1940), and finally using the synthesised optical soundtrack as synchronised visual source material in Synchromy (1971) (below).
Contemporary transcoded audiovisuals realise Fischinger’s experiments by similarly bypassing, or rather abdicating, the question of the map. This is not to say that the map disappears in an unmediated or inherent audiovisual connection. Instead, for Robin Fox and Andrew Gadow as well as Fischinger, the map is found, rather than constructed; it is embedded in the medium. Fox plugs his laptop into an oscilloscope, which maps the left and right channels of the audio signal into the x and y axes of its display. This audiovisual relation is in a sense a readymade, an existing cultural/technical artefact. Its process is literally hardwired, embodied in the analog electronics of the scope, just as Fischinger’s was in the optical technology of film.
Fused AV and Synesthesia
Do transcoded audiovisuals then realise the synesthetic ideal, or literalise the analogy? We can draw some correlations. To recap, the current scientific consensus is that synesthetic perceptions are real, automatic and involuntary, and caused by neural cross-connections at some level of the perceptual system. The cross-mappings of synesthetic perceptions are highly variable from one individual to another, but highly consistent for the individual. The transcoding approach of artists like Fox and Gadow seems fairly close: in Fox’s oscilloscope work for example, images are created “automatically” as Fox feeds audio to the oscilloscope, cross-wiring audio to vision; Fox uses the oscilloscope’s hardwired audiovisual map, which is fixed and consistent; but that mapping is different to, for example, Gadow’s equally automatic sound-to-image mapping, based on the interchange of analog video and audio signals. Even the visual aesthetics of these works could be likened to reports of colored hearing: in Cytowic’s terms these are not “elaborated” percepts, but simple, abstract elements.
The synesthetic affect that Cytowic’s study identifies is also suggestive. Fused audiovisuals can evoke (for some at least) a similar sense of revelation or noesis. Fox’s oscilloscope works show us something that feels both self-evidently “right” and surprising or unimaginable; the primal phosphorescent dot shows us its universe, a set of relations that are manifestly coherent and consistent, but whose implications are unforeseeable. Fox’s compositional structure emphasises the process of revelation at times: Photosynthesis (AOR) (above), the opening track on his Backscatter release, offers an initially gentle introduction, as the single point of the trace is buffeted by rhythmic subsonic clicks before slowly unfurling into harmonic pattern; but by the end of that track wave after wave of complex, nested forms have emerged and co-modulated; each point on the path is another noetic moment yet each is consistent and coextensive with the others. Mandala I, following, demonstrates almost the opposite approach, as Fox’s micro-switched digital twitches call up flickering variants on the circular carrier wave; to push the cosmological analogy, this is some kind of faster than light travel – we traverse many places at once – but again there is a revelatory quality as we witness accumulating relations, both momentary – between each sound, its corresponding form and movement – and sequential, between each sound/form/movement and the next, and the next.
How far can this line go, though, before it falls into the yawning gap in the analogy? Audiovisual works are artefacts; objects of perception, not perceptions. To put it bluntly, synesthesia, by definition, occurs in the perceptual system of a synesthete, not in the crossed connections of a video synth. Once again, we can use the gap as a provocation, rather than an obstacle. One response is to think of these works not as replicating human neurology, but rather something else. “Artificial synesthesia” is the term used by Dutch neuroscientist Peter Meijer (n.d.) to describe his work on sensory substitution; his vOICe system transcodes video from a small camera, into synthetic audio, in an attempt to use sound to provide visual information to those with little or no vision. In Meijer’s words, “we are interested in forms of learned synesthesia (acquired synesthesia) that might result from machine-generated crossmodal mappings.” Among other things Meijer’s work suggests that perception is not a fixed set of channels, but a reconfigurable network; over time, blind users of the vOICe seem to integrate image transcoded into sound, as functional vision. A recent paper shows that the lateral-occipital tactile-visual area of the brain, normally associated with the tactile and visual perception of shape, is activated by expert vOICe users (Amedi et al 2007). Other work in the field of sensory substitution suggests that different forms of synesthesia can also be acquired: Peter König’s feelSpace belt conveys orientation through vibrating touch, providing a augmented sense that some volunteers were able to integrate over time (see Bains 2007).
Are transcoded audiovisuals some form of sensory substitution or artificial synesthesia? There are two important differences. Sensory substitution operates by mapping an otherwise absent modality into an existing one; absent vision into existing hearing, in the case of the vOICe, and absolute orientation into touch, in the feelSpace belt. However for most, audiovisual transcoding links two existing modalities, “channels” already in perceptual use. Secondly, sensory substitution involves long-term integration and interaction with the environment; we can learn new “channels” but only by feeling out and (literally) incorporating their correlations with our existing sensory matrix. There are some striking parallels, and transcoded AV certainly hints at artificial synesthesia and a rewired sensorium, but as bounded aesthetic objects these works cannot realise that perceptual transformation.
The Pleasures of Binding
Correlation, key to artificial synesthesia, offers an alternative approach to the perceptual aesthetics of fused audiovisuals. At the core of transcoded and other tightly linked audiovisual forms is an experience based on a correlation between auditory and visual elements. While synesthesia offers a neurological analogy for the generation (poetics) of fused AV, this correlated quality leads into the neuroscience of perception, and thus offers a way to frame these works from the other side, the side of reception (or aesthetics).
The detection of correlations in the perceptual field is a normal, and crucial, perceptual task. From an ecological perspective, correlations underpin the recognition of objects in an organism’s environment. Our perceptual systems “bind” correlated elements into groups that often correspond to objects in our physical environment. A cat hiding in the garden might initially appear as an unrelated set of visual elements – a light grey splodge here, a dark shape there. When we “see” the cat, we detect correlations between those elements that enable us to interpret them as part of an underlying object. The image of a hidden Dalmatian dog (below) is often used to illustrate this phenomenon.
As Ramachandran and Hirstein (1999: 21-23) point out, this process of binding has some interesting features. Binding is “sticky” – we seem to hold on to bound perceptual elements. Once seen, the Dalmatian cannot be un-seen a without a conscious effort. Moreover, the act of making a binding is pleasurable in itself: “the discovery of the dog and the linking of the dog-relevant splotches generates a pleasant ‘aha’ sensation.” The authors offer an evolutionary rationale for this payoff: “The very process of discovering correlations and of ‘binding’ correlated features to create unitary objects or events must be reinforcing for the organism – in order to provide incentive for discovering such correlations.” Our limbic system apparently rewards us for detecting sensory correlations in our environment, even in advance of the final “recognition” of an object: “at every stage in processing there is generated a ‘Look, here is a clue to something potentially object-like’ signal that produces limbic activation and draws your atttention to that region or feature.” These incremental rewards “bootstrap” the final moment of recognition. Ramachandran and Hirstein work this perceptual pleasure principle into a neurological theory of aesthetic experience, suggesting that artists and designers seek out and intensify the pleasures of sensory binding, creating artefacts that “tease the system with as many of these ‘potential object’ clues as possible.”
Ramachandran and Hirstein also go further in proposing that the discovery of more abstract correlations is also reinforced by a limbic reward (1999: 31). They relate this to the ecological imperitive for classification – our evolved need to establish correlations that group and distinguish objects in our environment: say, edible versus inedible plants. This version of binding operates diachronically, rather than the synchronous binding of visual elements into a recognised form. “Being able to see the hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category… Consequently the discovery of similarities and the linking of superficially dissimilar events would lead to a limbic activation – in order to ensure that the process is rewarding.”
Cross-Modal Perception
How might processes of binding – the discovery of correlations – operate in fused AV, where the characteristic correlations are between, rather than within, sensory modalities? While studies of perception have traditionally focused on the senses in isolation, as independent neurological “modules", recent work has begun to explore the relations between sensory modalities. Media-based metaphors for perception encourage us to think of the senses as functionally distinct input channels. If sensory substitution shows that these channels can be re-wired, studies of cross-modal perception show that they are barely even distinct. The senses are involved in what Shimojo and Shams (2001: 506) describe as “vigorous interaction and integration,” mirroring Michel Chion’s description of the “mutual contamination” that characterises the audiovisual relationship in film sound ([1990] 1994: 9). Shimojo and Shams review experiments showing the range of these mutual influences: how vision can alter the content and spatial location of perceived sound; and how sound can alter the perceived intensity and timing of visual stimulus. We hear what we see, and see what we hear.
The perceptual trickery of these experiments is less interesting than what they suggest about normal perception. Just as the binding of visual percepts into a whole enables us to recognise objects in our environment, correlations in different sensory modalities cause us to bind those stimuli into a unified perception. This is illustrated with another trick, an experiment by Sekuler et al (1997), in which subjects were presented with two moving dots on intersecting paths. Two perceptual interpretations of this animation are possible: that the dots pass each other without touching, or that they collide and bounce off each other. Without sound, the former interpretation was dominant; however adding a brief sound at the crossing point biased perception strongly towards collision. This is an instance of cross-modal binding, where correlated stimuli in different modalities become fused into a coherent whole. It also suggests the ecological basis of cross-modal binding; that we interpret correlated events as cues to objects in the environment. The interpretation of sensory data seems to be shaped by pre-conscious processes that bind percepts into wholes; wholes that map onto ecologically plausible events. In the crossing dots experiment, sound binds with vision to alter our interpretation of the event. The correlated stimuli point to a common cause, a model that explains their coherence.
Fused audiovisuals are aesthetic objects founded on cross-modal binding. Ramachandran and Hirstein’s notion of the pleasures of binding applies here; in the transcoded AV of artists such as Fox and Gadow we experience sensory fields that are somehow entirely bound: completely self-consistent, devoid of extraneous elements. The affect that Ramachandran and Hirstein attribute to the moment of binding, the discovery of the Dalmatian – the ‘aha’ of recognition – seems to be intensified and prolonged here. It also suggests a connection between cross-modal binding and the noetic affect Cytowic identifies in synesthetic experience. If we accept the limbic payoff theory of aesthetics, then perhaps fused AV is a manifestation of this pleasure principle in the media arts.
Audiovisuals as Cross-Modal Objects
Cross-modal binding is not limited to experimental audiovisuals, however; in fact the opposite is true. Cinema and television constantly rely on our predilection for binding sound and image; this is the basis of Chion’s synchresis, a “spontaneous and irresistable mental fusion” caused by close synchronisation ([1990] 1994: 63). Lip sync is the archetypal example, where audiovisual correlation breathes life into the image of a body. Recall the ecological function of binding: to identify a common cause – an object in the environment. In most audiovisual media the objects are (all too) readily apparent. So if audiovisual correlations refer us to a shared cause, what is that cause in fused or transcoded AV? What is the underlying object, the cat hiding in the garden?
In a sound-to-image mapping, for example, it seems logical to propose that the cause is the source modality – sound. This involves a kind of reflexive redundancy; in Fox’s oscilloscope work, it would mean that the image is simply a pointer to the soundtrack, that it doubles or duplicates the sound. Subjectively at least, the relation seems richer and more complex than that; and it seems at odds with an ecological model of perception. Perhaps the common object is not the sound, but something more abstract: the signal. Signal here refers to a pattern of differences or fluctuations, a flux that, like data, must always be embodied but which, again like data, can be readily transduced between one embodied form and another. Fox’s laptop does not send sound to the oscilloscope, or in fact to the audio amp; it sends signal, a pattern of fluctuating voltage. That pattern is manifest on the scope as phosphorescent image, and when it leaves the speakers, as sound: but their common origin is the flux itself.
In transcoded audiovisuals sound and image perceptually triangulate a third point, the signal, that is imperceptible in and of itself. Signal maps to perception through the contingencies of both media technologies and sensory boundaries, but in itself it traverses these limits. This is apparent in Fox’s work, where subsonic fluctuations modulate the audible frequencies to create movements that are easily seen, but felt only as sharp thumps; the speakers struggle to transduce the signal into mechanical energy. Many of the complex, pointillist visual patterns are created by square-edged signal forms that again are acoustically impossible; the scope, more agile, is better able to trace them out. Similar trans-sensory signatures occur in Gadow’s work and that of other transcoders; Gadow’s Techne (2005) opens with a still blue screen and a raw, buzzsaw hum. The hum has no movement or form; as becomes clear as the piece develops, it corresponds to the blue video background. It is the sound of the 50 Hz scan-line structure of the video signal itself; so it looks like almost nothing. This is not to say that transcoded audiovisuals are reducible to the signal, an abstract or perhaps “higher” ideal, as in Goethe’s triangular model of color and sound. Here sensation and experience are foremost; these experiments feel out the ramifications of signal in specific circuits and transductions.
As Ramachandran and Hirstein suggest (1999:31), perceptual binding is both synchronic and diachronic, instantaneous and sequential. If the moment-by-moment audiovisual binding in these works refers us to their shared cause – the signal – how do these works operate in the diachronic axis? They often share a simple formal structure of establishment, development and elaboration, a successive playing-out of potential. Ramachandran and Hirstein state that perceiving “hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category.” We can think of the sequential similarities here as products of the constant, underlying structure that shapes all the outputs of the system. That structure is the map, the specific but abstract shape of the audiovisual correspondence. The map is an elusive entity; rather than an object we can think of it as a procedure, a verb or algorithm; a way of transforming between modalities and their shared signal. In Fox’s work the polar mapping of the oscilloscope is an algorithm that transforms phase – local relations in time – and amplitude into circular space. Considered as cross-modal objects, these works direct us to the underlying signal; and the signal is embodied audiovisually through the intermediary of the map. The map describes a space of potential, a range of possible correlations between domains; and it is that territory, I would argue, that these works reveal as they traverse it.
Inframedia and the Map
In earlier work on experimental sound I proposed the notion of inframedia, “a stratum below or within the mainlines of electronic media” (Whitelaw 2001:51). The noisy textures, resonant fuzz, glitches, crackles and pops of electronic music since the late 90s reveal “the sensory and affective textures of a media substrate, rather than media ‘content.’” That substrate is a critical domain; media infrastructures are more than technological artefacts; they are rapidly changing focii of power. Inframedia aesthetics reflect a consciousness of that domain, while in its processes such work often pursues local and particular manipulations, hacks or diversions of those media technologies.
Fused audiovisuals – a practice with close cultural links to experimental sound and music – can be approached along similar lines. Like hiss and hum, the audiovisual aesthetics of signal direct us to the abstraction and transduction occurring inside, or underneath, our media streams. Glitch-driven audio is founded on cracks in the surface – moments of interruption which allude to, and materialise, their own infrastructure. In a sense transcoded audiovisuals are a prolongation of those moments, leading to a flattening of the surface/depth dichotomy of glitch; the cross-modal coherence of this work is based on a sustained exploration of signal. Instead of mapping signal anthropomorphically onto perceptual “inputs”, these works show us where signal and affect meet or overlap, as well as where they diverge; they show us signal passing into, out of, and through perception.
These works also direct us to the map – the abstract space of possible transformations between signals. That domain of transformation is also inframedial, a key structure in digital media forms and cultures. Lev Manovich (2002) has described this question as the “built-in existential angst” of digital media: “By allowing us to map anything into anything else ... computer media simultaneously makes all these choices appear arbitrary….” In almost all digital media, the map – the pattern of relations between input and output – is imperceptible, obscured or encoded. This is clearest in the work of artists working explicitly with data inputs. In the work of Alex Drauglescu for example, spam email is used as the input to an algorithm that creates complex three dimensional forms. The mapping – the process that transforms spam into form – is never revealed, and so a concrete, specific process becomes a blank spot filled in with an impression of magical transubstantiation. In some computational work the artist provides source code, an explicit specification of the map but one that is highly encoded and unavailable, in itself, to perception. In most digital media objects, the map is inextricable from the residue or artefact it shapes. We perceive only the output, the image, sound or form, in which the input and its transformations are collapsed.
The wider significance of transcoded audiovisuals is that they approach a perceptual manifestation of the map, that space of transformation. We sense it, in these works, interpolated between each instant and every other. It’s perhaps not surprising that this characteristically digital figure is manifest through largely analog means; as well as a critical distance from the digital, analog signals offer transformations that are rich and immediate. Crucially the maps themselves are simple and static – highly reduced, compared to their digital counterparts – and so more available to the aesthetic and affective explorations of transcoded audiovisuals.
The prospect of somehow apprehending the map is both esoteric and pragmatic. The map is the inescapable intermediary, the necessary condition of our data-experience; but what is the map, what is its shape, how does it transform this into that? What are its conditions, limits, bounds? These works literally feel out the map, and in the process begin to address these questions, offering a sense of the abstract transformations that underpin contemporary digital culture.
Synesthesia is a powerful and persistent trope in the audiovisual arts. As shown here it offers some enticing parallels with the techniques and affects of audiovisual practice, yet as a techno-sensory analogy it has inherent limits. As in the visual music tradition, synesthesia plays a largely figurative role, and it demands critical scrutiny as such. In this investigation however, the synesthetic analogy has opened a path towards its more everyday converse, cross-modal perception, which offers a useful framework for a neuro-aesthetics of fused audiovisuals. These two approaches converge in the figure of the map, the space of correlation; the feeling of noesis or revelation common to both synesthesia and cross-modal binding, could be described as the affect of the map. That affect is central to the aesthetics of fused audiovisuals; though I would argue it offers more than a neurological hit; it brings us into contact with the abstract but culturally crucial terrain of the map itself.
References
Amedi, Amir, William M Stern, Joan A Camprodon, Felix Bermpohl, Lotfi Merabet, Stephen Rotman, Christopher Hemond, Peter Meijer and Alvaro Pascual-Leone. 2007. “Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex.” Nature Neuroscience 10: pp. 687-689.
Baker Fish, Bob. 2005. Review of Robin Fox, Backscatter. Cyclic Defrost 10. http://www.cyclicdefrost.com/review.php?review=795
Bains, Sunny. 2007. “Mixed Feelings.” Wired 15.04. http://www.wired.com/wired/archive/15.04/esp_pr.html
Chion, Michel. [1990] 1994. Audio-Vision: Sound on Screen. New York: Columbia University Press.
Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press.
Cytowic, Richard. 1989. Synesthesia: a union of the senses. New York: Springer Verlag.
Cytowic, Richard. 1996. “Synesthesia, phenomenology and neuropsychology: a review of current knowledge.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Fischinger, Oskar. 1932 “Sounding Ornaments.” Deutsche Allgemeine Zeitung (July 8). http://www.oskarfischinger.org/Sounding.htm
Harrison, John E., and Simon Baron-Cohen. 1996. “Synesthesia: an Introduction.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Hubbard, E.M., and V.S. Ramachandran. 2005. “Neurocognitive mechanisms of synesthesia.” Neuron 48(3): pp. 509-520.
Kandinsky, Wassily. 1977. Concerning the Spiritual in Art. London: Dover.
Manovich, Lev. 2002. “The Anti-Sublime Ideal in Data Art.” http://www.manovich.net/DOCS/data_art.doc
Marks, Lawrence. 1996. “On colored-hearing synesthesia: cross-modal translations of sensory dimensions.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Meijer, Peter. n.d. “Artificial Synesthesia for Synthetic Vision.” http://www.seeingwithsound.com/asynesth.htm
Peel, James. 2006. “The Scale and the Spectrum.” Cabinet 22. http://www.cabinetmagazine.org/issues/22/peel.php
Ramachandran , V.S., and E.M. Hubbard. 2001. “Synesthesia – A Window into Perception, Thought and Language,” Journal of Consciousness Studies 8(12): pp. 3-34.
Ramachandran, V.S., and William Hirstein. 1999. “The Science of Art: a Neurolgical Theory of Aesthetic Experience.” Journal of Consciousness Studies 6(6-7): pp. 15-51.
Sekuler, Robert, Allison B. Sekuler and Renee Lau. 1997. “Sound alters visual motion perception.” Nature 385: 308.
Shimojo, Shinsuke, and Ladan Shams. 2001. “Sensory modalities are not separate modalities: plasticity and interactions.” Current Opinion in Neurobiology 11: pp. 505-509.
Simner, Julia, Catherine Mulvenna, Noam Sagiv, Elias Tsakanikos, Sarah A Witherby, Christine Fraser, Kirsten Scott and Jamie Ward. 2006. “Synesthesia: the prevalence of atypical cross-modal experiences.” Perception 35(8): pp. 1024-1033.
Strick, Jeremy. 2005. “Visual Music.” In Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.
Whitelaw, Mitchell. 2001. “Inframedia Audio.” Artlink 21(3): pp. 49-52.
Zilczer, Judith. “Music for the Eyes: Abstract Painting and Light Art,” in Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.
Filmography
Fox, Robin. 2005. Backscatter. Videorecording. Melbourne: Synesthesia Records SYN012 DVD.
Gadow, Andrew. 2005. Techne. DVD-R courtesy of the artist.
Hodgin, Robert. 2007. “Trentemøller and Me.” http://www.flight404.com/blog/?p=52
McLaren, Norman. 1940. Dots. Animation. Available: http://www.youtube.com/watch?v=E3-vsKwQ0Cg
Mclaren, Norman. 1971. Synchromy. Animation. Available: http://www.youtube.com/watch?v=Jqz_tx1-xd4
Timberlake, Justin. 2007. “LoveStoned / I Think She Knows.” Zomba Recording. Music Video. Available: http://www.youtube.com/watch?v=GIYXHLlxD8U
Watz, Marius. 2005. “Video for @c: int.14/37.” http://www.unlekker.net/proj/cronica021/
