After the Screen: Array Aesthetics and Transmateriality

At the risk of some sort of blog-will-eat-itself situation, I'm posting this paper, presented at TIIC last November, which includes several threads developed here previously - arrays, transmateriality, and the work of HC Gilje. There are some new bits too however, on screens, projection mapping, and lots of tasty examples of a putative "post-screen" practice.

1. Glowing Rectangles

For all the diversity of the contemporary media ecology - network, broadcast, games, mobile - one technical form is entirely dominant. Screens are everywhere, at every scale, in every context. As well as the archetypal "big" and "small" screens of cinema and television we are now familiar with pocket- and book-sized screens, public screens as advertising or signage, urban screens at architectural scales. As satirical news site The Onion observes, we "spend the vast majority of each day staring at, interacting with, and deriving satisfaction from glowing rectangles."

Formally and technically these screens vary - in size and aspect ratio, display technology, spatiotemporal limits, and so on. They are united however in two basic attributes, which are something like the contract of the screen. First, the screen operates as a mediating substrate for its content - the screen itself recedes in favor of its hosted image. The screen is self-effacing (though never of course absent or invisible). This tendency is clearly evident in screen design and technology; we prize screens that are slight and bright - those that best make themselves disappear. Apple's "Retina" display technology claims to have passed an important perceptual threshhold of self-effacement, attaining a spatial density so high that individual pixels are indistinguishable to the naked eye (below - image Bryan Jones).


The second key attribute of contemporary digital screens is their tendency to generality. The self-effacing substrate of the screen is increasingly a general-purpose substrate - unlinked to any specific content type; equally capable of displaying anything - text, image, web site, video, or word-processor. This attribute is coupled of course to the generality of networked computing; since the era of multimedia the computer screen has led the way in modeling itself as a container for anything (just as the computer models itself a "machine for anything"). The past decade has simply seen this general-purpose container proliferate across scales and contexts, ushering us into the era of glowing rectangles.

However over the past decade in design and the media arts, a wave of practice has appeared which as this paper will argue, resists the dominance of the glowing rectangle. Given the near-total cultural saturation of the screen, this is unsurprising, given the ongoing cultural dance of fringe and mainstream in which this practice participates. This is not simply a story of resistance however. In proposing and describing two particular strains of "post-screen" practice, this paper aims firstly to outline the shared terms of their relationship with the screen, and in the process develop a more detailed sense of these conceptual devices of generality, outlined above, and its opposite, specificity. Secondly, and more briefly, it outlines a theorisation of this practice, invoking transmateriality, an account of the paradoxical materiality of (especially digital) media, and Gumbrecht's notion of presence.

2. Arrays

During the opening ceremony of the 2008 Beijing Olympics, a huge grid of drummers assembled in the stadium, each standing before a large square fou drum, a traditional Chinese instrument. Each drum was augmented with white LEDs mounted on its surface, triggered with each drum stroke. The drummers formed a vast array of discrete audiovisual elements, precisely choreographed in the style of these spectaculars. Human pixels, but coarse and resolutely human; at one point the drummers desynchronised entirely, forming a thunderous grid of flickering light. In a ceremony created for the (broadcast) screen - to the infamous extent of splicing computer-animated fireworks into its telecast in place of real ones - the drummers were a moment of involution. Their array echoed all the other, more conventionally self-effacing screens threaded through the event; but it also inverted some of their key attributes. Firstly its substrate, instead of receding behind "content", came forward; if anything substrate and content were one and the same. Secondly, while this array nods towards the generality of the screen in its choreographed patterns - which like the patterns on a screen could be "anything at all" - it veers strongly in the opposite direction, towards the here and now, what I will call specificity. As I argued at the time, the poetics of this array rely on the specificity of its elements - the drummers, drums, and their solid-state illumination - rather than the patterns that play across it.


The drummers are one popular example of a formal trope we can find throughout media arts and design practice over the past decade. Daniel Rozin's 1999 Wooden Mirror is one of the earlier examples. Wooden Mirror is an array of square wooden tiles embedded in a large octagonal frame, along with a bundle of custom electronics. The tiles are fitted with servomotors, so that each one can tilt up and down on its horizontal axis. As its angle to the light changes, each tile appears brighter or darker. Rozin wires up the array to a videocamera, to complete the mirror circuit: the brightness of pixels in the incoming image drives the angle of the tiles. Given the overtly visual logic of the work, it's interesting that its sound is equally striking: the wooden tiles clatter like mechanical rainfall, sonifying the rate of change of the image; as the image becomes still, the clatter dies off to a low twitching. Again, this array emphasises the material presence of its substrate. The tonal "generality" of the wooden mirror is functional enough to be familiar, but the coarse mechanical clattering of these pixels makes them inescapably specific.

Rozin has made many similar mirrors; notable is Trash Mirror (2001) where the individual elements - irregularly shaped pieces of rubbish - are packed into a freeform mosaic. This array moves one more step away from the homogeneous generality of the digital screen. Here the elements are irregular in size and shape, but also carry their own specific textures and colours. In Mirrors Mirror(2008) the regular grid returns, but the array elements are themselved replaced by mirrors; as these tilt they reflect different parts of the environment. Here the location of the tonal "content" in the array is, like the image source, deferred to the environment. In a familiar digital screen, image elements are luminous modules whose colour value is independent and absolute. In Rozin's Wooden Mirror that value becomes relative - tonality is based on self-shading, which depends on the lighting of the work. In Mirrors Mirror this relativity is multiplied; each element will reflect a different portion of the environment, depending on both its angle and the viewpoint of the observer.

In many cases these media art arrays depart from the two-dimensional grid entirely. Robert Henke and Christopher Bauder's ATOM (2007-8) (above) is an eight-by-eight grid of white helium balloons, each one fitted with LED illumination and tethered to a computer-controlled winch. The grid becomes a mobile, configurable light-form, tightly coupled with Henke's electronic soundtrack in live performance. This array lowers its resolution drastically, and limits its generality in one dimension (monochrome elements), but extends its reach (literally) into a third axis. ART+COM's 2008 kinetic sculpture at the BMW museum uses a similar configuration, but a higher "resolution" - in this case 714 metal spheres are suspended from motorised cables, forming a smoothly undulating matrix - a sort of programmed corporate ballet. Cloud (2008), a sculpture in Heathrow airport by London art and design firm Troika, illustrates another permutation: here a 2d array forms the skin of a large three-dimensional sculptural form. In this case the elements are electromagnetic flip-dots - components often used in airport signage before it was overtaken by glowing rectangles. As in Rozin's Mirrors, Troika consciously exploit the materiality, gestural character and the sound of these retro-pixels. rAndom International's 2010 Swarm Light demonstrates a "saturated" 3d array. The work consists of three cubic arrays of white LED lights, each ten elements per side; these cubic volumes host a flowing, flickering "swarm" of sound-responsive agents which traverse the space, brightening or dimming the array as they move.

The work of British designers United Visual Artists offers a useful longitudinal study in post-screen imaging; in particular their work addresses one of the central technical players in this field, LED lighting. UVA's first project involved a huge LED array that formed the stage set of Massive Attack's 100th Window tour. Unlike more screenful video backdrops, this low-res grid had an inescapable presence, hung directly behind the band and looming over the stage. Rather than an image machine, UVA treat the grid as a luminous dot-matrix for the twitching alphanumeric characters of real-time data. In subsequent work UVA develop this approach in a number of directions, but digitally articulated light - enabled by the LED - is a recurring theme. In Monolith (2006) UVA use a pair of large, full-colour LED screens, but treat them as a dynamic light source rather than a substrate for images; subtle gradients and washes of colour spill over the audience and into the installation environment, coupled with generated sound. In Volume (2006), another installation piece, the array elements are long vertical LED strips, again treated as generators of pattern, colour and sound; the work forms an interactive field as each element responds to nearby activity. In the context of this steady dismemberment of the screen, UVA's later work The Speed of Light is notable in that it leaves LED arrays aside entirely. Instead it uses installed lasers manipulated into dynamic, walk-in calligraphy, as if light had been finally prised away from its digital substrate, and turned loose in the environment.


Beyond their formal similarities, these arrays share some core approaches and contexts which provide a coherent portrait of a sort of post-screen practice. These works adopt one key feature of the screen - the "generality" of an articulated substrate - but trade it off to varying extents for more "specificity" - exploiting the local, particular materiality of the work and its environment. This specificity is also technological, reflecting a practice that crafts hard- and software into idiosyncratic configurations, rather than using off-the-shelf infrastructure. Light is a strong theme, in particular the solid-state, digitally addressable light of the LED (essentially a free-floating pixel). However the optical in these arrays is always tightly coupled with other modalities, especially sound, which is either a cherished byproduct of the array mechanism (as in Rozin's Mirrors and Troika's Cloud) or generated by the array elements themselves (as in the drummers and UVA's Volume). A quality of liveness is linked with the turn to specificity and being-in-the-environment; from the "live data" of UVA's Massive Attack show, to the live interaction and generation of their later installations, to the live video driving Rozin's Mirrors. Performance and temporary installation are the dominant forms here - emphasising the intensified moment, rather than the any-time of static content.

3. Projection Mapping and Extruded Light

In one sense these arrays present a disintegration of the screen - they pull its elements apart and embed them in the environment. In another strain of media arts practice, something like the converse occurs, though with what I will argue are similar interests and agendas. In this approach screen-like technologies are used intact, rather than decomposed; but their function and their relationship to the environment is transformed. These works reverse-engineer the digital image, exploiting its digital (general) malleability in order to fit it to a specific environment.

The work of Norwegian artist HC Gilje illustrates one trajectory of this second post-screen approach. Gilje's work from the late 90s was in live digital video, with his ensemble 242.pilots. This practice was linked to the burgeoning activity in experimental electronic music at the time; here again, performance, improvisation and the intensified moment - what Gilje calls an "extended now" - are central concerns, though the work is strongly screen-focused in its results . In Gilje's work over the following decade, he demonstrates another path towards the post-screen. Gilje's nodio (2005-) is a custom software system for distributing video content across collections of linked "nodes". In drifter (2006) these nodes are manifest as a ring of twelve screens which form a linked audiovisual interspace. With dense (2007) these nodes take on a more sculptural presence - hanging strips of fabric illuminated from both sides with a tailored video-projection. Here Gilje adapts the screen technology of the video projector to a sculptural environment, pushing it one step away from image and towards illumination. The work also depends on a specific material surface - the translucent weave of the fabric enables the double-sided layering of pattern.

shift (2008) (above) develops this approach: a technique known as projection mapping, in which the projected image is reverse-engineered to fit a specific surface. In shift Gilje's nodes are simple rectangular boxes, constructed from plywood. Using more custom software, the artist illuminates a cluster of these boxes with precisely mapped projected images. The coupled sound emanates from speakers housed in each box, so the objects are again audiovisual (and acoustically distinct) nodes; Gilje composes material for this environment in search of what he terms "audiovisual powerchords" - moments of intense juxtaposition and interplay. In blink (2009) Gilje dispenses with the boxes, instead treating the bare installation space. Simple, geometric elements - angular lines and bands of tone and colour - are reflected and modulated by the space itself, diffusing from irregular polished floorboards and painted walls. The work plays the room with articulated light, carefully matched to its geometry in way that heightens our awareness of the interplay of space, light and materials.

Projection mapping has recently flourished in "visualist" practice across art, design and performance contexts; trompe-l'oeil architectural facades are one popular genre, manipulating the built environment by rendering it with a tailored skin of articulated light (see for example Urbanscreen's Kubik 555). German designers Grosse 8 and Lichtfront demonstrate a logical extension of the technique, using multiple projectors to create an "augmented sculpture" in the round.

Another notable example is Scintillation (2009) (above) by Xavier Chassaing, a digital stop-motion film in which projection mapping is used to layer a domestic environment with luminous swirls of particles, igniting the petals of an orchid and tracing the curves of a moulded plaster cornice [24]. As in Gilje's blink, Scintillation emphasises the ambience of the projected light - reflections and diffusions are heightened by hand-held macro cinematography, artfully producing an impression of material texture. But in the process it raises some interesting problems for our analytical premise - a shift from the screenful image to something more live and specific. For Scintillation is absolutely a work of filmmaking; here projection mapping - the tailored materialisation of the image - is deployed as a technique for producing generalisable, substrate-independent image content.

The final example in this survey addresses the same tension. In their recent short film Making Future Magic (above), London design agency Berg give an ingenious demonstration of both the material turn of post-screen imaging, and its recuperation as image content. Berg developed an animation technique combining multiple-exposure stop-motion with a hand-held source of articulated light - specifically the glowing rectangle of the moment, Apple's iPad. 3d forms are digitally modelled and animated, then decomposed into sequences of 2d slices. These slices are then replayed into the environment, and thus recomposed into 3d forms, by moving an iPad screen over successive still frame exposures. As Berg term it, this is "extruded light" - as in UVA's latest work, it's as if light itself has been unpinned from its substrate. The results are a beguiling combination of loose, organic light painting with simple 3d geometry and DSLR imaging. As Berg frame the work, it fits entirely within the post-screen turn proposed here. Responding to a brief around "a magical version of future media", Berg are "exploring how surfaces and screens look and work in the world ... finding playful uses for the increasingly ubiquitous ‘glowing rectangles’ ...". Again the material embeddedness of this articulated light is emphasised - the way it reflects from puddles and diffuses through foliage. Screen as object in the world, rather than window to somewhere else. As in Scintillation however the inescapable irony is that the outcomes of this work are entirely bound up with screenful images - with the generalising infrastructures and distribution pipelines of social image sharing, print-on-demand and networked video.

4. Transmateriality and Presence Culture

To recap briefly: the ubiquitous digital screen is characterised by both generality - an ability to display any content at all - and self-effacing slightness - it tries to make itself disappear as a neutral substrate for content. In contrast to these tendencies this paper describes two distinct but parallel strains of "post-screen" practice in the media arts and design. Arrays mimic the grid configuration of the screen, but lower its resolution and emphasise the material presence of the array elements - their local and individual specificity is balanced with their malleable generality (their ability to carry anything-at-all). Projection mapping and "extruded light" practices also emphasise specificity, materiality and a local, performative being-in-the-world, but they do so by different means - exploiting the malleability of the digital screen (and the computational representations it hosts) in order to make it intensely site-specific. To the extent that they both adapt and resist the attributes of our familiar glowing rectangles, we could describe these practices as post-screen, but this "post" is nothing like a conscious critique, let alone a revolutionary break. However hard they may pull towards specificity and local materiality, they are readily - by design or necessity - recaptured as screen fodder.

Both these post-screen tendencies and their screenful recuperation can be usefully framed through the notion of transmateriality, a concept that attempts to capture a fundamental duality in digital (and other) media: they are everywhere and always material, yet often function as if they are immaterial. In a transmaterial view media always operate as local material instances (this is their aspect of specificity) yet retain the ability to hold specificity at bay - resisting the contingencies of flux - to create a functional generalisation in which this pixel is the same as that one, the email I send is the same as the one you receive, and one node on the network is much the same as any other.

In the glowing rectangle paradigm functional generality is entirely dominant. The work considered here, on the other hand, revels more in the pleasures and practices of specificity - the clatter of servo-actuated wood or the play of light on this particular wall. In their push towards liveness (of interaction or data), performativity, their integration of sound, and their emphasis on evanescent materiality, these works evoke what Hans Ulrich Gumbrecht would call "presence culture" - that mode of apprehending the world which is characterised by fleeting but intense moments of being, and a sense of being part of the world of things, rather than outside it, looking in. Gumbrecht constructs presence in opposition to a dominant "meaning culture", in which the essence of material things can be obtained only through interpretation. Gumbrecht describes the relationship between these poles as one of dynamic oscillation. "Presence phenomena" become "effects of" presence, "because we can only encounter them within a culture that is predominantly a meaning culture. ... [T]hey are necessarily surrounded by, wrapped into, and perhaps even mediated by clouds and cushions of meaning".

In exactly the same way we find an inevitable oscillation here between screen and post-screen. We can align the screen with generality and meaning culture, and the post-screen with specificity and presence culture; but here too the post-screen is evanescent and elusive, instead existing largely within the dominant screen culture. However this is not to discount the utopian aspirations of a post-screen practice, which might instead be located through the perspective of transmateriality. For in echoing the screen, or in literally bending it to the local, present and specific, these works operate as reminders of the ubiquitous and everyday materiality of our media, of the fact that depite appearances, every glowing rectangle is already local and specific. If that specificity is latent, then these works demonstrate practical strategies for making it explicit; from hardware hacking to modular LEDs and custom software, they participate in what might be called "expanded computing", using the malleability of digital media to reactivate its presence - and thus our presence, too - in the world of things.

Read More...

Summary only...

Right Here, Right Now - HC Gilje's Networks of Specificity

This essay was commissioned by Hordaland Kunstsenter in Bergen, Norway, to coincide with HC Gilje's solo exhibition blink (video below). It looks at Gilje's recent work - which spans audiovisual installation, performance, hardware, and networked forms - through the notion of specificity (developed earlier here).

The digital network, where we all spend ever more of our time, is a vast infrastructure of generality. It deploys a system which is standardised, formally defined, highly structured, and internally consistent. If I send you an email, I do it trusting that the interlinked systems of hard- and software, the protocols for data encoding and transmission, the network switches and servers, will all hold together so that the email you receive is the same as the one I sent. Perhaps I'm in Australia, and you are in Norway: we could say that the network generalises our two points in space - for the network, they are the same. As I draft my email it exists as a pattern of voltages and magnetic flux inside my computer. To transmit that pattern effectively, the digital network must erase or resist any local errors or inconsistencies that it might encounter along the way, so that it does not matter if the pattern travels by optical fibre or copper, or in radio waves, or if a boat anchor cut through a cable near Indonesia. It does not matter that your computer is made of different atoms to mine. Those are specificities - local, material events and instances. Digital culture, and networked space, absorbs specificities, compensates for them, rectifies them into generality. Wireless broadband and mobile computing make us into human nodes, bathing in shared connective protocols.

The aesthetics of digital media flow from a related generality, where sound and image are encoded as fields of data. If a pixel is a number, an image is a grid of pixels, video a stream of images, and each of these numbers can take any value at all, then formally, an aesthetics of digital video is only a matter of finding the right values - fishing around in a space containing all possible digital video. If digital media creates this generalised space, anything at all, the media arts are faced with unavoidable questions: not only what to make - which values to choose, but how to choose them, and why?

HC Gilje's work arises from a moment when the anything-at-all of digital video was just opening up, thanks to a combination of new real-time tools, cheap computing power, and some key interdisciplinary influences. Drawing on experimental sound and music, improvisation and performance became important solutions; working live in a specific situation, artists would gather, process, generate, and recombine material. In work from the late 1990s and early 2000s, from Gilje and his collaborators in 242.pilots, as well as video ensembles such as Granular Synthesis and Skot, the result is abstract and intense, a flow of layered digital texture. In performance it saturates the body and senses; big screens, big speakers. Instead of the narrative transport of cinema, which takes us somewhere else, this work creates - and is created in - an intensified sense of presence, what Gilje calls an "extended now". This methodology is vital; it focuses the open-ended generality of digital media in to a point: on this, rather than anything-at-all.

This moment relies on a circuit, a close coupling between artist and media; data flows become experienced events - sounds and images - which in turn inform new data flows, and so on. Audience and performers share a digital-material situation. The specificity of digital media comes forward; for of course these media are always specific, always local, always embodied; but that specificity is usually suppressed by the functional logic of generality. At the same time though, the processes underway here depend on exactly that generality, on the machine's ability to rapidly transform data and shift it between instantiations - from the voltages in video memory to the patterns of projected light.

In nodio (2005-) Gilje creates a system of networked audiovisual nodes that process and share image material. Each node generates sound derived from its image, in a process of automatic translation. On one hand this translation is another demonstration of the abstract pliability of the digital - its ability to transform anything into anything (generality); on the other, its tight audiovisual correpondences generate sparks of material intensity - real events, rather than digital effects (specificity). With these distributed nodes Gilje deploys audiovisual materials in space, creating flows and juxtapositions that function as dynamic sculpture. Of course the formal model of nodio echoes our most ubiquitous generalising paradigm: the network. Once again, the artist applies this digital tendency for generalisation in order to cultivate instances of specificity - the texture and sensation of the here and now.

From drifter (2006) (above) to dense (2006) and shift (2008) (below), Gilje's audiovisual nodes map out a developing exploration of specificity. drifter deploys standard computer hardware, formed into sculptural modules; in passing material between nodes Gilje begins to break the frame of the screen, creating an implicit inter-space. In dense, the hardware moves out of the sculptural field, and the screen is further deconstructed. Instead of the frontal configuration of the cinema / computer, these suspended fabric strips are illuminated from both sides with a video "weave". The familiar architecture of the screen as a blank (general-purpose) substrate containing or supporting image content, is reconfigured here; the specific materialities of screen and content overlap. Even more so in shift, where the nodes are now wooden boxes, illuminated with precisely controlled video projections. As in earlier nodio works sound and image are directly related. Here Gilje extends this fusion to the sculptural objects; each node is also its own speaker-box, so that the digital articulation of sound and image is realised, and grounded materially, in the nodes themselves.

These works drive towards a spatial materialisation of audiovisuals: dynamic constellations of AV intensity, fields for what Gilje calls "audiovisual powerchords". The projectors, speakers and networks of the nodio works present one means to this end, deploying existing media technologies. Again we find an interplay of generality and specificity, as Gilje adapts generalising systems - projectors, computers, networks - to realise materialised instances. The Wind-up Birds (2008) (below) represent another angle of approach; Gilje sets video aside, and creates materialised, local, sculpturally autonomous nodes from electronic and mechanical materials. In these robotic woodpeckers digital media and sculptural embodiment are further enmeshed. The birds communicate using digital radio, and their behaviour is programmed in a custom chip; but their sound is simply percussion - a mechanical switch, tapping on a specially constructed wooden slit-drum. Again this is specificity over generality: a loudspeaker is an acoustic shape-shifter, a technology which promises any sound, in the same way that the screen promises any image. By contrast the Birds produce only one sound, their sound, a specific conjunction of solenoid, timber and vibrating air.

The Birds will run for a month on their own batteries, strapped to trees, calling to each other and any other creatures nearby. These nodes are unplugged: they begin to come away from the technological support system of mains power and the shelter of the gallery or studio, and move out into the world. As in the artist's other work the engineering here is inseparable from the artistic agenda; the Birds are in that sense a realisation of Gilje's spatial and formal aims, an autonomous constellation of intensities. But they also literally expand from there; where the nodio works explore the composition of spaces within a network of intensities, the Birds move outwards, creating points of intensity in the wild, and evoking a spatial alertness - a way of being in and listening to the world - that extends beyond the well-marked edges of an artwork. The Birds are more like an experimental intervention, a digital-material overlay in a complex field of the living and non-living.

Similarly the Soundpockets works (both 2007) make small sonic interventions in urban spaces, pursuing local intensification and juxtaposition through directional soundbeams and micro-scale radio transmissions. Once again we find this interplay of the general - the anything-at-all of the digital - and the specific, the here and now. The "extremely local radio stations" of Soundpockets 2 form a sort of folded juxtaposition of three layers: globalised network infrastructures and protocols, the traced or mediated locations of field recordings, and the specific time and place of the transmissions. Just as Soundpockets 1 uses exotic soundbeam acoustics to perturb urban spaces, Soundpockets 2 shows how we can draw in technological infrastructures in order to reconfigure the real environment, creating flows and distributions that form intense moments of difference and specificity.


In this reading Gilje's work is partly critical. Pursuing specificity, and an intensified, material experience of the here and now, it pushes against the generalising tendencies of digital media. By the functional logic of the network, each node is formally identical, and must be effectively insulated from its environment. Ubiquitous computing promises us "everyware" - total connectivity, the complete interpenetration of the network and our lived environment [2]. But if the network is a generalising force, if it erases differences between places, what will life in "everyware" be like? Gilje's work suggests a utopian alternative: networks that are always local in time and space; nodes of right here, right now. Gilje's work strives for what Hans Gumbrecht calls "presence"; a way of knowing the world that is characterised by intense moments of encounter or revelation - aesthetic experiences that place us in the world, and of it, rather than observing from the intellectual distance of interpretation.

The beauty of Gilje's work though is that it not only suggests this prospect, but demonstrates it, makes it happen; and in that sense the work is constructive, rather than critical. In emphasising the specificity of media technologies, Gilje's work shows us a different way to frame those technologies; as always material, always in the world with us - a view I have called transmateriality. As Matthew Kirschenbaum writes, "computers ... are material machines dedicated to propagating a behavioral illusion, or call it a working model, of immateriality." Gilje shows us both sides of this statement, the functional illusion - generality - and its material foundation - specificity. It shows us a way to reframe the network, too; as always local, always specific; a tangle of real flows and propagating patterns; and endless possible ways of reconnecting the world with itself. Finally Gilje shows us one crucial role for the artist, in this context: seeking out configurations that intensify, rather than dilute, our sense of being in the world.

Read More...

Summary only...

Transduction, Transmateriality, and Expanded Computing

In common usage a transducer is a device that converts one kind of energy to another. Wikipedia lists a fantastic variety of transducers, mapping out links between thermal, electrical, magnetic, electrochemical, kinetic, optical and acoustic energy. In this form transducers are everywhere: a light bulb transduces electrical energy into visible light (and some heat). A loudspeaker transduces fluctuations in voltage into physical vibrations that we perceive as sound.

In analog media, transduction is overt (put the needle on the record...). But digital media are riddled with it too. Inputs and output devices all contain transducers: the keyboard transduces motion into voltage; the screen transforms voltage into light; the hard drive mediates between voltage and electromagnetic fields. A printer takes in patterns of voltage and emits patterns of ink on a page. Strictly transduction only refers to transformations between different energy types; here I want to extend it to talk about all the propagating matter and energy within something like a computer, as well as those between that system and the rest of the world. From this transmaterial perspective a computer is a cluster of linked mechanisms and substrates; a machine for shifting patterns through time and space.


If this sounds unfamiliar, it's only by historical accident. Mechanical computers, where these patterns are physically perceptible, predate electrical (let alone digital) ones, by centuries (above: a replica of Konrad Zuse's Z1, a mechanical computer from 1936. Image by rreis). Materially, our current computers are more or less black box systems. Their transductions come as a sort of preconfigured bundle or network, a set of familiar relations constructed again by mixtures of hard- and software, protocols, standards: generalising frameworks. I press a key, a letter appears; this is all I need to know. Click "OK". No user-serviceable parts inside.

Except that currently, across the media arts and a whole slew of other fields, the computer is undergoing a rich and productive decomposition. It's composting, to borrow a Sterlingism. This goes under all kind of different names: hardware hacking, device art, homebrew electronics, physical computing. Such practices mount a direct assault on the computer as a material black box, literally and figuratively cracking it open, hooking it up to new inputs and outputs, extending and expanding its connections with the environment. Microcontrollers like the Arduino present us with nothing but a row of bare I/O pins. Finally we can tackle the question of what should go in, and what should come out: of transduction. A whole generation of artists, designers, nerds and tinkerers are taking up soldering irons and doing just that. Below: the Spoke-o-dometer from Rory Hyde and Scott Mitchell's Open Source Urbanism project.


One side-effect of this decomposition of computing is that the ontological status of the digital starts to break down with it. As Kirschenbaum shows brilliantly, the digital is just the analog operating within certain tolerances or threshholds. Thomas Traxler's The Idea of a Tree (below) is a solar-powered system that fabricates objects from epoxy, dye and string, by turning a spindle. Solar energy generates electrical energy, which drives the motor, which draws the string through the dye and onto the spindle: a chain of analog transductions produce an object that manifests specific changes in its local environment. The work is a beautiful demonstration that variability doesn't have to be worked up with generative code: if the system is open to it, it's already there in the flux of the material field.

This is not to dismiss computing, only to recast it: an incredibly dynamic, pliable set of techniques for manipulating the material environment. Paradoxically the very generalities of computing - the abstractions and protocols that insulate it from local, material conditions - make it a powerful tool for transduction, that is, the propagation of specificities. Usman Haque's Pachube is a generalised infrastructure, a set of protocols and standards that rest in turn on wider standards like XML, and which assume a whole stack of functional layers: IP, HTTP, and so on. All in order to propagate material patterns and flows from here to there: this is an architecture of transduction whose utopian aim is to "patch the planet" into a translocal ecology of linked environments.

Digital fabrication is part of the same shift: an expansion and extension of the computer's range of material transductions. Digital pattern, to lasercutter instructions, to physical form. Fabbing shows how material matters. It's unsurprising that a piece of laser-cut ply is aesthetically different to a luminous pattern of pixels; more interesting is the way computation reaches out into the substrate's material properties, and the range of potential applications and domains it opens up. Fabbing has often presented itself with a narrative of materialisation, making the virtual real, translating bits into atoms - Generator.x 2.0 was subtitled "Beyond the Screen." Not so: because of course, the "virtual" never was, and the screen is material too. Fabbing does get us beyond the screen, but only because its processes and materials have different properties, different specificities, and they hook us up to new contexts, as well as new sensations. (Below: Andreas Nicolas Fischer & Benjamin Maus: Reflection - from 5 Days Off: Frozen)


Transduction suggests a way to link practices like physical computing, fabrication, networked environments, and many more. Data visualisation - in the broadest sense, from poetic to fuctionalist - is about creating customised transductions, sourcing new inputs and/or manifesting new outputs (even if they don't reach "beyond the screen"). We could add tangible interfaces, augmented reality, and locative systems. What does all this amount to? In 1970 Gene Youngblood observed a similar moment as the dominant cultural form diversified into a networked, participatory, interdisciplinary field of practices. He called it expanded cinema. So perhaps we can call this expanded computing: digital media and computation as material flows, turned outwards, transducing anything to anything else.

Read More...

Summary only...

Aspects of Transmateriality: Specificity

Transmateriality is a notion I'm working on that treats the digital as always and everywhere material - embodied from "end to end" - while maintaining a sense of how the digital functions as if it were immaterial. The core idea is well stated by Kirschenbaum (blogged earlier): "Digital systems are material systems designed to support an illusion of immateriality."

My proposal is that this view of the digital as material is particularly useful in looking at contemporary media arts. It resonates with practices across visual and sonic modes, generative art, data aesthetics; for me it also connects with Gumbrecht's presence and its tropes of manifestation and revelation. I'm trying to frame it through a handful of aspects or themes, provisionally: specificity, transduction, presence, ubiquity, materialisation and propagation. These aspects are inevitably connected - I'll add the links as they accumulate. In this post, specificity.

The digital is premised on generality; the ability to transduce a pattern from one instantiation to another, such that the pattern is effectively (but only effectively) independent of its substrate.
As Kirschenbaum points out, computing machinery works hard to support this generality, with the careful tuning of tolerances and threshholds, and the active interventions of error correction. Without these mechanisms a million entropic, material variations would creep in; dust motes, temperature variations, mechanical wear, noise. (Note how often these relate to the materiality of the substrate.) These would be incursions of specificity into the digital: local accidents, conditions of this or that substrate. The aesthetics of glitch reveal the material specificities of digital media systems by focusing on these incursions and cataloguing their qualities. So while the digital in general relies on holding specificity at bay, there seems to be a wave of creative interest in the specific material conditions of how the digital is manifest. Glitch is one clear example, but so is fabbing - more on that later.


The screen is the ultimate general-purpose substrate of the media arts: a homogeneous, uniform, dense, self-effacing surface. Yet recently we've seen a wave of arrays that can be read as anti- or post-screens: special-purpose displays that acknowledge their physical substrates. Think of Troika's Cloud (or indeed Rokeby's Cloud), Daniel Rozin's mirrors (above, his Wooden Mirror), or Art+Com's kinetic array for the BMW Museum (video). These "displays" show a renewed interest in the specific conditions of the manifestation of data - its local materiality (even presence) - rather than its abstract generality. They are also open displays of transduction: they tease apart the elements of the display to show how each one is discrete, addressable; a single micro-instantiation.

Digital sound and music - especially where it is real-time performed / improvised - also illustrate this turn towards specificity. A musician's rig is often a highly specific bricolage of hard- and software, acoustic and material sources, diverse technologies patched together. Oren Ambarchi's networks of effects pedals, motorised cymbals, and vestigial guitars for example. Performance in this genre is focused again on the conditions of instantiation, on specific transductions again, and how these circuits are materialised, how they vibrate in the air and in the assembled bodies, PA, room. Music also shows the interplay of specificity and generality at work here (and in the visual examples) - in Hayles' formulation this is incorporation and inscription. I can download Ambarchi's recordings and listen to them in my lounge room; I can make a faithful transduction, store it, back it up, copy it to my phone (always still materialised). The specificity that marks the artist's process recedes and instead becomes content for the functional illusion of digital generality. And then as it is reincorporated, materialised coming out of the speakers, it's specific again, folded into the everyday present of the lounge room and the evening.

Transmateriality is a useful concept, I'd argue, because among other things it can encompass this whole process without introducing ontological distinctions (or magical transformations) between one kind of thing and another - between data and matter. How does our view of computation - and the media arts - change if we think of it all as ultimately the propagation of material patterns? This involves throwing all kinds of useful abstractions out the window, at least initially - like data itself for example, or software. But my hunch is that if we can suspend them temporarily, they might return in a more interesting form. Your thoughts welcome, as ever.

Read More...

Summary only...

There is no Software - Kittler and Evolvable Hardware

I'm slowly developing this notion of transmateriality; in this post, some media theory and a nice example from computer science. In the next I'll try to connect all this with some current art work, and back to the notion of the transmaterial.

Thanks to a prod from my friend Brogan Bunt, I've been reading Friedrich Kittler, a literary and media theorist who has made some striking forays into computational media. In a paper from 1995 he grapples, like Kirschenbaum, with the grounding of computation in matter, distilling this position to a wonderful aphorism: "there is no software." Kittler begins by announcing the end of writing; that texts "do not exist anymore in perceivable time and space" but have been miniaturised to the scale of integrated circuits. This miniaturisation, in which writing escapes the bounds of human perception, is facilitated by Turing's core principle of computing, which sets out minimal conditions for computation and proves its independence from hardware - the ability for any number of different physical machines to implement a universal computer. This principle, Kittler says, "has had the effect of duplicating the implosion of hardware by an explosion of software." "Ordinary language" is overtaken by a new hierarchy of programming languages, layers that reach from the command line down to assembler and the very protocols embedded in the silicon itself.

Kittler plays out this "descent" in which each layer depends on the one below it; the word processor depends on DOS, which in turn rests on the hardwired BIOS. Ultimately "[a]ll code operations, despite their metaphoric faculties such as "call" or "return", come down to absolutely local string manipulations and that is, I am afraid, to signifiers of voltage differences." In other words, although they resemble "ordinary" language, programming languages return us to the imperceptible, inaccessable, too-fast and too-small world of the microprocessor. So, "there is no software at all," except as defined by the "environment of everyday language" that surrounds computation.

In a paranoid turn Kittler analyses the tendency of computer culture to "systematically obscure hardware by software, electronic signifiers by interfaces between formal and everyday languages." In the GUI, but also in high level languages such as C (let alone Java), the physical machine is increasingly concealed from its users, and its programmers, in the name of functionality and "friendliness."


Finally Kittler considers the limits and thresholds of "programmable matter"; he points out that current computing hardware relies on the isolation of discrete elements from each other and thus a limit to connectivity. This contrasts with the "maximal connectivity" of the physical systems - "waves, weather and wars" outside the computer. The current approach to computing hardware is essentially more of the same: more transistors, more elements, smaller circuits with better isolation. Kittler instead suggests that the only way to "keep up" with the physical complexity of the world is to match it with "nonprogrammable systems" made of "sheer hardware": "a physical device working amidst physical devices and subjected to the same bounded resources." In such devices once again "software as an ever-feasible abstraction would not exist any more."

Interestingly Kittler ends up close to where my earlier work on art and artificial life (published as Metacreation) came to rest. In considering the desire for emergence in a-life art, I wondered about the constraints imposed by the physical substrate - the "coarse, rigid grammar" of digital electronics. My favourite demonstration of what lies beyond this grammar is the evolvable hardware work of Adrian Thompson, in which circuit designs for programmable chips (field gate programmable arrays - as in the image above) are evolved using a genetic algorithm and tested in hardware for their performance in a particular task. It's perhaps not surprising that successful circuits were evolved over many thousands of generations; but the fun part is that when analysed, these circuits were completely unlike any human-designed computing machine. In Kittler's words they were "sheer hardware," treating the chip as a "maximally connective" physical substrate rather than an abstracted set of discrete elements. Some chips drew on external influences, such as electromagnetic radiation, to achieve their evolved ends; so the chip is not formally isolated (but to quote Kittler again) "a physical device working amidst physical devices." Not only that, they were often "tuned" to the physical specificities of a single chip, despite the FPGAs being notionally identical - the exact opposite to the hardware-indpendence of the Turing Machine.

Read More...

Summary only...

Notes on Transmateriality

At the recent UTS symposium I gave a short presentation titled "After Inframedia: Presence and Transmateriality." The presence stuff I covered earlier, but the second idea - which I touched on very briefly in this 2003 paper - is much less developed. So here goes.

The relationship between matter and "information" or "the digital" has been a recurring theme in new media theory for more than a decade. We could sketch it very roughly as follows. In the early to mid 90s, as digital hype was gathering pace, artists and cultural theorists began to critique the apparent drive towards disembodiment in technoculture. Simon Penny's 1991 text "Virtual Reality as the end of the Enlightenment Project" is a good (and early) example, even if VR now looks a bit like a straw figure in these critiques. This critical project of grounding the digital in the material (and the body) has continued. In 2000 Felix Stalder wrote of the "ideology of immateriality" underpinning the so-called "new economy." Around the same time Katherine Hayles published a more complex investigation in How We Became Posthuman, asking "how information lost its body" but also considering the inevitably embodied effects of this supposedly immaterial stuff (this is well covered in her paper The Materiality of Informatics).

Hayles introduces a conceptual pair: inscription and incorporation. Inscription is "normalized and abstract ... a system of signs operating independently of any particular manifestation" [Posthuman 198]. Inscription refers to the properties of a text, for example, that can be transcribed without regard to its specific embodied manifestation - digital computation thus relies on inscription, in moving patterns of data through various substrates. Incorporation is its flip-side, referring to the inescapably embodied aspect of a sign. Both inscription and incorporation are verbs - practices or processes - rather than ontological states; and they oscillate, a bit like presence and meaning for Gumbrecht: "incorporating practices are in constant interplay with inscriptions that abstract the practices into signs" [199].


Today I came a cross a more recent paper by Matthew Kirschenbaum, who pursues this investigation into the materiality of the digital, and like Hayles is approaching it from the perspective of textuality. In “Every Contact Leaves a Trace” (pdf) Kirschenbaum critiques the neo-Romantic, screen-focused tendencies of digital textual theory that tend to emphasise ephemerality and instability. He uses digital forensics to moves us from the screen to the hard drive, showing exactly how data is embodied (as in this image: a magnetic force microscopy image of a hard drive surface, from Pacific Nanotechnology). In the process he introduces another pair of concepts: formal and forensic materiality. Formal materiality refers to machine-readable data that reveals material specificities - in Kirschenbaum's paper, the use of a hex reader to discover traces of not-quite overwritten game code on an old Apple II floppy disc.

Forensic materiality refers to the material residues or byproducts that mark out one digital instantiation as different to another; for example the physical instantiation of copies of a file on two different hard drives will be different due to the material specificities of the drives - as when a misaligned write head again leaves traces of overwritten data. Yet these files are, for the computers concerned, formally identical. As Kirschenbaum writes, this shows how

"computers ... are material machines dedicated to propagating a behavioral illusion, or call it a working model, of immateriality."

This really nails it for me. It's exactly the functionality of this immateriality that earlier critiques of the disembodied digital overlook. It is an illusion, but it's an illusion that (mostly) works, and so is easily maintained: this is a hard-working model.

I'm developing an idea of transmateriality (sorry about the coinage), that draws on Hayles and describes exactly the "conundrum" that Kirschenbaum poses here; but that also has, I think, some wider implications, specifically for the media arts. Briefly, it proposes that the digital is, of course, always and inevitably embodied; that concepts like "data" are functional abstractions for describing the propagation of material patterns through material substrates. But that at the same time these material patterns - and here I mean everything from optical pulses to hard disk substrates, luminous screens and speakers pushing air - these material patterns, and the sensations and aesthetics that result are profoundly shaped by data acting as if it were symbolic and immaterial. Transmateriality is an attempt to "ground" the digital without losing sight of its (let's say) generative capacities. It also seems to resonate with a lot of current work in the media arts - but more of that later.

Read More...

Summary only...