This text was an invited contribution to Kerb 17: Is Landscape Architecture Dead? This looks like a rich volume with a sharp critical edge, and a swathe of interesting material spanning architecture, urbanism, art and landscape. Unfortunately my contribution was edited fairly severely; so here's the unabridged version. Redundancy warning for regular readers: there's a slight rehash of Watching the Sky in here; but afterwards there's fresh material on landscape / data projects by Driessens and Verstappen and Usman Haque.
Data is, we imagine, an immaterial thing; or at least ethereal, made of light and electricity, processed at superhuman speed, transmitted in real time. The everyday world we move in seems dense and slow by comparison. The landscape is slower again; thick, heavy and persistent. At the moment however those two domains, the fast lightness of data and the heavy slowness of the landscape, are urgently linked. We are faced with the prospect of momentous change in the landscape that is somehow both slow and fast; too slow for our real-time culture to grasp, and too fast for the living systems of the landscape to adapt to. This paper presents a handful of works that dwell in that disjunction, between landscape and data; not solving it at all, but at least forming links, complicating assmptions, and recasting the relationship between two terms that seem to neatly encapsulate our future.
In Watching the Sky a camera looks out my office window, at the sky and the landscape. A banal view over a university campus to a bushy ridge in Belconnen. The camera takes an image every three minutes; four hundred and eighty images in twenty four hours. Tethered to a computer, the camera records for weeks at a time; the computer accumulates thousands of images. I think of the images as data, traces of change in the world outside the office window. I visualise, or re-visualise, this image data in the simplest possible way; an automated process "cuts" a narrow vertical slit from the same location in each image, and compiles all these slits together (this is a digital imitation of an analog photographic technique known as "slit-scan"). In the rectangular visualisations the slices are tiled from left to right. In the radial visualisations slices are gradually rotated so that a twenty-four-hour period spans one complete revolution (the "seam" is at midnight).
In the resulting images the patterns of change within and between days are immediately visible. As I imagined, day and night, cloud and sky are obvious. The brief, delicate colour shifts of dawn and dusk were more surprising. Below the horizon, though, patterns appeared that complicated the work's nominal focus on the sky. It became clear that some of the richest and most revealing data here came from the landscape. In one of the earliest sketches I found small but distinct variations in the horizon line over the course of a day, and recurring on successive days. I eventually realised that this was caused by the afternoon breeze, shifting foliage by a few pixels within the frame. In other words, subtle changes in the material field of the landscape carried through to the image data. Moreover in many ways the landscape visualises its own internal structure: the trees blowing in the breeze are partly instruments, revealing material changes around them (the breeze); but also data, traceable as pixels. In many images the passage of a shadow across the ground appears as a recurring pattern, an enfolded or multiplexed representation of another set of material interactions; the landscape measures and reveals itself, but not as an object, image or view. It is a connective, dynamic, material system; what is revealed are the specific interactions of that system with itself. The image data acts as a kind of core sample, drilling through multiple spatial and material systems, but each is connected outwards, beyond the frame. The wind in the trees doesn't belong to this image, but like the angle of the sun revealed in the shadow, is an index of a wider system.
It also became clear that the landscape is densely packed with human, social data which is equally apparent in image data. In the rectangular visualisations presented here stripes of colour are visible towards the bottom of the frame. These are caused by cars, parked illegally under the trees; they form another ad-hoc graph that reflects cultural, institutional calendars and cycles, though again they are intermingled with other scales and structures.
Landscape is also cast as a self-revealing instrument in Driessens and Verstappen's Tschumi Tulips project. This landscape installation occupied the Tschumi Pavilion, in Hereplein, Groningen, during the northern Spring in 2008. The pavilion is a rectilinear glass container, rising at an angle from the surrounding park. In this installation the artists filled the base of this box with soil and planted over ten thousand white tulips. A matching array of tulips was planted outside, extending the line of the pavilion. Like scientists, the artists set up two identical subjects, but vary their environment: ten thousand tulips inside, ten thousand outside. A webcam reveals how these variations in environment are slowly materialised in the life of the tulips. The tulips inside grow, bloom and then, wonderfully, decay more rapidly than their twins outside. As in Watching the Sky, long time spans are compressed into human-scale time and space; and here too digital imaging plays a pragmatic role in that revelation. Deployed in rectangular masses we can easily read the flowers as abstract, sculptural materials; organic pattern and variation enframed and aestheticised. But at the same time the work has a kind of deadpan resonance, a rendition of life, and death, inside a greenhouse.
The Huey-Dewey-Louie Climate Clock, by Usman Haque and Robert Davis, addresses the long timescales of environmental change head-on in a proposal that further develops this articulation of slow data and landscape. The clock is a multi-layered system of autonomous machines and material processes. The "Huey" agent slowly builds "accretion mounds" using material extracted from the atmosphere and formed into accumulating conical stacks over the course of a year; like tree rings or geological strata these embed environmental materials directly into a designed representation. The "Dewey" element is a circular array of one hundred transparent containers, in which air and biomass samples are preserved year by year. Like Driessens and Verstappen's Tulips, Haque and Davis propose a biological instrument of one hundred genetically identical daffodils, which are sown and harvested each year, then entombed in the plinths - again a simple grid, a layer of invariance is imposed that allows the landscape to essentially represent itself, materially. Finally Louie, an autonomous solar-powered robot, gathers soil samples and compresses them into cubes, one per day. The surface of each cube is imprinted with some current data point - chosen by daily popular vote; perhaps oil price, or rainfall. So here fast, real-time, socially selected data comes to rest directly on the slow, material medium of the soil.
At one stage, not long ago, it may have seemed that we were leaving the landscape behind, or drafting it in only in as a support or substrate for the flickering patterns of real-time culture. Even now, that seems possible: the monthly figure for new housing construction, a bellwether for economic growth, is imposed on the landscape by earthmovers and roadbuilders, underscored by raw mounds of earth. The works presented here suggest an alternative role, perhaps an alternative future for the landscape; as slow data and slow instrument, a complex material system that can be subtly designed into self-revelation.
Friday, May 15, 2009
Landscape, Slow Data and Self-Revelation
Sunday, May 03, 2009
Transduction, Transmateriality, and Expanded Computing
In common usage a transducer is a device that converts one kind of energy to another. Wikipedia lists a fantastic variety of transducers, mapping out links between thermal, electrical, magnetic, electrochemical, kinetic, optical and acoustic energy. In this form transducers are everywhere: a light bulb transduces electrical energy into visible light (and some heat). A loudspeaker transduces fluctuations in voltage into physical vibrations that we perceive as sound.
In analog media, transduction is overt (put the needle on the record...). But digital media are riddled with it too. Inputs and output devices all contain transducers: the keyboard transduces motion into voltage; the screen transforms voltage into light; the hard drive mediates between voltage and electromagnetic fields. A printer takes in patterns of voltage and emits patterns of ink on a page. Strictly transduction only refers to transformations between different energy types; here I want to extend it to talk about all the propagating matter and energy within something like a computer, as well as those between that system and the rest of the world. From this transmaterial perspective a computer is a cluster of linked mechanisms and substrates; a machine for shifting patterns through time and space.
If this sounds unfamiliar, it's only by historical accident. Mechanical computers, where these patterns are physically perceptible, predate electrical (let alone digital) ones, by centuries (above: a replica of Konrad Zuse's Z1, a mechanical computer from 1936. Image by rreis). Materially, our current computers are more or less black box systems. Their transductions come as a sort of preconfigured bundle or network, a set of familiar relations constructed again by mixtures of hard- and software, protocols, standards: generalising frameworks. I press a key, a letter appears; this is all I need to know. Click "OK". No user-serviceable parts inside.
Except that currently, across the media arts and a whole slew of other fields, the computer is undergoing a rich and productive decomposition. It's composting, to borrow a Sterlingism. This goes under all kind of different names: hardware hacking, device art, homebrew electronics, physical computing. Such practices mount a direct assault on the computer as a material black box, literally and figuratively cracking it open, hooking it up to new inputs and outputs, extending and expanding its connections with the environment. Microcontrollers like the Arduino present us with nothing but a row of bare I/O pins. Finally we can tackle the question of what should go in, and what should come out: of transduction. A whole generation of artists, designers, nerds and tinkerers are taking up soldering irons and doing just that. Below: the Spoke-o-dometer from Rory Hyde and Scott Mitchell's Open Source Urbanism project.
One side-effect of this decomposition of computing is that the ontological status of the digital starts to break down with it. As Kirschenbaum shows brilliantly, the digital is just the analog operating within certain tolerances or threshholds. Thomas Traxler's The Idea of a Tree (below) is a solar-powered system that fabricates objects from epoxy, dye and string, by turning a spindle. Solar energy generates electrical energy, which drives the motor, which draws the string through the dye and onto the spindle: a chain of analog transductions produce an object that manifests specific changes in its local environment. The work is a beautiful demonstration that variability doesn't have to be worked up with generative code: if the system is open to it, it's already there in the flux of the material field.
This is not to dismiss computing, only to recast it: an incredibly dynamic, pliable set of techniques for manipulating the material environment. Paradoxically the very generalities of computing - the abstractions and protocols that insulate it from local, material conditions - make it a powerful tool for transduction, that is, the propagation of specificities. Usman Haque's Pachube is a generalised infrastructure, a set of protocols and standards that rest in turn on wider standards like XML, and which assume a whole stack of functional layers: IP, HTTP, and so on. All in order to propagate material patterns and flows from here to there: this is an architecture of transduction whose utopian aim is to "patch the planet" into a translocal ecology of linked environments.
Digital fabrication is part of the same shift: an expansion and extension of the computer's range of material transductions. Digital pattern, to lasercutter instructions, to physical form. Fabbing shows how material matters. It's unsurprising that a piece of laser-cut ply is aesthetically different to a luminous pattern of pixels; more interesting is the way computation reaches out into the substrate's material properties, and the range of potential applications and domains it opens up. Fabbing has often presented itself with a narrative of materialisation, making the virtual real, translating bits into atoms - Generator.x 2.0 was subtitled "Beyond the Screen." Not so: because of course, the "virtual" never was, and the screen is material too. Fabbing does get us beyond the screen, but only because its processes and materials have different properties, different specificities, and they hook us up to new contexts, as well as new sensations. (Below: Andreas Nicolas Fischer & Benjamin Maus: Reflection - from 5 Days Off: Frozen)
Transduction suggests a way to link practices like physical computing, fabrication, networked environments, and many more. Data visualisation - in the broadest sense, from poetic to fuctionalist - is about creating customised transductions, sourcing new inputs and/or manifesting new outputs (even if they don't reach "beyond the screen"). We could add tangible interfaces, augmented reality, and locative systems. What does all this amount to? In 1970 Gene Youngblood observed a similar moment as the dominant cultural form diversified into a networked, participatory, interdisciplinary field of practices. He called it expanded cinema. So perhaps we can call this expanded computing: digital media and computation as material flows, turned outwards, transducing anything to anything else.
Wednesday, April 15, 2009
Master of Digital Design / Grow Your Own Logotype
Over the past year or so I've been working on a major new offering here at UC. So, I'm delighted to finally launch the new Master of Digital Design online. This course will offer something quite unique in the Australian context: a trans-disciplinary coursework Masters focused on digital practice for designers and creative practitioners of all sorts. The key practical approaches are generative techniques, data visualisation and design, and physical computing; and we'll be using these to address three core themes or questions: the urban, the public, and the sustainable.
As readers of this blog will know, these themes and approaches are right in line with my own research and creative interests; so frankly, I'm thrilled to be leading this course. Teaching with me will be a crew of talented designers, artists and researchers including Stephen Barrass, Sam Hinton and Geoff Hinchcliffe. Finally, we'll be drawing on the wisdom and experience of an international advisory panel whose work exemplifies what we mean by digital design - a practice that engages deeply, and critically, with digital processes, digital materials, and digital contexts: Karsten Schmidt, Rory Hyde, Nervous System, Anthony Burke and foAM.
The course launch has also provided a great excuse (er, opportunity) to play with some ideas around generative branding and marketing. I've been tinkering with this logotype for ages; it uses the same basic algorithm as Limits to Growth but artificially constrains the growth to a letterform (in the guise of a hidden bitmap image). Lately I've extended the logotype into a little generative marketing gadget; a Processing applet that lets you grow endless variations, and receive the results as a PDF file, attached to an email. The aim is to provide a little taste of the power - and pleasure - of generative design.
Behind the scenes this project was yet another demonstration of the brilliance of Processing and its community. The key technical challenge was the upload-and-email functionality. Seltar's "save to web" hack provided the template; upload image data over HTTP, and have a PHP script catch and save the file. From there it was relatively straightforward to have PHP generate the email, with the help of the Pear MailMime package. The final step was uploading a PDF, rather than a bitmap. This seemed impossible, because the built-in PDF library needs to write a local file, which means the extra annoyance of a signed applet. I posted a query on the Processing forums and within 24 hours PhiLho saved me with a solution that extends the PDF class to allow access to the PDF data as a Byte array, without first saving the file. Amazing: thank you! Add the super-useful ControlP5 for the UI sliders and buttons, and the whole thing is built on, in and with free, open-source software. Again, a demonstration of why digital design is such an exciting field of practice right now.
Sunday, March 15, 2009
Watching the Street (Navigator) / citySCENE
Vague Terrain 13: citySCENE has just launched. As editor Greg J. Smith writes:
This issue of Vague Terrain is founded on two notions - that the city is a stage set for intervention and an engine for representation.The collection expands out from this premise in multiple directions: carto-mashups, projection-bombing, sound walks, psychogeographic imaging and ubicomp experiments. Early highlights for me included Crisis Fronts' Cognitive Maps and Database Urbanisms, which presents some impressive work on data visualisation and generative models as urban mapping strategies (below: Case Study: Los Angeles). Overall, on a first look, this collection is incredibly rich. It shows that a creative, wired-up, critical urbanism is not just a wisftul aspiration of the technorati, but a real practice.
Having said all that, it's a privelege to be a part of this collection. My contribution is Watching the Street (Navigator), a browsable visualisation of a single day of images from the Watching the Street dataset. It tests out the hunch that these time-lapse slit-scans can be used to read real patterns in the urban environment - that they are (or can be) more than just suggestive abstractions. It uses a simple interface to display both a single source frame, and a correlated slit-scan visualisation, with image-space and time-space sharing an axis, a bit like a slide rule. Greg Smith called it an "urban viewfinder", which sums the intention up nicely.
Playing with the navigator for a while seems to confirm that hunch. The composites reveal temporal patterns in the environment, but not the spatial context that allows us to identify their causes; the source frames show that spatial context, but not the change over time. Reading the two against each other involves chains and cycles of discovery, analysis and inference. These might be open-ended (spatiotemporal browsing) or more directed. What time do the sandwich-boards go out? How long does the delivery truck stay?
Building the navigator presented some interesting technical challenges: mainly, how to make a web-friendly interface to 1440 source frames (240 x 320) and 480 slit-scan composites (720 x 320). That adds up to about 75Mb of jpegs. Processing 1.0 came to the rescue, with its new built-in dynamic image loader. requestImage() pulls in an image from a given URL, on cue, without bringing the whole applet to a grinding halt; it provides some basic feedback on the state of that image - whether it's loading, loaded, or un-loadable. I also blundered into two other useful lessons: how to use the applet "base" parameter, and how to manage Java's local cache, which kept throwing up earlier versions of the applet during testing.
Having made a lean, mean, browser-friendly version, I'm now thinking of adapting the navigator into a full-screen, offline app, with the whole eight-day dataset, and perhaps some tools for annotation and intra-day comparison. Best of all would be a long term installation; a sort of urban space-time observatory, watching the street but also opening it up to ongoing interpretation. If you'd like it running in your foyer, let me know.
Friday, January 16, 2009
JCSMR Curls
This post is (belated) documentation of a project I worked on in 2007-8, creating an audio-responsive generative system for a permanent installation for the Jackie Chan Science Centre (yes, that Jackie Chan) at the John Curtin School of Medical Research, on the ANU campus. Along with some Processing-related nitty gritty, you'll find some broader reflections on generative systems and the design process. For less process and more product, skip straight to the generative applets (and turn on your sound input).
In mid 2007 my colleague Stephen Barrass and I were approached by Thylacine, a Canberra company specialising in urban art, industrial and exhibition design. Caolan Mitchell and Alexandra Gillespie were designing a new permanent exhibition, the first stage of the new Jackie Chan Science Centre, housed in a new building - a razor-sharp piece of contemporary architecture (below) by Melbourne firm Lyons. Instead of just bolting a display case and a few plaques to the wall, Mitchell and Gillespie (wonderfully) proposed a design that hinged on a dynamic generative motif - a system that would ebb and flow with its own life cycles, and echo the spiral / helix DNA structures central to the School's work, and already embedded in the building's architecture.
My initial sketches (below) took the spiral motif fairly literally, drawing vertical helices and varying their width with a combination of mouse movement and a simple sin function - the results reminded me of the beautiful spiral egg cases of the Port Jackson Shark. At that stage we were talking about the possibility of projecting back onto the facade of the building, which has big vertical glass panels; this structure informed the vertical format. I made a quick video mockup of the form on the facade - which was incredibly easy, thanks to the robust, adaptable, extendable goodness of Processing (a recurring theme in the process to come).
These sketches meet the simplest criteria of the brief (spiral forms) but do nothing about the more interesting (and difficult) ones: cycles of birth, growth and death, and dynamics over multiple time scales. Over the next couple of months I developed two or three different approaches to this goal.
The phyllotaxis model blogged earlier was one attempt. Spurred on by the hardcore a-life skills of Jon McCormack and co. at CEMA, I built a system in which phyllotactic spirals self-organised spontaneously. Because in Jon's words, anyone can draw a spiral, what you really want is a system out of which spirals emerge! The model worked, but I had trouble figuring out how phyllotactic spiral forms might meaningfully die or reproduce. Also, by that stage I had two other systems that seemed more promising.
From the early stages I wanted to make the system respond to environmental audio. The installation would be in a public foyer with plenty of pedestrian traffic, so audio promised a way to tap in to the building's rhythms of activity at long time scales, as well as convey an instantaneous sense of live interaction. In the two most developed sketches audio plays a key role in the life cycle of the system.
One sketch moved into 2d, and started with a pre-existing model for growth, by way of the Eden growth algorithm (this system would later be adapted again into Limits to Growth). I had already been playing with an "off-lattice" Eden-like system where circular cells could grow at any angle to their parent (rather than the square grid of the original Eden model). This system also made it easy to vary the radius of those cells individually. The next step was to couple live audio to the system; following a physical metaphor, frequency is mapped to cell size, so that larger cells responded to low frequency bands, and smaller cells to high frequencies. Incoming sound adds to the cell's energy parameter; this energy gradually decays over time in the absence of sound. Cell reproduction, logically enough, is conditional on energy.
The result is that cells which are best "tuned" for the current audio spectrum will accumulate more energy, and so are more likely to reproduce, spawning a neighbour whose size (and thus tuning) is similar to, but not the same as, their own; so over time the system generates a range of different cell sizes, but only the well-tuned survive. The rest die, which in the best artificial life tradition, means they just go away - no mess, no fuss. In the image below cells are rendered with stroke thickness mapped to energy level. The curves and branches pop out of rules sprinkled lightly with random(), resulting in a loose take on the spiral motif, which is probably the weak point in this sketch. I still think it has potential - nightclub videowall, anyone? Try the live applet over here (adjust your audio input levels to control the growth / death balance).
The third model takes this approach to energy and reproduction - about the simplest possible a-life simulation - and folds it back into the helical structures of the first sketches. In this world an individual is a 3d helix, built from simple line segments. Again each individual is tuned to a frequency band, which supplies energy for growth; but here "growth" means adding segments to the helix, extending its length. Individuals can "reproduce", given enough energy, but here reproducing means spawning a whole new helix, with a slightly mutated frequency band. All the helixes grow from the same origin point - they form a colony, something like a clump of grass.
This sketch went through many variants and iterations over the next month or so; in retrospect the process of working to a brief, within a design team, pushed this system further than I ever would have taken it myself. At the same time I was testing the system against my own critical position; I've argued earlier that the generative model matters, not just for its generativity but the entities and relations it involves.
From that perspective this system was full of holes. Death was arbitrary: just a timer measuring a fixed life-span. "Growth" was a misnomer: the number of segments was simply a rolling average of the energy in the curl's frequency band, so the curls were really no more than slow-motion level meters. Taking the organic / metabolic analogy more seriously, I worked out a better solution. An organism needs a certain amount of energy just to function; and the bigger the organism, the more energy it needs. If it gets more than it needs, then it can grow; if it gets less than it needs, for long enough, it will die. So this is a simple metabolic logic that can link growth, energy and death. Translated into the world of the curls: for each time step, every curl has an energy threshhold, which is proportional to its size (in line segments); if the spectral energy in its band is far enough over that threshhold, it adds a segment - like adding a new cell to its body; if the energy is under that threshhold, it doesn't grow; and if it remains in stasis for too long, it dies. Funnily enough, the behaviour that results is only subtly different to the simple windowed average. Does the model really matter, in that case? It does for me at least; if and how it matters for others is another question.
Next, the curls developed a more complex life-cycle - credit to Alex Gillespie for urging me in this direction. In line with the grass analogy, curls grow a "seed" at their tip when they are in stasis; when they die, that seed is released into the world. Like real seeds, these can lie dormant indefinitely before being revived - here, by a burst of energy in their specific frequency band. After several iterations, the seed form settled on a circle that gradually grows spikes, all the while being blown back "down" the world (against the direction of growth) by audio energy (below). As well as adding graphic variety, seeds change the system's overall dynamics. Unlike spawned curls, seeds are genetically identical to their "parent" - attributes such as frequency band are passed on unaltered. Because each individual can make only one seed, that seed is a way for the curl to go dormant in lean times; if it gets another burst of energy, it can be reborn. The curls demo applet demonstrates this best (again, adjust your audio input and make some noise).
A few technical notes. One big lesson here was the power of transform-based geometry. Each curl is a sequence of line segments whose length relates to frequency band (lower tuned curls have longer segments); each segment is tilted (rotateZ), then translated along the x axis to the correct spot. A sine function is used to modulate the radius of each curl along its length; frequency band factors in here too; this radius is expressed as a y axis translation. Then the segment is rotated around the x axis, to give depth. I iterate this a few hundred times to get one curl, and repeat this process up to twenty times to draw the whole world - each curl has its own parameters for tilt, x rotation increment, and frequency band.
In the live applet audio energy ripples up the curls, from base to tip. This was added to reinforce the liveness of the system and add some rapid, moment-by-moment change. It was implemented very simply. I used a (Java) ArrayList to create a stack of audio level values; at each time step, the current audio level value is added at the head of the list, and the ArrayList politely shuffles all the other values along. So each segment's length is a combination of three values; the base segment length, a function to taper the curl towards the tip, and the buffered audio level.
The graphics are all drawn with OpenGL - following flight404 I dabbled with GL blend modes, specifically additive blending, to get that luminous quality. The other key visual device here is the smearing caused by redrawing with a translucent rect(); instead of erasing the previous frame completely this fades it before overlaying the new frame. It's an easy trick that I've used before. But as Tom Carden explains, in OpenGL it leaves traces of previous frames. I discovered this firsthand when Alex and Caolan asked whether we could lose the "ghosts." I was baffled: on my dim old Powerbook screen, I simply hadn't seen them. Eventually, juggling alpha values I could reduce the "ghosts" to almost black (1) against the completely black (0) background - but no lower. Finally I just set the initial background to (1) instead of (0), and the ghosts were gone.
The adaptability of Processing came through again when it came to realising the installation. The final spec was a single long custom-made display case, with three small, inset LCD panels. These screens would run slide shows expanding on the exhibition content, but also feature the generative graphics when idle; the case itself would also integrate the curls as a graphic motif. For the case graphics, I sent Thylacine an applet that output a PDF snapshot on a key press; they could generate the graphics as required, then import the files directly into their layout.
The screens posed some extra challenges. The initial idea was to have the screens switch between a Powerpoint slideshow, and the curls applet; but making this happen without window frames and other visual clutter was impossible. In the end it was easier to build a simple slide player into the applet: it reads in images from an external folder, allowing JCSMR to author and update the slideshow content independently.
So to wrap up the Processing rave: it provided a single integrated development and delivery tool for a project spanning print, screen, audio, interaction, animation and even content management. Being able to burrow straight through to Java is powerful. Development was seamlessly cross-platform; the whole thing was developed on a Mac, and now runs happily on a single Windows PC with three (modest) OpenGL video cards. The installation has run daily for over six months, without a hitch (touch wood).
Some installation shots below, though it's hard to photograph, being a glass fronted cabinet in a bright foyer - reflection city. I'll add some better shots when I can get them. If you're in Canberra, drop in to the JCSMR - worth it for the building alone - and see it in person.
And very finally, photographic proof of the Jackie Chan connection - image from The Age.
Friday, December 19, 2008
Fabricated Growth Forms (Processing to Ponoko)
Like many others playing with generative techniques, I'm fascinated by the potential of digital fabrication. Getting beyond the screen and into the world of objects is a significant move for a field that has, until the last few years, reveled in its own immateriality. There's a lot to think about in this material turn, but that's for another post. Here, a quick report on my first experiment with generative fabrication.
I don't have a laser cutter handy at my workplace (though as William Turkel writes there are lots of good reasons why I should) so I decided to check out Ponoko; I wanted to see what was involved in generating, uploading and fabbing a small project. I started with the Processing sketch from Limits to Growth, and tweaked it to turn out much smaller forms (a few hundred nodes, rather than tens of thousands). I used the built-in PDF export, then opened the PDFs in Illustrator. (Illustrator is the only commercial/proprietary software step in this process, so I'd be interested to hear of any alternatives). The forms are drawn as linked line segments of varying stroke widths. Ponoko needs an EPS with only the outside edge of this form, so I used Illustrator to merge it into a composite path, then set the stroke colour and width as instructed (0,0,255 and 0.001mm).
The upload to Ponoko took a few tries - I was getting some strange errors as their system failed to "see" the cutting paths on the template - but after some swift and cheerful technical assistance it all worked. Pricing was also trial and error; the first design I uploaded was more complex than these, and of course these branching forms pack a long cutting path into a small surface area. I simplified the design, packed four forms onto a sheet, and opted for 4mm ply rather than acrylic. Final cost including (expensive) shipping to Australia was about $A60 (currently around $US40). Not what I'd call cheap, but not prohibitive either. There are intricate discussions of the economics of the business - shipping, exchange rates, local vs global, etc - on the Ponoko forums.
Eighteen days later, they arrived. Novelty counts for a lot here, but still, I'm totally charmed by these objects. A few surprises, but all good: they are smaller and finer than I imagined, and they smell very slightly of charred wood (excellent!). The cut edges are dark with a nice smooth, burnished surface, and the ply surface is clean. The scale and intricacy of the things seems to entice people to touch and handle them. I find them far more satisfying than the (much more detailed) laser prints I made with the same system.
Immediately it's clear how the fabbing process, and the materials, can reach back up through the production chain and influence the design and the generative system. One flaw in the design is a product of how I'm drawing the shapes: there are small rounded "shoulders" at the joints between line segments, caused by the overlap between one rounded line cap and the next segment - this is obvious in the physical forms. Better to draw the segments as tapered rectangles, and avoid the shoulders. Also, the branching topology is structurally risky; how to introduce more joins without breaking the generative model? This interplay, between computational process, manufacturing process, material and form, seems really promising. Ponoko seems to be an excellent, affordable way to try this out, and the built-in fab-on-demand shopfront is great, if you want to sell your wares. But it's still, ironically, working with a mass-production paradigm of one design, n copies. With hooks for a more dynamic, generative front end, it could get really interesting: designers like the wonderful Nervous System are doing this already. More documentation of the growth forms over on Flickr.
Thursday, November 27, 2008
Watching the Street
The recent Dorkbot show seemed to go off nicely - it was great to be part of such a strong show of local work (some documentation). I showed some prints from Limits to Growth, as well as a more experimental process piece, Watching the Street - a (sub)urban remake of Watching the Sky.
Credit to Nathan McGinness for the suggestion: use the same time-lapse / slit-scan technique to image change in an urban environment. Technically, the setup was fairly straightforward. Instead of a digital stills camera I used a webcam (in portrait orientation), and wrote a simple Processing script to save stills at one-minute intervals, while extracting and compiling one-pixel slices into 24-hour composites. The webcam was installed in a window box on the gallery street front, with a view across the road, under a street tree, to one of Manuka's low-rise shopping arcades (above). I also attached a printer to the installed rig, so that a new composite could be produced and pinned to the wall each day. So here, some of the resulting images, and a bit of commentary.
The image-gathering process got off to a rocky start. After a few hours, the webcam came unstuck from the side of the window-box, and lay forlornly on its side for the next 48 hours (here's what that looks like). I gaffed it back in place just before the opening, and restarted the capture in time to catch some gallery-goers loitering around out the front.
These two are the Frday the 7th and Saturday the 8th of November, the first two full day composites. Those striped rectangular chunks around mid-frame are cars, parked in the 30 minute loading zone accross the road. Some stay for a few minutes, a couple for what looks like an hour. Of course on the Saturday, the loading zone doesn't operate, and there's a single car parked in it from mid-morning to mid-afternoon. The single-pixel vertical shards give an indication of passing car and pedestrian traffic.
A quiet, sunny Sunday the 9th; the form hinted at on the 8th, reveals itself as the shadow of the big plane tree, creeping across the footpath. Then the following Friday the 14th. It's all happening; lots of car and pedestrian traffic, changes in sunlight, looks like an afternoon breeze in the foliage as well. The dominant, bluish horizontal stripe in all these images is the neon sign on the shopping centre - which runs all night. The orange rectangle that extends into the evening is the interior light of a shop - which you'll notice switches off at slightly different times each night.
So you'll notice that as in Watching the Sky, I'm persisting in reading these as visualisations of the environment, as well as digital images in themselves. I'm struck by how this simple, indiscriminate process reveals both expected and unexpected patterns, and continues to provoke new questions. This despite, or I would argue because of, its openness to multiple material / temporal systems. In an interesting bit of synchronicity, I was teaching in the UTS Street as Platform masterclass with Dan Hill (more on that soon) while this piece was running. Could a simple visualisation process like this function "informationally", as it were; to help answer real questions about a very specific slice of urban environment, in near-real time? More interesting for me, could it function in that way without prescribing the question in advance - that is, could it support an open-ended process of exploration and interpretation? I'm planning to build an interactive version of this piece, to try out these ideas. In these static visualisations there's a huge amount of data missing: I set the slice point more-or-less arbitrarily, so there are 479 other potentially interesting slices to browse. It would be nice to be able to change the slice point dynamically, as well as navigating through the source images. I notice that Processing 1.0 (yay!) now supports threaded loading of images: could come in handy. Meanwhile, the full set of composite images are up on Flickr.
Monday, October 27, 2008
Dorkbot CBR at Manuka CCAS
Thursday, October 02, 2008
Synesthesia and Cross-Modality in Contemporary Audiovisuals
Though written about a year ago, this essay has just been published in Senses and Society. It's related to the Synchresis project posted earlier but makes a more rigorous investigation of synaesthesia, as it is (so often) applied to fused or algorithmic audiovisuals. After a quick tour through the history of synaesthesia in the arts, it uses some nifty perceptual neuroscience to argue for an alternative model, of contemporary audiovisuals as cross-modal objects that reveal the space of relation between modalities - the map. It takes work by Andrew Gadow (below) and Robin Fox as case studies, but also touches on Oskar Fischinger, Robert Hodgin, Norman McLaren and others. The version here has plenty of pics and vids; for a more paper-based experience grab the pdf (and please use the print version for any citations). Oh and pardon the American spellings here - journal style.
In the age of ubiquitous digital media, synesthesia is everywhere. In human, neurological form, it is rare: for perhaps three in a hundred people, a stimulus in one sensory modality automatically induces a sensation in another. Auditory-to-visual synesthesia, or “colored hearing” is much rarer still. Yet now this phenomenon is realised, apparently, inside every digital music player, on VJ screens in every club, in robot lightshows. On these screens sound is transformed into visual pattern and form instantly and automatically; an exotic perceptual phenomenon becomes a technically mediated commonplace.
In fact digital synesthesia is a trope that occurs in the production and use of mainstream digital media, as well as the media arts. Computer users find audio visualisers built in to their music players; as this software shows, audiovisual relations can now be reduced to an algorithm, a formal procedure that interprets (sound) and emits (image) data; though in this case the results are mostly mundane psychedelia. In some recent music video audio visualisations are integrated into the narrative and performative conventions of the genre; in Justin Timberlake’s Lovestoned video (2007) the singer’s image is constructed from the flickering bands of a visualised audio spectrum. Here a technically guaranteed unity of sound and image is literally reinscribed on to the performing artist. In contemporary media arts practice the same techniques – computational analysis of sound driving generated visual elements – are widespread, and its aesthetics more diverse. In custom-coded audio visualisations artists such as Marius Watz (2005) and Robert Hodgin (2007) (below) construct visualisations tuned to specific soundtracks; the automatism of digital synesthesia animates specific, constructed worlds of form and image. The algorithm becomes an endlessly variable and dynamic intermediary between sound and image.
In a distinct but related approach, some media artists have opted instead to simplify or reduce that audiovisual relation, ofen bypassing computation altogether. In Carsten Nicolai’s Telefunken works (2000,2004) the stereo output of an audio CD player is connected to the audio and video inputs of a television screen; what is heard as synthetic tones and noisy drones, is seen on the screen as patterns of monochrome form and line. In an approach I will refer to here as transcoding, sound and image are linked through a direct transfer of signal, a simple cross-wiring.
Australian artist Robin Fox plugs audio from a custom-built digital synthesiser into an oscilloscope; in the resulting hybrid instrument, Fox explores a territory in which signal is simultaneously heard and seen; every sound is a form in motion, every form a sound (above). The connection, the cross-wiring of sound to image, literally manifests the sensory cross-over of synesthesia; more, the work itself seems to somehow induce synesthetic experience. The correspondence between sound and image is immediate, agile and intense; the audiovisual relation is completely consistent, somehow self-evident, yet continually surprising. There’s a feeling of something like revelation; one reviewer describes Fox’s Backscatter DVD (2005) as “mesmerizing” and “overwhelming,” and hints at a sense of “greater significance or higher purpose” (Baker Fish 2005). Andrew Gadow’s work approaches the same relationship from the other side; working with an old video synthesiser, he transfers its image signal directly into audio. The scan-line structure of the video signal becomes audible as modulations of a 50 Hz hum; flickering, disintegrating visual textures become abrasive but intricately detailed buzzsaw audio. Again the subjective experience can be powerful, a visceral sense of force or encounter; the audiovisual coupling is so close that it seems to disappear, distinct modalities fuse into raw sensation.
Synesthesia is widely used as an analogy around this work. The analogy provides a mapping that aligns subjective sensation with audiovisual signals; it maps perceptual or even neurological structures onto technical structures. The analogy also plays another role, foregrounding sensation in the reception of the artworks; proposing to operate, for the subject, at the level of direct sensation. Finally, synesthesia also connects this contemporary work with a historical artistic tradition. The new automatic or transcoded fusion of sound and image seems to mark the culmination of a practice spanning music, painting, film and electronic media and aspiring, as Jeremy Strick writes, to the ideal of synesthesia as “the unity of the senses, and, by extension, the arts.” (Strick 2005: 15) The 2005 Visual Music exhibition, curated by Strick, documents this tradition in detail, as well as making a bid for its continuation into the present:In digital media ... music and visual art are ... created out of the same stuff, bits of electronic information ... . [T]he aspiration to novel experience created by the compounding of sensation and association has never been more possible. (ibid.)
This paper’s main aim is to test this analogy, and the related historical drive that Strick suggests; to consider if, and how, such practice can be thought of as synesthetic, and examine structural parallels between synesthesia as a perceptual and neurological phenomenon, and the automatic or transcoded linking of audio and visual media? Following the tradition of artistic synesthesia that Strick invokes, the approach here is to provisionally ignore the glaring gap in this analogy, between subjective sensation and objective, technical artefact. Scientific work in perception and neuropsychology is drawn in for a more detailed account of synesthesia; but it also offers an alternative model for this practice, based on theories of cross-modal interactions in normal perception. Close correlations between sound and image are, after all, an everyday perceptual occurrence. From this perspective, tightly correlated audiovisuals direct us towards the abstract structures that are its generative materials: signal, as distinct from image or sound; and the map, the pattern of correlation between signals in different domains. Although artists such as Fox and Gadow use obsolete, analog technologies, their work is a sensory manifestation of these characteristically contemporary abstractions.
Synesthesia
Scientists studying synesthesia define it as occurring “when stimulation of one sensory modality automatically triggers perception in a second modality” (Harrison and Baron-Cohen 1996: 3). There are many documented forms of synesthesia; stimuli such as numbers, letters, words, days of the week, and musical tones may trigger perceived color and shape; taste, smell and pain can also trigger perceptions of shape. Estimates of the prevalence of synesthesia vary widely between 1 in 20 and 1 in 20 000; one recent study found a prevalence of around 3% and showed that calendar-color and letter- and number-color forms occur most often, while audiovisual synesthesia or “colored hearing” is comparatively rare (Simner et al 2006).
After long being debunked or treated as a curiosity, synesthesia has attracted increasing scientific attention, and validation, in recent years. Neuropsychologist Richard Cytowic undertook one of the first modern studies (Cytowic 1989). As well as a basic validation – finding that synesthesia is a real phenomenon with a neurological basis – Cytowic proposed a set of diagnostic and clinical features of the condition (Cytowic 1996: 23-31). He found that synesthesia is “involuntary but elicited,” an automatic perceptual experience that cannot be supressed or controlled. Synesthetic perceptions are “durable and generic,” meaning that an individual’s cross-sensory connections do not change over their lifetime, and that synesthetic perceptions are elementary and general, rather than “elaborated” – for example colors and simple shapes, rather than a detailed mental picture. Cytowic also points out that synesthetic perceptions are unusually memorable: some synesthetes use their triggered percepts as an index that aids their recall of the evoking stimulus. Moreover, Cytowic states, synesthesia is an emotional experience, “accompanied by a sense of certitude (the ‘this is it’ feeling)” that he links to William James’ description of religious ecstasy, and in particular the affect of noesis, “knowledge that is experienced directly, an illumination that is accompanied by a feeling of certitude.” It is in part this affective dimension that leads Cytowic to propose a linkage between synesthesia and the limbic brain – associated with emotion and a sense of “salience.”
More recent science has continued to investigate the neurology and psychology of synesthesia, using modern imaging techniques to show activation in the anatomy of the synesthetic brain, as well as behavioural experiments that seek out the parameters of synesthetic experience. Recent work has confirmed Cytowic’s finding that synesthesia is involuntary and perceptually real, though the results also suggest that there is significant variation between synesthetic individuals (see for example Hubbard and Ramachandran 2005). Discussion centers on where synesthesia occurs in the notional chain of perceptual processing; for a minority of synesthetes it seems to occur early in the chain, before cognitive processes such as attention; for the rest it seems to occur later; Ramachandran and Hubbard label these forms"lower” and “higher” (2001: 14). There is general consensus that synesthesia has a neural basis in the form of increased connectivity between normally separate neural regions or modules (contrary to Cytowic’s limbic model), though the connective mechanism and architecture is debated. Some, including Ramachandran and Hubbard, propose that this connectivity is a result of defective “pruning” of neural connections, suggesting that synesthetic cross-wiring is a normal early developmental stage. The notion of synesthesia as common, underlying or originary is supported by the correlations between synesthetic and “normal” cross-sensory associations; despite individual differences in color-tone mapping, non-synesthetes and those with “colored hearing” make similar mappings between pitch and lightness (Marks 1996: 72). Similarly Ramachandran and Hubbard (2001:19) cite the consistency of an (albeit simple) mapping between shape and sound to support the same point: asked to link two shapes, one round, one spiky, with two names, bouba and kiki, subjects overwhelmingly associate kiki with the angular form and bouba with the rounded one. The authors continue, in one of the more expansive examples of synesthesia science, to propose links between this “normal” synesthesia and the angular gyrus, an anatomical region associated with cross-modal association as well as numeracy, the neurology of metaphor, emotion, art and the evolution of language. In this formulation synesthesia is an extreme case that offers clues to the neurology of normal – and significant – human abilities to associate and synthesise disparate sensations and concepts.
Synesthesia in the Arts: Models and Maps
The history of synesthesia in the modern arts is well documented, and will not be recounted in detail here. More important for this argument is a sense of the major strains or variants of the concept, and their creative implications. This lineage has been traced to Romantic and Symbolist interests in the correspondence of the senses; poems of Rimbaud and Baudelaire correlated letters, colors, smells and sounds (see for example Cook 1998:25). In what Judith Zilczer (2005: 26) terms the mystical strain of artistic synesthesia, these sensory correspondences were held to refer to a higher, unitary reality, informed by Theosophy and Romantic philosophy. Kandinsky, reputedly a synesthete with “colored hearing,” is the best known of a group whose painterly abstraction was informed by musical and synesthetic analogies. In Kandinsky’s Concerning the Spiritual in Art (1977: 25) sensation, and especially color, can set the soul vibrating like a musical instrument: his aim was a form of absolute or “nonobjective” visual art, comparable to music. Cook (1998: 46) describes this model, informed by Goethe’s philosophy, as triangular. Both sound and color derive from the spiritual, or higher vibration, at the apex; thus sound and color have no inherent correspondence but “correspond to one another in so far as they embody the same ultimate meaning.” In a second wave of visual music between the wars, artists such as Paul Klee, Man Ray, Georgia O’Keefe and Arthur Dove adopted more concrete and structural models of correspondence, attempting to map harmony, counterpoint and rhythm into the abstract picture plane (see for example Zilczer 2005: 52-67). Synesthesia itself plays a shifting role in this context. Kandinsky and composer Alexander Scriabin seem to have experienced it, while many other artists were inspired by, or in some cases literally borrowed, synesthestic correspondences. Discussing the influence of Kandinsky’s note-color correpondences on Schoenberg, Cook (1998: 49) proposes a “cultural synesthesia” – where the idea of sensory correspondence can carry a cultural value independent of its actual experience.
In fact cultural synesthesia – evoked, suggested, implied or idealised synesthesia – dominates the visual music tradition; there are very few instances of actual, spontaneous, automatic audiovisual correspondences. In the work of Messiaen, Scriabin and perhaps Schoenberg (via Kandinsky), synesthetic experience formed the basis for a systematised set of pitch-color correspondences, though even these are not straightforward. The correspondences are different for each composer, as we would expect based on recent science. Moreover each is conditioned by what Cook (1998:46) argues is a mixture of subjective and cultural factors. Any correspondence between the continuous color spectrum and the discrete values of the Western twelve-tone scale is dubious – though these correspondences flourished in the early twentieth century in the “color organs” of Rimington and others (see for example Cook 1998:37 and Peel 2006). Later emblematic practitioners of the visual music tradition, John and James Whitney, used tightly composed but again ultimately arbitrary relations between sound and vision. If, as Strick argues, this creative tradition aspires towards synesthesia, when it comes to practically manifesting that sensory relation it founders on the problem of the map, the pattern of correspondences. Of all possible relations between sound and vision, what is it that makes one different, or preferable, to another? While recent science suggests some underlying perceptual commonalities, the devil, and the aesthetic, is in the detail.
Animator Oskar Fischinger provides a near-precedent for transcoded audiovisuals, and demonstrates one possible solution to the question of the map. Fischinger’s Ornament Sound experiments of 1932 (above) explored the double identity of the optical film soundtrack, printing regular visual patterns into the 3mm-wide sound strip at the edge of the frame, enabling them to be automatically rendered as sound. Fischinger (1932) emphasised the potential of this technique for composers: “control of every fine gradation and nuance is granted to the music-painting artist.” This form also promised a newfound “definitive” control over performance: “his creation, his work, can speak for itself directly through the film projector.” Fischinger also recognises the visual interest in recorded sound waves; and although he anticipated their use in conjunction with animation, he did not envisage the “sounding ornaments” as visual content in themselves. Nonetheless, Fischinger had found a space of audiovisual correspondence that was preexisting and “definitive,” yet seemingly had limitless creative potential. The later work of animator Norman McLaren developed Fischinger’s techniques, synchronising hand-drawn optical soundtracks with animation in Dots (1940), and finally using the synthesised optical soundtrack as synchronised visual source material in Synchromy (1971) (below).
Contemporary transcoded audiovisuals realise Fischinger’s experiments by similarly bypassing, or rather abdicating, the question of the map. This is not to say that the map disappears in an unmediated or inherent audiovisual connection. Instead, for Robin Fox and Andrew Gadow as well as Fischinger, the map is found, rather than constructed; it is embedded in the medium. Fox plugs his laptop into an oscilloscope, which maps the left and right channels of the audio signal into the x and y axes of its display. This audiovisual relation is in a sense a readymade, an existing cultural/technical artefact. Its process is literally hardwired, embodied in the analog electronics of the scope, just as Fischinger’s was in the optical technology of film.
Fused AV and Synesthesia
Do transcoded audiovisuals then realise the synesthetic ideal, or literalise the analogy? We can draw some correlations. To recap, the current scientific consensus is that synesthetic perceptions are real, automatic and involuntary, and caused by neural cross-connections at some level of the perceptual system. The cross-mappings of synesthetic perceptions are highly variable from one individual to another, but highly consistent for the individual. The transcoding approach of artists like Fox and Gadow seems fairly close: in Fox’s oscilloscope work for example, images are created “automatically” as Fox feeds audio to the oscilloscope, cross-wiring audio to vision; Fox uses the oscilloscope’s hardwired audiovisual map, which is fixed and consistent; but that mapping is different to, for example, Gadow’s equally automatic sound-to-image mapping, based on the interchange of analog video and audio signals. Even the visual aesthetics of these works could be likened to reports of colored hearing: in Cytowic’s terms these are not “elaborated” percepts, but simple, abstract elements.
The synesthetic affect that Cytowic’s study identifies is also suggestive. Fused audiovisuals can evoke (for some at least) a similar sense of revelation or noesis. Fox’s oscilloscope works show us something that feels both self-evidently “right” and surprising or unimaginable; the primal phosphorescent dot shows us its universe, a set of relations that are manifestly coherent and consistent, but whose implications are unforeseeable. Fox’s compositional structure emphasises the process of revelation at times: Photosynthesis (AOR) (above), the opening track on his Backscatter release, offers an initially gentle introduction, as the single point of the trace is buffeted by rhythmic subsonic clicks before slowly unfurling into harmonic pattern; but by the end of that track wave after wave of complex, nested forms have emerged and co-modulated; each point on the path is another noetic moment yet each is consistent and coextensive with the others. Mandala I, following, demonstrates almost the opposite approach, as Fox’s micro-switched digital twitches call up flickering variants on the circular carrier wave; to push the cosmological analogy, this is some kind of faster than light travel – we traverse many places at once – but again there is a revelatory quality as we witness accumulating relations, both momentary – between each sound, its corresponding form and movement – and sequential, between each sound/form/movement and the next, and the next.
How far can this line go, though, before it falls into the yawning gap in the analogy? Audiovisual works are artefacts; objects of perception, not perceptions. To put it bluntly, synesthesia, by definition, occurs in the perceptual system of a synesthete, not in the crossed connections of a video synth. Once again, we can use the gap as a provocation, rather than an obstacle. One response is to think of these works not as replicating human neurology, but rather something else. “Artificial synesthesia” is the term used by Dutch neuroscientist Peter Meijer (n.d.) to describe his work on sensory substitution; his vOICe system transcodes video from a small camera, into synthetic audio, in an attempt to use sound to provide visual information to those with little or no vision. In Meijer’s words, “we are interested in forms of learned synesthesia (acquired synesthesia) that might result from machine-generated crossmodal mappings.” Among other things Meijer’s work suggests that perception is not a fixed set of channels, but a reconfigurable network; over time, blind users of the vOICe seem to integrate image transcoded into sound, as functional vision. A recent paper shows that the lateral-occipital tactile-visual area of the brain, normally associated with the tactile and visual perception of shape, is activated by expert vOICe users (Amedi et al 2007). Other work in the field of sensory substitution suggests that different forms of synesthesia can also be acquired: Peter König’s feelSpace belt conveys orientation through vibrating touch, providing a augmented sense that some volunteers were able to integrate over time (see Bains 2007).
Are transcoded audiovisuals some form of sensory substitution or artificial synesthesia? There are two important differences. Sensory substitution operates by mapping an otherwise absent modality into an existing one; absent vision into existing hearing, in the case of the vOICe, and absolute orientation into touch, in the feelSpace belt. However for most, audiovisual transcoding links two existing modalities, “channels” already in perceptual use. Secondly, sensory substitution involves long-term integration and interaction with the environment; we can learn new “channels” but only by feeling out and (literally) incorporating their correlations with our existing sensory matrix. There are some striking parallels, and transcoded AV certainly hints at artificial synesthesia and a rewired sensorium, but as bounded aesthetic objects these works cannot realise that perceptual transformation.
The Pleasures of Binding
Correlation, key to artificial synesthesia, offers an alternative approach to the perceptual aesthetics of fused audiovisuals. At the core of transcoded and other tightly linked audiovisual forms is an experience based on a correlation between auditory and visual elements. While synesthesia offers a neurological analogy for the generation (poetics) of fused AV, this correlated quality leads into the neuroscience of perception, and thus offers a way to frame these works from the other side, the side of reception (or aesthetics).
The detection of correlations in the perceptual field is a normal, and crucial, perceptual task. From an ecological perspective, correlations underpin the recognition of objects in an organism’s environment. Our perceptual systems “bind” correlated elements into groups that often correspond to objects in our physical environment. A cat hiding in the garden might initially appear as an unrelated set of visual elements – a light grey splodge here, a dark shape there. When we “see” the cat, we detect correlations between those elements that enable us to interpret them as part of an underlying object. The image of a hidden Dalmatian dog (below) is often used to illustrate this phenomenon.
As Ramachandran and Hirstein (1999: 21-23) point out, this process of binding has some interesting features. Binding is “sticky” – we seem to hold on to bound perceptual elements. Once seen, the Dalmatian cannot be un-seen a without a conscious effort. Moreover, the act of making a binding is pleasurable in itself: “the discovery of the dog and the linking of the dog-relevant splotches generates a pleasant ‘aha’ sensation.” The authors offer an evolutionary rationale for this payoff: “The very process of discovering correlations and of ‘binding’ correlated features to create unitary objects or events must be reinforcing for the organism – in order to provide incentive for discovering such correlations.” Our limbic system apparently rewards us for detecting sensory correlations in our environment, even in advance of the final “recognition” of an object: “at every stage in processing there is generated a ‘Look, here is a clue to something potentially object-like’ signal that produces limbic activation and draws your atttention to that region or feature.” These incremental rewards “bootstrap” the final moment of recognition. Ramachandran and Hirstein work this perceptual pleasure principle into a neurological theory of aesthetic experience, suggesting that artists and designers seek out and intensify the pleasures of sensory binding, creating artefacts that “tease the system with as many of these ‘potential object’ clues as possible.”
Ramachandran and Hirstein also go further in proposing that the discovery of more abstract correlations is also reinforced by a limbic reward (1999: 31). They relate this to the ecological imperitive for classification – our evolved need to establish correlations that group and distinguish objects in our environment: say, edible versus inedible plants. This version of binding operates diachronically, rather than the synchronous binding of visual elements into a recognised form. “Being able to see the hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category… Consequently the discovery of similarities and the linking of superficially dissimilar events would lead to a limbic activation – in order to ensure that the process is rewarding.”
Cross-Modal Perception
How might processes of binding – the discovery of correlations – operate in fused AV, where the characteristic correlations are between, rather than within, sensory modalities? While studies of perception have traditionally focused on the senses in isolation, as independent neurological “modules", recent work has begun to explore the relations between sensory modalities. Media-based metaphors for perception encourage us to think of the senses as functionally distinct input channels. If sensory substitution shows that these channels can be re-wired, studies of cross-modal perception show that they are barely even distinct. The senses are involved in what Shimojo and Shams (2001: 506) describe as “vigorous interaction and integration,” mirroring Michel Chion’s description of the “mutual contamination” that characterises the audiovisual relationship in film sound ([1990] 1994: 9). Shimojo and Shams review experiments showing the range of these mutual influences: how vision can alter the content and spatial location of perceived sound; and how sound can alter the perceived intensity and timing of visual stimulus. We hear what we see, and see what we hear.
The perceptual trickery of these experiments is less interesting than what they suggest about normal perception. Just as the binding of visual percepts into a whole enables us to recognise objects in our environment, correlations in different sensory modalities cause us to bind those stimuli into a unified perception. This is illustrated with another trick, an experiment by Sekuler et al (1997), in which subjects were presented with two moving dots on intersecting paths. Two perceptual interpretations of this animation are possible: that the dots pass each other without touching, or that they collide and bounce off each other. Without sound, the former interpretation was dominant; however adding a brief sound at the crossing point biased perception strongly towards collision. This is an instance of cross-modal binding, where correlated stimuli in different modalities become fused into a coherent whole. It also suggests the ecological basis of cross-modal binding; that we interpret correlated events as cues to objects in the environment. The interpretation of sensory data seems to be shaped by pre-conscious processes that bind percepts into wholes; wholes that map onto ecologically plausible events. In the crossing dots experiment, sound binds with vision to alter our interpretation of the event. The correlated stimuli point to a common cause, a model that explains their coherence.
Fused audiovisuals are aesthetic objects founded on cross-modal binding. Ramachandran and Hirstein’s notion of the pleasures of binding applies here; in the transcoded AV of artists such as Fox and Gadow we experience sensory fields that are somehow entirely bound: completely self-consistent, devoid of extraneous elements. The affect that Ramachandran and Hirstein attribute to the moment of binding, the discovery of the Dalmatian – the ‘aha’ of recognition – seems to be intensified and prolonged here. It also suggests a connection between cross-modal binding and the noetic affect Cytowic identifies in synesthetic experience. If we accept the limbic payoff theory of aesthetics, then perhaps fused AV is a manifestation of this pleasure principle in the media arts.
Audiovisuals as Cross-Modal Objects
Cross-modal binding is not limited to experimental audiovisuals, however; in fact the opposite is true. Cinema and television constantly rely on our predilection for binding sound and image; this is the basis of Chion’s synchresis, a “spontaneous and irresistable mental fusion” caused by close synchronisation ([1990] 1994: 63). Lip sync is the archetypal example, where audiovisual correlation breathes life into the image of a body. Recall the ecological function of binding: to identify a common cause – an object in the environment. In most audiovisual media the objects are (all too) readily apparent. So if audiovisual correlations refer us to a shared cause, what is that cause in fused or transcoded AV? What is the underlying object, the cat hiding in the garden?
In a sound-to-image mapping, for example, it seems logical to propose that the cause is the source modality – sound. This involves a kind of reflexive redundancy; in Fox’s oscilloscope work, it would mean that the image is simply a pointer to the soundtrack, that it doubles or duplicates the sound. Subjectively at least, the relation seems richer and more complex than that; and it seems at odds with an ecological model of perception. Perhaps the common object is not the sound, but something more abstract: the signal. Signal here refers to a pattern of differences or fluctuations, a flux that, like data, must always be embodied but which, again like data, can be readily transduced between one embodied form and another. Fox’s laptop does not send sound to the oscilloscope, or in fact to the audio amp; it sends signal, a pattern of fluctuating voltage. That pattern is manifest on the scope as phosphorescent image, and when it leaves the speakers, as sound: but their common origin is the flux itself.
In transcoded audiovisuals sound and image perceptually triangulate a third point, the signal, that is imperceptible in and of itself. Signal maps to perception through the contingencies of both media technologies and sensory boundaries, but in itself it traverses these limits. This is apparent in Fox’s work, where subsonic fluctuations modulate the audible frequencies to create movements that are easily seen, but felt only as sharp thumps; the speakers struggle to transduce the signal into mechanical energy. Many of the complex, pointillist visual patterns are created by square-edged signal forms that again are acoustically impossible; the scope, more agile, is better able to trace them out. Similar trans-sensory signatures occur in Gadow’s work and that of other transcoders; Gadow’s Techne (2005) opens with a still blue screen and a raw, buzzsaw hum. The hum has no movement or form; as becomes clear as the piece develops, it corresponds to the blue video background. It is the sound of the 50 Hz scan-line structure of the video signal itself; so it looks like almost nothing. This is not to say that transcoded audiovisuals are reducible to the signal, an abstract or perhaps “higher” ideal, as in Goethe’s triangular model of color and sound. Here sensation and experience are foremost; these experiments feel out the ramifications of signal in specific circuits and transductions.
As Ramachandran and Hirstein suggest (1999:31), perceptual binding is both synchronic and diachronic, instantaneous and sequential. If the moment-by-moment audiovisual binding in these works refers us to their shared cause – the signal – how do these works operate in the diachronic axis? They often share a simple formal structure of establishment, development and elaboration, a successive playing-out of potential. Ramachandran and Hirstein state that perceiving “hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category.” We can think of the sequential similarities here as products of the constant, underlying structure that shapes all the outputs of the system. That structure is the map, the specific but abstract shape of the audiovisual correspondence. The map is an elusive entity; rather than an object we can think of it as a procedure, a verb or algorithm; a way of transforming between modalities and their shared signal. In Fox’s work the polar mapping of the oscilloscope is an algorithm that transforms phase – local relations in time – and amplitude into circular space. Considered as cross-modal objects, these works direct us to the underlying signal; and the signal is embodied audiovisually through the intermediary of the map. The map describes a space of potential, a range of possible correlations between domains; and it is that territory, I would argue, that these works reveal as they traverse it.
Inframedia and the Map
In earlier work on experimental sound I proposed the notion of inframedia, “a stratum below or within the mainlines of electronic media” (Whitelaw 2001:51). The noisy textures, resonant fuzz, glitches, crackles and pops of electronic music since the late 90s reveal “the sensory and affective textures of a media substrate, rather than media ‘content.’” That substrate is a critical domain; media infrastructures are more than technological artefacts; they are rapidly changing focii of power. Inframedia aesthetics reflect a consciousness of that domain, while in its processes such work often pursues local and particular manipulations, hacks or diversions of those media technologies.
Fused audiovisuals – a practice with close cultural links to experimental sound and music – can be approached along similar lines. Like hiss and hum, the audiovisual aesthetics of signal direct us to the abstraction and transduction occurring inside, or underneath, our media streams. Glitch-driven audio is founded on cracks in the surface – moments of interruption which allude to, and materialise, their own infrastructure. In a sense transcoded audiovisuals are a prolongation of those moments, leading to a flattening of the surface/depth dichotomy of glitch; the cross-modal coherence of this work is based on a sustained exploration of signal. Instead of mapping signal anthropomorphically onto perceptual “inputs”, these works show us where signal and affect meet or overlap, as well as where they diverge; they show us signal passing into, out of, and through perception.
These works also direct us to the map – the abstract space of possible transformations between signals. That domain of transformation is also inframedial, a key structure in digital media forms and cultures. Lev Manovich (2002) has described this question as the “built-in existential angst” of digital media: “By allowing us to map anything into anything else ... computer media simultaneously makes all these choices appear arbitrary….” In almost all digital media, the map – the pattern of relations between input and output – is imperceptible, obscured or encoded. This is clearest in the work of artists working explicitly with data inputs. In the work of Alex Drauglescu for example, spam email is used as the input to an algorithm that creates complex three dimensional forms. The mapping – the process that transforms spam into form – is never revealed, and so a concrete, specific process becomes a blank spot filled in with an impression of magical transubstantiation. In some computational work the artist provides source code, an explicit specification of the map but one that is highly encoded and unavailable, in itself, to perception. In most digital media objects, the map is inextricable from the residue or artefact it shapes. We perceive only the output, the image, sound or form, in which the input and its transformations are collapsed.
The wider significance of transcoded audiovisuals is that they approach a perceptual manifestation of the map, that space of transformation. We sense it, in these works, interpolated between each instant and every other. It’s perhaps not surprising that this characteristically digital figure is manifest through largely analog means; as well as a critical distance from the digital, analog signals offer transformations that are rich and immediate. Crucially the maps themselves are simple and static – highly reduced, compared to their digital counterparts – and so more available to the aesthetic and affective explorations of transcoded audiovisuals.
The prospect of somehow apprehending the map is both esoteric and pragmatic. The map is the inescapable intermediary, the necessary condition of our data-experience; but what is the map, what is its shape, how does it transform this into that? What are its conditions, limits, bounds? These works literally feel out the map, and in the process begin to address these questions, offering a sense of the abstract transformations that underpin contemporary digital culture.
Synesthesia is a powerful and persistent trope in the audiovisual arts. As shown here it offers some enticing parallels with the techniques and affects of audiovisual practice, yet as a techno-sensory analogy it has inherent limits. As in the visual music tradition, synesthesia plays a largely figurative role, and it demands critical scrutiny as such. In this investigation however, the synesthetic analogy has opened a path towards its more everyday converse, cross-modal perception, which offers a useful framework for a neuro-aesthetics of fused audiovisuals. These two approaches converge in the figure of the map, the space of correlation; the feeling of noesis or revelation common to both synesthesia and cross-modal binding, could be described as the affect of the map. That affect is central to the aesthetics of fused audiovisuals; though I would argue it offers more than a neurological hit; it brings us into contact with the abstract but culturally crucial terrain of the map itself.
References
Amedi, Amir, William M Stern, Joan A Camprodon, Felix Bermpohl, Lotfi Merabet, Stephen Rotman, Christopher Hemond, Peter Meijer and Alvaro Pascual-Leone. 2007. “Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex.” Nature Neuroscience 10: pp. 687-689.
Baker Fish, Bob. 2005. Review of Robin Fox, Backscatter. Cyclic Defrost 10. http://www.cyclicdefrost.com/review.php?review=795
Bains, Sunny. 2007. “Mixed Feelings.” Wired 15.04. http://www.wired.com/wired/archive/15.04/esp_pr.html
Chion, Michel. [1990] 1994. Audio-Vision: Sound on Screen. New York: Columbia University Press.
Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press.
Cytowic, Richard. 1989. Synesthesia: a union of the senses. New York: Springer Verlag.
Cytowic, Richard. 1996. “Synesthesia, phenomenology and neuropsychology: a review of current knowledge.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Fischinger, Oskar. 1932 “Sounding Ornaments.” Deutsche Allgemeine Zeitung (July 8). http://www.oskarfischinger.org/Sounding.htm
Harrison, John E., and Simon Baron-Cohen. 1996. “Synesthesia: an Introduction.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Hubbard, E.M., and V.S. Ramachandran. 2005. “Neurocognitive mechanisms of synesthesia.” Neuron 48(3): pp. 509-520.
Kandinsky, Wassily. 1977. Concerning the Spiritual in Art. London: Dover.
Manovich, Lev. 2002. “The Anti-Sublime Ideal in Data Art.” http://www.manovich.net/DOCS/data_art.doc
Marks, Lawrence. 1996. “On colored-hearing synesthesia: cross-modal translations of sensory dimensions.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.
Meijer, Peter. n.d. “Artificial Synesthesia for Synthetic Vision.” http://www.seeingwithsound.com/asynesth.htm
Peel, James. 2006. “The Scale and the Spectrum.” Cabinet 22. http://www.cabinetmagazine.org/issues/22/peel.php
Ramachandran , V.S., and E.M. Hubbard. 2001. “Synesthesia – A Window into Perception, Thought and Language,” Journal of Consciousness Studies 8(12): pp. 3-34.
Ramachandran, V.S., and William Hirstein. 1999. “The Science of Art: a Neurolgical Theory of Aesthetic Experience.” Journal of Consciousness Studies 6(6-7): pp. 15-51.
Sekuler, Robert, Allison B. Sekuler and Renee Lau. 1997. “Sound alters visual motion perception.” Nature 385: 308.
Shimojo, Shinsuke, and Ladan Shams. 2001. “Sensory modalities are not separate modalities: plasticity and interactions.” Current Opinion in Neurobiology 11: pp. 505-509.
Simner, Julia, Catherine Mulvenna, Noam Sagiv, Elias Tsakanikos, Sarah A Witherby, Christine Fraser, Kirsten Scott and Jamie Ward. 2006. “Synesthesia: the prevalence of atypical cross-modal experiences.” Perception 35(8): pp. 1024-1033.
Strick, Jeremy. 2005. “Visual Music.” In Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.
Whitelaw, Mitchell. 2001. “Inframedia Audio.” Artlink 21(3): pp. 49-52.
Zilczer, Judith. “Music for the Eyes: Abstract Painting and Light Art,” in Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.
Filmography
Fox, Robin. 2005. Backscatter. Videorecording. Melbourne: Synesthesia Records SYN012 DVD.
Gadow, Andrew. 2005. Techne. DVD-R courtesy of the artist.
Hodgin, Robert. 2007. “Trentemøller and Me.” http://www.flight404.com/blog/?p=52
McLaren, Norman. 1940. Dots. Animation. Available: http://www.youtube.com/watch?v=E3-vsKwQ0Cg
Mclaren, Norman. 1971. Synchromy. Animation. Available: http://www.youtube.com/watch?v=Jqz_tx1-xd4
Timberlake, Justin. 2007. “LoveStoned / I Think She Knows.” Zomba Recording. Music Video. Available: http://www.youtube.com/watch?v=GIYXHLlxD8U
Watz, Marius. 2005. “Video for @c: int.14/37.” http://www.unlekker.net/proj/cronica021/
Monday, September 22, 2008
Limits to Growth
A new generative work that has just fallen into place; I'll be showing prints at the upcoming Dorkbot CBR show (CCAS Manuka, in November). Made with Processing. More will accumulate here.
Economic growth is a central tenet of contemporary capitalism; but the logic of endless growth seems increasingly difficult to sustain. Limits to Growth, published in 1972 (the year I was born), was commissioned by the Club of Rome to report on the economic implications of exponential growth, and used an abstract "world model" to predict the behaviour of the global economic system. This artwork experiments with growth in another model world: a simple generative system in the form of a computer program. In this two-dimensional system, growth has the ability to constrain itself, creating boundaries that define a formal and graphical whole. These forms are utopian diagrams of self-limiting growth.
