This post is (belated) documentation of a project I worked on in 2007-8, creating an audio-responsive generative system for a permanent installation for the Jackie Chan Science Centre (yes, that Jackie Chan) at the John Curtin School of Medical Research, on the ANU campus. Along with some Processing-related nitty gritty, you'll find some broader reflections on generative systems and the design process. For less process and more product, skip straight to the generative applets (and turn on your sound input).
In mid 2007 my colleague Stephen Barrass and I were approached by Thylacine, a Canberra company specialising in urban art, industrial and exhibition design. Caolan Mitchell and Alexandra Gillespie were designing a new permanent exhibition, the first stage of the new Jackie Chan Science Centre, housed in a new building - a razor-sharp piece of contemporary architecture (below) by Melbourne firm Lyons. Instead of just bolting a display case and a few plaques to the wall, Mitchell and Gillespie (wonderfully) proposed a design that hinged on a dynamic generative motif - a system that would ebb and flow with its own life cycles, and echo the spiral / helix DNA structures central to the School's work, and already embedded in the building's architecture.
My initial sketches (below) took the spiral motif fairly literally, drawing vertical helices and varying their width with a combination of mouse movement and a simple sin function - the results reminded me of the beautiful spiral egg cases of the Port Jackson Shark. At that stage we were talking about the possibility of projecting back onto the facade of the building, which has big vertical glass panels; this structure informed the vertical format. I made a quick video mockup of the form on the facade - which was incredibly easy, thanks to the robust, adaptable, extendable goodness of Processing (a recurring theme in the process to come).
These sketches meet the simplest criteria of the brief (spiral forms) but do nothing about the more interesting (and difficult) ones: cycles of birth, growth and death, and dynamics over multiple time scales. Over the next couple of months I developed two or three different approaches to this goal.
The phyllotaxis model blogged earlier was one attempt. Spurred on by the hardcore a-life skills of Jon McCormack and co. at CEMA, I built a system in which phyllotactic spirals self-organised spontaneously. Because in Jon's words, anyone can draw a spiral, what you really want is a system out of which spirals emerge! The model worked, but I had trouble figuring out how phyllotactic spiral forms might meaningfully die or reproduce. Also, by that stage I had two other systems that seemed more promising.
From the early stages I wanted to make the system respond to environmental audio. The installation would be in a public foyer with plenty of pedestrian traffic, so audio promised a way to tap in to the building's rhythms of activity at long time scales, as well as convey an instantaneous sense of live interaction. In the two most developed sketches audio plays a key role in the life cycle of the system.
One sketch moved into 2d, and started with a pre-existing model for growth, by way of the Eden growth algorithm (this system would later be adapted again into Limits to Growth). I had already been playing with an "off-lattice" Eden-like system where circular cells could grow at any angle to their parent (rather than the square grid of the original Eden model). This system also made it easy to vary the radius of those cells individually. The next step was to couple live audio to the system; following a physical metaphor, frequency is mapped to cell size, so that larger cells responded to low frequency bands, and smaller cells to high frequencies. Incoming sound adds to the cell's energy parameter; this energy gradually decays over time in the absence of sound. Cell reproduction, logically enough, is conditional on energy.
The result is that cells which are best "tuned" for the current audio spectrum will accumulate more energy, and so are more likely to reproduce, spawning a neighbour whose size (and thus tuning) is similar to, but not the same as, their own; so over time the system generates a range of different cell sizes, but only the well-tuned survive. The rest die, which in the best artificial life tradition, means they just go away - no mess, no fuss. In the image below cells are rendered with stroke thickness mapped to energy level. The curves and branches pop out of rules sprinkled lightly with random(), resulting in a loose take on the spiral motif, which is probably the weak point in this sketch. I still think it has potential - nightclub videowall, anyone? Try the live applet over here (adjust your audio input levels to control the growth / death balance).
The third model takes this approach to energy and reproduction - about the simplest possible a-life simulation - and folds it back into the helical structures of the first sketches. In this world an individual is a 3d helix, built from simple line segments. Again each individual is tuned to a frequency band, which supplies energy for growth; but here "growth" means adding segments to the helix, extending its length. Individuals can "reproduce", given enough energy, but here reproducing means spawning a whole new helix, with a slightly mutated frequency band. All the helixes grow from the same origin point - they form a colony, something like a clump of grass.
This sketch went through many variants and iterations over the next month or so; in retrospect the process of working to a brief, within a design team, pushed this system further than I ever would have taken it myself. At the same time I was testing the system against my own critical position; I've argued earlier that the generative model matters, not just for its generativity but the entities and relations it involves.
From that perspective this system was full of holes. Death was arbitrary: just a timer measuring a fixed life-span. "Growth" was a misnomer: the number of segments was simply a rolling average of the energy in the curl's frequency band, so the curls were really no more than slow-motion level meters. Taking the organic / metabolic analogy more seriously, I worked out a better solution. An organism needs a certain amount of energy just to function; and the bigger the organism, the more energy it needs. If it gets more than it needs, then it can grow; if it gets less than it needs, for long enough, it will die. So this is a simple metabolic logic that can link growth, energy and death. Translated into the world of the curls: for each time step, every curl has an energy threshhold, which is proportional to its size (in line segments); if the spectral energy in its band is far enough over that threshhold, it adds a segment - like adding a new cell to its body; if the energy is under that threshhold, it doesn't grow; and if it remains in stasis for too long, it dies. Funnily enough, the behaviour that results is only subtly different to the simple windowed average. Does the model really matter, in that case? It does for me at least; if and how it matters for others is another question.
Next, the curls developed a more complex life-cycle - credit to Alex Gillespie for urging me in this direction. In line with the grass analogy, curls grow a "seed" at their tip when they are in stasis; when they die, that seed is released into the world. Like real seeds, these can lie dormant indefinitely before being revived - here, by a burst of energy in their specific frequency band. After several iterations, the seed form settled on a circle that gradually grows spikes, all the while being blown back "down" the world (against the direction of growth) by audio energy (below). As well as adding graphic variety, seeds change the system's overall dynamics. Unlike spawned curls, seeds are genetically identical to their "parent" - attributes such as frequency band are passed on unaltered. Because each individual can make only one seed, that seed is a way for the curl to go dormant in lean times; if it gets another burst of energy, it can be reborn. The curls demo applet demonstrates this best (again, adjust your audio input and make some noise).
A few technical notes. One big lesson here was the power of transform-based geometry. Each curl is a sequence of line segments whose length relates to frequency band (lower tuned curls have longer segments); each segment is tilted (rotateZ), then translated along the x axis to the correct spot. A sine function is used to modulate the radius of each curl along its length; frequency band factors in here too; this radius is expressed as a y axis translation. Then the segment is rotated around the x axis, to give depth. I iterate this a few hundred times to get one curl, and repeat this process up to twenty times to draw the whole world - each curl has its own parameters for tilt, x rotation increment, and frequency band.
In the live applet audio energy ripples up the curls, from base to tip. This was added to reinforce the liveness of the system and add some rapid, moment-by-moment change. It was implemented very simply. I used a (Java) ArrayList to create a stack of audio level values; at each time step, the current audio level value is added at the head of the list, and the ArrayList politely shuffles all the other values along. So each segment's length is a combination of three values; the base segment length, a function to taper the curl towards the tip, and the buffered audio level.
The graphics are all drawn with OpenGL - following flight404 I dabbled with GL blend modes, specifically additive blending, to get that luminous quality. The other key visual device here is the smearing caused by redrawing with a translucent rect(); instead of erasing the previous frame completely this fades it before overlaying the new frame. It's an easy trick that I've used before. But as Tom Carden explains, in OpenGL it leaves traces of previous frames. I discovered this firsthand when Alex and Caolan asked whether we could lose the "ghosts." I was baffled: on my dim old Powerbook screen, I simply hadn't seen them. Eventually, juggling alpha values I could reduce the "ghosts" to almost black (1) against the completely black (0) background - but no lower. Finally I just set the initial background to (1) instead of (0), and the ghosts were gone.
The adaptability of Processing came through again when it came to realising the installation. The final spec was a single long custom-made display case, with three small, inset LCD panels. These screens would run slide shows expanding on the exhibition content, but also feature the generative graphics when idle; the case itself would also integrate the curls as a graphic motif. For the case graphics, I sent Thylacine an applet that output a PDF snapshot on a key press; they could generate the graphics as required, then import the files directly into their layout.
The screens posed some extra challenges. The initial idea was to have the screens switch between a Powerpoint slideshow, and the curls applet; but making this happen without window frames and other visual clutter was impossible. In the end it was easier to build a simple slide player into the applet: it reads in images from an external folder, allowing JCSMR to author and update the slideshow content independently.
So to wrap up the Processing rave: it provided a single integrated development and delivery tool for a project spanning print, screen, audio, interaction, animation and even content management. Being able to burrow straight through to Java is powerful. Development was seamlessly cross-platform; the whole thing was developed on a Mac, and now runs happily on a single Windows PC with three (modest) OpenGL video cards. The installation has run daily for over six months, without a hitch (touch wood).
Some installation shots below, though it's hard to photograph, being a glass fronted cabinet in a bright foyer - reflection city. I'll add some better shots when I can get them. If you're in Canberra, drop in to the JCSMR - worth it for the building alone - and see it in person.
And very finally, photographic proof of the Jackie Chan connection - image from The Age.
Friday, January 16, 2009
JCSMR Curls
Monday, October 01, 2007
Impermanence - Life by Projector-Light at BEAP
Another quick catchup post from BEAP, before it recedes into the mists of time. The Impermanence show at the John Curtin Gallery is a beautifuly-installed collection of video and interactive works. The gallery is an impressive space and curator Chris Malcolm knows how to deal with media art: HD projections onto custom painted surfaces, well-contained hi-fi sound, careful design and layout, and tons of breathing space. Daniel Lee's Origin (below)was shown as still prints and high-def video loop, and looked quite amazing; though the work in itself didn't stun me. Originally created in 1999, it's almost retro on the new media scene's manic timescale, and to me it was showing its age. For 90s Photoshop organohybrids you can't go past Australian artists Patricia Piccinini, Murray McKeich or Linda Dement - all of whom deliver lush surfaces with a lot more bite than Lee's manipulations.
"Nice, but..." just about sums up my response to this show; Lynette Wallworth's Still:Waiting2 features amazing nature-doco style video, with thousands of small parrots coming to roost in some enormous red gums in the dawn light. But it only made me want to be actually watching the birds instead of sitting in a dark box in front of a HD projector screen. Bill Viola's slowmo screen Observance is very pretty, but I just can't watch the overacting. The most interesting thing in the place was, unfortunately, an outright failure.
When I visited, we were told that Christa Sommerer and Laurent Mignonneau's Eau de Jardin (above) was broken. This work reprises their 1992 classic Interactive Plant Growing, with sensors embedded in potplants driving artificial foliage on a panoramic projection. Ordinarily the artificial plants gradually recede back into their "pond," making space for new creations. But something was wrong and the water plants weren't dying, resulting in a kind of virtual algal bloom: the screen was locked up, choked with life. By contrast the real plants were not looking good at all. The ferns were shedding fronds onto the floor; I heard someone report that the soil in the pots was dry, while gallery staff explained the lengths they were going to in trying to keep the plants alive - wheeling in big UV lamps overnight, to compensate for the dim projector-light of their daytime life. The disjunction was stark; the polarity flipped on the happy techno/bio mix that characterises much of Sommerer and Mignonneau's work. It ocurred to me that a really useful project for all these bio-artists would be to engineer a form of plant life that could live happily under the light of a data projector.
Friday, February 16, 2007
Jonathan McCabe - Very Cellular Automata
A new year, and another exhibition from Jonathan McCabe at Canberra gallery/cafe The Front. The show, Travelling Wave, was shared with painter Luke Nilsen; it included some collaborative canvases, with Nilsen painting over McCabe's digital patterns, and new works from McCabe's Butterfly Origami and Nervous States processes. But also on the (very crowded) walls were images from a new McCabe process, based on cellular automata. In themselves the images are chunks of psychedelic maximalism, similar to McCabe's earlier work. But once again the real hook here is the mind bending and unusually rich generative process.
The generative system involves four linked cellular automata - think of them as layers. "Linked" because at each time step, a cell's state depends both on its neighbours in that layer, and on the states of the corresponding cells in the other three layers. Something like a three-dimensional CA, but not quite; the four layers influence each other through a weighted network of sixteen connections (a bit like a neural net). The pixels in the output image use three of the four CA layers for their red, green and blue values. (The images here show the full image on the left, and a 1:1 detail on the right)
As in a conventional CA, each cell looks to its neighbours to determine its future state. This is a "totalistic" CA, which means each simply sums the values of its neighbours, then changes its state based on a table of transition rules. Now for the really good part: each cell also uses its recent history as an "offset" into that transition table. In other words, the past states of a cell transform the rules that cell is using. The result is a riot of feedback cycles operating between state sequences and rulesets; stable or periodically oscillating regions form, bounded by membrane-like surfaces where these feedback cycles compete. Structures even form inside the membranes - rule/state networks that can only exist between other zones. 
The images reinforce the biological analogy, but philosophically (ontologically?) this system is even wilder. It's inspiring to see a formal system where "the rules" are local and variable, rather than global and static. The way things are (or have just been) controls the rules that determine how things will be next - that much is historical relativism I suppose. But here we see regions of self-perpetuating but incompatible realities, competing for space - even states of being that only emerge where two or more reality-attractors meet. An ecology of ontologies, if you will.
McCabe has put up a page with lots of these images, including full resolution (2048x2048) jpegs.
Sunday, July 16, 2006
Living with E-volved Cultures
Documentation of Driessens & Verstappen's E-volver has been floating around for some time - enough that I've already raved briefly about the work based purely on stills and a description of the generative system. I recently received a catalog on the work's installation at a Dutch medical research centre, which included a runtime / screen saver version of the work. Long story short, I've had the work running for a month or two now, at home and in the office... living with it, pretty much.
The generative system here is simple and superbly elegant; you can read about it briefly on the artists' site, though I think they under-explain it. Each image is generated by a set of eight single-pixel automata. Each automaton is a little state machine that moves around the picture plane, and alters pixel colours, based on the colour values of neighbouring pixels, and its own set of rules. A couple more things: each automaton has a unique rule set, so each set of eight forms an "ecosystem" or "culture" of interacting individuals; and each set is not hand-programmed, but evolved, based on the images they collectively produce. In the installation version visitors drive the evolution (as in Karl Sims' classic Genetic Images installation); in the screen saver version the artists have selected a set of pre-evolved "cultures".
I continue to be amazed by the images the system produces; watching it run live is especially illuminating. The word "organic" is overused in describing generative art, but it's unavoidable here; the forms that emerge have a fine-grained integrity and richness about them that inevitably recalls physical and biological processes. One of the "cultures" in the screen saver looks like time-lapse satellite photography of a rainforest: churning plumes of green threaded with dark river-like fissures. The rivers seem to silt up, forming classic serpentine paths, seeking out new channels. Another culture generates forms like the banded, differentiated structures of agate; another alien hot-pink and yellow clouds. Each one is in continuous process, eating itself, restructuring itself slowly from the inside.
The artists frame the work along the lines of their other practice, as an exploration of autonomous generativity. They also, like Karl Sims, talk about the interactive installation version in terms of human-machine collaboration. Again I think they underplay the significance of the work, which for me is in the structure of the generative system. The use of co-evolved communities is one element, but the key is the relation of the entities to their "environment", the picture-plane. It acts as a shared information space, a medium through which the agents interact, as well as a historical buffer; in this world the past is a rich source of structure for the present. Instead of a neutral "blank slate", the environment is a malleable habitat that in turn shapes the actions of its inhabitants. It plays out a kind of ecological fable of finely balanced coevolution, as one tiny fissure in the image forms a toe-hold for an influx of new forms: this is the world as teeming, shiftless self-structuring process.
So living with E-volved Cultures has been interesting; I had wondered if it would wear thin. It hasn't; I still love it, obviously, and I won't be switching it off any time soon. If anything this ambient exposure has given me space to keep thinking about the work in a back-of-the-mind way, and I'm aware that very few works in this genre get that opportunity. The people from Bitforms promote generative art for domestic consumption through Software {Art} Space, but I don't know how many takers they get. Anyone else out there living with generative art? Please report...
