In 2023 I worked on Pond Brain, a pair of installations by Jenna Sutela for Copenhagen Contemporary and the Helsinki Biennial.

I trained real-time neural synthesis models with recordings of marine mammals and water sounds and built them into a feedback ecology with the sounds of musical instruments and stars from the Kepler exoplanet survey. Microphones in the room spun the sounds of visitors into the web of human, whale and celestial body.

Since 2021 I'm a PhD researcher in the Intelligent Instruments Lab.

Synesthesia is VJ software for making audio-reactive visuals. It draws from the Shadertoy community, where everything is made with GLSL fragment shaders, but includes an easy way to add audio and video inputs and build simple MIDI-mappable control interfaces. Basically, everything a VJ needs to actually perform live with some shaders. The folks building Synesthesia seem super respectful of free-sharing ethos of Shadertoy -- they reached out to shader programmers like me and comissioned new works for Synesthesia, while keeping the actual shaders always hackable by users within the application. I wound up making a few fun video feedback and cellular-automata based scenes, and recently was goaded into using some of them to mess up the live stream from an algorave concert at the AIMC conference:

In 2019 I worked on a concert installation called Deviant Chain with Stefan Maier and Alan Segal. Stefan, a composer, and Alan, a filmmaker, created a sort of art haunted house in which the audience moved from room to room in a former bank vault, encountering a series of short videos and sound works with themes of transhumanism and machine alterity. One of these dealt with the development of language and the human voice, and for this I created neural text-to-speech software for Stefan to gather sound material from.

At this time, the WaveNet demo was in the air. This was one of the first neural generative models to work well with raw audio, and the uncanny babbling of the unconditional speech model was an arresting artifact of the ever-weirder capabilities of statistical autoregressive models. It had a few problems though: no open source implementation, very expensive training process, and extremely slow inference process. After some research, I found an open implementation of Tacotron2 from NVIDIA, which was state of the art for text-to-speech at the time, and substituted a WaveGlow vocoder for the original WaveNet. It turned out to still be quite slow and expensive to train WaveGlow, but it could actually generate audio with close to real-time throughput, and including a full text-to-speech model added many dimensions of control. This was a tool you could iterate with. I found the TTS could be induced to babble by letting it run past the end of its predicted alignment the the text, at which point it would get confused and start jumping around. Finally, I trained a few models on voices from Mozilla's CommonVoice project to get a much wider range of voice sounds. I built in some extra features to the CLI and got the whole thing running on Stefan's computer using PyInstaller.

I also developed a small glyph generator which was used in some of the videos -- this took in sounds, extracted features with some machine listening algorithms, and then mapped those features to a set of curves and strokes and stored them as 3D mesh files. I traveled to Oslo to install Deviant Chain, setting up the video players and cueing system. I was pretty happy with the results -- it was genuinely spooky working on it with just a few of us in the empty bank basement at night.

I implemented Rosca, Mihaela, et al. "Variational Approaches for Auto-Encoding Generative Adversarial Networks" using Pytorch. It's a modular implementation -- plug in any torch modules as encoder, generator, discriminator and code discriminator.

On GitHub

These are two animations I made for FS35 with Jodie Mack in winter 2015.

Bear Cult

Bear Cult was inspired by a collection of Joseph Campbell's essays I found in a used book store, specifically one entitled "Renewal Myths and Rites of the Primitive Hunters and Planters". It's about the prehistoric origins of myth. In Campbell's telling, the purpose of myth is to conquer death. Campbell cites preserved arrangements of cave bear bones as evidence for the ancient roots of a bear-baiting ritual still practiced by indigenous peoples in northern Japan. In the bear-baiting ritual, a young bear is captured, raised in captivity, and then ritually tormented, killed and eaten. The bear is believed to contain the soul of a demigod which yearns to be released from its fleshy bear-prison. For Campbell, this is the coping mechanism of an animal which kills to survive but understands death. It's the hunter's moral justification for killing: death isn't terminal, killing is a kindness. Ritualized death is a point of contact with the numinous; the fear of death is transmuted to awe. I found the deep connection Campbell makes between this sacred feeling and human capacity for cruelty heartbreaking.

I decided to depict a bear-baiting ritual. Not the specific cultural practice Campbell recounts, but the abstracted elements of a hypothetical prehistoric bear cult. The body of the bear would be made by hand with paper drawings and cutouts, while the soul would be made in software with virtual video feedback. These material and digital components would interact via fluorescent orange paper which could be keyed out in video editing software. A possible alternate title: "Mind-Body Chromakey". Originally I had storyboarded a longer narrative including human hands and spears piercing the bear and the escape of its soul, but only part proved feasible in the time I had.

I'm happy with how the bear soul material turned out. After shooting the paper animation, I exported just the key as a binary mask. As an input to the feedback process, the mask lets the roiling cloud-forms conform to the geometry of the bear eyes and nose-hole. Unfortunately the black-on-black paper was unwise; there's a lot of camera noise after adjusting the levels to make everything visible. A darker background of black felt or something might have worked better.

This project taught a lesson about how sound influences the perception of time in film. What felt like a good visual rhythm when I was silently animating seems weirdly abbreviated with the soundtrack added. The sound was a bit rushed, but I like the general effect. It's some recordings of bear vocalizations and carcass-mangling sounds chopped up into phase-rhythms with the same methods I used for SIBA. I'd like to revisit this project someday.

HOFSTADTERPILLAR

HOFSTADTERPILLAR is a many-looped negotiation between rules, materials and intuition, an effort to explore Hofstadter's idea of strange loop by drawing.

Drawing is an act of alternating abstraction and reification, recognition and construction of forms among materials. I draw, I see what I have drawn, I draw more. Animating by hand can be seen as a sort of image feedback loop; each frame is drawn with reference to previous frames, which are visible beneath transparent cels. I developed loose sets of rules to draw by, of the type "a silver line extends and curves more each frame". Other rules emerge from the materials; paint markers pool and smear when the next cel is placed on top. I used the "maximum cycle" technique of repeatedly drawing new material into the same set of frames to construct rich and evolving loops. The sound is another set of feedback loops, a Max doodle with chaotic noise generators stimulating banks of tuned filters.

In 2014 I spent a lot of time working through a Steve Reich fascination by making tape music. This piece was eventually presented at ICMC 2015. SIBA I|II|III stands for "Studies In Being Alive One Two and Three", which was a joke about the lack of purpose I felt at the time and also a genuine attempt to describe what I was groping toward, stylized to reflect the asemic repetitious sound-destruction therein. I don't know how to feel about it.

Douglas Hofstadter's Gödel, Escher, Bach contains about a hundred captivating ideas, and just one of them is the concept of video feedback: connect a video camera to a screen, and then point it at the screen. Weird stuff happens on the screen as light is projected though the air, captured by the camera and circled back to the screen, mutating each time. Zoom out to get a infinite hall of mirrors effect, zoom in to get kaleidoscopic pulsations, loosen the focus to get bubbling slime.

During college I became interested in computer graphics. Particularly, the idea of procedural graphics: producing images with only code, as opposed to virtually drawing (as with photoshop) or digitizing reality (as with a digital camera). Procedural graphics can translate some of the tantalizing infinites of mathematics into rich sensory experiences! A brief article by Íñigo Quílez introduced me to the idea of turning a random number generator into an endless alien terrain. One popular way to do this is with fragment shaders, specialized programs for doing pixel-by-pixel graphics with the GPUs found in most computers. For more than a few examples, see Shadertoy.

You can put a fragment shader into feedback just like a video camera. Pixels go in, pixels come out, pixels go back in 60 times per second. The shader determines how colors mutate and interact with nearby colors each time. Here are a few of mine:

Other things which resemble digital video feedback:

• 2D Cellular automata like Conway's game of life
• Finite element physical simulations of fluids and reaction-diffusion systems
• Convolutional neural network visualization techniques like deep dream

Other people working with this kind of stuff:

• Sabrina Ratté
• Alex Dupuis
• Cornus Ammonis
• Andrew Benson
• Felix Woitzel

I just can't stay away. See projects such as ABSTRACT/CONCRETE, Video Synthesis With Convolutional Autoencoders and Data Sonification Using a Cortical Representation of Sound.

In spring of 2015 I made a synthesizer called Bendy as a seminar project. A technical description and a Max/MSP implementation can be found on GitHub. Here's what is sounds like: