Musical scores are a precise set of actions: play this note, at this speed, at this volume, for this duration, with these other notes. They are read and then played. They are rules for the future, outlining the sounds to be played.
Sonograms (also known as spectrograms) and waveforms exist after the sounds have been played. They are visual artifacts of the sound. Sonograms show the spectrum of frequencies in a sound selection with time along the horizontal axis and frequency along the vertical axis. The colour intensity often represents the amplitude of the individual frequencies. When frequency is shown logarithmically, rather than linearly, the musical and tonal relationships are emphasized.
Scores and sonograms bracket the event of music being produced.
A sonogram is a visual analysis of a recording, of sound already played
By parsing out notes from the sonogram, I’m going through a translation of played > recorded > parsed > replayed in a new context
Using the sonogram to approximate a note and then building it into the composition
Bird songs are complex, polyphonic yet the score (In C) is composed of single notes, no chords – monophonic
Experimental Notation Precedents
Robert MoranLouis Andriessen, Blokken from “Souvenirs d’enfance”Brian Eno’s graphic notation for Music for Airports, published on the back of the album sleeveArtist John De Cesare’s rendition of Wagner’s “Ride of the Valkyries”Gyorgy LigetiKonstellationen by Roman Haubenstock Ramati
Aleatoric music is a form of composition in which a significant element is left to the determination of its performers. Terry Riley’s “In C” is a composition of 53 phrases to be played in sequence, but each performer individually determines the number of times they play each phrase. They form a collective conductor yet also act as many conductors as the role is distributed among each of them.
Terry Riley’s “In C”
Bird Sounds in C proposes to create an orchestra of birds to play “In C”. For each bird, the phrases of “In C” are recomposed by alternating and arranging an individual “C note” extracted from their bird song. These phrases are then further composed by a single user who chooses when to advances each bird through the composition.
Constructing Sounds
Rather than starting with notation and then playing the written composition as a “live” performance, Bird Sounds in C instead starts with recorded sounds and reorders extracted elements to form the phrases of Riley’s “In C”. By using recorded bird songs—sounds with specific structure and meaning within their species—and recomposing them into a new composition, the sounds are given new meaning and recontextualized outside of “wildlife”.
The written score for “In C” appears monophonic, consisting of a single melody without chords or harmonies. However, through Riley’s intended method of performance, the various phrases and patterns overlap to create a densely layered and heterophonic texture.
Reinterpreting this dense texturing, Birds in C instead uses the complex polyphonic sounds of bird songs. When visualized as sonograms, the layered frequencies of recorded bird songs are striking.
A Fox Sparrow’s Breeding Song (Source: The Bird Guide)
Using the sonogram as a visual artifact of the bird song, I identified a snippet which approximated “C” which I then transposed and arranged into Riley’s phrases. In this way, the translation of recorded sound to extracted and reordered elements attempts to use polyphonic sounds as monophonic approximations of single notes.
Bird Sounds Iteration-01
Click the circle to start and advance the bird sounds
Midi Sounds Iteration-01
Prior to working with bird sounds, I used midi to create the tones. One user controlled two “musicians” to advance through the piece using the keyboard. When purely using midi, the resulting audio was to distinguish into two “musicians”.
Questions of Playability
What is the allowable degree of asynchronization?
Does the computational aspect create a system which removes human variability (i.e. the code makes sure when a phrase is advanced, it comes in on the first beat of a bar – or does the musician need to keep track of this?
How much content from the original notation of the piece does the user need to know? What visual indicates are needed to make choices about when to advance one bird or another?
The (partial) text content is pulled from today’s front-and-center article on the New York Time’s website. Knowing that next week we’d be looking at APIs, I hope to continue building on this sketch and pull the article content automatically from the New York Times and use it to look across various reporting desks. Is the the Arts & Culture desk more descriptive, using more adjectives and adverbs? Or what are the most common proper nouns used in World News?
When hovering over any of the four buttons in the corners, the corresponding words are highlighted in the text. The contrast between the faded words and selected words provide a rhythm and instant visual indictor for the most common type of word or how this changes through an article. (link to code)
Synchrony occurs when events operate or occur in unison. At the core of synchrony is the passage of time and a change (the event) occurring within this. “Events” refer to a change in something—sound, motion, visuals, etc. Something was on, and then it wasn’t. Something was here, and then it was there. The other component of synchrony the relationship between multiple events; synchrony cannot be identified in isolation. When exploring synchrony, the questions to ask are:
What are the events, or what things are changing?
What is the duration of the relationship?
In the physical world around us rarely are events synchronized. We walk at a different pace that those around us, cars accelerate at difference speeds when the light switches from red to green. But this is why our attention can be captured by synchronicity. Since it’s so uncommon, we often take note when objects, people, actions or sounds around us are in sync. Sometimes this synchronicity is planned and part of a performance, such as an orchestra playing to the time kept by a conductor.
Sometimes it happens by chance. There’s a goofy scene in Scrubs in which the sounds playing in JD’s headphones seemingly align with the actions and movements of those around him.
I’m interested in exploring how changes in synchrony and asynchrony may affect the attention of a user or focal point of a piece of work. What interactions can be used to adjust the synchrony of events? Can users only participate in a piece if they perform synchronously? How does synchrony or asynchrony affect their behaviour?
Complete Audio and Visual Synchrony
Jono Brandel created Patatap, a keyboard-controlled sound and visual animation platform. Individual sounds and short animated elements are mapped to each key. Pressing the spacebar produces a new colour scheme as well as a new set of sounds. When keys are played systematically, the user can generate melodies and rhythm. At an event in San Fransisco, Brandel himself demonstrated this with the help of a looper (I think?)
What’s particularly evident in the performance video is how intertwined and inseparable the sounds and visuals are. Each time the colour background swipes across the screen, it reinforces the underlying beat. Perhaps this level of synchronicity is particularly suited for the type of electronic sounds available in the platform. Like each animation, each sound is triggered in isolation and appears/is heard precisely. There is overlap between sounds and visuals but the trigger and their presentation to the audience occurs individually.
Variable Synchrony and Asynchrony
In Bon Iver’s recent music video for 29 #Strafford APTS, the audio is accompanied by distorted visuals that are akin to seeing through a kaleidoscope. The visuals at the beginning of the song seem to take inspiration from Joseph Albers, composed of colour blocks with stark edges. Yet this clarity and precision is disrupted by the effects of the kaleidoscope—they become layered, multiplied and seemingly of a dream state. They transformation of visuals and effect of the kaleidoscope do not seem to be tied to the audio. Changes happen without regularity. The computer generated graphic compositions switch to recorded footage of birds, a bedroom, nature. Yet there is one striking moment of alignment between the music and visuals. After zooming into a psychedelic-like sunburst graphic that was above the bed, and seeing the pixel grain of screen which is showing this digital image, at 2:53 both the music and visual “snap out of it” – almost like waking up out of the dream. With the reappearance of words layered on the visuals, the viewer/listener is reminded that a story is being told. The dream state they were lulled in to through the use of asynchrony and blended changed is disrupted by the sudden alignment of sound and visual.
Work in Progress
To explore this idea of visual and auditory synchronization, I want to create a potentially-interactive animation to be coupled with a previous sound piece I worked on. In early prototyping, I’ve started looking at how to get objects moving independently.
I imagine building out a larger prototype in which multiple objects are synchronized to different aspects of an audio clip. Maybe changes in volume result in something changing size, or an object appears and disappears in line with the beat. Are all objects simultaneously synchronized with the audio or do they each come in and out of sync independently?
We recently learned how to transmit signals using serial communications from an Arduino to the browser and produce an “outcome” with P5.js. Physical interactions, such as turning a potentiometer or pressing a switch, resulted in signals being passed from the Arduino using Serial.print().
As a prototype to explore this functionality, my partner, Bryan, and I built a rain simulator that was controlled with a force sensor. More force produced more rain drops. (link to code)
An early iteration of the code used three bands of values from the force sensor to trigger varying amounts of rain. A sensor reading between 0 to 300 created a light drizzle, 301 to 800 created a moderate rainfall, and 800 to 1023 resulted in a monsoon. However, we wanted a more nuanced gradient of falling raindrops. By using a “greater than” condition for the modulo operation was able to produce the desired effect.
Interacting with the computer using a device other than the traditional mouse, trackpad, or keyboard was thrilling. Using an unexpected mechanism can inspire interactions originally overlooked or not considered. After building the rain-maker, I’ve been considering what other mechanisms I could use to control a game and how unexpected sensors may challenge a user’s preconception of a familiar game. For example, what if:
interacting with a photo resister moved the paddle for pong?
two force sensors controlled left-to-right movement and up-and-down movement respectively?
a potentiometer was used to control an object balancing on something else?
