The Shape of Sound: Using mixed realities to bridge music and architecture
Amy Gharst | Faculty of Architecture, Design and Planning, University of Sydney, Australia
CAADRIA 2008, [Proceedings of the 13th International Conference on Computer Aided Architectural Design Research in Asia] Chiang Mai (Thailand) 9-12 April 2008, pp. 494-500
Abstract. There are structural and aesthetic components in architectural design that mirror the foundational components of musical compositions. In recent years, both architects and musicians have taken advantage of the advances in technology, allowing for new designs and compositions that would not be possible without computers. This paper discusses the issues and possible outcomes of a system that would allow for collaboration between architects and musicians.
Keywords: Collaboration, Communication, Interaction, Architecture, Music, 3D Visualizations, Composition, Modeling.
There is a certain fluidity and beauty to post-contemporary architecture that is not found in the stoic classical buildings of the past. While the buildings of the past are beautiful in their own right, it is with the recent advances in technology that more experimental designs and structures have been made possible. The same is true for music. With the aid of computer technologies, musicians are able to experiment with the shape of sound and the way in which it is produced.
Architecture and music are linked by their similar characteristics. The structures, repetitions, patterns, harmonies and variations (among other elements) are based on the same mathematical and aesthetic principles that can be found in nature.  With the digitization of design (including musical composition as a design form), the latest trends in architecture and music have included producing impressive new techniques. Using technology to create a feasible collaborative connection between architects and musicians can potentially produce an enriching design environment where the relationship between space and sound will be able to evolve, just as each of the separate fields have advanced with the aid of technology.
2.1 Collaborative Design
Using technology for collaborative design purposes is typically thought of from two approaches, the designer-designer collaborative relationship, and the designer-client instructive relationship. Two designers can use technology to collaborate with each other at different stages of a design. In a designer-client relationship, the designer can use the technology to explain confusing or difficult concepts to the client. But for the purpose of collaborative design between architect and musician, the technology needs to be appropriate for each of the separate skill sets and design techniques brought to the collaborative environment (similar to the designer-designer collaborative environment, except with the addition of a different type of design.) It also needs to be able to translate what each person is into a language that the other person can understand (similar to the designer-client instructive environment.) To do this, there must be similar attributes between the two separate fields that are translatable, at the very least, on an abstract level.
2.2 Compatibility of Architecture and Music
“There has always been a close relationship between music and architecture, experimental or otherwise, in terms of structure, pattern and aesthetics, even though sound ultimately describes immaterial space. Plainchant, for example, somehow belongs to Romanesque abbeys, even though its origins are much older, just as Bach is all but synonymous with baroque churches. For better or worse, Wagner conjures images of the fairy-tale, alpine fantasmagoria of Neuschwanstein, the Sleeping Beauty castle built by Wagner's indulgent patron, Ludwig II.” Jonathan Glancey 
Music is usually thought of as an auditory experience, but just as architecture has rules, musical composition also is guided by structure. A basic rhythm tree shows the very mathematical and precise nature of musical composition. The musician is bound to rules without which music cannot exist. Architecture, likewise, can not physically exist as real spaces without set rules that are dictated by gravity and other forces outside of the architect’s control. Architectural structure is composed of beams, columns, trusses, foundations, and other physical objects which already exist in previous buildings and designs, but which the architect can change, according to the rules of architecture, to meet the needs of each space. Musical structure is composed of notes and rests within measures. These elements of music are given, but it is in the way the musician places them, according to the rules of music, that individual pieces of music are produced.
An interesting element is the element of surprise. In music this is the climax of a piece and in architecture is often referred to as the attractor or a focal point of a space. This element of surprise is not necessarily a loud, shocking moment in time or space. It can be tranquil - an open atrium flooding a space with unexpected ambient light, or a several measure long trill building up to a pleasant, flowing resolution in a particular piece of music.
Where music is guided by time, architecture is seen by the movement through space. At different points in time, a musical piece will fluctuate, but the overall atmosphere of a good piece will stay consistent with its main theme. The different spaces in a building will also change depending on context, but the overall atmosphere of a good space will also stay consistent with itself.
Further, music is appealing to the emotions of the listener or player through expression, while architecture provokes a certain experience from the user. These might be considered the most influential part of the piece or space, for it is the expressive experience that lasts with the listener or user and that is used to describe of the work.
Both music and architecture use repetition of elements as a structural or aesthetic design component. Repetition can create movement through space or time, and produces pleasing patterns and shapes. Musical repetition is created both with repetitive individual notes and rests, but also with repetitive patterns of notes and rests. Architectural repetition can be created with structural elements such as columns or trusses, but can also be created with repetitive patterns designed by the architect.
Harmony in music is equivalent to proportion in architecture.  A good harmony is proportional, just as a well proportioned space has a certain and pleasing visual harmony. Additionally, a musician can use dissonances to change the effect of the harmony. A dissonance might sound like a mistake, but if it follows the structural rules of music and it is as the musician intended, it is not a mistake. Similarly, architectural dissonances, such as many deconstructivist buildings, might appear wrong but are not. Designed well, dissonances have their value, and to many are aesthetically pleasing.
The intimacy of a physical space is similar to the amplitude, or volume, of music. A small space might appropriately match a soft pianissimo volume, while a large space could express the feelings of a loud fortissimo. However, each space and musical piece is different and the amplitude matching the intimacy is certainly dependent on each unique example.
Music is meant to be performed, to be enjoyed by an audience, by a single listener or by the musician performing the piece. Likewise, architecture is meant to be used and lived in; it is meant for interaction. “Music is part of architecture -- the acoustics of the building, the sound of a city, inspire you and give you kind of a connection. And certainly, even the way that architecture is produced is very similar to music. You have to write an abstract score. The plans, the sections, elevations, but in the end it has to be performed by others and it has to… be harmonious."  Like the expression and experience, music and architecture are similar in their dependency on human interactivity.
Finally, all music has a concept or story. The music itself is a story, it tells a story, it goes along with a story, or it is a story to be determined by each individual listener. The story might be a specific one, a fairy tale, an opera, a pop culture reference; there are no limits on the stories that music tells. Similarly, architecture starts off with a concept. There are specific meanings behind the design decisions that an architect makes. The design begins in the conceptual stage with ideas that might or might not be made public, but those ideas often carry through the entire design process.
2.3 Space Meets Sound
There are many examples of architects using their musical backgrounds as inspiration for their designs. Two notable musician architects are Daniel Libeskind and Iannis Xenakis. Daniel Libeskind, formerly a professional pianist, has spoken often about the connection between music and architecture. “Music is certainly part of my life and it is part of architecture. When I was designing the Jewish Museum in Berlin, the acoustics of the building, the sound of the building was one of the primary dimensions of creating that space of the void.”  Libeskind has made abstract and aesthetic connections between sound and space and designed buildings based on his own interpretations of those connections.
Iannis Xenakis was a 20th century Greek composer and an architect who took a more literal approach to designing spaces based on the shape of sound. He drew theoretical musical elements and then translated those into physical shapes. Using elements from music, Xenakis worked with Le Corbusier on projects such as the chapel at Ronchamp. He was intrigued by the nature of the problems and solutions within architecture and music , and his architectural design strongly reflected his musical abilities.
There are also examples of architects collaborating with musicians, such as Santiago Calatrava’s Milwaukee Art Museum opening to a musical piece written by Philip Glass specifically for the opening. While this is a unique collaborative example showing the compatibility of architecture and music, there are still issues involved with this type of collaboration. Most notably, as Philip Glass has spoken about, the design of the building was nearly finished by the time Glass was asked to write music for the opening. “The designer or the architect or the artist has to actually be willing to get into the complex business and the complex encounter of collaboration. And usually, you have to remember, by the time the inauguration of a building happens, the building is done, it’s up… By the time they think of the composer, the work is done... The trouble is that real collaboration has to begin very early and by the time they think of the composer, it’s very late.”  True collaboration between architect and musician would enable them to work together at the beginning of the design process, placing equal importance on both the music and the building.
2.4 Sound meets Space
There are several artistic projects and computer programs that use the similarities between architectural space and music to create new sounds or spaces. Symbolic Composer  is a computer program for musicians that aids in musical composition. It has the capability of generating 3D models in VRML format based on the music. Instant City  is an interactive table that allows users to create sounds by manipulating space in the form of blocks. Audiopad  is a system that facilitates the composition and performance of electronic music via the movement of objects on a tabletop. AudioCubes  allows the user to create musical patterns which are then exchanged by the cubes. As the user moves the cubes around, the patterns evolve and music is created. Each of these systems is unique and innovative, but they give greater importance to the music than the space. If space and sound are given equal importance, new ideas and possibilities can be generated from the collaborative work between architects and musicians.
3.1 A Marriage Between Experimental Space and Experimental Sound
A technology that allows for a digital marriage between architecture and music, one in which both will have equal importance, will have certain issues with which to be dealt. Primarily would be the question of music being taken as either an abstract or literal definition of space or if a median between the two (depending on the context of each individual situation) should be used. The dynamics of a real space translated from a virtual music will be considerably different depending on the degree of abstraction in the translation. Additionally, if a unique, real architectural space can be produced from a virtual music, then reversing the process should produce real music out of virtual three dimensional spaces. The more experimental the dialogue between real and virtual, the more interesting the resulting spaces and sounds will be.
3.2 Platform for Collaboration
To bring musicians and architects together to a collaborative environment that places equal importance on both music and architecture will require the collaboration to take place at the beginning, conceptual and experimental stage of design for both. Each will require a unique system and interface that is appropriate for the details of their work. The musician will need digital instruments, inputs, audio equipment and music editing software, while the architect will require CAD and modeling programs. But the systems must also be able to read, translate and display the work of both. The architect must hear the music and see the two dimensional graphical representation, and the musician should be able to see the spaces generated not only by the computer (from the music) but the spaces edited by the architect.
With this system, a musician would be able to compose real music, send it through the computer to the architect, who receives the digital audio and video files as well as a VRML model. The architect would then be able to manipulate the model into a space which can then be developed further towards the realization of a physical building, or it can be sent back to the musician, the process reversing and the manipulations made by the architect becoming new sounds with which the musician can work.
The intended purpose of a system like this is not to produce great works of music or architecture. The technology is a long way from being perfected for this type of collaboration. Instead, it is to facilitate and encourage architects and musicians to share their skills with each other and produce experimental new spaces, sounds and ideas that can then be used as inspiration in their work.
4 Literature Review
In Navigation via Continuously Adapted Music , the authors describe a system of musical navigation where information about location and space are given to the user through musical cues. They believe using music as a navigational guide to location is a promising approach to tracking mobility.
Three dimensional music interfaces are the key components to Sutoolz 1.0 Alpha: 3D Software Music Interface.  The user controls music by navigating through virtual architectural spaces. Sound becomes a visual experience as the user creates an auditory performance with visual elements.
The authors of Musical Creativity in Collaborative Virtual Environments  review several virtual projects that use collaboration in musical creativity. They investigate what collaborative virtual music looks and sounds like, and try to bring awareness to the lack of music related research and projects within collaborative virtual environments.
In Multisensory Musical Entertainment Systems , the authors explain two visual musical programs. One supports collaborative musical composition. In Bubble Bumble, the users pop virtual bubbles which contain music inside of them, and then place the music on a virtual timeline. In The Magic Music Desk, the emphasis is on visual 3D sound, where sound is linked to virtual 3D objects.
Rethinking the design of traditional acoustic instruments, and questioning why digital instruments (such as the keyboard compared to the piano) mimic their acoustic counterparts, is the basis of the design described in Augmented Reality Interface for Electronic Music Performance.  The authors created a new digital instrument that is not a technological copy of any existing acoustic instrument. The users create sound by moving digital objects, because the authors want to investigate the way in which music is played, to match the advances in the way music is made.
PODIUM is a virtual environment which facilitates collaborative musical performance, as described in Incorporating Co-presence in Distributed Virtual Music Environment . The authors looked the quality and effectiveness of co-presence in their environment for several people to share in musical creativity.
In What You See Is What You Get: On Visualizing Music,  the author discusses visualization techniques used for music discussion. He discusses Music Information Retrieval (MIR) and the ways in which music is represented visually. There is information gained or lost depending on the visualization style or technique.
With the aid of technology, architects and musicians have generated impressive new works. Because the two fields are closely related, a collaborative design system linking the two can facilitate an environment for shared exploration of the shape of sound and the sound of space and has the potential to allow architects and musicians to produce innovative new techniques and designs that would not be possible without this technology.
Tangled Web | Amy Gharst | Faculty of Architecture, Design and Planning, University of Sydney
This paper reviews the ideas of Irit Rogoff and her influence on the fields of digital design.
Theorist Irit Rogoff is known for her critical thought on the subject of geographies. A map is a visual device that has within it information chosen by someone to be displayed to another. What Irit Rogoff questions is the information that is not being displayed. She dissects the mapping process with the purpose of finding information lost, either intentionally or otherwise, that can be reconstructed to define the identities from the resulting alternative geographies. These concepts of identity and mapping are prevalent in the fields of design computing and digital media, where projects are often about the idea of an alternative map (or environment) to what is already known and how it relates to the people using the space.
2. IRIT ROGOFF
2.1 On The Geographies of Humanity
History tells the story of displacement. All things eventually fall victim to the never ending struggle for power, where an elite few make decisions for the trudging masses. The geographies of nations change, but the story remains the same. Empires grow and empires collapse, but the real story of humanity, “the actual business of identity construction gets done in the cracks which mark repeated displacements, the inter-discursive spaces.”  It is in questioning and exploring how society chooses to represent these identities, whether they are erased, memorialized or actively remembered, that Irit Rogoff has devoted her attention.
Belonging - and with that, the question of identity - is a recurrent subject in Rogoff’s field of thought. To belong is to be safe and, to some degree, be complacent with the events shaping society.  It is to have an identity recognized by a greater entity (for instance, the State) that has the influence (according to its own standards) to identify the individual. But what Rogoff explores is how identities are represented and, more importantly, what is being lost in these representations. It is the unseen story behind the representations of the ‘unbelonging’ where “the very condition of critical theoretical activity”  begins and where layers such as journey, significance and revolution can be found.
The displaced victims of history, after suffering the injustices brought about by struggles for space, ideology and power, are often later memorialized in stories or visual exhibits by a need for the guilty to ease their collective conscience. But “the culmination of violent historical acts and the aftermath of void demand to be…compensated for”  and the cartographies that display historical events as representing change, of loss for one side and gain for the other, become a means of ‘staging memory’ , of justifying past behaviors or settling past wrongs. What is overlooked is an “alternative set of relations between subjects and places”  that the State is unable to determine. An honest representation is one that acknowledges that a loss for one is a loss for both, and to compensate for those losses is to recognize “contemporary parallels  and take an active responsibility for them.
In addition to the question of belonging, Rogoff also investigates the struggle between power and liberation and the consequent fragmentation of space. War has no boundaries; it “is de-territorialized”,  and for generations the effects of war are inadvertently displaced geographies apart.  What begins in one place seeps into another, and a ‘relational geography’ is formed that the State, “with its illusions of being seamlessly bound from the outside and solidly coherent from the inside” , is not capable of mapping.
Visual geographies should convey more than a calculated assignment of information. Where geography asks “who has the power and authority to name”, “critical activity which…pursues an active form of unnaming, renaming and the revising of such power structures”  exposes an ‘otherness’ behind the many different layers of information and erasure (deliberate non-information). This ‘otherness’ is what instigates Rogoff in her field of thought.
2.2 And The Questioning of the Given
Using the definition that digital media is “formed by the braided interplay of technical invention and cultural expression” , it becomes evident where Irit Rogoff’s ideas on the entwined nature of people and the representation (or expression) of resultant geographies are influential to the field of digital media (and with it, design computing.)
Digital designers often create unique uses for existing technologies by rethinking not only the given properties of those technologies but also the way people move within and react to their environments (the interaction between person and place or person and object). This is similar to Rogoff’s ideas about finding meanings and stories that are not at first evident or that have been removed (either intentionally or mistakenly). Critical thinking, or developing a ‘counter viewing position’  that encourages curiosity, allows for new dimensions of digital design.
The erosion of the conventional notion of how location represents a clearly defined position  relates to the locative arts model of “considering geographical space to be a canvas” . Cartography is not static. It is active and malleable, as are the people and events that give geography a visual form. As such, locative arts have become a significant element in digital design.
It is also necessary to take into consideration that maps made to express or represent information are not always literal geographical maps. Within the different digital design fields, this openness to the concept of what a “map” looks like allows for not only more creativity, but for more of an opportunity to truly convey information that has otherwise been absent or distorted.
3. WORKS REVIEW
3.1 World Skin
World Skin is a virtual environment created by Jean-Baptiste Barriere and Maurice Benayoun that immerses a person into a landscape of war. The visitor acts as a reporter and uses a camera to take pictures that would (hypothetically) be relayed back to distant places. When a picture is taken, the subject of the image captured turns to a shadow and eventually fades to white, representing the effect of audience desensitization to the realities of war.
Rogoff’s statement that “war is deterritorialised”  is nearly the same as one made by Barriere and Benayoun, “the land of war has no borders.”  More than simply meaning war can and does happen everywhere, these statements are also referring to the way war affects people who may not be directly involved. As Rogoff discussed, traces of war can be found generations apart or in locations far away.  This idea that other times and distant places are exposed to the consequences of war is applied in World Skin by the gradual erasing of the environment to symbolize “war reportage’s effect of…relieving media consumers of the duty to take action.”  The gradual disappearance of the world, both as it slowly fades to white and as the different pieces of the world are captured (beginning the process of erasure) at different times, gives the impression of a multi-generational experience, while the media consumers are being influenced by the war from distant locations.
Although the location shift in World Skin reflects a specifically negative geographical event, the very environment that simulates the stages of erasure as a direct consequence of user action is also a form of “active engagement rather than pious genuflection.”  Instead of turning the horrors of war into some artistically representative space that asks the viewer to associate displaced victims of history with the properties of the space (something Rogoff criticizes for its apathetic nature while there are “contemporary parallels” in need of attention)  World Skin aims to commemorate the lost and ruined lives by turning the passive spectator into an active participant. 
Another aspect of World Skin that incorporates Rogoff’s ideas would be the displaced person in the environment. The user is a person stepping into an interactive warlike environment from what can be assumed is a safe office, lab or studio environment. This user is instantly displaced from his or her comfortable civilian life into a chaotic war zone where he or she does not belong. Given the liberty to move through the space with the task of taking pictures, what this person does, how he or she moves through the space, is going to develop a new identity during these moments not only for the displaced user but for the World Skin environment as well.
World Skin’s most powerful aspect is that it uses an immersive space complete with images, audio and user participation to give the user an experience that Rogoff would commend for its responsible way of remembering the victims of history.
3.2 They Rule
They Rule was created by Josh On as a tool to inform users of the interconnectedness of American corporations and the US Government. It is a website that allows the user to choose from a list of corporations and board members then see the links that connect them. Corporations are represented by table icons, while board members are little men (the more corporations they are involved with, the larger the man’s waist.) As the user chooses companies and people, a map of the relationships between American corporations is made, thus creating a visual cartography of corporatism.
“To recognize some of the present culture’s thorniest problems from yet another angle”  is why the field of visual culture is important to the process of mapping. Rogoff has chosen to work in geography because she wants to find alternative information (relations) between subjects and places.  They Rule is not a traditional “map” – in fact, it starts off as a blank space. How the user compiles the information and the connections can affect the way the information is processed by a viewer. If the map is cluttered and disorganized, it will give off a sense of anarchy, and if it is structured and rigid, a sense of (power) control (see Figure 2).
The aesthetics of the map is entirely up to the user’s preference and creativity. The mapmaker on They Rule becomes the powerful entity (i.e. the State) that decides what information is to be included or hidden from view in the making of their map. They Rule only has a limited type of information to be displayed, but the user can certainly manipulate that information to reflect what they desire. The intended user of They Rule would be an anti-corporatist person who wants to explore the links between corporations, board members and the United States government. But a user with a more pro-corporatism stance could make a map that has an entirely different meaning from their perspective and for those they would have view their map. Perhaps one of the very board members who has a little fat man icon on the site would find it an interesting (and amusing) tool to map his career. After all, “they are proud to rule.” 
An interesting side to this project is that the maps are intended as a tool for the displaced of today’s corporate society (the common person) to study the structure of the wealthy ruling class. Rogoff proposes that we take the “privilege of mapping away from the nation-state where…it had produced one of our most unshakable authorities” and give it to someone else.  What They Rule does is just that. “If any individuals could be seen as having control it is ‘them’, not ‘us’.”  It gives control to the commoners who are oppressed and ruled by corporations and the State, and lets them map out the truth behind their current displacement in a corporatist society with the goal of working towards “a world in which we can honestly say: We rule.” 
By challenging the generally accepted methods of information gathering and representation, Rogoff and digital designers who share her ideas are reinventing the cartographic process and discovering identities and truths that might have always been available but without proper representation. With new light being shed on valuable information, the mapmaker will be able to give the displaced a voice, and given and accepted ideas or “truths” will be challenged. When these things take place, that which “belongs” might find itself displaced, and a new search for identity begins again. Displacement will tell the story of history.