Final project proposal

Continuing with Alex Fast on our mixed reality quest to augment LPs, we will pursue new features not yet explored in our midterm and refine things like API integration and video surfaces that are already key ingredients.  

We have discussed putting this project into the ITP Winter Show, presented as a sort of AR record store pop-up shop, so we'll need to augment a handful of records.  So we'll add to the list of albums and pursue refining the Radiohead and Bowie records that we began with.

Big New Things: we want to integrate 3D objects to provide contextual information in more spatial ways, as well as implement some element of the Google Resonance spatial audio SDK.

Augmented Reality + NYC Open Data API integration

I'd like to make a project called "NYC - What Went Wrong."  Incorporating complaint data from NYC's Open Data set, I want to make an experience that allows users to walk around a neighborhood and see the story of the place through the gripes of it's residents.

Not all citizens feel comfortable lodging official complaints with the city, and their silence tells almost as descriptive a story as the places where residents are comfortable being more vocal.  But the complaints that do exist in the data set can help us trace back the changes and activity in a neighborhood, which is a story well suited to being told while actually walking around the neighborhood.

Midterm - Kid AR, Augmenting the Record

Alex Fast and I continued our exploration of augmenting the physical object of the vinyl record.  This time we chose Radiohead's Kid A, the 2000 release that showcased the first instance of the band's now famously clever use of social media promotion.

To promote the record, Radiohead released a series of animation videos they dubbed "blips" which featured glimpses of the forthcoming songs.  Alex and I repurposed these videos to augment the initial experience of holding the cover in your hands.  The mountains of the cover art appear to be replaced and begin to shift as red dots enter.  Then a TV set playing more of the "blips" lowers into the frame and we begin to see two spheres floating out of the TV set towards the users' face.  This element of spatializing the experience at the outset is a key way of making our ultimate goal - to entertainingly provide extra context and information - to playfully engage the user experience.

Further documentation video is forthcoming, but as we showed during the in-class live demo of our Android app, the user opens the gatefold record to find that the artwork works as a image target that triggers the integrated music api to display real time data on the number of Kid A listeners, album summary, and related artists.

The back cover is also an image target which attempts to emulate the physical experience of flipping through records in a bin at record store by using virtual buttons to move through a slideshow of other record covers which the band has sited in interviews as the central reference points during the making of the album.

It would be great to continue with this project and continue to make spatialized elements and sounds become a part of the experience, perhaps by placing objects around the room that could more fully immerse the user in the experience of literally 'unpacking' the record.

calendAR - Week 7

We like to say that time is our most precious commodity.  And calendars become highly personal documents that reflect all sorts choices and responsibilities about how we live our lives.  As a graduate student, I'm in a constant state of juggling large and small projects, with looming timelines and weekly assignments that quite literally feel like hurdles.  Because of this context, more than I ever I think about time spatially, and the 7-day or Monthly grid of the calendars I use to stay organized are frustratingly confined to an arbitrary way to chart our lives.

So this piece is about trying to incorporate the idea for a calendar into the notebook that I use to take notes in, augmenting the tangible object that I use everyday in school to create a super-notebook.  

I had some trouble building for iOS after my last iOS update now apparently does not jive with my Xcode.  I also wasn't able to harness the raycasting necessary to make this as musical and aural as I intend.  I think as a proof of concept though it does a nice job of using a little tune I made (playfully featuring groups of children cheering) to create a fun, spatial, and linear caledar - exactly the kind of calendar I want.

AR Bowie (week 4, "tell a story")

AR Bowie.  inspiration...loving Bowie, Record collections, the Rauschenberg retroospective at MoMA that had an audio guide with lots of explanations and anecdotes by Rauschenberg himself.

The idea here is of course to fill in the context surrounding an historic album, with footage from the time in which the music was born and performed, and as well from the position of retrospect.

I look forward to working more on this after having been mulling this idea over the summer.  I see great potential in this sort of product in the world.  My partner Alex Fast is eager to keep pursuing it as well.

play MirroR

Expressing our creativity allows us to see the different sides of ourselves.  What if playing music allowed us to literally look into a mirror?  What if the mirror could capture different perspectives of us and different moments in time based on when we made sound? 

This week, in response to Rui’s prompt to make a piece of interactive projection that takes some physical input from the user, I created a piece called “play MirroR.”  This piece aspires to a dual purpose: turn the xylophone into both the playful, interactive controller and also the user’s mirror. 

A MAX patch listens to the the SM-7 microphone and detects which note of the xylophone you are striking.  For as long as that note rings out above a certain volume threshold the patch will allow that corresponding vertical slice of the laptop’s camera feed to be revealed.  This output is then run through MadMapper so that the output from Max is neatly contained to just the various rectangles of the xylophone’s 9 pads. 

Week 1 - A gif to represent yourself

I produced a gif of my grandfather's old Shure SM-55 microphone from the original Disney studios.  He worked there as an animator and director for years and ended up with this great piece of gear, which I luckily inherited.  As a musician and storyteller it holds a lot of significance and value for me, and I was happy to dangle it from the ceiling and give it a spin in front of the camera.