[Leap Motion x Music] Interactive Media for Live Music Events (Thesis, Vassar '16)

An interactive media installation demo for live music events. It experiments multiple audience and artists interacting with each other and create collaborative soundscape.

Click Here for Thesis Paper (English)

Pop-up Showcase of Media Thesis Project @ Vassar College Campus (뉴욕 바싸대학교 미디어 전공 졸업 논문 프로젝트 팝업 쇼케이스 영상)

Short Description

“Interplaying” is an interactive digital installation to encourage audience member’s live social interaction with one another in live music events – concerts/festivals. By allowing the audience to virtually mash up the music with the musician and other audience members, “interplaying” demo aims to show the potential of technology building people’s ‘social’ capital.

Project Summary

In the 21st century, there is a strong trend of the indivdual audience's personal technology-dependent behavior (constantly checking a cellphone, trying to record everything via camera) in live music events.

This project investigates the way to encourage the audience’s human-to-human interaction in live music events environment. It hopes to create better experience of socialization via technology, not by removing technology.

It found a lesson from organic and physical human-to-human interaction in cultural/community festivals. Also, it does not criticize the presence of technology itself in the music events. Interplaying experiments if technology can help and encourage live social interaction amongst audience members, bringing people together in real world.

How It Works

1. Put your hands on top of Leap Motion device
2. Try to touch each red cube, which is similar to DJ LaunchPad button
3. Each cube has a stem file extracted from the music going on in the live music events
4. Play it with other people around you
5. Feel free to collaborate and create the sound together
6. Feel free to share eye contact, physical interaction, smile with your co-players

Thesis Presentation Preview

Background

While exploring the technology in live music concerts and music festivals, I found these interesting images - sarcastically commenting on how audience members are tuning into their own personal devices, like digital cameras or cellphone, while being surrounded by hundreds of people around them. (Of course, we would need a drone to support our WiFi-s and maybe a 30-inch LCD screen for better view).

And due to this device-attached behaviors, we are missing something—face-to-face interaction, language and flesh. There are already many concerns about our social capitals negatively influenced by technology in sociology, psychology, and many other fields (especially, Sherry Turkle). They provided me new critical perspectives and inspiration to think about meaningful interactions for human beings.

While focusing on the interactions in live music events, I also searched local events, regional or cultural festivals. Why? Although they might have completely different target audience, music style, set up location, they are so full of organic human interaction between the audience members. I decoded what makes this possible in cultural festivals is that they have shared medium.

Coming back to live music events, I thought there should be a reason why people visit public live music events instead of just listening to music at home, live-streaming Spotify or Apple music. Similar to how regional/cultural festivals create humane interactions amongst diverse participants, socialization is one of the main reasons why people still visit live music events.

Socialization is an important part of the live music event experience. It has shaped the history of how shared medium successfully encouraged social interaction in festivals. This is why I created the shared medium (using hand motion detector and gaming program) to generate communal activity that the audiences can play with one another.

There were two big questions that came along with the usage of technology. First, why not get rid of it? Commercially and industrially, it is realistically hard to completely stop using it. Easier said than done. Second, technology does not inherently isolate people. As the quotes imply (3rd to the last photo above), it is how we use the technology. Thus, I created the interactive media installation that would bring people together in real life, initiating eye contacts, short conversations, even small physical contacts.

배경

인터플레잉은 라이브 음악 공연(콘서트, 페스티벌)에서 21세기의 관객이 핸드폰이나 카메라와 같은 개인의 디지털 기기에만 집착하는 현상을 타파하고 서로 다른 관객과 라이브 현장을 즐길 수 있는 경험을 선사하기 위한 프로젝트입니다.

(서로 알지 못하는) 참여자 간의 자연스러운 교류가 활발하게 이루어지는 지역 축제들에서 다양한 영감을 받았습니다. 성공적인 지역 축제들은 모두가 공유하는 하나의 미디엄, 매체를 가지고 있었습니다. 그 아이디어를 바탕으로 라이브 음악 이벤트/공연/축제에서도 사람들이 직접 가지고 놀 수 있는 매개체가 있어야 한다는 의견을 제시하고 음악과 기술을 이용한 미디어 아트 작품을 만들었습니다.

본 논문 프로젝트는 기술을 배제함으로써 현장에 대한 집중도와 라이브 음악 현장에서의 소셜라이징을 유도하는 것이 아닙니다. 역으로, 기술을 활용해서 사람들의 공동 활동을 유도하는 매체를 만든다면 관객과 관객 간의 인간적인 소통을 이끌어낼 수 있다는 가능성을 제시합니다.

작품 설명 및 구동 방법

본 설치 미디어 작품은 DJ들이 쓰는 Launchpad의 형태에서 영감을 받아 버튼과 같은 비쥬얼로 만들었습니다. 손동작 감지 하드웨어인 Leap Motion (립 모션) 위에 손을 올리면, 본인의 손을 화면에서 볼 수 있습니다. 손을 움직여 화면 상의 큐브를 버튼처럼 생각하고 터치하며 사운드가 재생됩니다.

본 데모는 뮤지션이 라이브 공연을 진행하고 있다는 가정 하에, 흘러나오고 있는 음악에서 직접 쓰이는 하모니 및 필러 (3~5초)를 각 버튼에 삽입했습니다. 터치할 때마다 버튼은 노란색으로 바뀌어 재생되고 있음을 알리고 해당 사운드가 끝나면 다시 빨간색(본래 색깔)으로 돌아옵니다.

이 미디어 아트는 기술을 이용해 사람과 사람 간의 인터렉션을 증폭시키기 위한 것이 목적인만큼 여러 명이 함께 사용할 수 있는 것이 가장 이상적입니다. 다만 이번 프로젝트에서는 재정 지원의 문제로 립 모션 하드웨어를 1개만 지원받았고, 1개의 립 모션은 4개의 손 이상을 감지할 수 없는 관계로 2~3명의 참여자가 최대였습니다.

그래도 서로 다른 사람들이 (모르는 사이여도) 이런 매체를 통해 같이 놀 수 있는 환경이 제공되어 대화나 눈빛 교환 등 인간적인 교류가 오고 갔다는 의의를 남기는 프로젝트 쇼케이스였습니다.

Credits

Partially inspired by Sherry Turkle, a professor of Social Studies of Science and Technology at MIT, who wrote multiple books about the relationship between humans and computers/technological devices. Two of her books, Alone Together and Reclaiming Conversation: The Power of Talk in the Digital Age, were my go-to-books during my thesis period.

Special Thanks to Vassar College Media Studies department & TimeWaveFest folks & Unity & Leap Motion

Thanks to everyone who supported me. I would be glad to see this project being useful to any hacker/maker projects.