[Leap Motion x Music] Interactive Media for Live Music Events (Thesis, Vassar '16)

Position: Director] Producer, Researcher, Developer
Topic: Interactive Installation, Social power dynamic, Playful technology
Venue: Vassar College
Type: Media Studies Major Thesis Project, Class of 2016
Showcase: Vassar College Main Building
Tools: Unity Engine, Leap Motion, ์œ ๋‹ˆํ‹ฐ, ๋ฆฝ๋ชจ์…˜
Date: 2016. 05. 09

An interactive media installation demo for live music events. It experiments multiple audience and artists interacting with each other and create collaborative soundscape.

Click Here for Thesis Paper (English)

Pop-up Showcase of Media Thesis Project @ Vassar College Campus (๋‰ด์š• ๋ฐ”์‹ธ๋Œ€ํ•™๊ต ๋ฏธ๋””์–ด ์ „๊ณต ์กธ์—… ๋…ผ๋ฌธ ํ”„๋กœ์ ํŠธ ํŒ์—… ์‡ผ์ผ€์ด์Šค ์˜์ƒ)

Short Description

โ€œInterplayingโ€ is an interactive digital installation to encourage audience memberโ€™s live social interaction with one another in live music events โ€“ concerts/festivals. By allowing the audience to virtually mash up the music with the musician and other audience members, โ€œinterplayingโ€ demo aims to show the potential of technology building peopleโ€™s โ€˜socialโ€™ capital.

Project Summary

In the 21st century, there is a strong trend of the indivdual audience's personal technology-dependent behavior (constantly checking a cellphone, trying to record everything via camera) in live music events.

This project investigates the way to encourage the audienceโ€™s human-to-human interaction in live music events environment. It hopes to create better experience of socialization via technology, not by removing technology.

It found a lesson from organic and physical human-to-human interaction in cultural/community festivals. Also, it does not criticize the presence of technology itself in the music events. Interplaying experiments if technology can help and encourage live social interaction amongst audience members, bringing people together in real world.

How It Works

  1. Put your hands on top of Leap Motion device
  2. Try to touch each red cube, which is similar to DJ LaunchPad button
  3. Each cube has a stem file extracted from the music going on in the live music events
  4. Play it with other people around you
  5. Feel free to collaborate and create the sound together
  6. Feel free to share eye contact, physical interaction, smile with your co-players
thesis20
Main Image
thesis6
May 2016, Interplaying, Media Studies senior thesis project showcase, Vassar College Main Building, New York.
thesis8
thesis7
DemoDay hearstTower
Demo Day (May 13) at Hearst Tower, New York City
thesis1
Facebook Event for On-Campus Showcase
thesis3
Facebook Event for On-Campus Showcase
thesis4
Facebook Event for On-Campus Showcase

thesis9

Thesis Presentation Preview

thesis11
thesis12
thesis13
thesis14
thesis15
thesis16
thesis17
thesis18
thesis19
thesis10

Background

While exploring the technology in live music concerts and music festivals, I found these interesting images - sarcastically commenting on how audience members are tuning into their own personal devices, like digital cameras or cellphone, while being surrounded by hundreds of people around them. (Of course, we would need a drone to support our WiFi-s and maybe a 30-inch LCD screen for better view).

And due to this device-attached behaviors, we are missing somethingโ€”face-to-face interaction, language and flesh. There are already many concerns about our social capitals negatively influenced by technology in sociology, psychology, and many other fields (especially, Sherry Turkle). They provided me new critical perspectives and inspiration to think about meaningful interactions for human beings.

While focusing on the interactions in live music events, I also searched local events, regional or cultural festivals. Why? Although they might have completely different target audience, music style, set up location, they are so full of organic human interaction between the audience members. I decoded what makes this possible in cultural festivals is that they have shared medium.

Coming back to live music events, I thought there should be a reason why people visit public live music events instead of just listening to music at home, live-streaming Spotify or Apple music. Similar to how regional/cultural festivals create humane interactions amongst diverse participants, socialization is one of the main reasons why people still visit live music events.

Socialization is an important part of the live music event experience. It has shaped the history of how shared medium successfully encouraged social interaction in festivals. This is why I created the shared medium (using hand motion detector and gaming program) to generate communal activity that the audiences can play with one another.

There were two big questions that came along with the usage of technology. First, why not get rid of it? Commercially and industrially, it is realistically hard to completely stop using it. Easier said than done. Second, technology does not inherently isolate people. As the quotes imply (3rd to the last photo above), it is how we use the technology. Thus, I created the interactive media installation that would bring people together in real life, initiating eye contacts, short conversations, even small physical contacts.

๋ฐฐ๊ฒฝ

์ธํ„ฐํ”Œ๋ ˆ์ž‰์€ ๋ผ์ด๋ธŒ ์Œ์•… ๊ณต์—ฐ(์ฝ˜์„œํŠธ, ํŽ˜์Šคํ‹ฐ๋ฒŒ)์—์„œ 21์„ธ๊ธฐ์˜ ๊ด€๊ฐ์ด ํ•ธ๋“œํฐ์ด๋‚˜ ์นด๋ฉ”๋ผ์™€ ๊ฐ™์€ ๊ฐœ์ธ์˜ ๋””์ง€ํ„ธ ๊ธฐ๊ธฐ์—๋งŒ ์ง‘์ฐฉํ•˜๋Š” ํ˜„์ƒ์„ ํƒ€ํŒŒํ•˜๊ณ  ์„œ๋กœ ๋‹ค๋ฅธ ๊ด€๊ฐ๊ณผ ๋ผ์ด๋ธŒ ํ˜„์žฅ์„ ์ฆ๊ธธ ์ˆ˜ ์žˆ๋Š” ๊ฒฝํ—˜์„ ์„ ์‚ฌํ•˜๊ธฐ ์œ„ํ•œ ํ”„๋กœ์ ํŠธ์ž…๋‹ˆ๋‹ค.

(์„œ๋กœ ์•Œ์ง€ ๋ชปํ•˜๋Š”) ์ฐธ์—ฌ์ž ๊ฐ„์˜ ์ž์—ฐ์Šค๋Ÿฌ์šด ๊ต๋ฅ˜๊ฐ€ ํ™œ๋ฐœํ•˜๊ฒŒ ์ด๋ฃจ์–ด์ง€๋Š” ์ง€์—ญ ์ถ•์ œ๋“ค์—์„œ ๋‹ค์–‘ํ•œ ์˜๊ฐ์„ ๋ฐ›์•˜์Šต๋‹ˆ๋‹ค. ์„ฑ๊ณต์ ์ธ ์ง€์—ญ ์ถ•์ œ๋“ค์€ ๋ชจ๋‘๊ฐ€ ๊ณต์œ ํ•˜๋Š” ํ•˜๋‚˜์˜ ๋ฏธ๋””์—„, ๋งค์ฒด๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ ์•„์ด๋””์–ด๋ฅผ ๋ฐ”ํƒ•์œผ๋กœ ๋ผ์ด๋ธŒ ์Œ์•… ์ด๋ฒคํŠธ/๊ณต์—ฐ/์ถ•์ œ์—์„œ๋„ ์‚ฌ๋žŒ๋“ค์ด ์ง์ ‘ ๊ฐ€์ง€๊ณ  ๋†€ ์ˆ˜ ์žˆ๋Š” ๋งค๊ฐœ์ฒด๊ฐ€ ์žˆ์–ด์•ผ ํ•œ๋‹ค๋Š” ์˜๊ฒฌ์„ ์ œ์‹œํ•˜๊ณ  ์Œ์•…๊ณผ ๊ธฐ์ˆ ์„ ์ด์šฉํ•œ ๋ฏธ๋””์–ด ์•„ํŠธ ์ž‘ํ’ˆ์„ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค.

๋ณธ ๋…ผ๋ฌธ ํ”„๋กœ์ ํŠธ๋Š” ๊ธฐ์ˆ ์„ ๋ฐฐ์ œํ•จ์œผ๋กœ์จ ํ˜„์žฅ์— ๋Œ€ํ•œ ์ง‘์ค‘๋„์™€ ๋ผ์ด๋ธŒ ์Œ์•… ํ˜„์žฅ์—์„œ์˜ ์†Œ์…œ๋ผ์ด์ง•์„ ์œ ๋„ํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹™๋‹ˆ๋‹ค. ์—ญ์œผ๋กœ, ๊ธฐ์ˆ ์„ ํ™œ์šฉํ•ด์„œ ์‚ฌ๋žŒ๋“ค์˜ ๊ณต๋™ ํ™œ๋™์„ ์œ ๋„ํ•˜๋Š” ๋งค์ฒด๋ฅผ ๋งŒ๋“ ๋‹ค๋ฉด ๊ด€๊ฐ๊ณผ ๊ด€๊ฐ ๊ฐ„์˜ ์ธ๊ฐ„์ ์ธ ์†Œํ†ต์„ ์ด๋Œ์–ด๋‚ผ ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฐ€๋Šฅ์„ฑ์„ ์ œ์‹œํ•ฉ๋‹ˆ๋‹ค.

์ž‘ํ’ˆ ์„ค๋ช… ๋ฐ ๊ตฌ๋™ ๋ฐฉ๋ฒ•

๋ณธ ์„ค์น˜ ๋ฏธ๋””์–ด ์ž‘ํ’ˆ์€ DJ๋“ค์ด ์“ฐ๋Š” Launchpad์˜ ํ˜•ํƒœ์—์„œ ์˜๊ฐ์„ ๋ฐ›์•„ ๋ฒ„ํŠผ๊ณผ ๊ฐ™์€ ๋น„์ฅฌ์–ผ๋กœ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ์†๋™์ž‘ ๊ฐ์ง€ ํ•˜๋“œ์›จ์–ด์ธ Leap Motion (๋ฆฝ ๋ชจ์…˜) ์œ„์— ์†์„ ์˜ฌ๋ฆฌ๋ฉด, ๋ณธ์ธ์˜ ์†์„ ํ™”๋ฉด์—์„œ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์†์„ ์›€์ง์—ฌ ํ™”๋ฉด ์ƒ์˜ ํ๋ธŒ๋ฅผ ๋ฒ„ํŠผ์ฒ˜๋Ÿผ ์ƒ๊ฐํ•˜๊ณ  ํ„ฐ์น˜ํ•˜๋ฉฐ ์‚ฌ์šด๋“œ๊ฐ€ ์žฌ์ƒ๋ฉ๋‹ˆ๋‹ค.

๋ณธ ๋ฐ๋ชจ๋Š” ๋ฎค์ง€์…˜์ด ๋ผ์ด๋ธŒ ๊ณต์—ฐ์„ ์ง„ํ–‰ํ•˜๊ณ  ์žˆ๋‹ค๋Š” ๊ฐ€์ • ํ•˜์—, ํ˜๋Ÿฌ๋‚˜์˜ค๊ณ  ์žˆ๋Š” ์Œ์•…์—์„œ ์ง์ ‘ ์“ฐ์ด๋Š” ํ•˜๋ชจ๋‹ˆ ๋ฐ ํ•„๋Ÿฌ (3~5์ดˆ)๋ฅผ ๊ฐ ๋ฒ„ํŠผ์— ์‚ฝ์ž…ํ–ˆ์Šต๋‹ˆ๋‹ค. ํ„ฐ์น˜ํ•  ๋•Œ๋งˆ๋‹ค ๋ฒ„ํŠผ์€ ๋…ธ๋ž€์ƒ‰์œผ๋กœ ๋ฐ”๋€Œ์–ด ์žฌ์ƒ๋˜๊ณ  ์žˆ์Œ์„ ์•Œ๋ฆฌ๊ณ  ํ•ด๋‹น ์‚ฌ์šด๋“œ๊ฐ€ ๋๋‚˜๋ฉด ๋‹ค์‹œ ๋นจ๊ฐ„์ƒ‰(๋ณธ๋ž˜ ์ƒ‰๊น”)์œผ๋กœ ๋Œ์•„์˜ต๋‹ˆ๋‹ค.

์ด ๋ฏธ๋””์–ด ์•„ํŠธ๋Š” ๊ธฐ์ˆ ์„ ์ด์šฉํ•ด ์‚ฌ๋žŒ๊ณผ ์‚ฌ๋žŒ ๊ฐ„์˜ ์ธํ„ฐ๋ ‰์…˜์„ ์ฆํญ์‹œํ‚ค๊ธฐ ์œ„ํ•œ ๊ฒƒ์ด ๋ชฉ์ ์ธ๋งŒํผ ์—ฌ๋Ÿฌ ๋ช…์ด ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒƒ์ด ๊ฐ€์žฅ ์ด์ƒ์ ์ž…๋‹ˆ๋‹ค. ๋‹ค๋งŒ ์ด๋ฒˆ ํ”„๋กœ์ ํŠธ์—์„œ๋Š” ์žฌ์ • ์ง€์›์˜ ๋ฌธ์ œ๋กœ ๋ฆฝ ๋ชจ์…˜ ํ•˜๋“œ์›จ์–ด๋ฅผ 1๊ฐœ๋งŒ ์ง€์›๋ฐ›์•˜๊ณ , 1๊ฐœ์˜ ๋ฆฝ ๋ชจ์…˜์€ 4๊ฐœ์˜ ์† ์ด์ƒ์„ ๊ฐ์ง€ํ•  ์ˆ˜ ์—†๋Š” ๊ด€๊ณ„๋กœ 2~3๋ช…์˜ ์ฐธ์—ฌ์ž๊ฐ€ ์ตœ๋Œ€์˜€์Šต๋‹ˆ๋‹ค.

๊ทธ๋ž˜๋„ ์„œ๋กœ ๋‹ค๋ฅธ ์‚ฌ๋žŒ๋“ค์ด (๋ชจ๋ฅด๋Š” ์‚ฌ์ด์—ฌ๋„) ์ด๋Ÿฐ ๋งค์ฒด๋ฅผ ํ†ตํ•ด ๊ฐ™์ด ๋†€ ์ˆ˜ ์žˆ๋Š” ํ™˜๊ฒฝ์ด ์ œ๊ณต๋˜์–ด ๋Œ€ํ™”๋‚˜ ๋ˆˆ๋น› ๊ตํ™˜ ๋“ฑ ์ธ๊ฐ„์ ์ธ ๊ต๋ฅ˜๊ฐ€ ์˜ค๊ณ  ๊ฐ”๋‹ค๋Š” ์˜์˜๋ฅผ ๋‚จ๊ธฐ๋Š” ํ”„๋กœ์ ํŠธ ์‡ผ์ผ€์ด์Šค์˜€์Šต๋‹ˆ๋‹ค.

Credits

Partially inspired by Sherry Turkle, a professor of Social Studies of Science and Technology at MIT, who wrote multiple books about the relationship between humans and computers/technological devices. Two of her books, Alone Together and Reclaiming Conversation: The Power of Talk in the Digital Age, were my go-to-books during my thesis period.

Special Thanks to Vassar College Media Studies department & TimeWaveFest folks & Unity & Leap Motion

Thanks to everyone who supported me. I would be glad to see this project being useful to any hacker/maker projects.