[AR] MIT media lab: VR/AR Hackathon with Hololens

Position: [Director] Team Lead, Producer, Developer
Topic: AR / VR/ XR, Playful Technology
Team: Augmented Beatz (Randall Spence, Tom Strissel)
Venue: MIT Media Lab, 3-days Reality Virtually Hackathon 2016 - VR/AR
Tools: Unity Engine ์œ ๋‹ˆํ‹ฐ, Hololens ๋งˆ์ดํฌ๋กœ์†Œํ”„ํŠธ ํ™€๋กœ๋ Œ์ฆˆ
Date: October 2016

Hololens App demo of XR DJ-ing application that imagines a AR/MR DJ equipment one can bring anywhere.


mit4
Hackathon participant enjoying our demo
mit5
Testing assets of Hololens
mit6
One of the judges enjoying our Hololens DJ
mit7
Screenshot of making process
mit8
Hololens workshop by Microsoft
mit9
Workshop by IDEO
mit10
Workshop at the hackathon

Hackathon in MIT

In October 2016, I traveled to Boston from New York to attend 'Reality, Virtually, Hackathon' hosted by MIT Media Lab with numerous sponsors. It was 4 days events including workshops, hacking, and demo day.

I met two other developers at the event and formed a team. We created the DJ mixer set in Hololens (augmented reality) platform. We used gaze control/gesture control/audio trigger/color change for User Experience.

  • Simplified design of DJ mixer; 2 circles are main soundtracks & 5 square buttons are sound fillers.
  • First version: changes into skyblue when clicked, back to white when sound file ends.
  • Second version: changes into blue when gazed, red when clicked, back to green when sound file ends.

It is often useful to set a goal in your mind when you attend a hackathon, which is a short-term/hyper-stressed/unpredictable event. I set it up as 'Let's try something new that I will not be able to easily play around with', and thus, I chose Microsoft Hololens, augmented reality device with the latest update.

DJ-ing in Augmented Reality

As I wanted to continue my experience of using music with VR/AR programs, I pitched my idea of 'DJ-ing in AR'. My initial idea was that if DJ/musicians are able to play mobile and ubiquitous, more street music shows and outdoor gigs can prosper, and hopefully such music x tech application may increase the communal musical experience in public spaces.

Click here for more info about 'Augmented Beatz' project or Github Repo

Luckily, few people contacted me with my pitch and with a slight change in members, I ended up with 3 junior developers. They never had Unity experience, which is crucial and helpful for any VR/AR hackathon. So I was half-teaching them how Unity front/back-end is constructed and communicate with one another. Then, we broke down our tasksโ€”modelling, rendering, Unity x Hololens scripting for gaze control, finger pinch control, and etc.

It was overall a lot of fun and trying out new gears/tech skills. Since Hololens is extremely expensive, this event was a great opportunity for me to hack it and build something. Microsoft not only has quite a number of hololens built-in apps and games, but also has an incubator program which seems to produce AR-related tech/products. If you're getting indifferent with VR, AR would be good to look into.

Hackathon Side-notes

  • Great workshops on Friday, 24 hours before the actual hackathon starts
  • My favorites were UI UX, Samsung Web VR, Msft Hololens workshops.
  • 3D Modelling, Unity (C# or javascript), Design UI UX are all great handy skills
  • Winner contents: Kids Education/Game x VR, Pharmaceutical x AR, Dynamic Game/Music Activity x VR, Psychological (social anxiety) x AR
  • Hololens has a very limited gesture controls, so something too dynamic wouldn't be ideal for now. You can only pinch, drag, and zoom with your fingers.
  • Unity and Hololens teams are working on better synchronization and building more documentation and reference materials. Currently, two platforms still have some conflicting scripts and functions that make harder to hack.

MIT media lab์ด ์ฃผ์ตœํ•œ ํ•ด์ปคํ†ค์—์„œ ๋งˆ์ดํฌ๋กœ์†Œํ”„ํŠธ ํ™€๋กœ๋ Œ์ฆˆ ๊ธฐ๊ธฐ๋ฅผ ๋Œ€์—ฌํ•ด์„œ ์ฆ๊ฐ•ํ˜„์‹ค (AR) ํ”Œ๋žซํผ์— ํ”„๋กœ์ ํŠธ๋ฅผ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ์Œ์•… ๊ณต์—ฐ์ด ๋” ๋‹ค์–‘ํ•œ ๊ณณ์—์„œ ์ด๋ค„์ง„๋‹ค๋ฉด ์–ด๋–จ๊นŒ๋ผ๋Š” ์•„์ด๋””์–ด๋ฅผ ๊ธฐํšํ•ด์„œ 3๋ช…์˜ ๊ฐœ๋ฐœ์ž๋ฅผ ๋ชจ์•„ AR ๋””์ œ์ž‰ ์„ธํŠธ๋ฅผ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ์ด ํ”„๋กœ๊ทธ๋žจ์ด ์ œ๋Œ€๋กœ ๊ฐœ๋ฐœ๋œ๋‹ค๋ฉด DJ ์žฅ๋น„๋ฅผ ํ’€๋กœ ๊ฐ€์ง€๊ณ  ๋‹ค๋‹ˆ์ง€ ์•Š์•„๋„ ํ™€๋กœ๋ Œ์ฆˆ ํ•˜๋‚˜๋กœ ๋” ์œ ๋™์ ์œผ๋กœ ์Œ์•…์„ ์—ฐ์ฃผํ•˜๊ณ  ๊ด€๊ฐ๋“ค๊ณผ ๋งŒ๋‚  ์ˆ˜ ์žˆ์ง€ ์•Š์„๊นŒ์š”?

์ด ์˜์ƒ์€ ์ €ํฌ ํ”„๋กœ์ ํŠธ๋ฅผ ์œ„์ฃผ๋กœ ํŽธ์ง‘ํ•œ ์งง์€ ๋ฐ๋ชจ ์˜์ƒ์ž…๋‹ˆ๋‹ค. 3์ผ๋™์•ˆ ํ”„๋กœํ† ํƒ€์ž…์„ ์—ฐ๊ตฌํ•ด์„œ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ํ™€๋กœ๋ Œ์ฆˆ๊ฐ€ ํ˜„์žฌ ๊ธฐ์ˆ ์ ์œผ๋กœ ๋ถ€์กฑํ•œ ๋ถ€๋ถ„๋“ค์ด ์žˆ์–ด ์†์œผ๋กœ ์ž์œ ๋กญ๊ฒŒ ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ์กฐ์ข…ํ•˜๋Š” ๋ฐ์— ํ•œ๊ณ„๊ฐ€ ์žˆ์—ˆ์ง€๋งŒ, ์‹œ์•ผ ์ปจํŠธ๋กค, ์†๊ฐ€๋ฝ ์ปจํŠธ๋กค๋งŒ์„ ์ด์šฉํ•ด ๋””์ œ์ž‰ ์„ธํŠธ๋ฅผ ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ๊ฐ€์ƒํ˜„์‹ค์ด ์•„๋‹Œ ์ฆ๊ฐ•ํ˜„์‹ค๋กœ ์ด ํ”„๋กœ์ ํŠธ๋ฅผ ๋งŒ๋“  ์ด์œ ๋Š” ๊ฐ€์ƒํ˜„์‹ค์˜ ๊ฒฝ์šฐ VR ํ—ค๋“œ์…‹์„ ์“ฐ๋ฉด ๋ณธ์ธ๋งŒ์ด ์ฆ๊ธธ ์ˆ˜ ์žˆ๋Š” ์ปจํ…์ธ ๊ฐ€ ๋˜์ง€๋งŒ, AR๋กœ ํ•œ๋‹ค๋ฉด ๋ฎค์ง€์…˜์ด ๊ด€๊ฐ๋“ค์„ ๊ทธ๋Œ€๋กœ ๋ณผ ์ˆ˜ ์žˆ๊ณ , ์†Œ๋ฆฌ๋„ ๋ธ”๋ฃจํˆฌ์Šค๋ฅผ ์ด์šฉํ•ด ์‰ฝ๊ฒŒ ์Šคํ”ผ์ปค๋กœ ์ „๋‹ฌํ•  ์ˆ˜ ์žˆ๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

MIT ํ•ด์ปคํ†ค์„ ํ†ตํ•ด ๋ฐฐ์šด ๊ฒƒ

  • ํŠธ๋ Œ๋“œ๋ฅผ ์•ž์„œ๊ฐ€๋Š” ๊ธฐ์ˆ ๊ณผ ์—ฐ๊ตฌ์— ๋Œ€ํ•œ ๋ฐœํ‘œ ๋ฐ ์›Œํฌ์ƒต๋“ค
  • VR/AR ์‚ฐ์—…์—์„œ 3d ๋ชจ๋ธ๋ง, Unity (C#), ๋””์ž์ธ UI UX ๊ธฐ์ˆ ์˜ ์ค‘์š”์„ฑ
  • ์šฐ์Šน ์ž‘ํ’ˆ: ์–ด๋ฆฐ์ด ๊ต์œก ๋ฐ ๊ฒŒ์ž„ ์ปจํ…์ธ (VR), ์•ฝ/์˜ํ•™(AR), ๋‹ค์ด๋‚ด๋ฏนํ•œ ๊ฒŒ์ž„/์Œ์•… ํ™œ๋™ (VR), ์‹ฌ๋ฆฌํ•™/์ •์‹ ์‹ฌ๋ฆฌ์น˜๋ฃŒ(AR)

Thanks to ...

๋งˆ์ดํฌ๋กœ์†Œํ”„ํŠธ ๋ณธ์‚ฌ์˜ ์ง์›๋“ค์ด ์•„์ง ๋ถˆ์•ˆ์ •ํ•œ ํ™€๋กœ๋ Œ์ฆˆ์™€ ์œ ๋‹ˆํ‹ฐ ์—”์ง„ ๊ฐ„์˜ API ์—ฐ๋™์—์„œ ๋ฐœ์ƒํ•˜๋Š” ๋ฒ„๊ทธ ํ•ด๊ฒฐ์— ์ ๊ทน์ ์œผ๋กœ ๋„์™€์ฃผ์–ด ๊ฐ์‚ฌํ–ˆ์Šต๋‹ˆ๋‹ค.