Touching Virtual Reality Musical Instruments
by Daniel Schwartz daniel.schwartz630824@tufts.edu
SUMMARy (TLDR)
The incorporation of touch feedback in virtual reality musical instruments opens up lots of questions about how to best apply it. Namely, is it best to be more realistic or more abstract/unrealistic? Either way, it seems clear that VRMIs have the power to create lots of new musical opportunities and shape the future of music.
I love listening to music and keeping up with the music industry. My favorite artists include Ariana Grande, Kacey Musgraves, Beyoncé, Megan Thee Stallion, and more! When it came to this project, I knew that I wanted to connect virtual reality to music, and see how the two intersected. I stumbled upon virtual reality musical instruments and was intrigued by the implications of being able to play an instrument in VR. My first thought is that it might give me the opportunity to learn an instrument since (despite my love for music) I don’t actually know how to play any instruments! Then I started to think bigger, about how virtual reality musical instruments could shape the future of music, contribute to democratizing music education, and offer out-of-this-world musical experiences.
Why I FIND IT COOL
My spotify code, if you want a window into my musical world!
WHAT IS A VR MUSICAL INSTRUMENT?
Virtual reality instruments, or VRMIs for short, are what they sound like: musical instruments that are experienced in virtual reality. (I also broadly consider VRMIs to be VR music-making applications, include VR DJ'ing and music production.) VRMIs can come in many forms, spanning from recreations of real instruments to new, abstract instruments that don't look like anything you'd ever see in an orchestra!
... AND HOW CAN WE TOUCH THEM?
Touch is incorporated into VRMIs using haptic feedback. Haptic feedback is to touching as visual feedback is to seeing. VR is usually defined its visual and auditory aspects, and haptic feedback relatively underexplored in VR. Haptic feedback is super important to real instruments, though. I mean, other than the theremin, can you think of any instrument that you don't have to touch? And as O’Modhrain and Gillespie point out, "It is also no accident that the parts of the body that interact with instruments—lips, fingers, hands—are the most highly populated by haptic receptors.” Haptic feedback can be accmomplished in VRMIs using gloves that apply pressure to the user's hands, controllers, and other methods.
TYPES OF haptic feedback in vrmis
Throughout the course of my research, I developed different categories of haptic feedback found in VRMIs, as follows...
Concrete = tries to be realistic to what real instruments feel like, and it is found in VRMIs that try to replicate real instruments
Abstract = the other broad category, which is essentially just not concrete haptic feedback
True = absrtact haptic feedback that is not related to what the user touches, and is just information communicated through haptic feedback
Hypothetical = abstract haptic feedback that the user feels when touching abstract objects in VRMIs
To better understand these categories, let's consider some example of each. Concrete haptic feedback could be found in a realistic VR piano, which has haptic feedback that is supposed to feel like what a real piano feels like. True abstract haptic feedback could be found in a VRMI that has controllers that vibrate to let the player know when they play the wrong notes. In this case the haptic feedback is being used to communicate information about the instrument that is not directly related to what the user is feeling in the VRMI application. Hypothetical abstract haptic feedback could be found in an abstrtact VRMI that consists of floating orbs that the user swipes their hands through to produce certain notes, and the haptic feedback could be vibrations to simulate what the floating orb could feel like hypothetically.
THE QUESTION: TO ABSTRACT OR NOT TO ABSTRACT?
Considering that VR itself is relatively new, that VRMIs are even newer, and that haptic feedback in VRMIs is even newer than that... there's still a lot of potential for developing haptic feedback in VRMIs! With that in mind, moving forward which direction should developers go: emphasizing concrete haptic feedback or abstract haptic feedback? Concrete haptic feedback seems like it would be the most beneficial for using VRMIs as replacements/alternatives for real instruments. Abstract haptic feedback, though, seems like it has the most promise for creating new instruments and musical experiences that might only be possible in VR. Maybe they should meet somewhere in the middle?
I think this is also indicative of a larger question surrounding VR. That is, what is the best way to take advantage of VR as a medium? Is it to create hyper-realistic experiences or to create the most unreleastic and impossible experiences?
THE relevance of the synthesizer
This kind of question is definitely not new to the history of musical development. As a historical parallel, in the 1960s when electronic synthesizers were being developed, there was a debate over whether it should have a piano-like keyboard interface or not. On the one hand, having the keyboard would make synthesizers more familiar, which makes it easier for composers and performers and helps on the commercial front because people are more likely to buy something that they recognize and that they feel comfortable with. (For my comparison, this would be comparable to the case for concrete haptic feedback.) On the other hand, the keyboard kind of holds the synthesizer back by making it conform with existing instruments and not embracing the newness of it. (And this would be comparable to case against concrete, and for abstract haptic feedback.) As the picture to the left demonstrates, most synthesizers today do have some form of keyboard, as well as other controllers and knobs. Based on this precedent, we might predict that a more concrete form of haptic feedback will prevail in VRMIs... but only time will tell!
THE FUTURE OF THE VRMI
VRMI developers certainly have their work cut out for them when it comes to experimenting with different forms of haptic feedback. I'll certainly be excited to see VRMIs develop, and I foresee lots of cool opportunities being created by VRMIs. For one, VRMIs could make instruments and music more accessible. This is probably most applicable in schools, where it could be more cost effective to have VR sets that contain 10+ VRMIs on them than to buy 10+ real instruments. If this is the goal of VRMIs, then concrete haptic feedback would probably be most useful. It's also worth noting that VR headsets are becoming increasingly accessible, but are still not totally there yet, so it might take a while before this becomes a reality. VRMIs could also become grounds for musical experimentation with all-new, abstract musical instruments that could defy our laws of physics to create sound in cool new ways. Abstract haptic feedback would probably work best in this case, because new abstract instruments would demand new ways of interacting with them. This would also most likely lead to lots of musical innovation. For example, maybe VRMIs will become a new recognized class of instruments that are celebrated through VR concerts. Maybe VR music will become it's own genre. Future music could be played, produced, and streamed excusively in VR. Artists could release VR music videos.
The possibilties are both endless and exciting, so be on the lookout for VRMIs in the future! And don't forget to pay attention to the haptic feedback (or lack thereof) in them as they develop.