Early Ideas
While most of my experiences with controller development up to this point were related to Stickless, I have always wanted to try out new and creative means of integrating custom hardware directly with a game engine. My custom controllers use Arduino microcontrollers to send input messages to video game consoles (like the PS4 and Xbox One) and computers. However, Arduinos can receive input data from far more than simple buttons and joysticks. So I thought to myself: why not try using something like a heart-rate monitor to influence gameplay and audio systems in a game engine?
The Hardware
Before we can just plug an Arduino into Unreal Engine, we’ll need to set up the hardware to communicate basic information in the first place. There are a number of pulse monitors designed for use with Arduinos, but I specifically used this one. This pulse sensor, from the aptly named PulseSensor.com, includes a custom library for streamlining the scripting process on Arduino.
As a musician, my first instinct upon receiving the pulse sensor was to set it up as a MIDI controller. The clip here demonstrates the pulse sensor sending MIDI data via the Arduino to a software synthesizer called VCV Rack. My pulse is setting the MIDI note value for the base pitch of each chord.
The Software
While the final version does not communicate data via MIDI, this basic demonstration offers a great opportunity for seeing how the foundation of the system works. Read through the comments in the watered-down code below to see how data is read from the pulse sensor every tick, or more specifically, at a baud rate of 9600 bits per second!
Controlling A Game With Your Heart
With the hardware and software set up for basic communciation between the Arduino and PC, it’s time to move on to working in a game engine. The following videos are longer breakdowns at each step through the development of this project, so feel free to explore this section at your own pace. Enjoy!
Part 1: The Whitebox
Whitebox-ing is a process level designers undertake when initially building levels for games to highlight and test gameplay mechanic ideas. In this demonstration, my whitebox level is set up specifically to highlight some of the basic potential for the heart-rate system:
Controlling a classic heartbeat sound with the BPM from my current heart-rate
Controlling the chords in a song by using my heart-rate to set a position in time
Controlling the intensity of each instrument in the soundtrack
Artificially manipulating heart-rate values to increase intensity in addition to the current rate.
Part 2: The Blueprints
In Unreal Engine 4, code can be written using C++ or a visual scripting language called Blueprints. This video breaks down how the Blueprints in the game engine receive information from the Arduino then pass that off to FMOD Studio, the audio middleware tool I used for setting up adaptive music and sound in the level.
Part 3: The Prototype
This final video demonstrates a fully playable demo featuring heart-rate as a core part of both the audio system and the mechanics for the game: the likelihood that the patrolling enemy AI will spot the player as they stealthily travel up a perilous mountain-top temple. While the development of this system likely ends here, I rather enjoyed working on the project and having played a working prototype, would love to see future applications of heart-rate controlled audio systems in other games (especially horror games). Maybe I’ll make a mobile game that utilizes heart-rate data from a connected smart watch…