SpaceKeys

About The Project

Description

Space Keys is a captivating game that challenges your rhythm and reflexes. Your mission is to catch as many celestial music objects as possible, by navigating two UFOs through space using webcam-based hand tracking technology. Scores ranging from "Stellar!" for perfect catches to "Lost in space" for the occasional miss add a dynamic layer to your interstellar journey. Each successful catch not only contributes to your score but also enhances the song, creating a harmonious symphony that evolves with your skill. Are you ready to listen to some great music on your journey through space?

Motivation

For this project we wanted to do something that was just for fun and could be taken in any creative direction that we wished for it to go, and which didn’t speak to any bigger cause or was meant to inform people. Another requirement we had was to not be limited by access to hardware, as our previous project had provided some limitations in that regard. Finally, we wanted to work with game logic and include sound from the very beginning, instead of it being an addition towards the end.

The Process

Technology Used

Unity is a real-time development engine made for facilitating collaboration and building games and other interactive experiences. It was used to host the game world, because we had previous experience with Unity within the team.
Blender is a free and open source software commonly used for 3D modeling and animation. We used Blender to create our own 3D models which we could import into Unity, and we opted to use this tool because we had experience with this within the team too.
FMOD is an audio system that is easily integrated with Unity, and this was used for adding music to our game.
Mediapipe for Unity is an existing solution which enables, among other things, hand tracking using webcam. On top of this solution we also built our own algorithms to decide the position of the center of a tracked hand, as well as to detect when the user makes fists with their hands.

Challenges and Obstacles

As with any project, we of course faced difficulties along the way with Space Keys. Initially, we intended to have people interact with the game using a smartphone, and we thought this could be accomplished by using data from the built-in accelerometer sensor. We tried this approach, but had to let go of that idea as we couldn’t find a suitable way to accomplish the interaction we wanted using this method. It was challenging to spend a lot of time and effort on researching how to firstly connect the smartphone to a laptop using a client-server architecture, and then how to use the accelerometer data in a reasonable way, only to have to scrap the entire idea in the end. It has also sometimes been challenging to coordinate the work on this project, as our group was comprised of 6 people, who all had very busy schedules.

Lessons learned

One thing we learned from this project is that it can be a good idea to limit the amount of time spent on researching something, so that there is still time for whatever else needs to be done. Additionally, it is good to try to plan when to meet ahead of time, or set a recurring time each week for example. Because although we continuously used Discord to keep each other updated on progress made or issues we faced, we would sometimes need to have a more thorough discussion to be able to make decisions. And that can be incredibly hard to find time for, when you are this many people, who are all very busy with other things as well.

Possible future development

The only thing players struggled with was the point of view of the camera which made it hard to understand what was in front and what was behind which made it difficult to target objects accurately and led to missing many easy targets. One of the suggestions we got was to take the camera further away and to add orthographic projections of shadows or "shinings" of the stars/notes on the grid walls to allow players to better orient themselves.
There could be more advanced effects when a game object is interacted with, e.g. something that indicates even more clearly that you managed to catch it.

The making of Space Keys

Team

Akshata Murdeshwar

akshata@kth.se

User Interface

Since the overarching theme of the game was neon and retro, much like old arcades, we followed the same for the GUI of the game. The UI's main purpose was to provide instructions, remind players of possible actions, show progress, and give feedback. There were two elements that received special attention:

Text

Each text element was mostly white with 2 undertones of other colors - one darker and one lighter - to give it a 3D feel. The font chosen was bold and in capitals. This choice was based on the fact that a lot was happening on the screen at all times in terms of light as well as movement. Hence, the text needed to be clearly visible without a lot of effort.
Icons

The icons were instrumental in communicating instructions clearly. Since our interactions were mostly based on hand gestures, we used bold white line icons of hand positions and gestures with text to support the same.

General Art Direction

I made mood boards and collected resources to anchor our graphic elements, as well as made sketches of possible stage arrangements, the first mock-up of decoration around the stage, and developed emissive materials and explored their use within the game for beautification and graphic enhancement.

3D Modeling of Elements

I created the prism shaped game objects, and a 3D model of an asteroid field, which was accompanied by a Unity script to animate the asteroids. I also created a black hole, by using a free Unity shader and modifying it to suit our needs. I also used particle systems restricted within a sphere for the star clusters.

Evaluation and Documentation

I also conducted evaluation during testing, and edited the making-of video.

Jacky Liu

hjjliu@kth.se

Server-Client Architecture

Initially, we wanted the player to be able to control the UFO by using their phone. When the player moves their phone up-and-down or left-to-right, so would the UFO in-game. A server was set up so the player could connect to it. Once connected, the player’s phone would send its gyroscope and accelerometer data to the server and update the UFO’s position accordingly. However, we quickly found out that this method is too imprecise, as the data contains too much noise.

Hand Tracking

In the end, we decided to use hand-tracking for controlling the player movement. We made use of MediaPipe’s Framework and their pre-trained solution for Hand-landmark detection. Using MediaPipeUnityPlugin as the base, we further modified it to suit SpaceKeys needs. I was in charge of importing this framework, making sure that we can accurately control the UOF’s position using hand-tracking and then handed it off to the other teammates for further adjustments.

3D Modelling and Visual effects

The shield was modeled in Blender, by taking a sphere and then subdividing the surface to get individual hexagons and pentagons. The UV shield texture was then further modified in Krita, to get a highlighted outline. Using Unity’s ShaderGraph, we control the color and the oscillation effect of the shield. To render the following music objects: Nebula, Quasar and Supernova, a method called Volumetric Raymarching was used. By using volumetric textures, we can determine the density of the cosmic clouds at each pixel. Using this density, we can then deduce the color for that pixel. To get the volumetric textures, we first modeled a cloud in Houdini. Houdini provides a convenient function to create clouds out of geometries using noise (Cloud geometry node). After tweaking the noise parameters, we can then export the model as a Volume Texture using the Volume Texture Export geometry node.

Linnéa Gustafsson

linneag2@kth.se

Audio-Visual Mapping

Since the audio is a central part of this game, it needed to be properly and intuitively integrated into the experience. The melody pitch and brightness were spatially mapped to the horizontal and vertical positions of the melody objects (stars). The colors of music objects also had an intuitive mapping based on audio-color mapping research. The choice of which space objects to use for which music object was also made with logical reasoning behind it: hard objects (asteroids) for drums, single stars for melody and multiple stars for harmony, a black hole for the low-pass filter (absorbing the song’s top frequencies), etc.

Audio Composition

We created our own music and background audio. It comprises 18 audio files created in Logic Pro and four audio effects applied in Unity. The song was designed in Logic Pro, and exported in shorter samples that were then looped and triggered in Unity at the right positions in time.

Audio Programming

For the audio programming, the Unity audio system FMOD was used. The audio can be divided into background audio and game audio. The background audio is triggered regardless of the user's actions – spaceship sound, drums, bass, and some harmony. Game audio is triggered as the user collides with specific music objects. The generation of music objects was programmed as a low-level version of a MIDI editor, with a function triggering a specific music object at a specific time and (x, y) position. The game audio consists of melody, harmony, “space probe” audio, and sound effects (reverb, low-pass filter, high-pass filter, and flanger).

Saga Palmér

sagapa@kth.se

Game Logic

Responsible for the overall logic of the game itself, and for putting it all together. I created the main scene with a star particle system as a background, and generated the procedural circular grid. I also added and generated the music objects in the given sequential order, was responsible for creating the collision logic, and syncing it all with the music.

Visuals

I added the UFO player object, and created light rings from the music objects and the UFO on the circular grid, to provide players with support to understand the positions of the UFO and the music objects coming towards them.

Game Testing

Tested the game and fixed miscellaneous bugs throughout the project.

Shiqi Gong

shiqig@kth.se