Trabajo Fin de Grado Final-Year Project

(1)

Universidad Politécnica de Madrid

Escuela Técnica Superior de Ingenieros Informáticos

Grado en Ingeniería Informática Bachelor in Computer Engineering

Juego de Baile Tradicional Malasio Game for Malaysian Traiditional Dance

Autor / Author: Juan Gaspar García

Madrid, Septiembre, 2022

(2)

This Final-Year Project has been deposited in the ETSI Informáticos de la Universidad Politécnica de Madrid.

Trabajo Fin de Grado / Final-Year Project

Grado en Ingeniería Informática

Bachelor in Computer Engineering

Título: Juego de Baile Tradicional Malasio

Title Project: Game for Malaysian Traditional Dance Septiembre 2022

Autor /Author: Juan Gaspar García

Tutor / Supervisor: Co-Tutor / Co-supervisor:

Nombe y apellidos / Name and Surname/s: Mr. Mohd Azam Osman Título Académico / Academic degree:

BCompSc (Hons.), MSc.

Universidad del Título Académco / University of the academic degree:

Universiti Sains Malaysia

Nombe y apellidos / Name and Surname/s: ---

Título Académico / Academic degree:

Universidad del Título Académco / University of the academic degree:

Escuela / Facultad Department / School Universidad / University

Escuela / Facultad Department /School Universidad / University

(3)

School of Computer Sciences

CAT400 - Undergraduate Major Project

Final Report

GM21220176: Game for Malaysian Traditional Dance

Juan Gaspar Garc´ıa 778714

Supervisor: Mr. Mohd Azam Osman

Examiner 1: Assoc. Prof. Dr. Wan Mohd Nazmee Wan Zainon

Examiner 2: Dr. Syaheerah Lebai Lufti

Academic Session

2021/2022

(4)

DECLARATION

“I declare that the following is my own work and does not contain any unacknowl- edged work from any other sources. This report was undertaken to fulﬁl the requirements of the Undergraduate Major Project for the Bachelor of Science in Computer Science (Honors) program at Universiti Sains Malaysia”.

Signature:

Name: Juan Gaspar Garc´ıa

Date: October 3, 2022

(5)

CAT400 Analysis Report Academic Session: 2021/2022

ABSTRAK

Menari, aktiviti sosial yang telah ada di sekeliling kita selama beribu-ribu tahun.

Ia adalah salah satu perkara yang paling universal di dunia ini. Dengan teknologi terkini, anda boleh membawa tarian ke peringkat seterusnya dan menyesuaikannya sebagai permainan video. Dan, ia telahpun dilakukan. Hadnya ialah permainan tarian sedia ada hanya mengandungi beberapa lagu untuk menari dan orang ramai serta pembangun permainan sering melupakan lagu atau tarian tradisional yang ku- rang popular. Sebaliknya, tarian tradisional adalah sangat penting dan ia adalah cara kita untuk memberitahu orang lain apa yang mereka patut tahu tentang bu- daya itu. Ia juga bagaimana orang lain boleh mempunyai rasa hormat, pengetahuan dan kepentingan kepada tradisi dan norma kita. Walau bagaimanapun, dengan ke- majuan teknologi dan perubahan masa, orang ramai cenderung melupakan tarian tradisional. Oleh itu, projek yang dinamakan Games for Malaysian Traditional Dance dicadangkan untuk memperkenalkan salah satu tarian tradisional Melayu yang dinamakan dikir barat. Permainan ini akan membolehkan pengguna mereka bentuk dan berkongsi tarian baharu serta menambah bilangan lagu yang tersedia dan dengan itu dapat menyelesaikan masalah. Ia juga akan membantu orang ramai mencari tarian kegemaran mereka dan menari bersama dalam permainan yang men- jejaki pergerakan mereka menggunakan kamera. Tambahan pula, orang ramai boleh menggunakan alatan editor untuk mencipta tarian baharu untuk dinikmati semua orang. Semoga projek ini akan memperkenalkan dan melestarikan Tarian Tradi- sional Malaysia,dikir barat kepada masyarakat di seluruh dunia melalui permainan ini.

Kata kunci: Tarian, Permainan, Penglihatan Komputer, Kinect

(6)

ABSTRACT

Dancing, a social activity that has been all around us for thousands of years. It is one of the most universal things in this world. With the latest technology, you can take dance to the next level and customize it as a video game. And, it has already been done. The limitation is that existing dance games only contain a few songs to dance to and people as well as game developers often forget about less popular songs or traditional dances. On the other hand, traditional dance is very important, and it is our way of telling others what they should know about the culture. It is also how others can have respect, knowledge and importance to our traditions and norms. However, with advances in technology and the changes of time, people tend to forget about traditional dances. Therefore, a project called Games for Malaysian Traditional Dance is proposed to introduce one of the traditional Malay dances calleddikir barat. The game allows users to design and share new dances as well as increase the number of songs available and thus solve the problems. It also helps people ﬁnd their favourite dances and dance with them in a game that tracks their movements using a camera. Furthermore, people can use the editor tools to create new dances for everyone to enjoy. Hopefully this project will introduce and preserve the Malaysian Traditional Dance, dikir barat, and introduce it to the community around the world.

Keywords: Dance, Game, Computer Vision, Kinect

(7)

RESUMEN

Bailar, una actividad social que nos rodea desde hace miles de años. Es una de las cosas más universales de este mundo. Con la última tecnolog´ıa, puedes llevar el baile al siguiente nivel y personalizarlo como un videojuego. Y ya se ha hecho.

Sin embargo, los juegos de baile existentes solo contiene algunas canciones para bailar y la gente, as´ı como los desarrolladores de juegos, a menudo dejan atrás las canciones menos populares o tradicionales. Por otra parte, el baile tradicional es muy importante y es nuestra forma de comunicar nuestra cultura con el resto del mundo. También es cómo otros pueden tener respeto, conocimiento e importan- cia para nuestras tradiciones y ra´ıces. Este proyecto, llamado Juego de Practicar Baile Tradicional Malasio tiene como objetivo introducir una de los bailes mala- sios tradicionales llamado dikir barat a las nuevas generaciones. El juego permite a los usuarios diseñar y compartir nuevos bailes. As´ı aumentan el número de bailes disponibles y se solucionan estos problemas. También ayuda a las personas a encon- trar sus bailes favoritos y practicar con ellos en un juego que rastrea su movimientos usando una cámara. Además, las personas pueden usar las herramientas del editor para crear nuevos bailes para que todos disfruten. Esperemos que este proyecto preserve el baile tradicional de Malasia,dikir barat.

Palabras Clave: Baile, Juego, Computer Vision, Kinect

(8)

DECLARATION i

ABSTRAK ii

ABSTRACT iii

RESUMEN iv

TABLE OF CONTENTS v

LIST OF TABLES vii

LIST OF FIGURES viii

LIST OF ABBREVIATIONS AND SYMBOLS ix

1 INTRODUCTION 2

1.1 Background . . . 2

1.2 Problem Statements . . . 2

1.3 Motivation . . . 2

1.4 System Objectives . . . 3

1.5 Proposed Solutions . . . 3

1.6 Beneﬁts and Uniqueness of the Proposed Solutions . . . 4

1.7 Organization of the Report . . . 4

2 BACKGROUND & RELATED WORK 6 2.1 Status of the Project . . . 6

2.2 Existing Related Projects . . . 6

3 SYSTEM REQUIREMENTS / ANALYSIS 8 3.1 Project Scope . . . 9

3.2 System Capabilities . . . 9

3.3 System Limitations . . . 10

3.4 Development Methodology . . . 10

3.5 Tools and Technologies . . . 11

3.5.1 Motion Tracking . . . 11

3.6 System Requirements . . . 12

4 SYSTEM DESIGN & IMPLEMENTATION 13 4.1 Dance File Format . . . 13

4.2 Server Implementation . . . 14

4.2.1 API Design . . . 15

4.2.2 Database Design . . . 15

4.3 Client Implementation . . . 16

4.3.1 Graphics . . . 16

(9)

4.3.2 Gesture Detection . . . 18

5 SYSTEM TESTING & EVALUATION 20 5.1 Server Testing . . . 20

5.2 Client Testing . . . 20

6 CONCLUSION & FUTURE WORK 21 REFERENCES 22 APPENDICES 24 APPENDIX A . . . 24

APPENDIX B . . . 25

APPENDIX C . . . 27

APPENDIX D . . . 28

APPENDIX E . . . 31

(10)

1 Existing Dancing Games . . . 6 2 List of API endpoints . . . 15

(11)

LIST OF FIGURES

1 Let’s Dance! Dikir Barat logo. . . 3

2 Project Structure . . . 8

3 Agile Development Methodology Diagram . . . 10

4 C# code for the deﬁnition of the gesture RiseRightArm . . . 18

5 Initial Project Structure . . . 24

6 Entity-Relationship Diagram . . . 27

7 Let’s Dance! Dikir Barat - Title Screen . . . 28

8 Let’s Dance! Dikir Barat - Dance Explorer Screen . . . 28

9 Let’s Dance! Dikir Barat - Dancing Screens . . . 29

10 Let’s Dance! Dikir Barat - Results Screen . . . 30

11 Just Dance Game Images . . . 31

12 Dance Central Game Images . . . 32

13 StepMania Game Images . . . 33

14 Dance Dance Revolution Game Images . . . 33

(12)

LIST OF ABBREVIATIONS AND SYMBOLS Glossary

Alembic a lightweight database migration tool for usage with the SQLAlchemy Database Toolkit for Python. (See [15]). 14

dance pad ﬂat electronic game controller used for input in dance games. 6, 11, 12 FastAPI modern, fast (high-performance), web framework for building APIs with

Python 3.6+. (See [8]). 11, 14, 20

Godot Engine open-source multi-platform 2D and 3D game engine. (See [10]). 11 Kinect for Windows SDK enables developers to create applications that support gesture and voice recognition, using Kinect sensor technology on computers running Windows. (See [11]). 11

Performous open-source karaoke, band and dancing game where one or more players perform a song and the game scores their performances. (See [12]). 6 PostgreSQL a powerful, open source object-relational database system with over

30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance. 11

Python a programming language that lets you work quickly and integrate systems more eﬀectively. (See [7]). 14

SQLAlchemy Python SQL toolkit and Object Relational Mapper that gives application developers the full power and ﬂexibility of SQL. (See [13]). 11, 14 SQLite relational database management system contained in a C library. (See [14]).

11

Unity Engine a cross-platform game engine developed by Unity Technologies.

(See [9]). 11, 16

Acronyms

API Aplication Programming Interface. 3, 4, 8, 11, 14, 15, 20, 24 DB Database. 8, 9, 11, 14, 15, 27

GUI Graphical User Interface. 7

HTTP Hypertext Transfer Protocol. 20

(13)

SQL Structured Query Language. 11 URI Uniform Resource Identiﬁer. 15 URL Uniform Resource Locator. 14

(14)

1 INTRODUCTION

1.1 Background

Both games and dances have thousands of years of history [1][2] around the word.

Mixing these two areas together isn’t a new thing. Dancing games have been around for a very long time and are very popular among kids. With the latest technologies, video games have raised, allowing games to have more complex rules and behaviors.

Dancing video games aren’t a new thing either, with many titles and diﬀerent options existing already in the market [3].

These games are a great way to introduce people into new music and dances, while also measuring and ranking the more experienced users. Because dance is interna- tional and for everyone, it makes dancing games appealing to a very wide audience.

The project I propose would make more people experience new types of dance, in particular the Malaysian traditionaldikir barat. Dikir barat is a style of Malay choral singing. It can be interpreted in ﬂexible formats that incorporate singing, poetry, movement and music [4]. It started as a form of entertainment, but nowadays it’s also used to help discuss current issues and to educate. It is performed in groups, often in a competitive manner. A dikir barat group is composed of four main elements: the tok juara (leader), tukang karut (song initiator), awok-awok (10-15 people chorus) and a percussion ensemble. The project will focus in the role of theawok-awok, that is, to respond to the tok juara and tukang karut by doing rhythmic movements of their arms, hands and upper torso while seated cross-legged on the ﬂoor [5].

Users will also be able to contribute with new dances and seek for other peoples’

dances. In this document I will state why the project is a demanded idea and how I plan to achieve the proposed objectives, including the tools and technologies to get there.

1.2 Problem Statements

Most dancing video games oﬀer a limited range of dance types and songs [6]. They usually focus in popular western music, and don’t allow for new dances to be added by users. Traditional dances are often forgotten. For this reason, I believe that an expandable open dancing game that allows dances to be easily added by everyone would beneﬁt its users. This project will facilitate players to dance to their favorite music or create a custom dance for everyone to enjoy.

1.3 Motivation

As a video game and music enthusiast, I am very excited about a project that involves both of them. Game development has always been one of my favorite things, and combining it with other ﬁelds such as computer vision looks far more interesting. In my opinion, open software is always the right option, so contributing with a system like this feels very special.

(15)

1.4 System Objectives

The objectives of this project are the following:

• To develop a ﬁle format that deﬁnes a dance.

• To develop an editor to easily create and share these ﬁles.

• To create some examples of dance ﬁles based on traditional Malaysian dance dikir barat.

• Be able to track the user’s movements and rank them according to an already deﬁned dance.

• To integrate all previous points together in a game-like interface.

1.5 Proposed Solutions

Figure 1: Let’s Dance! Dikir Barat logo.

The solution I propose isLet’s Dance! Dikir Barat, a new dancing game that includes some examples of dikir barat dances. People can dance to the included dances and improve their scores. To keep track of the user dancing, the system will use a camera.

The user can also choose to see himself dancing while playing.

To make the project expandable, new dances can be easily created and added to the game with an editor. The editor will be a developed tool that facilitates the creation of new dances, as well as the editing of existing ones.

Finally, to help people sharing and getting new dances, everyone can submit their dance ﬁles created with the editor to a centralized server. This server can be accessed in game through a public Aplication Programming Interface (API).

The application to develop can be separated in the following modules:

(16)

• Dance screen: shows the next moves to perform by the player, the rating of the current move (bad, okay, good or perfect) and the accumulated score as a natural number. If the player so chooses, he will also be shown performing the dance on screen. Other particles and eﬀects following the music might be added too.

• Motion tracking: receives input from the camera and encodes it into a series of human moves. This module is separated to the next one to facilitate the inclusion of other methods for motion tracking as a possible feature in the future.

• Dance evaluation: receives the moves processed by the motion tracking module as an input and ranks them according to the current dance ﬁle. This rank is then displayed on the dance screen.

• Navigation and pause menus: a simple menu system to allow navigation through the other modules.

• Dance explorer: a paged list of published dance ﬁles found in the database.

Dances will be searched by name. Available dances are received via an API and can be downloaded to be played.

• Editor: the included tool to create or edit diﬀerent dance ﬁles.

More on the technical details of the proposed solution in section 3.

1.6 Beneﬁts and Uniqueness of the Proposed Solutions

The project has the potential to create a community around it. It can make dis- covering new music and dances easier. People can spend their free time dancing, or maybe practising to improve their dance moves. Dedicated people can choose to design a dance ﬁle for their favorite songs, and share them for others to enjoy.

1.7 Organization of the Report

This document is organized in six main sections.

Section 1, Introduction, is an introduction to the project, explaining its background, the problem it aims to solve and the proposed solution itself.

In section 2, Background & Related Work, I compare this project to existing others and research on diﬀerent intakes to the same idea. I also compare those approaches to mine, emphasizing on the diﬀerences.

An advanced analysis about the system, its requirements, development, used tools and a deep analysis of the diﬀerent alternatives is what section 3 contains, System Requirements & Analysis.

Going forward to section 4, System Design & Implementation. It consists of a detailed summary of the implementation of the project, the techniques used and the system documentation.

(17)

In section 5, Testing & Evaluation, there is a brief description of how the system has been tested and what methods have been used for that matter.

A ﬁnal conclusion is done to summarize every aspect of this report and what can be improved moving forward in section 6, Conclusion & Future Work.

(18)

2 BACKGROUND & RELATED WORK

2.1 Status of the Project

The project has ﬁnished development for the scope deﬁned in this report. However, there is a lot of potential for new features and improvements. These will likely be developed in future updates to highly improve the quality.

2.2 Existing Related Projects

Game Title Developer Tracking method

Number of

songs

Max players

Custom songs?

Just Dance

(2021)

Ubisoft Gyroscope

and Camera

44 6 No

Dance Central Harmonix Music Systems

Camera 10 2 No

StepMania StepMania Team

Dance pad - 2 Yes

Dance Dance Revolution

Konami Dance pad 303 2 No

Performous Community driven

Dance pad - 2 Yes

Table 1: Existing Dancing Games

There are many existing dancing games (table 1) (See Appendix E). They all share the same core idea, but develop it in many different ways. Aside from what songs are included or visuals, one of the main differences from one game to another is what method they use to track movement. This has a huge impact on how the game is played and its limitations. Back in the arcade days, a dance pad was the most popular option for motion tracking. Later, with consoles arriving to customer’s houses, cameras started to popularize at performing this task. Finally, with technology evolving at such a fast pace, gyroscopes started to appear inside game controllers and smartphones, making for a good quality alternative to cameras at a cheaper cost. These three are the main methods, but other alternatives exist. More on the different tracking methods on section 3.5.1.

Out of all the listed games in table 1, Performous stands out for being open source.

It is a package of three types of musical games, including dancing. Users can supply and make their own dance files for others to enjoy, similarly to how it is planned to be the case with this project. However, because the tracking method used in this game is a dance pad, dance moves are very limited. In this project, a camera will be used for motion tracking to evade this limitation. Performous will then be a huge influence for the dance description file system, but not for the game visuals and mechanics themselves.

(19)

Just Dance is probably the biggest title of the genre, with yearly multi-platform releases and probably the largest soundtrack. As seen on ﬁgure 11 in the Appendix E, the visual aspect of this game is very iconic. There are scenes with many vibrant colours and a centred character that leads your current dance. On a corner of the screen you can see depictions of the next moves to perform, and when to do so.

Other aspects can be seen on the Graphical User Interface (GUI) such as the score, the lyrics of the song, and a smaller silhouette representing what the player is doing.

The game also supports multiplayer, allowing multiple people to dance at the same time to the same music. The GUI and other aspects of Just Dance will for sure serve as inspiration when working on this project.

(20)

3 SYSTEM REQUIREMENTS / ANALYSIS

DB

API

Game Dance Explorer

Camera

Client Server

Figure 2: Project Structure

To accomplish the objectives and ideas of this project, I have structured it as re- ﬂected in ﬁgure 2:

• The game is the main part of this project. It is, in essence, a program that allows the user to choose a song and dance to it.

• The dance explorer is an integrated screen inside the client application. It’s propose is to fetch and list all available songs in the server, so the user can later choose one.

• An API handles communication between the Database (DB) and the application. Dance ﬁles are uploaded together with their respective music in an audio ﬁle (i.e. .mp3, .wav, .ogg) making use of the API and external tools.

Any user can download existing dance ﬁles to play them or edit them through the game interface. The DB can serve to store additional information such as global leader-boards or ratings, although this is not the case within the scope of CAT400.

The game has menus that can be navigated with a mouse, a keyboard or a game pad. During the dancing sessions of the game, a camera is used to capture the player’s movements.

Due to unexpected setbacks during development, some of the initial intended features were dropped for this release, such as the dance editor. Because of this, dances have to be manually created using a text editor. The initial proposal of the project structure can be found at ﬁgure 5 in the Appendix A.

(21)

3.1 Project Scope

When looking at this project, there are two main distinguished blocks:

• A game that allows users to individually dance to many available songs and dances. The game tracks the player’s movement to give them a score and help them visualize how they performed on every session. The game also has some basic menus to select a song, view rankings, explore online songs and pause while in a dance session.

• A server application used to manage, store and provide access to media and information about songs, dances, users, etc.

This implies that the following, smaller and speciﬁc tools will be developed or used:

• A file format to define dances. The file will contain different moves associated with timestamps, similar to how subtitle files for videos work [17].

• A game engine that allows displaying graphics and playing sounds easily.

• A fully responsive game-like interface to interact with.

• A system that translates the video input from a camera into a set of human moves for one person.

• An algorithm to rank and compare the previously obtained moves to the ones deﬁned in a dance ﬁle.

• A way of storing local values such as leader-boards, settings and local dance ﬁles.

• An externalized DB to store shared dance ﬁles and user accounts.

• Documentation for every public aspect of the project.

Anything not included in this list remains out of the scope of this project, and as such, will not be implemented unless a radical design change is made.

3.2 System Capabilities

As a dancing game, the main feature seems obvious. The player can dance to a selected song and receive a grade indicating how they performed. Nonetheless, there are many other capabilities. They all are:

• Explore songs published by other users.

• Dance to the beat of many songs and dances, including those from other users.

• Get and save a score for each performance.

• Create new dances or edit existing ones in the form of dance ﬁles.

• Share the created dance ﬁles.

(22)

Test Implementation

Design

Req uirem

ents

Analysis Plan

Figure 3: Agile Development Methodology Diagram

3.3 System Limitations

For complexity reasons, dances will have to be performed individually. Dances that intended to be performed in groups can be performed individually if the group factor is expendable. Otherwise, they wont be easily adapted to the game.

To follow the copyright laws, protect the creator’s properties and fight piracy, copy- righted songs with distribution limitations will not be accepted in the database. For now, this will be manually verified before the song is later approved and published to everyone. If this happens, the dance file will be published without an attached song, meaning the player can play it himself while dancing or provide his own copy of the song.

Because of the tools used (see section 3.5), the camera will be limited to a Microsoft Kinect device in the context of the subject CAT400. Later work may be done on this project to support generic camera devices.

3.4 Development Methodology

I’m approaching this project with an Agile-Scrum [18] development methodology in mind. As visualized in figure 3, Agile is an iterative process. It starts as a set of ideas that have to be planned into a project. Planning includes defining the most important aspects of the project, its requirements and a timetable. When the plan is complete, an extensive analysis is done on the requirements that have been established. Later, the final product is designed, implemented and tested. If more ideas or necessities emerge after this iteration, a new iteration can start from the planning phase.

Scrum is a framework process for Agile which facilitates scheduling, organization and a fast workﬂow. It is designed for groups of developers, having diﬀerent roles for the members of the development team, but can be used in individual projects to help manage time and priorities. Development is divided in cycles called sprints.

Each sprint will have the duration of two weeks. At the start of a sprint, its goals

(23)

are deﬁned. At the end of a sprint, a conclusion of how the last sprint went is made. This is all done using existing tools such as the ones in GitHub. To facilitate organization, week 1 of the project is set to match week 1 of Academic Session 2021/2022.

3.5 Tools and Technologies

As this project has many different requirements, there are also many different tools to work with. I selected them based on how well they fit for this project, how familiarized I am with them and how they would allow for the project to expand in the future.

• Unity Engine for the game and dance explorer, which assists in easily creating a unique and responsive graphic experience. Because of it’s popularity, the community has developed many tools that are useful for this matter.

Godot Engine was initially planned as the game engine for the project, but was later replaced due to lack of compatibility and existing tools.

• Kinect for Windows SDK for the camera motion recognition. This means that, at least for now, the camera will have to be a Microsoft Kinect device and the game will only support Windows as a platform.

• FastAPI for the API service. The API needed for this project is simple, so there is no need to go for something very complex.

• PostgreSQL as a DB engine to store and access dance ﬁles and account information using Structured Query Language (SQL).

SQLite was initially thought for this task, but was later replaced due to a lack of compatibility with SQLAlchemy.

• SQLAlchemy to facilitate and sanitize the communication between the API and the DB. Using SQL queries without protection can lead to query injection attacks that can cause serious damage [19].

3.5.1 Motion Tracking

Every dancing game uses a particular way of monitoring the movement of the player.

Each tracking method has its advantages and limitations:

• Dance pads offer a very game-like look, as in the end it consists of many buttons on the flor that the user has to press in order to progress, just like in other games with a game pad. Their main disadvantage is they only monitor where the players steps, leaving the arms and torso free of tracking. Also, as it is such a specific hardware, not many people own it, which makes it harder for people to play a game that requires it.

• Cameras can monitor the whole body. Thanks to Computer Vision techniques, human movements can be recognized and analyzed in real time. They can be very dependant of good light conditions to work properly. Aside from that,

(24)

they are easy to install. Also, as a camera, it can record the user dancing for later evaluation or other uses.

• Gyroscopes bring a very precise motion tracking, but only to the part of the body they are attached to. The light restriction from cameras is gone, but the player can’t see himself dancing on screen. The user is forced to hold or wear a device with the gyroscope, which may break the immersion in the dance.

When thinking about what tracking method to use for this project, there are many diﬀerent variables. The reasons why I have chosen to use camera tracking are:

• Cameras are easy to access devices that many people already have.

• Dance pads restrict possible movements too much.

• Gyroscopes can be annoying to hold or carry while dancing.

• The user can see and evaluate their own dancing on screen while performing it.

3.6 System Requirements

This game will be released for Personal Computers running Windows Operative Systems. Hardware requirements are not very demanding. For example, a laptop with integrated graphics is enough. The application itself won’t be any larger than 1 Gigabyte, although more space would be required for every dance files and songs the user downloads. Dance files are nothing more than plain text files, and songs are usually in the range of Megabytes. Maybe one or two downloaded dances don’t require a significant amount of disk space, but large quantities certainly can.

An internet connection is required for exploring online dances, as well as to upload created dance ﬁles.

Another important requirement is the camera, which is needed for the main aspect of this project, motion tracking. However, the rest of the project can absolutely be used without a camera.

(25)

4 SYSTEM DESIGN & IMPLEMENTATION

As stated in section 3, the whole project is divided in many components. These components end up forming two main programs: the server and the client. Each of them had its own development, with different technologies and environments used to adapt to the requirements. Aside from these two softwares, a file format had to be designed from scratch to represent dances. In this section, we will cover the development and implementation of this different parts.

4.1 Dance File Format

The Dance File format, or DNC, has been design with the following necessities in mind:

• Easy to create, modify and understand.

• Easy to distribute.

As mentioned before, the biggest inspiration for this format has been the globally used filetype for subtitles, SRT (See [17]). As such, DNC files are just plain text files that follow a ver specific syntax. For a dance file example, please check Appendix B. The file can be divided in two sections:

1. Header - Contains data about the ﬁle and song.

2. Moves list - A list of dance moves that constitutes the dance itself.

The header consists of three lines, each with a speciﬁc meaning:

1. Version number - An integer representing which of the DNC format speciﬁ- cations is used. Latest version is number 2 and is the one explained in this report.

2. Duration of the dance - The dance will stop after this many seconds, even if the song hasn’t ﬁnished yet.

3. Song oﬀset - If it’s a positive number, the song will seek in this many seconds before playing. If it’s a negative number, the song will delay this many seconds after the dance starts. This value can be used to easily ﬁx a possible desynchronization between the moves’ timestamps and the song itself.

After the header, an empty line must be included to indicate the beginning of the next section, the moves list. The moves list is a series of moves, ordered chronologically as they appear in the dance. Each move has the following syntax:

1. Move index - An integer indicating which move this is in the timeline. Because the moves are ordered chronologically, this information is redundant but still used as a validity measure.

2. Start and end timestamps - Two ﬂoat values indicating when to when the move is valid to be perform. This values are separated by an arrow (-->). During this time window, the move will be scored as good if performed correctly.

3. Gesture name - The name of what gesture to expect for this move.

(26)

4. Max score - An integer that determines the maximum possible score for performing this speciﬁc move.

The structure of a dance move can be repeated many times to form the moves list.

After every move in the list, an empty line is required.

4.2 Server Implementation

The server side of the project has the very speciﬁc function to centralize all created dances, songs, users, etc. Ideally, future users could upload their own made dances to the server and explore other peoples’ dances.

Technically speaking, the server consists of two modules: the DB and the API. Both communicate to each other as they are implemented in the same base program.

However, the main program may be hosted in a diﬀerent machine than the DB. The implementation is a simple Python project that uses the FastAPI and SQLAlchemy modules among others.

The folder structure of the server is as follows:

• / - Root of the project. Contains some configuration and a README file explaining how to configure the virtual environment and how to run the application.

• /lddb - Root of the application. Contains the main ﬁle.

• /lddb/alembic - Contains conﬁguration ﬁles for Alembic.

• /lddb/core - Contains database.py, where the connection to the database is performed.

• /lddb/core/models - Contains deﬁnitions for the SQLAlchemy models. In other words, these are the database tables.

• /lddb/core/schemas - Contains deﬁnitions for the serializable data classes used for integration with the API.

• /lddb/core/services - Contains deﬁnitions for atomic operations involving the DB.

• /lddb/api/v1/endpoints- Contains definitions for every endpoint defined in the database. See figure 2 for all endpoints.

The server runs in a python virtual environment to ensure that all libraries are running the appropriate version. A list of all the python modules is found in the file /requirements.txt. It’s also necessary to create a /.env file with some environment variables dictating where the media will be stored and the DB Uniform Resource Locator (URL). Instructions on how to setup the virtual environment and the environment variables are included in the /README.md file.

(27)

4.2.1 API Design

Type Uniform Resource Identiﬁer (URI) Description

GET /status Returns the status of the server

POST /dance Creates a new dance

GET /dance/{dance_id}/info Returns the metadata of a dance GET /dance/{dance_id}/data Returns the ﬁle of a dance GET /dance_list Returns a list of existing dances

POST /song Creates a new song

GET /song/{song_id}/info Returns the metadata of a song GET /song/{song_id}/data Returns the ﬁle of a song GET /song_list Returns a list of existing songs

POST /song_author Creates a new song author

GET /song_author/{song_author_id}/info Returns the info of a song author GET /song_author_list Returns a list of existing song authors

POST /user Creates a new user

GET /user/{user_id}/info Returns the info of an user GET /user_list Returns a list of existing users

Table 2: List of API endpoints

The API has been designed with simplicity in mind in order to meet the established requirements of the project. It’s main propose is to communicate the DB with the client. The API must be capable of retrieving information about existing dances and creating new dances, with everything that this entails. Each dance has a linked song, and a user that created it, so this information should also be managed through the API. See table 2 for a list of the API endpoints.

As a note, song files are uploaded to the server as files. Contrary to this, dance files are uploaded as plain text, which has to be base64 encoded beforehand. The dance is stored encoded and sent back encoded when requested. It has to be decoded by the client when received.

4.2.2 Database Design

When designing the DB, it’s very important to first create an entity-relationship diagram [20], first to see how many entities are needed in our DB, and second to know how they relate to each other. The diagram can be found at figure 6 in Appendix C.

After designing the diagram, it’s easier to know how many tables our DB needs. In the case of this project, the tables are:

• User - For now, this table has no authentication or identiﬁcation attributes.

Only a username is associated with each user.

• Song - As many dances may use the same song, creating a speciﬁc table for songs will later allow for better expandability and less storage being consumed.

(28)

• Song Author - Even tho it appears to be a simple attribute, having song authors as a separate table will allow in the future for better song ﬁltering, sorting and searching.

• Dance - Each dance has it’s creator (user who designed the dance) and song.

It also has the name and the dance ﬁle associated.

4.3 Client Implementation

When designing the client, these are my top priorities:

• The application should have the appeal of a game, and as such, look like one.

• Menu navigation through the program and the dancing experience should be intuitive and doable by most people without being taught.

With these aspects in mind, I started development. When developing for Unity Engine I like to think in the term of screens or scenes. Each scene has it’s own specific setup to fulfill one specific action. This project has the following scenes:

• Title - A very simple menu to welcome the user and allow them to start playing or close the game. See ﬁgure 7 in Appendix D.

• Dance explorer - A scrolling menu that shows every dance available right now in the server. Selecting a dance directly downloads it and changes to the dancing scene. See ﬁgure 8 in Appendix D.

• Dancing - A dance is currently being played, with it’s song and timeline of dance moves. When this screen is loaded, the parser will scan the dance ﬁle before playing said dance. With this information, it generates a list of icons representing each dance move or gesture. These icons are displayed at the bottom of the screen. When every icon is ready, a countdown will appear and the dance will soon start taking action. While playing, the icons scroll from right to left. When the move icon is right at the center of the indicator in the middle of the screen, that’s when the user should perform the move. The avatar is seen in the middle of the screen replicating the user movements. See ﬁgure 9 in Appendix D.

• Results- A recap of how the dancer performed is shown to the user, showing the score obtained as well as the maximum score possible. Options are given to play again or go back to the title screen. See ﬁgure 10 in Appendix D.

4.3.1 Graphics

As the core essence of the project is the one of a game, there has to be some care taken into the graphical aspect of it. Because this is a more artistic ﬁeld and not the main aspect to take in consideration from a Computer Science point of view, I will not dig into much detail.

As a style, I went very minimalist and plain. This is important because it helps me develop and design good quality assets even tho I’m not an artist myself. I focused

(29)

in using a very simple color language, with the main colors varying between shades of magenta and blue. Some text, highlights and accents are colored in a pale yellow, to have a good contrast with the main colors.

I designed the main logo (See ﬁgure 1 in section 1) with this stated style in mind, but also thinking about what dikir barat resembles to me during all my research.

This performances have a very theatrical look to them, so a sign that resembles that one of a theater is the perfect ﬁt. It also gives that impression about it being traditional, but modernized at the same time.

The game scenes have all a very similar setup. A magenta background with some snow falling around. The wall is litten with diﬀerent spotlights, giving it the feeling of a big stage. The menus have buttons that lid up when hovered to provide some feedback to the user.

The gesture icons are one of the hardest graphical part of the project. They have a crucial function: to instruct the player what to do next. Achieving this is not trivial, but with simplistic shapes and some arrows I would say that I’ve got to a level of quality that I’m happy with.

Finally, the dancing avatar is a low polly model of a human, made to ﬁt many peoples appearances and to be inclusive for as most people as possible.

(30)

4.3.2 Gesture Detection switch(gestureData.state) {

case 0: // gesture detection

if (jointsTracked[rightHandIndex] &&

jointsTracked[rightShoulderIndex] &&

(jointsPos[rightHandIndex].y -

jointsPos[rightShoulderIndex].y) > 0.1f) {

SetGestureJoint(ref gestureData, timestamp, rightHandIndex, jointsPos[rightHandIndex]);

} break;

case 1: // gesture complete

bool isInPose = jointsTracked[rightHandIndex] &&

jointsTracked[rightShoulderIndex] &&

(jointsPos[rightHandIndex].y -

jointsPos[rightShoulderIndex].y) > 0.1f;

Vector3 jointPos = jointsPos[gestureData.joint];

CheckPoseComplete(ref gestureData, timestamp, jointPos, isInPose, KinectWrapper.Constants.PoseCompleteDuration);

break;

}

Figure 4: C# code for the deﬁnition of the gesture RiseRightArm

This has surely been the most challenging part of the whole project. Converting taken depth images by the Kinect device into useful, human readable information about what movements were performed. Before starting this project, my background about this matter was not competent at all. After all the research and work that I went through, I ﬁnd myself in a comfortable position about this ﬁeld of Computer Science.

I based my gesture recognition out of the library published by RF Solutions [16].

This provides me with a way to transform the depth images into a human skeleton, and to detect diﬀerent gestures and movements that are performed by this skeleton. It also includes other useful utilities such as rigging animation and Kinect calibration.

In figure 4 we can see an example of how gestures are defined. This example is for the gesture RiseRightArm, which detects when the user rises their right arm. A gesture has a state parameter, which is first evaluated to know how advanced the gesture has gone. This gesture has two states (0 and 1). State 0 checks if the initial conditions for said gesture are met, in this case, the relative positions of the right

(31)

arm’s joints. State 1 veriﬁes that this condition is still true after some time, and then, marks the gesture as complete.

There are many more gestures programmed in the game: RiseArm, Clap, Squat, SwipeLeft, PointRight... All of them are deﬁned following this simple approach. It allows to track the eﬃciency of a gesture through time, and as such, evaluate how each state of the gesture was performed.

(32)

5 SYSTEM TESTING & EVALUATION

As the project consists on two main programs, both had to be properly tested. The correct behavior of each application is crucial to avoid unexpected errors in other places. As such, each module has been approached the best way possible taking in consideration its utility and use cases.

5.1 Server Testing

Server testing has been tested in a methodical manner. API endpoints were evaluated making very speciﬁc Hypertext Transfer Protocol (HTTP) requests and comparing the response with an expected result. The requests were made using the default documentation page (/docs) provided by FastAPI. After comparing the response, the database is checked for the expected changes made.

Repeating this methodology with every endpoint and every thought case scenario, we can conclude that the backend side of the project is done and working as expected.

This will make later testing of the client side much easier and less troublesome.

5.2 Client Testing

As it is often common with video game development, automated testing is oﬀ limits [21]. To test the proper functionality of the system, manual testing had to be done for menu navigation and the dance player. However, gesture recognition could be easily repeated automatically thanks to Kinect Studio, an oﬃcial tool designed, developed and delivered by Microsoft to easily record Kinect depth images and play them over a connected application. This way, the exact same depth movie can be send to the game, and as such, the same dance can be tested without having to perform it multiple times.

When evaluating the quality of this side of the project, the most crucial aspect is the gesture recognition. Gestures have to be detected precisely, unequivocally and quickly. If this is not achieved, the dance can be wrongly interpreted and, as such, the whole main objective of this project would be a failure. To ensure these conditions, I determined that it is better to deﬁne dance moves (gestures) with simple and not many rules. Also, when dancing, it’s better for the game to listen for the expected gesture only, rather than listen to every gesture and then succeed if the expected gesture was performed.

Testing menu navigation, graphics, and interface is simple for a scenario like this, where there are not many possibilities.

Thanks to the testing of the project being separated for both client and server, no problems were found regarding API requests, as it was properly tested beforehand.

(33)

6 CONCLUSION & FUTURE WORK

There are a lot of aspects to keep in mind when designing a big project. In a game like this, where a lot of different things have to interact with each other, it’s important to have a well defined structure and organization for everything. Having prepared every tool and library to use beforehand can save many development issues and headaches later. Comparing your future work to existing projects can not only serve you as an inspiration, but also help you stay aware of the problems that they were surprised with. Moving forward from this position assured a fluid and fast development.

In this case, I was able to consider very diﬀerent solutions to this particular project, evaluate them, and choose the one that best adapted to the project requirements and ideas. The selected tools are not only convenient enough for this system, but also to help it expand in a future.

For a proper commercialization and distribution of this application, there are several things that would have to be improved or added ﬁrst:

• User authentication.

• In-game editor.

• Ability to bookmark, sort, ﬁlter and search for dances.

• Online scoreboards and dance ranking.

• More gestures, including maybe those of other dance types.

With these features and some others, I truly believe that this project has the potential to become a standard platform for dancing, entertainment and training.

(34)

REFERENCES

[1] Jon Radoﬀ. (2010) History of Social Games. [Online].

Available: https://web.archive.org/web/20100527090108/http:

//radoff.com/blog/2010/05/24/history-social-games/

[2] Eva Frederick. (2019) Dancing chimpanzees may reveal how humans started to boogie. [Online].

Available: https://www.science.org/content/article/dancing- chimpanzees-may-reveal-how-humans-started-boogie

[3] Stephen Lagioia. (2021) The 15 Best Dancing (And Rhythm) Video Games Ever Made, Ranked. [Online].

Available: https://www.thegamer.com/best-dancing-rhythm-video- games-ever-made-ranked/

[4] Stephanie Ho. (2015) Dikir barat. [Online]

Available: https://eresources.nlb.gov.sg/infopedia/articles/SIP_

2015-02-12_101207.html

[5] Feel Malaysia. (2016) Dikir Barat Performance. [Online]

Available: https://www.youtube.com/watch?v=lOHh4Lh0Zh0

[6] Ryan Craddock. (2020) Just Dance 2021 Launches On Switch Today, Here’s The Full Track List. [Online].

Available: https://www.nintendolife.com/news/2020/11/just_dance_

2021_launches_on_switch_today_heres_the_full_track_list [7] Python website. [Online].

Available: https://python.org/

[8] FastAPI website. [Online].

Available: https://fastapi.tiangolo.com/

[9] Unity Engine website. [Online].

Available: https://unity.org/

[10] Godot Engine website. [Online].

Available: https://godotengine.org/

[11] Kinect for Windows SDK website. [Online].

Available: https://www.microsoft.com/en-us/download/details.aspx?

id=40278

[12] Performous website. [Online].

Available: https://performous.org/

[13] SQLAlchemy website. [Online].

Available: https://www.sqlalchemy.org/

[14] SQLite website. [Online].

Available: https://www.sqlite.org/

(35)

[15] Alembic website. [Online].

Available: https://alembic.sqlalchemy.org/

[16] Kinect with MS-SDK for Unity, RF Solutions. [Online].

Available: https://assetstore.unity.com/packages/tools/kinect-with- ms-sdk-7747

[17] Tally. (2018) Understanding and Creating SRT Files. [Online].

Available: https://hungrytally.medium.com/beyond-the-subtitles- the-magic-of-srt-files-c4b99e042819

[18] Apoorva Srivastava, Sukriti Bhardwaj, Shipra Saraswat. (2017) SCRUM model for agile methodology. [Online].

Available: https://ieeexplore.ieee.org/abstract/document/8229928 [19] Justin Clarke. (2009) SQL Injection Attacks and Defense. [Online].

Available: https://ns2.elhacker.net/descargas/manuales/Hacking%20y%

20Seguridad%20informatica/Offensive%20Security/SQL%20Injection%

20Attacks%20and%20Defense,%202nd%20Edition,%20Justin%20Clarke.pdf [20] Il-Yeol Song, Mary Evans, E.K. Park. (1995) A Comparative Analysis of Entity-

Relationship Diagrams. [Online].

Available: https://www.cin.ufpe.br/~in1008/aulas/A%20Comparative%

20Analysis%20of%20Entity-Relationship%20Diagrams.pdf

[21] Cristiano Politowski, Fabio Petrillo, Yann-Gaël Guéhéneuc. (2021) A Survey of Video Game Testing. [Online].

Available: https://ieeexplore.ieee.org/abstract/document/9463010

(36)

APPENDICES

APPENDIX A

Legacy Figures DB

API

Application

Game Editor

Camera

Figure 5: Initial Project Structure

(37)

APPENDIX B

DNC File Example 2

42,000 -1 0

2,120 --> 2,200 RightArmLeft 50

1

4,200 --> 6,600 SwipeRight 100

2

6,700 --> 9,100 SwipeLeft

100 3

9,200 --> 11,600 SwipeRight

100 4

11,700 --> 14,100 SwipeLeft

100 5

14,200 --> 16,600 SwipeRight

100 6

16,700 --> 19,100 SwipeLeft

100 7

19,200 --> 21,600 SwipeRight

100

(38)

8

21,700 --> 21,800 LeftArmRightDouble 150

9

24,100 --> 26,500 SwipeLeftDouble 200

10

26,600 --> 29,000 SwipeRightDouble 200

11

12

13

14

(39)

APPENDIX C

DB Design

User Username

Creates Plays

Dance

Name File path Number of downloads

Song

Name File path

Author 1

m m

1

Note that the following parameters common to every entity were excluded from the diagram for simplicity reasons. Those parameters are:

• ID (primary key)

• Creation date

• Modiﬁcation date

Figure 6: Entity-Relationship Diagram

(40)

APPENDIX D

Let’s Dance! Dikir Barat Client Images

Figure 7: Let’s Dance! Dikir Barat - Title Screen

Figure 8: Let’s Dance! Dikir Barat - Dance Explorer Screen

(41)

Figure 9: Let’s Dance! Dikir Barat - Dancing Screens

(42)

Figure 10: Let’s Dance! Dikir Barat - Results Screen

(43)

APPENDIX E

Similar Projects Images

Figure 11: Just Dance Game Images

(44)

Figure 12: Dance Central Game Images

(45)

Figure 13: StepMania Game Images

Figure 14: Dance Dance Revolution Game Images