Prior works [62, 87] as well as our survey insights (described earlier in Section 3.1.3 of Chapter 3) reveal that gym-goers are interested in automatically tracking their ex- ercise behavior and prefers to obtain personalized feedback on their performance. Past literature is, however, silent on the preferred timescales and frequency for such feedback–e.g., whether users would prefer to receive feedback during the ongo- ing repetitions in a set, at the end of individual sets or collectively at the end of an entire exercise session. As such, a future smart gym application should have the following capabilities: (i) distinguish between multiple people exercising si- multaneously in the gym, (ii) unobtrusively monitor exercises performed by each individual and obtain deeper insights on various facets of exercising, (iii) provide personalized feedback to the individuals to improve the exercise effectiveness and prevent injuries.
For realization of such a smart gym application, we assume that individuals exer- cising in the gym are using earables and the exercise equipment/machine is attached with cheap IoT sensor devices. The earables are equipped with a microphone, iner-
tial sensors (accelerometer, gyroscope), bio-sensors (heart rate, body temperature) and are paired to a smartphone. The IoT device attached to the exercise equip- ment (e.g., dumbbells, barbells, weight machines) have embedded accelerometer, gyroscope and magnetometer sensors. A custom built smartphone application has a Sensor Recorder process that records the sensor data from both the devices and a Send Data module that periodically transmits the sensor data to a backend server over the WiFi network. This App also has a Feedback Receiver that receives audio inputs/feedback from the server and relays it to the earables.
The backend server executes the required smart gym analytics components. In the backend, there is a Sensor Listener module for obtaining sensor data from both the earable and the equipment-sensor. Once the sensor data is obtained, the Sig- nal Correlatormodule checks for the correlation between the earable sensor stream and equipment sensor stream to determine who is working out with which exercise equipment. The correlated sensor data pairs are then fed to the Exercise Analytics module, which identifies the type of the exercise performed and determines more fine-grained aspects such as the exercise intensity, correctness, heart rate variation for different exercises. Then, the Feedback Determiner module utilizes these ana- lytics to determine the appropriate timing and the audio feedback to be sent to the earable device.
Figure 4.1 illustrates the architecture of the system with the sensor devices, server components and flow of the analytics pipeline. In this work, we mainly focus on the two components outlined in red-dotted lines. Note: For a clear repre- sentation, the figure depicts only a single-user scenario. In a practical setting, there will be multiple people exercising and thus multiple streams of both dumbbell and earable sensor will be streamed simultaneously to the backend sever.
4.3
Dataset
We conduct real-world studies, at our University gym, in which the participants performed a variety of weight-based exercises. The studies were approved by our Institutional Review Board (IRB-19-088-A078(919)). For the study, we recruited 12 (8 males, 4 females) university students and staff. Each study session involved multiple individuals performing exercises concurrently.
Sensor Devices Used: For obtaining sensor data, we used the following devices: (i) eSense Earable device1, which the subjects wore on their left ear, (ii) Cosinuss
One2 earphone, worn by subjects on their right ear and (iii) a multi-sensor device
(DA14583 IoT Sensor3) to attach to the exercise equipment (e.g., dumbbells, ex-
ercise machines). For the eSense earable, we used only the left-side earbud which has the capability to stream inertial sensor (accelerometer and gyroscope) data as well as receive audio inputs. The Cosinuss One device has in-built sensors to record heart rate and body temperature. These devices are paired with a smartphone and we developed an android application that simultaneously connects to these devices over Bluetooth Low Energy (BLE) and records sensor data and ground truth labels such as exercise performed, set count and amount of weight lifted.
Targeted Exercises: For the study, we focused on collecting data for 9 different ex- ercises (listed in Table 4.1). This involved six free-weights exercises performed with dumbbells (both upper and lower body exercises) and three exercises performed on weight-based machines (we utilize a multi-purpose cable pulley machine).
Overall Study Procedure: Prior to data collection, the gym equipment (dumbbell and weight machine) was instrumented with the DA14583 IoT Sensor device. The subjects who consented to participate in the study visited the gym and they were first briefed about the study procedures. The participants were given the eSense
1eSense– http://www.esense.io/
2Cosinuss One– https://www.cosinuss.com/products/one/
Table 4.1: List of exercises and targeted primary muscle groups
Exercise Name Primary Muscle Group Exercise Equipment
Biceps Curls Biceps Dumbbells
Triceps Extension Triceps Dumbbells
Lateral Raise Shoulders Dumbbells
Side Bend External/Internal Obliques Dumbbells
Goblet Squats Quadriceps Dumbbells
Lunges Glutes, Hamstrings, Quadriceps Dumbbells
Standing Cable Lifts Abs Cable Pulley Machine
Bent Over Side Lateral Shoulders Cable Pulley Machine
Upright Cable Row Traps Cable Pulley Machine
earable (to be worn on their left ear) and the Cosinuss One (to be worn on their right ear).
A study session involved multiple users (varying from 2 to 4) who performed each exercise set concurrently. In a session, the subjects performed 3 sets of 10 repetitions of each of the 9 exercises. Note: for the cable pulley machine exercises, data was collected only when two people were exercising concurrently. Out of the three sets of each exercise in a session, the subjects concurrently performed the “same” exercise for 2 sets and for the last set, they alternated between “different” exercises. When performing each exercise set, all the subjects (exercising simulta- neously) started exercising at the same time. However, the exercise set ending times varied depending on each individual’s exercise pace. Overall, we collected 680 sets (of 10 reps each) of exercise data. All exercises performed by participants were video recorded for obtaining the ground truth. On an average, an exercise session per subject lasted for about 48 minutes. For participating in the study, we provided each participant a monetary compensation of $10. Table 4.2 summarizes the details of the user study.
Additional Small-scale Study: In addition to the actual user study, we also con- ducted a small-scale study at the gym to collect data for additional variety of free- weights exercises as well as for heavier weights. The main motivation for this study is to understand the role of earables in distinguishing between exercises with simi-
Table 4.2: Summary of real-world multi-user concurrent exercise dataset collected from University gym
Study at University gym No. of participants 12 (8 males, 4 females)
Age Variation 21-40 years
Self-rated expertise 5 (Novice); 7 (Intermediate); 3 (Expert)
No. of exercises 6 dumbbell exercises (3 upper-body, 3 lower-body) and 3 weight-machine exercises
No. of concurrent users
Concurrent user count varied from 2 to 4 2 users only (374 sets)
3 users only (162 sets) 4 users only (144 sets) No. of sets of same/different
exercise performed concurrently
Same exercise (452) Different exercises (228) Total no. of exercise sets 680 sets (10 repetitions each) Average duration of exercise
session across subjects 48 minutes
lar dumbbell kinetics (e.g., Squats and Deadlifts) as well as exercises with different body postures (e.g., lying down for Weighted Crunch). For this study we recruited two people (in different sessions) who were well-experienced in weight-based train- ing. In this session, they performed 6 different exercises namely, (a) Seated Bar- bell Shoulder Press, (b) Inclined Chest Flyes, (c) Dumbbell Triceps Kickback, (d) Weighted Crunch, (e) Barbell Deadlifts and (f) Alternating Bicep Curls. Compared to the previous set of dumbbell exercises which all had a “standing” posture, these exercises either have a “seated” or “lying down” posture or uses barbells instead of dumbbells. Both subjects performed 3 sets of 10 repetitions of each exercise. Additionally, they also performed 2 sets of 8 reps each of both Biceps Curls and Lateral Raiseexercises with heavier weights (both 10kg and 14kg). In this study, we collected a total of 44 sets of data.