Human tracking can now be done in a wide variety ways, ranging from simple geolocation by Global Positioning Satellite (GPS) or other sensors such as known WiFi and cell tower locations, to full body motion tracking. An in-depth look at different sensor types and tracking methods is
Figure 7.2: Four main constituent parts of the DAAC
covered in AppendixB.
At the cross roads of cost and tracking solutions sits the Microsoft Kinect v2 sensors. These sensors are particularly attractive for use within manufacturing cells as they do not require any fiducial markers or other sensors to be worn by the tracked user. This allows them to move unencumbered within the manufacturing cell, and does not require preparation or prior knowledge before a user can be introduced to the manufacturing cell. The sensors also have a large group of enthusiast developers writing software and programs for the Kinect sensors. This provides an large knowledge base that can be drawn on while developing new applications for the sensors.
Microsoft has ceased to support and develop for the Kinect systems, so although the existing Software Development Kit (SDK) and hardware will still work well together, any improvements will have to come from another source. Its possible that the Free and Open Source Software (FOSS) community will come together and write post process wrappers for the existing SDK but it is more likely that future work will encompass existing FOSS alternatives like libfreenect (librefreenect). Which are discussed in the next section.
7.2.1.1 Kinect V2 and the Robot Operating System (ROS)
The Kinect sensors are a conglomeration of different sensors that are coordinated and provide the end application with the required data sets (Figure7.3). The sensor data is controlled by the Kinect Drivers, and then combined and interpreted by the Kinect services layer. In turn, this provides the Kinect SDK with the combined data sets, such as skeleton data, that can then be harnessed by the application (Figure7.4).
One of the limitations of the Kinect v2 sensors is that a single computer can only host a single sensor, severely limiting the area that can be tracked to a single sensors capture area. If a user is to be tracked across an entire manufacturing cell, multiple sensors would be required, and
7.2 Methodology 119
Figure 7.3: Sensor Components of the Kinect v2 (IFIXIT,2018)
consequently, multiple computers each running one sensor.
A Microsoft product using multiple Kinects to track a large area would require multiple Windows licenses and hardware able to run at a correspondingly high enough specification. For something which would only be a sensor controller, this at first seemed expensive and somewhat excessive. Fortunately, the Open Source community presented an alternative in the form of an open Kinect project that ran on Linux (H. MARTIN, 2018). The libfreenect Linux project mirrored the
abstraction layers of the Windows software suite up to the driver level (JOSHBLAKE,2018), which
would provide the requisite data streams, but the service layer and the SDK layer would still need to be developed.
Using Linux means a reduction in cost, as most Linux distribution is FOSS, and can be run with much smaller processor overheads. Since libfreenect is open source itself, then the data stream could be culled at the driver level, which would free up even more processing resource, potentially reducing the hardware requirements even further.
There remains an ever prevalent issue within the nuclear industry about using open source code within the nuclear industry (VENTÄ et al., 2007). However even though the manufacturing
processes are strictly controlled by regulation and must conform to a wide range of codes and standards, there is little to suggest that Open Source software could not be used as part of the technological solutions for manufacturing. In fact many CPS and other systems are already running variations of Linux and other open source technologies.
That is not to say that providing the source code for the programs you are running would be entirely safe or recommended if these systems had access to any exposed networks. The ability to see the source code of these systems could allow external operators unique and precise attack vectors into the systems. This is normally mitigated within the open source community by having a critical mass of users and maintainers that update security issues as and when they are identified. This may not be possible for systems that are bespoke or have a small user base since the number of people reviewing and updating the code would be too low to ensure a satisfactory level of security.
The use of open source software in areas where the code would be widely used and supported would allow for anyone to update the software from anywhere and would also hugely increase the likelihood of identify and fixing vulnerabilities not just for your own company but for all companies that use that software. Clearly the selection of what systems are suitable for low cost and open source solutions will be difficult, and only made harder by ensuring that the proprietary systems a company use can communicate with the open source ones. These are issues for networking and computer system specialists and at this level outside the remit of this thesis.
Figure 7.4: Kinect v2 abstraction layers on Windows and Linux
A low powered brix computer (GIGABYTE, 2018) running ubuntu was connected to a Kinect
sensor. This provided the raw data which needs to be interpreted into a skeletal data set and could then be combined with other Kinect data sensors before being imported into the VE. To achieve this a Robot Operating System (ROS) wrapper program that harnessed the OpenNI NiTE 2.2 libraries was used to convert the sensor data into joint sets with Cartesian position vectors and rotation quaternions. The joint sets were then broadcast across a ROS network. The KinectTracker program (STEPHENREDDISH,2018) was capable of tracking multiple people, and transmitting
their skeletal data as individual frames out across the ROS network.
A single Kinect can only track over a limited area and from a single direction, which means a lot of human movement is occluded by the body itself. To track a larger area multiple Kinects and brix computers were used across a network (Figure7.5). The ROS was used to broadcast each frame of skeletal data from each brix on the network. Each frame was then imported into a global space hosted on one of the brix themselves. As each sensor was positioned independently of each other to maximise coverage it meant that the global space would have as many skeleton data sets as there were Kinect sensors.
The solution to this problem was attempted by creating areas of overlap for the kinect sensors themselves, which would mean they would be recording the skeleton data at the same time and location. The difference in the skeletal position and rotation could then be used to find the vector differences in the sensors location to each other. When taken across the network this meant the
7.2 Methodology 121
sensors relative positions and rotations to each other could be calculated. This would also remove the common issue of line of sight trackers that is encountered through object occlusion, preventing a sensor from recording a users location. Multiple sensors from multiple angles should ensure that enough data is collected on the user to ensure that their location or location of their joints is known. This only becomes unreliable if the tracked individual becomes occluded from all angle, which could happen if the user climbed within another object such as a box or tube. At this point the tracking systems would fail completely, and the user would be untracked.
The brix that was combining the skeletons sets into a global tracked space, could then run those position and rotation vectors as transforms on the data sets from those sensors, and a global space with multiple skeletons aligned on top of each other would be accomplished. The final step of taking the average result of all the transformed data sets for each frame would render a single skeleton data set in a global frame of reference.
Unfortunately despite the Research Engineers (REs) best efforts the final rotation quaternion calculations proved more than the hardware could complete in real time, and the optimisation techniques beyond his abilities. A global space with multiple aligned skeleton data sets was obtained and was readily viewable across the ROS network using the programme reviewer, but without the homogenisation into single user skeleton data sets it would prove impossible to import into a VE.
The RE decided to use Quaternions as a method of storing the rotational joint data as it is industry standard to do so within the world of computer game development. The reason for this is that cartesian coordinate methods suffer from something known as gimble lock. This is where, through rotation of the object, two axis become aligned and as such a rotation in one angle is now the same as in another. Quaternions avoid this issue, but using a fourth angle, which is in the imaginary rotation plane. The imaginary quaternion plane, can be thought of in a similar fashion to that of the imaginary plane in complex numbers. The fourth imaginary rotation is often annotated as w i n a coordinate set of i,j,k,w and is used to prevent the possibility of gimble lock.
Despite its early promise of network utility and lightweight platform, the problem of integrating multiple Kinect 2 sensors into a single global space proved insurmountable. Fortunately a group at the university of Ulm were developing a similar tool on the windows platform (RIETZLERet al.,
2016). The team were kind enough to grant the RE permission to use the early development binaries for the DAAC project. The system worked in a very similar fashion to the ROS networked solution but ran on the Windows platform (Figure7.5).