Smart IoT monitoring and real-time control based on autonomous robots, visual recognition and cloud/edge computing services

(1)

vrbvnjnnbbttbbtbt

"SMART IOT MONITORING AND REAL-TIME CONTROL BASED ON AUTONOMOUS ROBOTS, VISUAL

RECOGNITION AND CLOUD/EDGE COMPUTING SERVICES"

Programa de Doctorado: ENERGÍAS RENOVABLES Y EFICIENCIA ENERGÉTICA

Autor: Marouane Salhaoui

Cartagena (2021)

(2)

Smart IoT Monitoring and Real-Time Control Based On Autonomous Robots, Visual Recognition and Cloud/Edge Computing Services

ABDELMALEK ESSAADI UNIVERSITY FACULTY OF SCIENCE AND TECHNIQUES

TANGER / MOROCCO

POLYTECHNIC UNIVERSITY OF CARTAGENA UPCT / SPAIN

DOCTORAL THESIS

(Year 2021)

Presented By:

MAROUANE SALHAOUI

Directors:

Antonio Guerrero-González, Mounir Arioua

Co-Directors:

Francisco J. Ortiz, Ahmed El Oualkadi

Thesis Title:

"SMART IOT MONITORING AND REAL-TIME CONTROL BASED ON AUTONOMOUS ROBOTS, VISUAL RECOGNITION

AND CLOUD/EDGE COMPUTING SERVICES"

Accredited research institution:

• Laboratoire des Technologies de l’Information et de la Communication de ENSA de Tanger (Morocco)

• Departamento de Automática, Ingeniería Eléctrica y Tecnología Electrónica, UPCT Cartagena (Spain)

(3)

Abstract

In the fourth industrial revolution in which we are immersed, new technologies are being introduced in production processes, such as the use of Unmanned Vehicles (UVs) data collection in large surfaces, and the use of the Industrial Internet of Things (IIoT). The main keys to integrate this new technology in the industry is to face the challenge of making the IT network compatible with its machines, including interoperability, fog and cloud computing, security, decreasing latency and improving data accuracy and quality of service.

Smart industrial platforms require multiple synchronized processes that require low latency and higher reliability to achieve the necessary performance.

In addition, Artificial Intelligence (AI) methods applied to IIoT must be able to address these issues as well as other parameters such as network deployment and resource management.

The issues of high-latency and unreliable links between the cloud and Industrial IoT endpoints are significant challenges. Each fog and edge application may have different latency requirements and may generate different types of data and network traffic. Such generated data can be photos received from an UV system. The latter can be connected to other control system, being used both to perform enhancements and to make decisions based on the captured photos.

This type of connection is sensitive in terms of accuracy and latency, as the whole platform must decide quickly and with certainty.

One of the solutions to overcome the latency challenge is the fog/edge architecture. This architecture can also be a viable solution regarding the interoperability barrier between interconnected systems. Fog computing extends computation and storage to the edge of the network and presents an effective tool for integrating new complex interconnected processing systems.

The constraint of interoperability can be overcome by adopting advanced software deployed in the edge and fog installed in an IoT gateway. This software interacts simultaneously with the different systems involved through different protocols. However, the choice of an IoT gateway is crucial in terms of latency and accuracy, as it is at the heart of processing and transmitting data to the different systems and platforms and considered the interface of junction between the physical level and cloud. The latter also affects performance as it must ensure that data is transferred, processed and returned at speeds that meet the needs of the application.

(4)

We address all these challenges by considering appropriate protocols and software for interoperability and connectivity constraints and we discuss the performance some appropriate IoT devices capable of providing minimal response time.

Deep Learning (DL) services can be deployed near requesting users and the cloud only intervenes when additional processing is required, significantly reducing the latency and cost of sending data to the cloud for processing. In this thesis, we propose novel approaches to solve the latency issue by deploying intelligence at the edge that pushes DL computations from the cloud to the edge enabling various distributed, low-latency and reliable intelligent services.

The main benefit of the proposed approaches is the integration of cloud services into a control loop to improve a platform’s decision making and the performance of an industrial control system. Cloud AI services are also integrated into a drone control loop as an input that helps improve the monitoring capability to find and track stationary and mobile objects.

In this work, we evaluate the latency and accuracy of different systems involved and we propose an intelligent algorithm to select the appropriate AI technology for the scenario to be monitored. This proved to be crucial in deciding the best source of artificial intelligence to be used to achieve the specified goals at each stage in real time. The proposed intelligent algorithms offer a compromise between latency and accuracy.

(5)

Resumen

En la cuarta revolución industrial en la que estamos inmersos, se están introduciendo nuevas tecnologías en los procesos productivos, como el uso de vehículos autónomos (UVs) para recogida de datos en grandes superficies y el uso del Internet Industrial de las Cosas (IIoT). Las principales claves para integrar esta nueva tecnología en la industria es afrontar el reto de compatibilizar la red informática con sus máquinas, incluyendo la interoperabilidad, computación en la niebla/nube/borde (fog/cloud/edge computing), la seguridad, la disminución de la latencia y la mejora de la precisión de los datos y la calidad del servicio.

Las plataformas industriales inteligentes requieren múltiples procesos sincronizados que exigen una baja latencia y una mayor fiabilidad para lograr el rendimiento necesario. Además, los métodos de Inteligencia Artificial (IA) aplicados a la IIoT deben ser capaces de abordar estas cuestiones, así como otros parámetros como el despliegue de la red y la gestión de recursos.

Los problemas de alta latencia y enlaces poco fiables entre la nube y los puntos finales del IoT industrial son retos importantes. Cada aplicación de niebla y borde puede tener diferentes requisitos de latencia y puede generar diferentes tipos de datos y tráfico de red. Estos datos generados pueden ser imágenes recibidas de un sistema UV, por ejemplo. Este sistema puede a su vez conectarse a otro sistema de control, utilizándose tanto para realizar mejoras en el proceso como para tomar decisiones basadas en las imágenes capturadas. Este tipo de conexión es sensible en términos de precisión y latencia, ya que toda la plataforma debe decidir con rapidez y seguridad.

Una de las soluciones para superar el reto de la latencia es la arquitectura basada en la niebla/borde (fog/edge). Esta arquitectura también puede ser una solución viable en cuanto a la barrera de interoperabilidad entre los sistemas interconectados. La computación en la niebla extiende la computación y el almacenamiento al borde de la red y presenta una herramienta eficaz para integrar nuevos sistemas complejos de procesamiento interconectados.

La limitación de la interoperabilidad puede superarse adoptando un software avanzado desplegado en el borde y la niebla instalado en una pasarela de IoT. Este software interactúa simultáneamente con los distintos sistemas implicados a través de diferentes protocolos. Sin embargo, la elección de una pasarela IoT es crucial en términos de latencia y precisión, ya que está en el centro del procesamiento y la transmisión de datos a los diferentes sistemas y plataformas y se considera la interfaz de unión entre el nivel físico y la nube. Esta última también afecta al rendimiento, ya que debe garantizar que los datos se

(6)

transfieran, procesen y devuelvan a velocidades que satisfagan las necesidades de la aplicación.

Abordamos todos estos retos teniendo en cuenta los protocolos y el software apropiados para la interoperabilidad y las restricciones de conectividad, y analizamos el rendimiento de algunos dispositivos IoT apropiados capaces de proporcionar un tiempo de respuesta mínimo.

Los servicios de Deep Learning (DL) pueden desplegarse cerca de los usuarios que los solicitan y la nube solo interviene cuando se requiere un procesamiento adicional, reduciendo significativamente la latencia y el coste de enviar los datos a la nube para su procesamiento. En esta tesis, proponemos enfoques novedosos para resolver el problema de la latencia mediante el despliegue de inteligencia en el borde que empuja los cálculos de DL desde la nube hasta el borde permitiendo varios servicios inteligentes distribuidos, de baja latencia y fiables.

La principal ventaja de los enfoques propuestos es la integración de los servicios en la nube en un lazo de control para mejorar la toma de decisiones de una plataforma y el rendimiento de un sistema de control industrial. Los servicios de IA en la nube también se integran en un lazo de control donde interviene un dron como una entrada que ayuda a mejorar la capacidad de monitorización para encontrar y rastrear objetos estacionarios y móviles.

En este trabajo, evaluamos la latencia y la precisión de los diferentes sistemas implicados y proponemos un algoritmo inteligente para seleccionar la tecnología de IA adecuada para el escenario a vigilar. Esto resulta crucial para decidir cuál es la mejor fuente de inteligencia artificial que debe utilizarse para alcanzar los objetivos especificados en cada escenario en tiempo real. Los algoritmos inteligentes propuestos ofrecen un compromiso entre latencia y precisión.

(7)

صخلملا

يف عمج مادختسا لثم ، جاتنلإا تايلمع يف ةديدج تاينقت لاخدإ متي ، اهيف كراشن يتلا ةعبارلا ةيعانصلا ةروثلا ةيسيئرلا حيتافملا لثمتت .ءايشلأل يعانصلا تنرتنلإا مادختساو ، ةعساو قطانم يف ةلوهأملا ريغ تابكرملا تانايب

ةهجاوم يف ةعانصلا يف ةديدجلا ايجولونكتلا هذه جمدل ةقفاوتم تامولعملا ايجولونكت ةكبش لعج يف لثمتملا يدحتلا

ينمزلا ريخأتلا ليلقتو نملأاو ةيباحسلاو ةيبابضلا ةبسوحلاو يليغشتلا قفاوتلا ةيناكمإ كلذ يف امب ، اهتزهجأ عم ةمدخلا ةدوجو تانايبلا ةقد نيسحتو )نومكلا(

ةددعتم ةنمازتم تايلمع ةيكذلا ةيعانصلا تاصنملا بلطتت ىلعأ ةيقوثومو ضفخنم ينمزريخأت بجوتست يتلاو ،

ءايشلأا تنرتنإ ىلع ةقبطملا يعانطصلاا ءاكذلا بيلاسأ نوكت نأ بجي ، كلذ ىلإ ةفاضلإاب .مزلالا ءادلأا قيقحتل دراوملا ةرادإو ةكبشلا تيبثت لثم ىرخأ تاملعم ىلإ ةفاضلإاب تلاكشملا هذه ةجلاعم ىلع ةرداق

اورلا تلاكشم دعت ءايشلأا تنرتنإو ةيباحسلا ةياهنلا طاقن نيب يلاع ينمزريخأت تاذ و اهب قوثوملاريغ طب

عاونأ هنع جتني دقو ةفلتخم نومك تابلطتم ةفاحلاو بابضلا تاقيبطت نم قيبطت لكل نوكي دق .ريبك يدحت ةيعانصلا يتلا تانايبلا هذه نوكت نأ نكمي .تاكبشلا ربع ةلوادتملا تانايبلا نم ةفلتخم نم ةملتسملا روصلا نم اهؤاشنإ مت

تانيسحت ءارجلإ همادختسا متي ثيح ، رخآ مكحت ماظنب ريخلأا اذه ليصوت نكمي .ةلوهأملا ريغ تابكرملا ماظن بجي ثيح ، نومكلاو ةقدلا ثيح نم ساسح لاصتلاا نم عونلا اذه .ةطقتلملا روصلا ىلع ًءانب تارارقلا ذاختلاو فنملا يساسلأا ماظنلا ىلع دكؤم لكشبو ةعرسب ررقي نأ هلمكأب ذ يف ةتبثم ، بابضلاو ةفاحلا يف ةرشتنم ةمدقتم جمارب دامتعا للاخ نم يليغشتلا قفاوتلا دويق ىلع بلغتلا نكمي تلاوكوتورب للاخ نم ةكراشملا ةفلتخملا ةمظنلأا عم نمازتم لكشب جمانربلا اذه لعافتي .ءايشلأا تنرتنإ ةباوب كلذ عمو .ةفلتخم إف ،

اهنإ ثيح ، ةقدلاو ينمزلاريخأتلاب قلعتي اميف ةيمهلأا غلاب رمأ ءايشلأا تنرتنإ ةباوب رايتخا ن

ةقبطلا( ةيداملا ةقبطلا نيب عطاقتلا ةهجاو ربتعت يتلاو ، ةفلتخملا ىنبلاو ةمظنلأا ىلإ اهلقنو تانايبلا ةجلاعم بلق باحسلاو )يرايعملا لاصتلاا جذومن يف ةقبط ىندأ،ىلولأا نمضت نأ بجي هنلأ ءادلأا ىلع اًضيأ ةريخلأا هذه رثأت .ة

قيبطتلا تاجايتحا يبلت تاعرسب اهتداعإو اهتجلاعمو تانايبلا لقن لاصتلاا دويقو يليغشتلا قفاوتلل ةبسانملا جماربلاو تلاوكوتوربلا يف رظنلا للاخ نم تايدحتلا هذه لك عم لماعتن

يشلأا تنرتنإ ةزهجأ ضعب ءادأ شقاننو ينمزلا ريخأتلا نم ىندلأا دحلا ريفوت ىلع ةرداقلا ةبسانملا ءا ىلإ ةجاحلا دنع لاإ ةباحسلا لخدتت لاو ، ةبولطملا نيمدختسملا ةمظنأ نم برقلاب قيمعلا ملعتلا تامدخ رشن نكمي

ملل ةباحسلا ىلإ تانايبلا لاسرإ ةفلكتو ينمزلا ريخأتلا نم ريبك لكشب للقي امم ، ةيفاضإ ةجلاعم هذه يف .ةجلاع

تاباسح عفدت يتلا ةفاحلا ىلع ءاكذلا رشن للاخ نم ينمزلا ريخأتلا ةلكشم لحل ةديدج بيلاسأ حرتقن ، ةحورطلأا ضفخنم ينمز ريخأت تاذو ةعزومو ةقوثومو ةعونتم ةيكذ تامدخ حيتي امم ةفاحلا ىلإ ةباحسلا نم قيمعلا ملعتلا

ةحرتقملا قرطلل ةيسيئرلا ةدئافلا لثمتت يف رارقلا عنص ةيلمع نيسحتل مكحتلا ةقلح يف ةيباحسلا تامدخلا جمد يف

كذلا تامدخ جمد اًضيأ متي .يعانصلا مكحتلا ماظن ءادأو يساسلأا ماظنلا يف مكحتلا ةقلح يف ةيباحسلا يعانطصلاا ءا

ةتباثلا ءايشلأا ىلع روثعلل ةبقارملا ةردق نيسحت ىلع دعاسي لخدمك ةلوهأملا ريغ تابكرملا

اهعبتتو ةكرحتملاو

ءاكذلا ةينقت ديدحتل ةيكذ ةيمزراوخ حرتقنو ةينعملا ةفلتخملا ةمظنلأا ةقدو ينمزلا ريخأتلا مييقتب موقن ، لمعلا اذه يف يعانطصلاا ءاكذلل ردصم لضفأ ديدحت يف مساح رمأ اذه نأ تبث .هتبقارم دارملا ويرانيسلل ةبسانملا يعانطصلاا لأا قيقحتل همادختسلا اطسو لاح ةحرتقملا ةيكذلا تايمزراوخلا مدقت .يلعفلا تقولا يف ةلحرم لك يف ةبولطملا فاده

ةقدلاو نومكلا نيب

(8)

List of Acronyms

AI Artificial Intelligence

AIS Automatic Identification System ANN Artificial Neural Network

API Application programming interfaces ASV Autonomous Surface Vehicle

AUV Autonomous Underwater Vehicle CAN Controller area network

CCM Cloud custom model CGM Cloud General Model

CoAP Constrained Application Protocol CPS cyber-physical systems

CPU Central processing unit

CSMA/CD with NDBA Carrier Sense Multiple Access / Collision Detection with Non-Destructive Bitwise Arbitration

CV Computer vision

DAyRA División de Automatización y Robótica Autónoma DNNs Deep Neural Networks

DP Deep Learning

DPM Dynamic Position Mode DVL Doppler Velocity Logger

(9)

ECM Edge custom model EI Edge Intelligence

ERP enterprise resource planning FN False Negative

FP False Positive

GPS Global Positioning System GPU Graphics Processing Unit HD High-definition

HVAC Heating, Ventilating, and Air Conditioning IaaS Infrastructure-as-a-Service

IETF The Internet of Engineering Task

ILSVRC ImageNet Large Scale Visual Recognition Challenge IM Inspection Mode

IMARS IBM multimedia analysis and retrieval system IoS Internet of services

IoT Internet of things IoU Intersection on union

IPM Image Processing Algorithm

IUCN International Union for the Conservation of Nature IUNO Interface for Unmanned Drones

M2M machine-to-machine

MASS Maritime Autonomous Surface Ships MES manufacturing execution systems

(10)

ML Machine Learning

MLaaS Machine Learning as a service MMM Main Mission Mode

MPA Marine Protected Area

MQTT The Message Queuing Telemetry Transport ND Not detected

NFC Near Field Communication NIC network interface controller

OPC UA Open Platform Communications Unified Architecture OWD One-way delay

PaaS Platform as-a-Service PC-G PC Gateway

PID Proportional–Integral–Derivative PV Process Variables

QoS Quality of Service

RFID Radio frequency identification RPI-G Raspberry PI Gateway

RTD Round-trip delay time S-G Siemens Gateway

SAAO Smart algorithm for autonomy optimization SaaS Software-as-a-Service

SAR Synthetic Aperture Radar

SHDL ScatterNet hybrid deep learning

(11)

SISO single-input–single-output SOA Service-oriented architecture SP Set Points

SVM Support vector machine TAS Time-aware scheduler TM Tracking Mode

TP True Positive

TSN Time sensitive networking

UAV Unmanned Autonomous Vehicle UVs Unmanned vehicles

VPL Visual Programming Language WFQ weighted fair queuing

WSN Wireless Sensor Network WVR Watson Visual Recognition

(12)

List of Publications

The work presented in this thesis has appeared in the articles reported below.

Journal papers:

(J1) Salhaoui Marouane; Guerrero-González Antonio; Arioua Mounir; Ortiz, Francisco J.; El Oualkadi Ahmed; Torregrosa Carlos L. 2019. "Smart Industrial IoT Monitoring and Control System Based on UAV and Cloud Computing Applied to a Concrete Plant" Sensors 19, no. 15: 3316. https://doi.org/10.3390/s19153316 (J2) Salhaoui Marouane; Molina-Molina J. C.; Guerrero-González Antonio; Arioua Mounir; Ortiz Francisco J. 2020. "Autonomous Underwater Monitoring System for Detecting Life on the Seabed by Means of Computer Vision Cloud Services"

Remote Sens. 12, no. 12: 1981. https://doi.org/10.3390/rs12121981

(J3) Molina-Molina J. C.; Salhaoui Marouane; Guerrero-González, Antonio; Arioua, Mounir. 2021. "Autonomous Marine Robot Based on AI Recognition for Permanent Surveillance in Marine Protected Areas" Sensors 21, no. 8: 2664.

https://doi.org/10.3390/s21082664

(J4) Benbarrad, Tajeddine; Salhaoui Marouane; Kenitar Soukaina B.; Arioua Mounir. 2021. "Intelligent Machine Vision Model for Defective Product Inspection Based on Machine Learning" J. Sens. Actuator Netw. 10, no. 1: 7.

https://doi.org/10.3390/jsan10010007

International conference papers:

(C1) Marouane Salhaoui, Mounir Arioua, Otman Chakkor, and Jihane Elaasri. 2017.

Performance Evaluation Survey of WSN Physical Layer. In Proceedings of the 2nd International Conference on Computing and Wireless Communication Systems (ICCWCS'17). Association for Computing Machinery, New York, NY, USA, Article 68, 1–5. DOI: https://doi.org/10.1145/3167486.3167557

(C2) Marouane Salhaoui, Mounir Arioua, Antonio Guerrero-González, María Socorro García-Cascales, "An IoT Control System for Wind Power Generators", 17th International Conference, IPMU, Published in Information Processing and Management of Uncertainty in Knowledge-Based Systems. Applications, Springer, Cádiz, Spain, 2018. https://doi.org/10.1007/978-3-319-91479-4_39

(13)

(C3) Soukaina Bakhat Kenitar, Salhaoui Marouane, Arioua Mounir, Ali Younes, and A. Guerrero Gonzalez. 2018. Evaluation of the MQTT Protocol Latency over Different Gateways. In Proceedings of the 3rd International Conference on Smart City Applications (SCA '18). Association for Computing Machinery, New York, NY, USA, Article 87, 1–5. DOI: https://doi.org/10.1145/3286606.3286864

(C4) Soukaina B.K., Ali Y., Mounir A., Marouane Salhaoui. (2019) Latency Assessment of MQTT Protocol in Transferring Data from the Field to the Cloud Over Different Gateways. In: Ben Ahmed M., Boudhir A., Younes A. (eds) Innovations in Smart Cities Applications Edition 2. SCA 2018. Lecture Notes in Intelligent Transportation and Infrastructure. Springer, Cham.

https://doi.org/10.1007/978-3-030-11196-0_71

(C5) S. B. Kenitar, M. Arioua, A. Younes, M. Radi and Marouane Salhaoui,

"Comparative Analysis of Energy Efficiency and Latency of Fog and Cloud Architectures," 2019 International Conference on Sensing and Instrumentation in IoT Era (ISSI), 2019, pp. 1-5, doi: 10.1109/ISSI47111.2019.9043738.

(C6) Yassine Yazid, Imad Ez-zazi, Marouane Salhaoui, Mounir Arioua, El Oualkadi Ahmed, Antonio Guerrero González. Extensive Analysis of Clustered Routing Protocols For Heteregeneous Sensor Networks. Third International Conference on Computing and Wireless Communication Systems, ICCWCS 2019, April 24-25, 2019, Faculty of Sciences, Ibn Tofaïl University -Kénitra- Morocco.

http://dx.doi.org/10.4108/eai.24-4-2019.2284208

(C7) Marouane Salhaoui, Molina-Molina, J. C, A. Guerrero-González, Antonio;

Arioua, Mounir; Ortiz, Francisco J.; El Oualkadi, Ahmed. Edge-Cloud Architectures Using UAVs Dedicated To Industrial IoT Monitoring And Control Applications.

IEEE- International Symposium on Advanced Electrical and Communication Technologies ISAECT2020, November 25th-27th, 2020 Ibn Tofail University, Morocco

(C8) Benbarrad Tajeddine; Salhaoui Marouane; Arioua Mounir. Impact of Standard Image Compression on the Performance of Image Classification with Deep Learning. ICDATA21 (International Conference on Digital Age & Technological Advances for sustainable development), 2021. 29 - 30 June 2021 Marrakech, Morocco.

(14)

List of Tables

Table 2.1. Main protocols used in the IoT field

Table 2.2. Comparison of Internet of Things (IoT) protocols Table 4.1. Confusion matrix.

Table 4.2. Specification of each machine environment.

Table 4.3. RTD test of 5200 samples from the OPC UA client to the OPC UA server (PLC) over different clients through different machines.

Table 4.4. RTD Test of 200 photos sent from the IoT gateway to the AR.Drone 2.0.

Table 4.5. RTD test of 100 samples from the IoT gateway to IBM Watson over different machines.

Table 4.6. Speed Test over the three gateways (S-G, RPI-G, PC-G).

Table 4.7. GPS coordinates of the area explored.

Table 4.8. Accuracy measurement in different platforms.

Table 4.9. Latency measurement in different platforms.

Table 4.10. Definition of mission stages.

Table 4.11. AI source preferences according to mission stage.

Table 4.12. RTD test of 300 samples of the Edge and Cloud model.

Table 4.13. Experimental SAAO results

Table 4.14. Summary of SAAO logs during the experiment

(18)

List of Figures

Figure 1.1. A three-layer IoT architecture based on: Device, Edge and Cloud for Predictive Maintenance (PM) [8].

Figure 1.2 Major limitations of current IoT platforms Figure 1.3 Mapping of interoperability levels to OSI layers Figure 1.4. Automation Pyramid vs Automation Network [43]

Figure 1.5. Capabilities comparison of cloud, on-device and edge intelligence [40]

Figure 2.1. Most common IoT architectures Figure 2.2. UV-IoT Architecture

Figure 2.3. Wiring diagrams in vehicles before and after the appearance of CAN Figure 2.4. Controller area network bus node

Figure 2.5. Node of the CAN bus system

Figure 2.6. Comparison of protocols for the exchange of messages: (a) MQTT; (b) MODBUS TCP.

Figure 2.7. The IEEE model (a); compared to the HTTP (b); the CoAP (c); the MODBUS TCP (d); and the MQTT (e).

Figure 2.8. OPC UA in the automation pyramid Figure 2.9. Architecture of the OPC UA Server Figure 3.1. Node-Red Platform

Figure 3-2: Deep learning in the context of artificial intelligence

Figure 3-3. Connections to a neuron in the brain. xi, wi, f (·), and b are the activations, weights, nonlinear function, and bias, respectively

Figure 3.4 Simple neural network example and terminology. (a) Neurons and synapses.

(b) Compute weighted sum for each layer.

Figure 3.5. Training and inference comparison Figure 3.6. Six-level rating for edge intelligence

Figure 3.7. IoU equation, Red is ground truth bounding box and green is predicted bounding box

Figure 3.8. Latency between two terminals in a network

Figure 3.9. OPC UA delay in OPC UA client server in an Ethernet network Figure 3.10 Video transmission system delay sources.

Figure 4.1: Proposed UAV-IIoT Platform

Figure 4.2. Development design of autonomous IIoT flight

Figure 4.3. Node-RED flow in the IoT gateway including the path from the PLCs to the UAV, from the UAV to IBM Watson, and from Watson to the control center.

Figure 4.4. SCADA Industrial concrete plant with a typical concrete formula.

Figure 4.5. AR.Drone 2.0 mission in the concrete plant.

Figure 4.6. Communication process in the fog layer.

Figure 4.7. Path used by the drone to execute the mission in a concrete plant.

Figure 4.8. Dataset used to train the custom model in WVR service: (a) Shows images used to train the Mixed class; (b) Shows Images used to train the Normal class.

Figure 4.9. Watson visual recognition test of new images not used in the training phase.

Figure 4.10. Node-RED flow and WVR results of an UAV photo.

Figure 4.11. OPC UA delay in OPC UA client server in an Ethernet network.

(19)

Figure 4.12. Node-RED flow used to calculate round trip latency (OPC UA Client to the OPC UA Server).

Figure 4.13. OPC UA client-server RTD to read one bit through different machines.

Figure 4.14. (a) Simulation results of CPU load (%) versus OPC UA RTD (ms) in the S-G;

(b) Simulation results of CPU load (%) versus OPC UA RTD (ms) in the RPI-G.

Figure 4.15. Probability density function of the delay of the drone connected to the gateway when successive pictures from PC-G and RPI-G are taken.

Figure 4.16. Probability density function of the delay of the drone connected to the gateway when successive pictures from S-G are taken.

Figure 4.17. CPU Load while taking successive photos and writing them in a folder in the PC-G.

Figure 4.18. CPU Load while taking successive photos and writing them in a folder in the RPI-G.

Figure 4.19. CPU Load while taking successive photos and writing them in a folder in the S-G.

Figure 4.20. Probability density function estimation of IBM WVR latency to classify an image located in the IoT gateway.

Figure 4.21. Proposed AUV-IoT Platform Figure 4.22. Proposed hardware architecture.

Figure 4.23. Node intercommunications and concurrent threads in the IoT gateway.

Figure 4.24. Communication between platforms.

Figure 4.25. Fan mussel recognition training: defining a fan mussel bounding box in different cloud services.

Figure 4.26. Pictures used for custom CV model training.

Figure 4.27. New specimen detection using the IBM Python API.

Figure 4.28. Triangular similarity using a single camera [47].

Figure 4.29. Closed control loop for object detection and tracking.

Figure 4.30. Basic closed-loop system with sensor and actuator delays.

Figure 4.31. Mission generated in IUNO and uploaded into AUV.

Figure 4.32. Deploying the platform to initiate the mission. AUV submarine connected to a buoy via a DSL cable.

Figure 4.33. Specimen detection and positioning in IUNO.

Figure 4.34. Communication edge cloud. (a) Training and inference in the cloud; (b) training in the cloud, inference in the edge.

Figure 4.35. Latency in the proposed platform within the cloud architecture.

Figure 4.36. Edge architecture latency in the proposed platform.

Figure 4.37. Cloud-based custom models for detecting new specimens.

Figure 4.38. BUSCAMOS-VIGIA framework.

Figure 4.39. BUSCAMOS-VIGIA ASV.

Figure 4.40. Platform’s communications in the tracking algorithm.

Figure 4.41. SAAO diagram.

Figure 4.42. Calculation of acceptable latency limits. Main ASV camera point of view.

Figure 4.43. Comparison of three different clouds vision API detection of boat in Los Nietos port (Murcia, Spain).

Figure 4.44. Types and number of vessels used to train the vision custom models.

Figure 4.45. Performance of the cloud custom model object detection in discerning different boat types.

(20)

Figure 4.46. Performance differences between the Edge and the cloud custom models.

Figure 4.47. Cloud and edge custom models for detecting new vessels.

Figure 4.48. Latency of more than 300 samples.

Figure 4.49. Images analysed. Cloud/edge results comparison

Figure 4.50. Scale experiment. Equivalence of area and distance from integral reserve (Islas Hormigas) to base station (right) and equivalent area in Mar Menor (left).

Figure 4.51: Edge (left) / cloud (right) trained model recognition tests.

Figure 4.52. Start of mission (MMM) of surveillance of area equivalent to integral reserve.

Figure 4.53. (a) Stopped vessel detected. Start TM mode. (b) Tracking Mode (TM) test during the experiment.

Figure 4.54. Wind energy IoT communication architecture Figure 4.55. Hardware Setup

Figure 4.56. Data flow between different systems and across different protocols.

Figure 4.57. Checking OPC UA connection using UaExpert Software

Figure 4.58. Communication between the PLC 1512 and IBM Cloud through OPC UA protocol using Node-RED installed the industrial Gateway IOT2040.

Figure 4.59. Dashboard Data of wind Sensors in the IoT2040 Gateway Figure 4.60. Dashboard data wind sensors in the IBM Watson Platform.

(21)

CHAPTER 1 ---

Introduction

---

This chapter presents the background, motivation and main contributions of this thesis. It presents an overview of using computer vision and AI in IoT monitoring and its applications. The limitations are highlighted of using AI in IoT monitoring and control and its main constraints as motivations for the presented work. Subsequently, the main contributions of this thesis are presented. Finally, the organization of this thesis is detailed.

2.3 Background

Emerging new market demands and autonomous technologies such as IoT are moving the environment of manufacturing companies towards smart factories. The fundamental idea of IoT is a system where physical objects are enhanced with embedded electronics (RFID tags, sensors, etc.) and connected to the Internet. Thus, IoT is based on both smart objects and smart networks [1]. The devices in the IoT network can be employed for collecting information based on the use cases. These include retail, manufacturing industries, autonomous vehicles, smart grid, etc. These IoT devices can be used for tasks such as tracking items and objects, remote monitoring, and fully autonomous robots. It is reported that the amount of IoT devices has been growing every year with the predicted number of devices by 2025 reaching 75.44 billion [2].

(22)

The use of IoT has become ubiquitous and IoT devices are common in many fields. The integration of IT and Operational Technology (OT) in the Industrial Internet of Things (IIoT) enables the “smart factory” concept. IIoT uses IoT devices and sensors to monitor machines and environments to ensure the highest performance of equipment and processes.

In practice, IoT has stimulated the factories and the governments to launch an evolutionary journey toward the fourth industrial revolution called Industry 4.0. The first industrial revolution started with the introduction of mechanical manufacturing equipment, followed by a second that entailed the mass production of goods. Since the beginning of the 1970's and until today, the increasing automation and control of manufacturing processes through the use of electronics and computers is considered the third revolution (digital revolution). Leveraging IoT technology in the manufacturing environment leads to the fourth stage of industrialization [3].

At the heart of IoT and smart manufacturing is the basic principle of Industry 4.0, products being manufactured, components and production machines must collect and share data in real time. This leads to a shift from centralized factory control systems to decentralized intelligence.

The exchange of real-time data and information between different devices and parties is the key element of smart factories; this data could represent the state of production. Therefore, the next generation of smart factories will need to be able to adapt, almost in real time, to constantly technological options and regulations [4].

To perform effective predictive maintenance (PM), massive amounts of data are collected, processed and ultimately analyzed by machine learning (ML) algorithms. ML can be used on collected data to make predictions. Indeed, the accuracy of ML models depends primarily on the data collected.

Traditionally, IoT sensors transmit their data readings to the cloud for processing and modeling. Processing and transmitting massive amounts of data between IoT devices and infrastructure comes at a cost. Edge computing, in which sensors and intermediate nodes can process data, can reduce data transmission costs and increase processing speed. These techniques can reduce the amount of data sent to the cloud for processing, however there are potential accuracy trade-offs when ML algorithms use reduced data sets. Another approach is to move ML algorithms closer to the data to reduce the amount of data transmitted [5].

Visual recognition technologies based on artificial intelligence (AI) and the Internet of Things (IoT) can offer a detection capacity close to human capabilities [6]. The AI cloud services allows training of customized ML models that are able to classify the received data or detect individual objects in a given image along with their bounding box and label. There are many different cloud APIs for computer vision, e.g., IBM, Google, Microsoft Azure and Amazon. They all

(23)

provide fairly similar capabilities, although some emphasize object recognition, Amazon, or building custom models, like Microsoft Azure and IBM.

The strength of these cloud APIs is their ability to develop custom models rapidly and download trained custom models to deploy them on the edge for real-time applications and low-latency requirements [7-8].

When computing is deployed at the edge for real-time data processing, accuracy is also of paramount importance. Further, it is also clear that for data reduction, the edge or device is mainly exploited. However, since the initial training is computationally intensive, the cloud is still used in most of the proposed techniques for model training. In cases where a dedicated edge node is not available, network devices can also be exploited.

1.1.1 APPLICATIONS

Many fields and industries are using IoT as part of their architecture today.

In the following, we will look at some of them and how IoT can be used to further improvements.

A. Smart Factory

The main concept of Industry 4.0 is smart manufacturing and IIoT, where the component, product and machine will exchange data on the basis of real time [9].

Since exchange of data between different devices in real time is the main element of smart factory, this information can be considered as control, production status, supplier and customer order feedback information, material movement, simulation. Smart factory will provide the customer with service and smart product, which will be connected to a network based on IoT. The smart factory gathers and scans data from a related smart application and the product.

B. Smart Vehicles

A fully autonomous vehicle can be defined as a vehicle that is capable of perceiving its environment, deciding on a route to its destination and driving it.

It is a smart car or robocar that uses a variety of sensors, computer processors and databases such as maps to take over some or all of the driving functions of human operators. In a few words, an autonomous (or driverless) car can also be defined as a vehicle that relies on a combination of Internet of Things (IoT) devices (e.g., sensors, cameras, and lidars), appropriate software, and artificial intelligence to move without a human operator [10].

C. Smart grid

(24)

Today most of the power supply system is manually operated, and due to some human error, there is loss of power. These small losses result in massive outrage of power supply. This loss can be brought under control, and a 100%

efficient power transfer system can be designed using IoT, and it is known as the Smart grid. It is a fully automated system based on blockchain technology, which is entirely robust & encrypted. This power is divided into channels for each individual, and this channel is wholly encrypted with its stash key, which is to be decrypted. This results in an equalized power supply throughout the grid without any power loss [11]. Given that millions of end users will be involved in the processes and information flows of smart grids, the high scalability of these methods becomes an important issue. To solve these problems, cloud computing services emerge as a viable solution by providing reliable, distributed and redundant capabilities on a global scale. Moreover, the implementation of an intelligent network application on top of mixed cloud and edge processing middleware is able to achieve higher performance by leveraging edge node processing and data aggregation to reduce communication with the cloud environment [12].

D. Monitoring environmental parameters

Environmental monitoring, as an integral part of smart campuses, is an application that describes any activity in a surrounding area to monitor the quality of an environment [13]. It is used in the assessment of any risk that may be posed to the environment and humans. The applications of IoT in environmental monitoring are vast: Industrial site monitoring, seabed monitoring, sea or ocean monitoring, environmental protection, extreme weather monitoring, water safety, endangered species protection, commercial agriculture, etc. In these applications, sensors detect and measure every type of environmental change [14].

E. Smart Waste Management

One of the major issues that modern cities are facing is waste management.

It consists of multiple processes like managing and monitoring waste, transport, collection, disposal, etc. These processes are costly and time-consuming. One can optimize these processes to reduce cost, which can be used for solving other issues that smart cities can be deal with. For instance, various sensors can be installed in places like trucks or cans of garbage, which can detect type and amount of garbage [15].

F. Smart agriculture

(25)

Thanks to the IoT, it is possible to meet the food needs of a constantly growing population. The analysis of smart agriculture data, i.e., land condition, weather situation, and soil type, collected from the IoT network, can provide practical information if used in combination with the data captured by sensors, which measure the level of water resources, heat, humidity, chemicals, water stress, pump condition, etc. This allows farmers to use fertilizers, water and pesticides in the most accurate amounts, at precise positions and with efficient scheduling to improve crop yields. Smarter water use, such as monitoring and supervising the capacity, location, timing and period of water flow based on data analysis, increases irrigation efficiency and thus reduces costs [16].

G. Smart Home

To reduce user’s interference in controlling and monitoring home settings as well as home appliances, smart home is an emerging application [17]. A smart home provides many features for the user like measuring home conditions (i.e., light intensity, temperature, heating, etc.), operate home’s Heating, Ventilating, and Air Conditioning (HVAC) appliances and control them with reduced human interaction [18]. Paper [19] presents an example of procedure to develop a smart home by combining IoT with cloud computing and web services, use a platform for implanting intelligence in actuators as well as in sensors and facilitates interaction within smart things using cloud services.

H. Weather Forecasting

To predict the state of the atmosphere for a future time and for a given location, weather forecasting is very important. Weather forecasting and monitoring consist of a collection of data, assimilation of data, and forecast presentation. Sensors at weather station used to sense humidity, temperature wind speed, the moisture of soil, the intensity of light, radiation, etc. Data coming from these sensors is huge in size and difficult to monitor. The integration of this sensor infrastructure with cloud increases its storage and computational capabilities. It also provides effective solutions for monitoring and presentation of data [20].

I. Health Care

Sensors of pervasive healthcare applications make use of cloud computing and IoT to allows a machine-to-machine communication regardless of the location [21]. Nowadays, in modern hospitals, various body sensors are used to measure and monitor physiological data of the patients. This sensitive collected data is maintained for future diagnosis of patient. In some hospitals, this data is maintained at the local database. Due to this, doctors who are called to handle

(26)

critical cases unable to analyze disease. After visiting to the patient only they can give proper treatment. However, using the cloud service, this issue can be solved i.e., data of patients can be maintained and shared with doctors who are abroad, so that they can treat the patient, independent of location [22].

1.1.2 IoT monitoring and controlling

The rising trends of the Internet of Things (IoT) paradigm are attracting increasing attention from both academia and industry for their highly emerging applications of smartly connecting the surrounding things or objects without human intervention [23]. Collecting information from the surrounding environment to analyze, control, and making correct action is the main idea for IoT. To interchange data, IoT resources using internet makes use of multiple interconnected technologies like wireless sensor network (WSN) and radio frequency identification (RFID). IoT consists of smart objects, which can be read, locate, address, and control through the internet using RFID, wireless LAN, or some other means [24]. In recent time, IoT is getting more attention due to the advancement of wireless technology. The basic idea is due to variety of objects such as Sensors, RFID, actuators, Near Field Communication (NFC), mobile phones, etc., which can interact with each other by having a distinct address.

Artificial Intelligence (AI) may greatly support Internet of things in different ways for physical (PHY), data link (MAC), network, transport, and application layers. AI cloud computing is the fusion of machine learning (ML) capabilities of AI with cloud-based computing environments, enabling connected, and intuitive experiences. AI has become more affordable through the use of cloud platforms.

The affordable cost, coupled with cloud providers promoting AI as having a widespread value, leads to concerns that the technology will be misapplied.

There are different IoT architectures adopted in research and development works. The three-layer IoT architecture shown in Figure 1.1, consists of a sensing or device layer, which is also call as physical object layer whose main purpose is sensing and data collection [25]. The second layer is the edge layer, its role is to perform data transmission over the network, including sometimes being responsible for preprocessing and data storage before sending the data to the cloud. The edge layer is also an appropriate place for ML deployment, allowing the frameworks to be implemented using hybrid architectures.

A layer exists for the primary processing of data. Data can be stored and processed by high-performance servers. Predictive maintenance (PM) can monitor machine health to determine likely component failure. ML optimization models are deployed to help make intelligent decisions about which production parameters to monitor.

(27)

Figure 1.1. A three-layer IoT architecture based on: Device, Edge and Cloud for Predictive Maintenance (PM) [25].

1.1.3. Advantages of Using AI in the cloud

AI technology is being applied in most cloud services to drive interest in application development. Typical offerings combine the ability to leverage AI services at a lower cost with important data management systems that provide the source of the data, and thus the source of the models.

Cloud-based AI solutions are different; however, they have some commonalities, as well as benefits and limitations. First of all, the great benefit is that these systems are inexpensive to operate. To drive an AI application, payment can be made per hour used of each service. This is perhaps the largest benefit of cloud AI, really bringing AI down to the level of a basic enabling technology.

Public clouds also offer cheap data storage. Real databases or storage systems can be leveraged as data input into AI applications. Finally, they all provide SDKs and APIs that allow us to integrate AI features directly into applications, and they support most programming languages.

While AI is a technology that allows a machine to simulate human behavior, ML is a subset of AI that allows a machine to automatically learn from prior data without explicit programming. ML as a service (MLaaS) is an umbrella concept

(28)

of various cloud-based platforms that cover most infrastructure issues such as data pre-processing, model training, and model evaluation, with further prediction.

ML offers several advantages, including accurate predictions, speed, automation, and scalability [26]. The ML method uses algorithms to analyze data, find rules and abstract the rules into models to classify and predict unknown data. It can significantly enhance the efficiency of data processing and the accuracy of prediction results by applying machine learning methods to monitoring complex IIoT data. Furthermore, it can also detect abnormal conditions of the IIoT to the greatest extent possible and reduce the loss of properties and lives [27]. On the one hand, Deep learning (DL) structures the algorithms into multiple layers in order to create an “artificial neural network (ANN)”. Complex DL models are being developed, and in addition, CE research is accelerating to provide more computational resources for DL models to support more applications [28]. Prior to the use of ML in IIoT, the cognitive ability (to learn the environment) of machines was simply a predefined heuristic.

However, sophisticated ML algorithms have improved the cognitive capability by finding patterns in the data and making predictions [29].

1.1.4. Constraints

IoT is a novel paradigm to interconnect massive wireless devices, which has the potentials to be applied in diverse applications and fields. However, due to a huge number of wireless devices in IoT networks, many technical challenges need to be addressed, Figure 1.2 presents some limitations of current IoT platforms.

Figure 1.2 Major limitations of current IoT platforms

(29)

Based on the presented limitations, there are imperative directions that have to be considered in the future for IoT research studies.

A. Scalability

The growing idea of IoT which generates a tremendous amount of data for processing and storage guide to enormous leap in the forthcoming year, and hence it becomes insistent on making the scalable system. The vast application of IoT has increased the number of devices being connected to the internet, which meets the concern to consider various complications that are arising in connectivity [25]. Different sources like the internet, social media, machine, and many other devices generate data. Thus, special attention must be given for transportation, access, storage, and processing of these large sets of structured and unstructured digital data [25].

B. Interoperability

As the data sharing among smart devices is increasing day by day, it is necessary to manage these data transfer properly among the system [30].

Interoperability can be considered as the potentiality of two systems to communicate, exchange information, or program, or transfer the data among each other and to implement the given data [31]. It is the exchange of information among different computers through wide area networks or local area networks.

It is critical for IoT as most of the communication takes place as a machine to machine [32].

C. Real-Time (Delay)

Meeting real-time latency requirements depends on how data is collected and processed [33]. This becomes more severe as IoT applications that involve rich data types such as images evolve. In addition, developing real-time analytics in the cloud is nearly impossible to achieve.

A variety of IoT applications require local analytics. For instance, in the context of IIoT, to quickly turn on or off a piece of equipment in a production environment can prevent a catastrophic situation. Analytics depend on ML algorithms that are computationally expensive for some tiny sensors. In addition, the power consumption of small sensors has been one of the main concerns even before the emergence of ML in IoT. Thus, achieving the goal of real-time with a sensor cloud architecture seems ambitious.

The limited computational capacity of sensor nodes is a major challenge.

Therefore, a hybrid architecture to implement computationally intensive tasks such as training on the cloud and model deployment for prediction on the sensing node has emerged. However, this approach also presents challenges in

(30)

the case where models require retraining based on new data. In this case, all new data must be moved to the cloud, which incurs costs in terms of latency, energy consumption, and network resource usage [34].

D. Accuracy

There are many possibilities for designing a Neural Network (NN) model, provided that different hyperparameters in the network provide a different level of accuracy. Particularly, a high accuracy model requires more memory than a low accuracy model due to the number of parameters. The metric used to measure accuracy depends on the domain in which the ML algorithm is applied.

In IIoT, a traditional approach to data collection is to stream data from sensing devices to the cloud where it is processed and modeled. Sensing devices generate huge amounts of data, continuously or periodically, often in a very short period of time. For instance, in one second, thousands of records can be generated by one machine [35]. According to the Cisco cloud index (2013-2018), an automated facility can generate one terabyte of data per hour. For this purpose, approaches such as sampling, compression, filtering are used to reduce the size of data. These techniques reduce the amount of data transmitted to the cloud.

However, there are potential accuracy trade-offs for ML models that use reduced datasets, as the accuracy of ML models depends primarily on the data collected.

The accuracy of models trained on reduced data should also be a concern when optimizing for energy consumption and latency. This is more important in deep learning approaches that require more data to be trained.

In current IIoT systems based on edge computing, edge devices can only perform lightweight computing tasks. To enable edge devices and servers to perform more complex tasks with higher data processing performance and lower latency, edge intelligence (EI) is applied to the IIoT edge to make the devices and servers intelligent. However, an AI model can be trained to make predictions and decisions with high accuracy; however significant amounts of training and verification data are required. For edge devices, training and operating the AI model is challenging due to limited computing and storage resources.

E. Security

As the IoT trend inflates multiple interconnections and device heterogeneity, it eventually generates cyber-attacks. Thus, data security is one of the most critical issues. As there is an increase in the number of connected devices, there are possibilities of cyber-physical security vulnerabilities that can be exploited by various attackers [36]. Security is a key pillar of the internet, which is the main challenge of IoT.

Therefore, it is necessary to learn from these incidents: industries are now the target of attackers and there is an urgent need to address this issue. IIoT is

(31)

sometimes thought of as the integration of operational technology (OT) and information technology (IT), with OT comprising the factory network where manufacturing is performed and IT comprising the enterprise network [37].

These two components have different security requirements, which must be addressed to prevent compromise of the IIoT infrastructure.

1.2 Motivation

Although IoT has been widely used in the above applications, a number of critical limitations have hindered the use of AI in various implementations. These limitations tend to mainly affect the system response time and the efficiency of the proposed solution. The various aspects of DL that need to be improved in an IoT concept include smart algorithms with improved efficiency and support for better platforms. For this reason, the following issues had to be investigated to overcome these limitations of using AI in IoT architecture.

A. Interoperability

Recent advances in manufacturing technology, such as industrial internet, cyber–physical systems, AI (Artificial Intelligence), and machine learning have driven the evolution of manufacturing architectures into integrated networks of automation devices, services, and enterprises. One of the resulting challenges of this evolution is the increased need for interoperability at different levels of the manufacturing ecosystem. Interoperability means the ability of two or more parties, machine or human, to make a perfect exchange of content.

The IoT is a dynamic global network infrastructure with self-configuring capabilities based on interoperable, standard communication protocols, enabling all objects to communicate with each other. The application of IoT to the industrial domain has given rise to a new research area called the Industrial Internet of Things (IIoT). IIoT is a new service-oriented industrial ecosystem that uses networked interconnection of industrial resources, data interoperability, and system interoperability to enable flexible resource allocation, rational process optimization, on-demand process execution, and rapid adaptation of environments [38]. In general, interoperability is defined as the ability of heterogeneous networks, applications, or computer components to exchange and use information, i.e., to speak and comprehend each other [39]. Suited to the IoT context, this refers to the ability of heterogeneous components to exchange and share data and functions to achieve a desired goal (defined by a use case or application), regardless of the communication protocol used or the format of the exchanged data.

Up to only a few years ago the communication systems for industrial automation aimed only at real-time performance suitable for industry and

(32)

maintainability based on international standards [40]. The Industry 4.0 concept has the flexibility to achieve interoperability between the different industrial engineering systems. To connect the different industrial equipment and systems, the same standards and safety levels are required. Open Platform Communications Unified Architecture (OPC UA) is a machine-to-machine (M2M) communications protocol developed to create inter-operable and reliable communications and is now generally accepted as standard in industrial plant communications [41].

The domain extends from software, devices, and control systems on the factory floor to Internet-based cloud platforms that provide various on-demand services.

A successful implementation of interoperability in smart manufacturing would therefore result in efficient communication and error-free data exchange between machines, sensors, actuators, users, systems and platforms. The architecture and platforms used by machines and software packages are a major challenge in this regard. A better understanding of the subject can be obtained by studying industry-specific communication protocols and their respective logical semantics.

To be more precise and in accordance with [42], three levels of interoperability can be specified to achieve horizontal interoperability between different components, each level covering a different facet of interoperability and communication:

• Technological interoperability is given if the two components are physically connected to each other and can transmit any type of information on the transmission layer [42]. This level of interoperability concerns the lower transmission layers of the OSI model (i.e., the physical, data link, network and transport layers) and the corresponding protocols.

• Syntactic interoperability is ensured if the data format, including encodings, as well as the communication interface format are clearly defined and agreed upon between the two components [42]. The abstract term "communication interface format" refers to the protocol used on the application layer and provided as communication interface by the respective component. As with technological interoperability, it is not necessarily required that the two components agree on the same protocol, as long as there is a possibility of adapting two different protocols (the same applies to the format and encoding of the data embedded in the protocol(s)).

• Semantic interoperability is ensured if both components agree on the same information model describing the topic of the exchanged information as well as the provided communication interfaces [42] (i.e., the meaning of the

(33)

Figure 1.3 depicts the mapping of these three interoperability levels defined in [42] to the OSI layers. It should be noted that this mapping is not strict and that the boundaries between the OSI layers may vary depending on the implementation of the application concerned.

Figure 1.3 Mapping of interoperability levels to OSI layers

The evolution from traditional hierarchical models of enterprise control system integration, or automation pyramid, to integrated networks of smart devices, cloud services, and extended enterprises requires seamless communication and information exchange between heterogeneous and traditionally silent systems (Figure 1.4).

Different types of interoperability problems may arise, due to the diverse nature of interactions in the emerging automation networks. Thus, there is a need for vertical interoperability between devices and shop floor automation services, as well as horizontal interoperability between enterprises and cloud service platforms.

(34)

Figure 1.4. Automation Pyramid vs Automation Network [43]

B. Low-latency and ultra-Reliability

The industrial smart facility requires multiple synchronized processes that require low latency and higher reliability to achieve the necessary performance [44]. Furthermore, AI methods applied to IIoT must be able to address these issues as well as other parameters such as network deployment and resource management [45]. Nevertheless, the competence and usefulness of DL-based IIoT scenarios are still in the evolutionary phase, requiring exclusively the strict low- latency and ultra-reliability requirements of IIoT. Therefore, further research efforts are needed in this direction to establish a theoretical and practical background for DL-IIoT to ensure low-latency and ultra-reliable communication.

Fast and efficient computing is another key feature that can affect not only latency and reliability but also many other performance parameters in smart industries. As mentioned earlier, IIoT requires powerful and useful tools to compute big data obtained from various processes and analyze them on specific platforms including servers, transmission media, and storage.

Intelligence at the edge should push DL computations from the cloud to the edge as much as possible, enabling various distributed, low-latency and reliable intelligent services, as shown in Figure 1.5.

Smart IoT monitoring and real-time control based on autonomous robots, visual recognition and cloud/edge computing services