New trends and challenges in the data world

27  Download (0)

Full text


New trends and challenges in the data world

March 2018


This study has been developed within the framework of the Aporta Initiative, developed by the Ministry of Energy, Tourism and the Digital Agenda, through, a Public Corporate Entity, and in collaboration with the Ministry of Finance and Civil Service. Contents and points of view expressed in this publication are the exclusive responsibility of its author, Carlos Iglesias. Aporta team does not guarantee the accuracy of the data included in the study. The use of this document implies the express and full acceptance of the general reuse conditions included in the following legal notice:




1.1. Big data and Artificial Intelligence 1.2. The decision algorithms

1.3. New data sources and Internet of Things 1.4. Distributed records and blockchains 2. CONCLUSIONS





We live in a data-driven present, where a significant part of our lives is online. Every aspect of our activity can be turned into a bunch of bits that travel through networks and add fuel to our day to day. Human beings have become great producers and data

collectors. Every second we send millions of emails and hundreds of thousands of

instant messages, we make thousands of purchases on the Internet or we produce tens of hours of multimedia content. To this we must add the increasing number of connected devices, such as wearables or sensors linked to Internet of Things initiatives.

90% of the data created in the history of humanity were produced during the last year and a 40% annual growth is estimated for the next decade. Experts expected that, by the year 2020, we will be producing every day a quantity of data equivalent to all the conversations among humans made throughout our history.

If we add to these future forecast our growing dependence on information, we could

assure without fear of error that the influence that the data have on our lives,

governments and businesses will continue to grow in the future.


In this report we will review some of the main technological and social trends related to

this new data-rich environment, as well as their opportunities and challenges. All of that

through the analysis of the conclusions drawn from the reports and recommendations

provided by several entities of reference in the data sector, such as the United Nations,

the World Economic Forum, the International Development Research Center, the

McKinsey & Company consulting or the World Wide Web Foundation, among others.



Linked data, enterprise data, data models, big data streams, neural networks, data infrastructures, deep learning, data mining, web of data, signal processing, smart cities, unstructured data, NoSQL, small data, data mining, data science, cloud computing, business intelligence, wearables, data scrapping, cluster analysis, real time data, open data, blockchain, transparency, predictive modelling, semantic web, data schemas, internet of things, pattern recognition, natural language processing, network analysis, personal data, data structures, sentiment analysis, open science, data visualisations, machine learning, pattern recognition, spatial analysis…

... the new trends related to open data are innumerable and we could fill hundreds of pages analysing their challenges. That is why, for this report, we have decided to make a selection of four specific trends that, although they already have a certain degree of maturity, still need a significant evolution, and where governments will also play a decisive role, as shown below.

In addition, all these new trends are developed in a data-rich environment that mean new challenges in order to maintain the control and privacy of personal data while technologies evolve. That is why, throughout this report, we will also highlight the privacy issues associated with these trends.


1.1 Big data and Artificial Intelligence

The combination of Big Data and artificial intelligence offers governments and organizations the possibility of using large amounts of data, both structured and unstructured, to improve their capacity to diagnose, understand and deal with their daily problems. To consider that we are working with Big Data, we need to meet specific criteria that include a large volume and variety of information that must be processed at high speed and through efficient analysis techniques that allow us to extract better knowledge, make better decisions and automate certain processes.

It is also important to point out that, in order to have an efficient management of Big Data, it will not be enough to simply access to large amounts of data, but it will also be necessary to have the technical infrastructure that facilitate data to be efficiently processed and the skills and knowledge necessary for the analysis of the obtained information.

1.1.1 Risk and opportunities

We are at the beginning of a new digital and social revolution. The possibility of analyse huge amounts of data from very diverse sources and almost immediately facilitate to define problems from a new perspective, which opens up a whole new world of possibilities that would not be possible using more traditional methods. Big Data also allows for easily detection of anomalous effects in large and complex systems, given us the possibility to detailed analyse and correct them, if necessary.

If we add to this the capacity that artificial intelligence gives to machines to adapt and use algorithms, so that they are able to make their own decisions according to the previously indicated parameters - or at least to clearly define the different scenarios for the final decision taking -, the fields of application and the benefits will be multiple. Several examples are listed below:


Employment and economic growth: although there is widespread concern about the potential damage that task automation can cause on the current labour market, the new data economy combined with the benefits of artificial intelligence is already giving rise to a new a wave of innovation in different economic sectors, such as digital home assistants. It is also a very useful tool to optimize other more traditional services such as medical services.

Improvement of public services: thanks to the combination of data and artificial intelligence, you not only achieve improvements in the planning of services, but also improve and simplify the transactions on which the public services operate and are sustained. However, there are also doubts about the convenience of public services that are too automated due to the limitations of artificial intelligence itself and the dependencies that this could cause.

Strengthening democracies: if it is used properly, artificial intelligence can also be a very useful tool in simplifying the complexity of governments and legal systems for citizens and facilitating their active participation. On the contrary, we can also find opinions warning about the danger that could be caused by governments and other third parties´ abuse, precisely, with the opposite aim.


1.1.2 Next challenges

The main challenges that governments will have to face in the near future, within this area, could be summarized in the following table.

Now we will explain each of the challenges shown in the figure:

1. Capabilities: the experience in Big Data and artificial intelligence field is still limited, scarce and difficult to find. Currently, the talent demand far exceeds the available supply. In addition, the limited existing talent is worryingly concentrated in certain countries and companies, restricting the possibilities of many governments and economies to benefit from the great upcoming revolution. Strengthening capacities in science, technology, engineering and mathematics will be a priority to resolve this current lack.

2. Infrastructures: both Big Data techniques and artificial intelligence have a high demand on infrastructures, including the permanent connectivity required to guarantee the ubiquity of data and processes, the storage of huge amounts of data and a high processing capacity for working with them. These dedicated infrastructures are generally complex and expensive to maintain, which is why many times they opt for outsourcing, creating strong dependencies and giving rise to certain non-trivial security risks.


3. Protection of privacy: as the number of new benefits and services available increases, thanks to the exploitation of Big Data and artificial intelligence, potential users will be pushed to expose more personal data in order to not be excluded. In general, there is a growing feeling of lack of protection and concern about the loss of control of our data and the lack of reliability in the companies and organizations that manages data. That is why we must promote completely transparent practices related to how personal data are managed and used by artificial intelligence, guaranteeing their security and privacy against the new possibilities offered by these techniques when inferring conclusions.

4. Ethical challenges: mainly caused by the current concentration of infrastructures and knowledge that generates reasonable doubts about the representativeness of ongoing developments. In addition, in most cases the technologies used behave like real black boxes whose operation is impossible to analyze. If the data and the artificial intelligence are destined to direct several aspects of our daily lives, it will be critical to be able to guarantee the transparency and accountability of these systems, including all perspectives.

1.2 The decision algorithms

As we have seen in the previous section, the amount of data produced constantly by humanity is growing impressively. At the same time, governments and organizations are also beginning to adopt a new decision-making strategy based on data and evidence. This is why it is increasingly common to rely on automated agents to extract the necessary knowledge of these data in an agile and efficient way, and, as a consequence, also facilitate - or even completely automate - the decision-making process.

These automated agents use algorithms and different formulas with a large number of application areas: search engines, classification and reputation systems, predictive models, content filtering, recommendation systems, etc. The algorithms have now the responsibility to make decision making not only more efficient but also more equitable. At the same time, governments and organizations must also take responsibility for maintaining final control over


the algorithms and the decisions they make or report, ensuring transparency throughout the process and the reliability of the results.

1.2.1 Risk and opportunities

The risks that we face when automating tasks and decisions using algorithms is mainly related to the possible discrimination of these decisions if we do not have the necessary guarantees.

This discrimination can appear in two different ways:

 Two individuals that should be considered equal for some aspects evaluated by the applied algorithm, but in practice they obtain different results.

 Two individuals that are manifestly different for the evaluation, but, because they share some other common indicators, are finally treated identically by the algorithm in a wrong way.

The main causes for this discrimination generally have their origin in some of the different stages during the algorithms conceptualization and development:

The input data used to feed the algorithms are incomplete, incorrect or low quality. If the data does not faithfully represent reality, then the results will not be the expected ones.

The rules applied to the input data analysis may not be sufficiently refined, or perhaps the algorithms have not been adequately trained in these rules, due to an insufficient variety of data. Therefore, once again, the result will not be correct.

Lack of the necessary contextualization to its correct operation, being adequate in some contexts but not in others.

 Lack of adaptation to the environment in which the algorithm operates to continue remaining relevant as it evolves.

We should add to the above the problem of a growing users’ distrust regarding the conditions of use and implicit consents that they are not able to interpret, increasing the doubts and


reticence about the functioning of these algorithms. In addition, there is a the lack of documentation and little transparency related to how these algorithms work, being even impossible for its own creators to explain its operation, in some cases.

With the aim of addressing all these problems, there is a global debate focused on the accountability and transparency of algorithms, as well as on the principles that should be followed for the development of responsible algorithms.

1.2.2 Next challenges

We are rapidly approaching an era in which the impact of algorithms on our daily activities will be a constant, in practically all areas. In order to make the algorithms transparent and fair, it will be necessary a series of simultaneous interventions in different technical, legal, social and political spheres.

The main challenges that governments will have to face in the near future, within this area, could be summarized in the following table.


Now we will explain each of the challenges shown in the figure:

1. Encourage social dialogue on the responsible management of algorithms and the risks associated with automated decision making, including the fundamental principles and values that public administration should always incorporate into the code and the algorithms operation.

2. Improve the quality and availability of the data produced by the administration so that they can be used to train and / or feed these algorithms. In this way, they can guarantee the reliable and neutral origin of the data and a greater transparency in their conceptualization and operation.

3. Establish a series of reference guidelines, standards and codes of conduct when developing algorithms for administration and public services that guarantee their integrity, ethics and independence.

4. Invest in audits and quality controls for all the algorithms - whether they are self- development or provided by third parties - with the aim of guaranteeing that their operation is equitable and corresponds to what is expected. Introduce other additional security measures such as human approval for any key decision finally.


1.3 New data sources and Internet of Things

The concept of Internet of things is based on direct communication between a variety of technological devices, all of them connected to each other. In today's environment, hardware is increasingly powerful and more affordable, while permanent connectivity is also more ubiquitous and cheaper. Thanks to the combination of both variables at the same time we are living the current great boom of a new generation of intelligent everyday objects connected, from light bulbs to thermostats, appliances or the increasingly popular personal assistants.

The growth rate of this trend is such that it is expected that by the year 2020 there could be an average of up to six devices connected for every person on earth, and in a very short time the communications between objects will have far surpassed the communications between humans. Expert also expected that these devices will generate a considerable qualitative change in the way we interact with each other and with the environment.

1.3.1 Risk and opportunities

In this case, both risks and opportunities are given by the same common cause: the hyperconnection offered by this set of elements and the associated overexposure of our lives and personal data to the external world. In addition, we must add that we are in a still incipient market and the maturity of these products is not very high.

On the one hand, we will enjoy a greater variety of services because our possibilities and selection options multiply. In addition, we will also improve the accessibility to these services, providing new and multiple interaction forms. This will facilitate the universalization of services regardless of the skills of the person interacting with them or the environment in which such interaction takes place.

On the other hand we will be opening a huge number of doors with direct access to our privacy through these devices, since many of them do not have just the security measures necessary to ensure the security of the information they manage, introducing a whole universe of potential new vulnerabilities in our homes.


In this scenario we can propose some general strategies that will be useful to reduce risks, while seeking to maximize the benefits:

 Minimize the collection of personal data to those that are essential, in order to ensure the provision of the specific service, always avoiding unnecessary data exposure.

 Allow data self-management, where the user has the ownership, control and final decision-making related to the publication of his information, never exposing data without his prior consent.

 Make sure that data are adequately anonymized by the devices, carrying out a dissociation process to avoid that users can be identified through the services provided, either individually or through the combination of several services.

 Ensure the appropriate information storage and protection, both on the device and in any associated cloud, always using encryption for any sensitive data and secure communication protocols for data exchange.

1.3.2 Next challenges

In the very near future, will have a growing number of sensors, devices and micro devices connected to each other to make life easier, which will generate a whole set of new social and technological challenges that must be addressed. The main challenges that governments will have to face in the near future, within this area, could be summarized in the following table.


Now we will explain each of the challenges shown in the figure:

1. The ownership of an increasing number of connected devices also carries a greater inherent risk because the potential security and privacy vulnerabilities also increase as our exposure through these devices grow. As a consequence, maintaining control over our data security and privacy can become a very complex task. Therefore, we should work with suppliers to strengthen consumers trust and transparency, improving the clarity of the terms of service offered to users and the flexibility in order to customize our privacy preferences and recover our personal information when we want. Clear regulations and guidelines related to information security and privacy will be necessary, in line with the change in the European Union through the implementation of the new General Regulation of Data Protection (GPDR). Other reference works to take into account in this area are the ISO/IEC 27000 standards family or the ISF Standard of Good Practice for Information Security.

2. As the number of services available to those connected through Internet of Things continues growing, the social gap between the most favored classes (largest data producers and consumers of services) and the least favored ones lowest data producers and less access to services) can also increase. This would generate new flows of both, data and services, import and export. The government will also be responsible for taking the necessary measures to avoid this gap and ensure equitable access to any new available service. There are several organizations that are already promoting initiatives in this area, such as UN Global Pulse, Data-Pop Alliance, Datax2 initiative, OD4D network, DataShift or GovLab.


3. The universe of existing sensors and connected devices from different manufacturers is continuously growing at an impressive rate. However, as long as they are not able to easily communicate with each other, their possibilities will be severely limited. We are at the right time to ensure the scalability of the internet of things and ensure that all these devices can easily communicate with each other thanks to the creation of common protocols and standards that enable the necessary interoperability of platforms. That is why the major standardization authorities ,such as W3C, ISO, IETF, ITU, IEEE and the European Commission, are already working on their different strategies to deal with this problem.

4. New challenges in infrastructures to face the growing number of elements connected simultaneously into networks. This growth could, at least initially, be assumed in terms of bandwidth, due to their generally limited position regarding the volume of this kind of communications. However, this large number of concurrent micro-requests could generate a saturation of the networks and, even, the wave space if the infrastructures do not adapt and evolve properly.


Despite the fact that blockchain technology has become popular in recent times, because of its relationship with the most popular cryptocurrency of the moment - that is bitcoin -, the reality is that this technology is applicable to any use case whose main purpose is data chains storage and management in a decentralized and distributed way.

Blockchain has a series of features that will provide advantages, as a useful technology in various fields of application: privacy, (quasi) anonymity, integrity, trust distribution, transparency, security, sustainability and open source.

While it is clear that its current most widespread application is related with the field of finance, specifically cryptocurrencies, blockchain can also be very useful for many other fields within governments, particularly for everything related to personal identification or the protection of personal data through the decentralization of privacy.


1.4.1 Risk and opportunities

Blockchain is a technology with great potential to totally transform our political systems and enable profound social changes, at the same time. This technology offers the possibility of transparent and immutable transactions, also guaranteeing their origin and traceability in a transparent, verifiable and accessible -for anyone whit an Internet connection- way. These characteristics make the blockchain to have great potential, in addition to contributing significantly to a greater availability of public data.

The improvements offered by this technology cover very diverse areas, such as the provision of public services, the authenticity of public records, the management of public sector data, the fight against corruption or electoral processes transparency, among others. There are already dozens of examples of entrepreneurs applying the concept of blockchain to innovate in fields as important as health or agriculture.

But, as also happens with any other disruptive technology in the maturation phase, not everything is advantageous, and we will also find some disadvantages and limitations to be overcome:

 The scalability problems, mainly due to the high computational and interconnection costs that each of the operations must support, and which requires high investments in hardware. To this we must add the high number of connections necessary in some use cases and the technology nature, which increases exponentially with no limits.

 The environmental impact associated with the high computational cost of calculations.

For example, just in the case of bitcoins as a whole, the cost currently exceeds the total energy consumption of a majority of the countries in the world, and it grows continuously.

The excessive centralization of each chain and the governance problems of the most successful use cases, since there is no authority or group officially responsible for maintaining and evolving the technology in order to face the challenges arised in the near future, such as deciding the ideal size of the blocks in the chain.


 The high complexity of the cryptographic processes, which, on the one hand, requires highly specialized talent to work with the basic technologies and, on the other hand, also means a barrier of entry for the potential end users, due to the difficulty to understand some of the underlying technology concepts.

1.3.2 Next challenges

Although blockchain has quickly become the fashionable technology and, at first glance, it give the impression of apparent simplicity, the reality is that, in the present, it is one of the most cryptic and misunderstood technologies regarding its potential beneficiaries.

In order for these decentralized data management technologies to become popular in the near future, it will be necessary to face another type of structural barriers to entry, related to several aspects:

Now we will explain each of the challenges shown in the figure:

1. The need to invest in training and development of new capabilities to meet the specific technical and cryptographic knowledge requirements that will be needed, and whose demand would increase significantly once the associated technologies became more popular and the use cases increase in proportion.


2. Raise awareness and improve citizens' electronic security practices, since one of the requirements for the correct functioning of blockchain technology is the proper management of end users' passwords (public and private). Citizens already show a clear trend towards bad practices in terms of technological security in general. If we add the new and complex concepts associated to blockchain technologies - that can be a bit intimidating at the beginning-, the result is that many organizations prefer to delegate security management to third parties or to devices created specifically for this purpose - as for example the physical or electronic wallet in the case of cryptocurrencies-. This gives rise to a new breach in the confidence chain blockchain is based on - some people is already taking advantage of it -, decreasing security at the end.

3. Blockchain is basically a decentralization technology. For that reason, it has a great potential to support the concept of highly decentralized government towards many of the governments of the world have been evolving. However, to make this possible, it will be necessary a greater capacity for institutional adaptation, both in the technological area - where significant reforms in infrastructures and hardware would be necessary - as well as in the organizational area.

4. Development of the regulatory changes necessary to support the growth of these technologies. As happen with other leading technologies, more and more countries have developed a specific legislation to protect their citizens' data, but emerging use cases are advancing many times ahead of regulation. Furthermore, this technology is destined to have a very important role in the use cases related to the identification and authenticity of the information, so it will also be fundamental to have the necessary legal validity.



Nowadays, each and every one of us generates an unprecedented amount of data through any of our daily activities. Thanks to this great availability of new data, in combination with the enormous evolution that is taking place in all technology areas, there is a whole world of new possibilities and tendencies to explore, like those analyzed in this report.

Big data, artificial intelligence, decision algorithms, internet of things and blockchain share a great unexploited potential to improve our lives, but as we have seen throughout this report, they also share a large number of new challenges to face before we can take advantage of its full potential.

At European continent level, the European Commission has already taken action and is launching various initiative to respond to many of these challenges.

Big data

As part of its strategy to build a European data economy, the Commission recognizes the role of big data in current technological innovation in order to improve public sector efficiency and productivity, while at the same time it provide a better response to current society challenges. Not surprisingly, data technologies and standards, including big data policies as well as open data and open access policies, are one of the five priority areas within the digitization initiative of European industry.

For this reason, the Commission expect ensuring free access and circulation of any non-personal data throughout the European Union, while at the same time safeguarding personal information in an appropriate manner. In addition, there are also plans to strengthen access to all public or publicly funded information, including data from private sources that could be considered as public interest goods, during 2018.

On the other hand, the Commission also recognizes the limitations that present infrastructures have to work with large amounts of data in the near future and, through communication on


Connectivity for a European Gigabit Society, they also consider the challenges that must be solved at structural level.

Artificial Inteliggence and algorithm

The Commission has launched the largest public financing plan for research into artificial intelligence existing so far through the Horizon 2020 program, with more than 700 million euros invested, and we must add an additional contribution of more than 2 billion from the industry. It is not surprising, therefore, that artificial intelligence is also one of the three fundamental technological pillars of digitalization strategy of European industry.

On the other hand, ethical considerations are also a recurrent topic within the European Robotics Forum. For this reason the Commission has launched a study on the challenges of automated algorithms for decision making, and, at the same time, we begin to see notorious cases in which the Commission declares against the conditioned and interested use of algorithms.

In addition, the comiccion try to respond to the challenge of public acceptance of artificial intelligence, bringing people closer to the current and real use cases, through the launch of the European Robotics Week.

Internet of things

According to a study from the Commission, the market value of internet of things in the European Union will exceed one trillion euros in 2020.

Over the last two years, the Commission has been conducting a series of initiatives in several complementary areas with the intention of taking advantage of all its potential, such as:

 The preparation of an internal working document on the strategy to drive Internet of Things area in Europe with the aim of improving its security, privacy and trust, working in three specific areas that include (1) the creation of the ecosystem to support it; (2) a vision focused on people and (3) the implementation of a single market.


 The creation of The Alliance for Internet of Things Innovation (AIOTI) to collaborate with all the agents involved in the market and facilitate the emergence of new business models.

 The revision of telecommunications rules in the European Union in order to accommodate a new open system for the identification and authentication of the objects that make up this new network.

All these activities are already becoming effective,driving interesting cross-cutting initiatives such as the ActivAge project, which aims to improve the quality of life of older people thanks to the possibilities of Internet of Things.


The Commission recognizes blockchain technologies as a key emerging area. They have also recently launched the EU Blockchain Observatory and Forum and it is expected that

investment in the area may exceed 300 million euros in the next two years. There is also an open call to reward the innovative social uses of this technology.

Currently the Commission has launched a study to assess the creation of a common blockchain infrastructure for the whole Union. In the near future, the Commission also plans to start exploring the potential of this technology to improve pan-European services in areas as diverse as taxes, customs, registers, the environment and health.

With this review we wanted to offer an example of what actions you need to perform in order to start facing pending challenges in the European context. However, each European Union country and government also has an important and independent work to be done with the objective of adapting their society, legislation, procedures and infrastructures in accordance with their own internal challenges. In this way, you can benefit from the social and economic improvements offered by these new technological trends.



Gartner, Open for businesses: Learn to profit by Open Data. Gartner, Open Data is coming to the enterprise.

Web Foundation, A Smart Web for a more Equal Future – Artificial Intelligence.

McKinsey, Big Data: The next frontier for innovation, competition, and productivity. next-frontier-for-innovation

ACM, Statement on Algorithmic Transparency and Accountability. policy/2017_usacm_statement_algorithms.pdf

IEEE, Ethically aligned design – Version One: Request for input. World Economic Forum, The future of jobs. Analysis Group, Global economic impacts associated with artificial intelligence. nomic_impact_of_ai.pdf

Accenture, Why artificial intelligence is the future of growth. us-en/_acnmedia/PDF-33/Accenture- Why-AIis-the-Future-of-Growth.PDF

Nesta, Digital Democracy – The tools transforming political engagement. Cathy O’Neil, Weapons of Math Destruction.

Web Foundation, A Smart Web for a more Equal Future – Algorithmic Accountability. ProPublica, Big Data’s disparate impact.


RFID Journal, That “Internet of Things” thing.

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, Data for Development – What’s Next?

Web Foundation, A Smart Web for a more Equal Future – Personal Data.

World Economic Forum, Unlocking the Value of Personal Data: From Collection to Usage. rt_2013.pdf

United Nations Development Group, Big Data for Achievement of the 2030 Agenda: Data Privacy, Ethics and Protection.

Oxfam, Responsible Data Management. management

ICT Works, How to Develop and Implement Responsible Data Policies. UN Global Pulse, Risks, Harms and Benefits Assessment.

United Nations, Resolution on the right to privacy in the digital age. Privacy International, Tracking the global state of surveillance.

International Data Responsibility Group, People First in a Digital Age. United Nations, Data protection regulations and international data flows. McKinsey, How blockchains could change the world. the-world

IDRC, Blockchain - Unpacking the disruptive potential of blockchain technology for human development.

(26) European Union, General Data Protection Regulation (GDPR)

European Commission, Communication on Data-Driven Economy.


European Commission, Communication on digitising European Industry.



Figure 1. The importance of data in the future.

Figure 2. Trends in open data sector.

Figure 3. The challenges of big data and artificial intelligence.

Figure 4. Principles for the development of responsible algorithms.

Figure 5. The challenges of decision algorithms.

Figure 6. The challenges of new data sources and Internet of Things.

Figure 7. The challenges of Distributed records and blockchain.




Related subjects :