Our society is a typical complex system composed of many components, interacting on diferent scales and levels.391 For example, human beings, the main components of
our society, are complex systems themselves.392 They communicate with others in
diferent forms of conversations, such as calling and emailing. Humans form larger- scale composites (organizations, communities and societies) with specifc complexities; these composites then associate and communicate with each other and with humans for both fun and proft.393 As a result of datafcation, more and more
such interactions are recorded and digitized by ubiquitous sensors (e.g., in smart phones), log fles of internet infrastructure elements, commercial transactions etc., basically turning many aspects of the lives of people and societies into big data. This data provides near real-time measures of the aforementioned complex systems. It contains patterns that can be used to fnd attributes that were previously difcult to detect.394
Many complex systems display so-called emergent properties. An emergent property is a property displayed by a complex system, that is not directly predictable from the
390 “(A)ny natural person who, in contracts covered by this Directive, is acting for purposes
which are outside his trade, business or profession”: article 2(b), Council Directive 93/13/EEC on unfair terms in consumer contracts (Unfair Terms Directive), [1993] OJ L 95, p. 29-34.
391 Claudio J Tessone, ‘The Complex Nature of Social Systems’ in Bernardo Alves Furtado,
Patricia AM Sakowski and Marina H Tóvolli (eds), Modeling Complex Systems for Public Policies (IPEA 2015); John H Holland, Complexity: A Very Short Introduction (Oxford University Press 2014) ch 3.
392 Simon A Levin, ‘Ecosystems and the Biosphere as Complex Adaptive Systems’ (1998) 1
Ecosystems 431, 432.
393 Orlando Gomes, ‘The Economy as a Complex Object’ in Bernardo Alves Furtado, Patricia
AM Sakowski and Marina H Tóvolli (eds), Modeling Complex Systems for Public Policies (IPEA 2015).
properties of that systems’ elements.395 In the context of this chapter, relevant
examples of an emergent property are distinguishable patterns like a behavioural convention among members of a group, the response of the human body to illness that resembles other individuals’ responses, or the segregation of individuals into groups. Emergent properties of a complex system can become apparent by observing the entire system or by observing interactions between the elements in the system. The interactions within human societies are carried out through the behaviour of its individual elements: human beings. Human behaviour in turn is based on both our cognition and our biology.396 Human cognition perceives our environment, including
others’ behaviour, values, beliefs, attitudes and intentions, and then to a large extent shapes our traits and social conventions, mainly through implicit or even automated processes; our biology guides other responses to our environment.397 Evidence shows
that our cognition works “efortlessly, and even unintentionally”.398 Human behaviour
is an emergent property of the human organism as a complex system; it lies at the root of conventions and segregation just as human biology lies at the root of our response to temperature changes or illness. Therefore, as datafcation covers more and more aspects of our lives and society, behavioural patterns of cognition and biology are encoded in data.
One of the fastest-developing techniques for the processing of data is Artifcial Intelligence (AI).399 The performance of AI has increasingly been proven to beat
human performance in certain felds, like playing Go and poker, making predictions,
395 Holland (n 391) ch 6; Here, “emergence” is used as shorthand for “higher-level order”. Note
that emergence has been called a “notoriously murky notion”. It is undecided whether it is related purely to human understanding or to underlying causality. Still, “If a system doesn’t exhibit higher-level order (…), it is not complex.” Ladyman, Lambert and Wiesner (n 374) 40–41, 58–59.
396 Biological and psychosocial systems of humans count as complex systems. Bar-Yam (n 372)
2–4.
397 James S Uleman, S Adil Saribay and Celia M Gonzalez, ‘Spontaneous Inferences, Implicit
Impressions, and Implicit Theories’ (2008) 59 Annual Review of Psychology 329, 330 <http://www.annualreviews.org/doi/10.1146/annurev.psych.59.103006.093707> accessed 21 March 2019 and the referenced literature.
398 James S Uleman, Leonard S Newman and Gordon B Moskowitz, ‘People as Flexible
Interpreters: Evidence and Issues from Spontaneous Trait Inference’ (1996) 28 Advances in experimental social psychology 211, 211
<http://www.sciencedirect.com/science/article/pii/S0065260108602397> accessed 5 August 2017.
399 Nils J Nilsson, The Quest for Artifcial Intelligence (1 edition, Cambridge University Press
and judging human character.400 Machine learning, a major subfeld of AI, provides a
number of cost-efective algorithms aiming to make sense of big data.401 Such
algorithms are roughly divided into three diferent categories: supervised learning, unsupervised learning and reinforcement learning.402 Supervised learning comprises
techniques that predict the value of a target variable given an input variable. An example is the automated recognition of handwriting in US ZIP codes: since each element of a ZIP code is a digit, predictions for each digit can be limited to an integer with a target value between 0 and 9. In unsupervised learning, the aim is to fnd patterns in the data such that certain variables can be identifed. An example is the analysis of a large customer database to fnd groups of “similar” customers for achieving market segmentation, without identifying those segments in advance. Finally, in reinforcement learning, a certain goal is pursued in a dynamic process without knowing beforehand whether or not the approach will lead to reaching the goal, and the learning process is driven by feedbacks. An example would be developing an algorithm predicting the best possible next move in a turn-based game like Go by playing a large number of games to their conclusion.
Just like the complex systems they analyse and represent, machine learning algorithms can exhibit emergent properties in their output. This appears to be the underlying cause of algorithms’ perceived bias: if an algorithm is trained using biased
400 AR Guess, ‘Artifcial Intelligence Had a Breakthrough Year in 2015’ (DATAVERSITY, 9
December 2015) <http://www.dataversity.net/artifcial-intelligence-had-a-breakthrough- year-in-2015/> accessed 20 March 2019; Tonya Riley, ‘Artifcial Intelligence Goes Deep to Beat Humans at Poker’ (Science, 3 March 2017)
<http://www.sciencemag.org/news/2017/03/artifcial-intelligence-goes-deep-beat- humans-poker>; BBC News, ‘Artifcial Intelligence: Google’s AlphaGo Beats Go Master Lee Se-Dol’ (BBC News, 12 March 2016) <http://www.bbc.com/news/technology-35785875> accessed 19 March 2019; Navin Sharma and others, ‘Predicting Solar Generation from Weather Forecasts Using Machine Learning’, Smart Grid Communications
(SmartGridComm), 2011 IEEE International Conference on (IEEE 2011) 551
<http://ieeexplore.ieee.org/abstract/document/6102379/> accessed 20 March 2019; Stephen F Weng and others, ‘Can Machine-Learning Improve Cardiovascular Risk Prediction Using Routine Clinical Data?’ (2017) 12 PLOS ONE e0174944, 9
<http://journals.plos.org/plosone/article?idd10.1371/journal.pone.0174944> accessed 21 March 2019; Wu Youyou, Michal Kosinski and David Stillwell, ‘Computer-Based Personality Judgments Are More Accurate than Those Made by Humans’ (2015) 112 Proceedings of the National Academy of Sciences 1036, 1039
<http://www.pnas.org/content/112/4/1036> accessed 21 March 2019.
401 Peter Flach, Machine Learning: The Art and Science of Algorithms That Make Sense of Data
(1 edition, Cambridge University Press 2012).
data, this bias can be reflected in its output.403 This can be understood as follows.
Human behaviour is characterised by traits and views at the scale of individuals, groups and entire societies. Shared traits lead people to associations, or the presumption of attributes, in accordance with those traits. These associations can develop conventions that are, in turn, expressed in individual and collective human behaviour. When applied to data sets resulting from datafcation, machine learning algorithms can build a mathematical representation of these traits, conventions and behaviours. Therefore, if these algorithms are then used to assign attributes to data subjects, these attributes may reflect these traits, conventions and behaviours. If controllers use the assigned attributes in such a way that data subjects sharing protected traits are treated diferently, this can have discriminatory efects. In those cases, algorithms can be said to be “racist”, “sexist”, or otherwise biased against groups of data subjects sharing protected traits in the sense that these data subjects are treated diferently when compared to other data subjects with otherwise similar traits. For example, extending special rebates to consumers who have bought alcohol but not to others, could be seen as discrimination against consumers who do not drink alcohol for religious reasons.
Thus, we propose that the concept of emergence can provide insights relevant to the subject of EU anti-discrimination law in relation to data processing algorithms. These insights are especially relevant to data subjects: being unaware of the information that controllers can obtain through algorithms can make it difcult to detect or escape discriminatory efects. Several instances of the successful deduction of sensitive traits from non-sensitive data have been published. For example, Kosinski et al. found that gender, racial origin, sexual orientation, political opinions and religious beliefs can be predicted by Facebook “likes” with more than 80% accuracy.404 Seneviratne et al. show
that they can predict users’ religion (and some non-sensitive traits) with over 90% precision in some cases by taking a snapshot of the apps that data subjects have downloaded to their smartphones.405
403 Jieyu Zhao and others, ‘Men Also Like Shopping: Reducing Gender Bias Amplifcation
Using Corpus-Level Constraints’, arXiv:1707.09457 [cs, stat] (2017) s 3 <http://arxiv.org/abs/1707.09457> accessed 21 March 2019.
404 Kosinski, Stillwell and Graepel (n 384) 5803.
405 Suranga Seneviratne and others, ‘Predicting User Traits from a Snapshot of Apps Installed
on a Smartphone’ (2014) 18 Mobile Computing and Communications Review 1, 6 <http://dl.acm.org/citation.cfm?idd2636244> accessed 20 March 2019.
Avoiding possibly discriminatory efects of algorithms is difcult for both data subjects and controllers because emergence is a fundamental property of complex systems. It is virtually impossible to eliminate all possible emergent properties from a data set because these properties may not be known in advance and they are captured in innocuous individual data points. External triggers are not necessary for emergence.406 Extensive eforts have been made to understand emergence, resulting
in theories and tools for complex systems such as nonlinear dynamics, fractal theory, and agent-based modelling.407 Based on these eforts, Feng et al. proposed a
theoretical framework to understand why machine learning algorithms can successfully identify patterns from the data containing the information of interactions within the complex system. They argue that by introducing non-linear interactions and optimization, machine learning algorithms themselves are complex systems, assimilating the dynamics of pattern formation from the complex system they represent.408 They also pointed out that the more exhaustive the available data,
the more accurately the patterns will be identifed: increasing datafcation will therefore increase the risk of the discovery of patterns coinciding with sensitive traits. Thus in the era of datafcation, it may be unavoidable that large data sets will contain patterns coinciding with sensitive traits because they cover “activities resulting from (protected) opinions or beliefs”.409 If these patterns can be found in collections of non-
sensitive data, the safeguards against the processing of sensitive data may become less efective.
406 Damon Centola and Andrea Baronchelli, ‘The Spontaneous Emergence of Conventions: An
Experimental Study of Cultural Evolution’ (2015) 112 Proceedings of the National Academy of Sciences 1989, 1989 <http://www.pnas.org/content/112/7/1989> accessed 19 March 2019 and S5 in the Supporting Information.
407 For examples, see Stephen H Kellert, In the Wake of Chaos: Unpredictable Order in
Dynamical Systems (University of Chicago press 1994); Jean-Francois Gouyet and B Mandelbrot, Physics and Fractal Structures (1 edition, Springer 1996); Volker Grimm and others, ‘Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology’ (2005) 310 Science 987 <http://science.sciencemag.org/content/310/5750/987> accessed 20 March 2019.
408 Qing Yi Feng and others, ‘An Exploratory Statistical Approach to Depression Pattern
Identifcation’ (2013) 392 Physica A: Statistical Mechanics and its Applications 889, 894 <http://linkinghub.elsevier.com/retrieve/pii/S0378437112009211> accessed 19 March 2019. For a formal treatment of the same thesis: Bar-Yam (n 372) ch 2 (especially pages 296-297).