FUNCIONES DEL PERSONAL DEL COLEGIO

The Shannon Diversity Index proved to be a reliable statistic for selecting a subset

of continuous numerical signals from data produced by networks representing several

Original Equipment Manufacturers (OEMs), model years, and vehicle models. It

measures the number of unique members in a population and the proportion of the

total population each represents [48]. Equation 7 is a method for calculating this

statistic given a population of decimal numbers.

H = −

n

X

i=1

x ∗ log2(x) (7)

The variable x represents the proportion of payloads in a signal which have

100 observed payloads. A particular numerical interpretation is observed 10 times

in the total of 100 observations. Then that particular part of the sum would be

(10/100) × log2(10/100) ≈ −0.332. The upper limit n represents the total number of

unique numerical values present in the population of payloads the signal is composed

from. H is the Shannon Diversity Index. The base of the logarithm used affects the

units which H is expressed. H is expressed in bits when using log2, decimal digits with

log10, and natural units with loge; a decimal digit is about 31₃ bits [116]. The range

of H when using log2 is 0 to the bit width of the signal. The minimum occurs when a

signal is only one value repeated in every observation. The maximum Shannon Index

occurs when two conditions are met. First, the number of unique values n observed

in the signal but be 2 to the power of the signal’s bit width: 2nbit width_{. Second, each}

unique value xi must represent an equal proportion of the total observations. Ex-

pressed in another way, the signal must be composed of 2nbit width _{unique numerical}

values and each value must be observed the same number of times.

The Shannon Index is used in this context with the assumption that signals de-

scribing continuous numerical processes are more diverse than signals related to cate-

gorical data like turn signal activation. Selecting a subset of only continuous numerical

signals can be achieved by sorting the total population of signals produced during

tokenization by Shannon Index. A tunable percentage parameter tsubset is used to

select a percentage of the total population with the largest Shannon Index values. In-

creasing this percentage parameter increases the quantity of signals that are labeled

as continuous numerical signals in the signal clustering phase presented in Section

5.3. However, this also increases the possibility that signals which are not actually

continuous numerical signals become mislabeled. As with all unsupervised clustering

methods, there is no ‘magic’ setting for a parameter which is optimal for all data sets.

plots of Shannon Index values for studied vehicles shown in Fig. 27. The critical piece

of information is the Interquartile Range (IQR) expressed by the box in each plot.

The box represents 50% of observed Shannon Index values for signals produced by

the vehicle and extracted during lexical analysis. Each whisker is either 1.5 × IQR

from the nearest edge of the box or the minimum or maximum observed Shannon

Index. Observations that exceed either whisker are considered potential outliers [117].

Fig. 27 may help provide insight into how to best leverage the Shannon Index

statistic to perform subset selection. This chapter focuses on using a population

percentage threshold to select a subset of signals generated during lexical analysis

which represent the most diverse signals. An alternative approach may be to perform

subset selection by excluding signals which do not meet a minimum Shannon Index

threshold. Fig. 27 shows approximately what percentage of the total population of

signals extracted from each vehicle would be excluded for different threshold values.

The percentage based approach proposed in this section ensures a relatively consistent

quantity of signals are included during semantic analysis. The alternative minimum

threshold approach may cause the majority of signals from one vehicle to be included

in the subset while excluding the majority in another vehicle. Comparing how subset

selection would be effected by using a minimum Shannon Index threshold of three

with the signal populations from vehicles five and eleven demonstrates how such a

scenario might occur in practice.

It is not immediately clear whether a percentage based, ‘fixed’, a hybrid thresh-

old approach for Shannon Index based signal subset selection technique is superior.

Lexical Analysis of vehicle five produced a population of signals with approximately

75% containing less than two bits of information. It is unlikely that a continuous

numerical time series can be expressed using only two bits. Thus, a percentage based

75

_Percentile

25

_Percentile

Interquartile

Range IQR

Upper Whisker

Possible Outlier

Notched Area

95% Confidence Interval of

the Median

Lower Whisker

Median

Mean

Figure 26. Features of a Notched Box Plot

The varied distribution of Shannon Index values across the 17 vehicles studied also

highlight that it is also unlikely that a ‘fixed’ minimum threshold is effective for all

vehicles; some sort of adjustment may be necessary based on the vehicle OEM or

make. Such adjustments are expected to require domain specific heuristics or a priori

knowledge which this research is explicitly avoiding in order to remain as generaliz-

able as possible. Thus, the percentage based approach is used in this chapter based

upon the assumption that it is a more generalizable thresholding strategy.