Objetivo 3: Ajustar la Estrategia de Comunicación de “Hábitat 2” en Cumanayagua

Due to the huge volume of daily posts on Twitter, a large percentage of them can be seen as redundant, as they only report on the sub-events that are already reported by other tweets. In [122], researchers observed that when people were discussing a product, the vocabulary that they used converged for important factual aspects; on the other hand, the vocabulary for personal reviews was often diverse. In [262], researchers distinguished near-duplicate tweets on 5 levels, which were: exact copy,

near exact copy,strong near-duplicate,weak near-duplicate andlow-overlapping: for

exact copy, near exact copy, strong near-duplicate and weak near-duplicate tweets, the main parts are identical or almost the same; for low-overlapping tweets, they

only have a couple of common key terms, but greatly vary in word usages and expression patterns. My sampled tweet dataset illustrates that tweets reporting on the same sub-event are near-duplicate tweets, which is also consistent with the findings in [122].

In [262], researchers treated the near-duplicate detection as a classification prob- lem and the classifier had to make a decision on every pair of tweets that were possible to be near-duplicates. Their near-duplicate detection strategy worked well on a small scale, but it needed human annotation of tweets from various domains to train the classifier and its computation complexity was thus really high. On the other hand, traditional online incremental clustering algorithms, based on the simi- larities oftfidf vector representations (discussed in Section 3.2.1) of tweets’ textual content [12, 46, 298], are often employed to process tweet streams online. The tra- ditional incremental online clustering algorithms have low complexity and do not need prior knowledge of the number of clusters. However, they have the following drawbacks: (i) The inverse document frequency (idf) information, either iteratively updated or extracted from auxiliary corpus, can be biased, depending on the dif- ferences between the term distributions of the processed tweets/auxiliary corpus and the real term distribution; (ii) Tweets are short texts, and therefore the role of some rare and novel terms can be dominating, when calculating the similarities between tweets using theirtfidf vector representations; (iii) Researchers usually re- duce the dimensionality of the vector representations by selecting the terms with high idf values only, but it is questionable to equal rareness with importance, es- pecially when theidf information is not reliable, as some valuable information can be easily lost; (iv) By setting a reasonable threshold, this kind of online clustering algorithms may have acceptable performance onexact copy,near exact copy,strong near-duplicate and weak near-duplicate tweets, but they can hardly deal withlow- overlapping tweets, which account for 18.8% of all kinds of near-duplicate tweets, according to [262].

Another drawback shared by most current online incremental clustering algo- rithms for event detection in tweets is that they do not consider the textual variants of terms, which are highly frequent, because of tweets’ short and informal charac- teristics and their rich syntax features. For example, tweet t1: “#Senegal sends

medical teams to border with #Guinea after an outbreak of Ebola there. #Sierra Leone, much closer to the epicenter, hasn’t.” and tweet t2: “Senegal has sent a

of Ebola... http://fb.me/6WqOZ2b3h” are talking about the same sub-event. They contain some key terms that vary in representation forms, but have the same mean- ing, such as #Senegal and Senegal, medical teams and medical team, Guinea and

#Guinea, which should be treated as the same terms. Other examples include

#Liberians and liberia,#EbolaFree and ebola-free, etc. However, traditional online clustering algorithms ignore this phenomenon, they treat these key terms, which share the same meaning, but only vary slightly in representation forms, as different terms. As a result, the similarity of these two tweets decreases and they cannot be included in the same cluster when the clustering threshold is high. However, alternatively blindly lowering theclustering threshold can cause the decrease of the

clustering precision (defined in Section 5.3.3).

Algorithm 1:Sub-event detection in tweet stream

input :T, tweet stream; E, target entity(ies)

output:P rocessingClusters, clusters of tweets reporting the samesub-event

P rocessingClusters=∅;

foreachTweet t_∈T mentioning e_∈E do

Preprocess t;

if t is reporting a real-world sub-event then

Initialise a cluster ct withtusing Ut (useful URLs in t) and Kt (key

terms in t);

foreach cluster c in P rocessingClustersdo

if cluster chas common useful URL with ctthen MergeClusters(c, ct);

else if GetSimilarity(c, ct) >clustering threshold then MergeClusters(c, ct); else addct toP rocessingClusters; end end end end

To solve the above problem and reduce the dimensionality of tweets’ vector repre- sentations, as well as increase theclustering precision and decrease thecompression ratio (defined in Section 5.3.3), I propose a variant online incremental clustering algorithm, as shown in Algorithm 1. Algorithm 1 incrementally clusters the tweets based on common URL(s) and key terms sharing the same meaning.

in real-time. When a new tweet comes, after preliminary target entity(ies) filtering, preprocessing and classification, I obtain the tweets reporting real-world sub-events belonging to the pre-known major event. To eliminate the noise, I only use somekey terms and URLs in the tweet to construct its vector representation, as these parts are crucial to deciding what sub-event the tweet is discussing. Thekey terms are noun phrases, verbs, hashtags, numbers and at-mentions. The choice is made both empirically, based on the observation that for tweets reporting the same sub-event, these key terms would be the same or textually similar, but the other parts of the tweets, such as conjunctions, adjectives and adverbs, often vary; and experimentally, based on the performance of various choices. In this way, the dimensionality of the tweets’ vector representations is reduced. Since I already have the POS-tags after the preprocessing step, I only need chunking to extract the key terms, and lemmatisation to transform the verbs from their various inflected forms to their original forms.

It is of high probability that tweets containing URLs are closely related to the content of the linked webpages [3]. Some studies have used this kind of tweets as the summaries or highlights of the sub-events reported by the linked webpages [285]. This has shown that the benefits of the assumption that tweets containing URLs represent highlights of the sub-events reported by the linked webpages, overweigh the risks. Based on the above assumption, new tweets are incorporated into the processing sub-event cluster with which it shares common URL(s). Two sub-event clusters are considered to report on the same sub-event if they contain common URL(s). Similar approach was also employed in [51]. I do not take the full actual content of the webpages into account, to avoid the inclusion of noisy information. Because of the characters limitation of Twitter, the URLs contained in the tweets are mostly shortened in various ways, to save space. After retrieving the original URLs, I consider the URLs that contain nothing but domain and category information, such as http://NBCNews.comand http://www.nbcnews.com/news to be useless, as this kind of URLs provide no information about the sub-events. One original URL would only be consider asuseful, if it contains the concrete address of a real-world sub-event reporting webpage.

The prioritisedURL-based clustering strategy can help to enrich the processing sub-event cluster with the key terms that report the sub-event from different angles. On the other hand, if an incoming tweet does not contain any common URL with any processing sub-event cluster, it is still possible to be incorporated into one processing sub-event cluster. This is achieved by the threshold-based clustering strategy. As

mentioned before, the problem that key terms appear in slightly different forms, but have the same meaning, widely occurs in tweets. For example, the occurrences of “#Liberia”, “Liberian” and “Liberia’s” have the same effect as the occurrence of the country name “Liberia”. To deal with this problem, I treat two different key terms as the same key term, as long as the Jaro-Winkler metric between them is above a threshold. This method can further reduce the dimensionality of the tweets’ vector representations. Similar to [250], each dimension in the tweets’ condensed vector representations denotes a cluster of key terms that share the same meaning, rather than one individual key term. The Jaro-Winkler metric is specially designed for short strings matching, which is based on the length of the longest common prefix, the number and order of the common characters between two strings [62]. In [62], researchers replaced the exact term matching with approximate term matching, based on the Jaro-Winkler metric; in [61], researchers compared various personal name matching techniques, and the Jaro-Winkler metric was one of the techniques with the best performance. I set the Jaro-Winkler metric threshold to 0.9, following [62]. Based on the above description, I defineGetSimilarity(c1, c2) in Algorithm 1

as follows:

GetSimilarity(c1, c2) =JJW(K1, K2) =

K₁′ _∩K₂′

K₁′ ∪K₂′ (5.1)

where: c1 and c2 denote two clusters, K1 and K2 denote the key term sets in c1

andc2. I replace the traditional Jaccard similarity metric based on exact matching

(J) with a Jaccard similarity metric based on the Jaro-Winkler matching (JJW). In

Equation 5.1, K′ represents key term clusters, with all the key terms in the same cluster sharing the same meaning. A new key term can be incorporated into one of the key term clusters, as long as the Jaro-Winkler metric between this new key term and any one of the key terms that are already in the cluster is above 0.9. For two key term clusters, ki and kj, they are viewed as the same key term cluster if

the Jaro-Winkler metric between one key term fromki and one key term fromkj is

above 0.9.

All information about the processing sub-event cluster, such as the above men- tioned key terms and URL(s), will be updated as long as it incorporates new tweets; the information about individual tweets in the sub-event clusters will be discarded to save space.

In [304], researchers pointed out that clustering algorithms utilising the Jaccard similarity metric achieved better performance than the ones utilising the cosine simi-

larity metric, because of the sparsity of tweets. Similar to [315], I choose the Jaccard similarity metric when evaluating the similarity between two tweets’ condensed vec- tor representations. Because of the usage of Jaccard similarity, I only need to store the tweet’s vector representation as a binary vector, which means the tweet’s con- densed vector representation is a variant of the binary-based bag-of-words (BOW) vector representation discussed in Section 3.2.1. Since only key terms are consid- ered, I do not have to face the problem that the Jaccard similarity metric cannot deal with terms with different levels of importance.