Nivel macro: perspectiva económica y de emprendimiento Colombia vs USA

In the base method described above, the result of the cosine similarity be- tween two bags of words is considered directly to estimate their absolute (i.e. not conditioned from others) probability of being related. Anyway, this is used as a rough approximation. For example, this would imply that a document and a category which are very related to each other have rep- resentative vectors with a mutual cosine similarity very close to 1, but this is unlikely to happen in practice. Similarly, vectors of unrelated documents and categories should have a cosine similarity close to 0; but often there are some words which are present in almost all documents, granting a minimum degree of similarity to all documents and categories, although lower with respect to related ones.

As an example, the histograms in Figure 4.7 show how related and unrelated document-category couples of the source domain for the rec vs talk dataset used in the experiments are distributed across various ranges of cosine similarity. While the average value for unrelated couples is near to 0, that for related couples is very far from the ideal value of 1. Couples extracted from other datasets generally follow similar distributions.

A possible solution is to apply some sort of correction to the “raw” co- sine similarity value, so that typical similarity values for couples of related document and categories are mapped to sensibly high probabilities and con- trarily typical similarities for unrelated couples are mapped to consistently low probabilities. This can be formalized as applying a suitable function π : [0, 1] _{→ [0, 1] to the cosine similarity between vectors of a document}

92 Chapter 4. Cross-Domain Text Categorization 0 0.1 0.2 0.3 0.4 0.5 0 500 1,000 1,500 Cosine similarity Related couples (µ ∼= 0.144) 0 0.1 0.2 0.3 0.4 0.5 0 500 1,000 1,500 Cosine similarity Unrelated couples (µ ∼= 0.058)

Figure 4.7 – Distribution of cosine similarity for couples of source docu- ments and categories in the 20 Newsgrops rec vs talk dataset, dashed lines indicate averages

d and a category c to find their final relatedness probability si_{(d, c) (with}

reference to a generic iteration i of the process). si(d, c) = π(cos(wd, wci))

To pick a function which adheres to the aforementioned conditions, typ- ical values for cosine similarity for related and unrelated couples should be known. The trivial chosen option is to sample values from couples of the source domain: as the labeling of the documents is known, it is possible to measure the cosine similarity cos(wd, w0c) for all couples (d, c) ∈ DS × C,

which constitute samples of its distribution for both related (those where CS(d) = c) and unrelated couples (the remaining ones).

To generalize these observations, univariate logistic regression is used. In general, logistic regression is a probabilistic classification model which can be used as a machine learning technique. In the specific univariate case, a value π(x)_{∈ [0, 1] is returned from a single predictive feature with a value} x∈ R, according to a function with the following form.

π(x) = 1

1 + e−(β0+β1x)

4.4. Iterative refining of category representations 93 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Cosine similarity Relatedness

Figure 4.8 – Logistic function extracted by regression on data of Figure 4.7, for which summary information is reported through box plots (whiskers are total range, box lines are quartiles, diamond is average)

parameters β0 and β1. Given n observations x1, x2, . . . , xn∈ R respectively

labeled with y1, y2, . . . , yn ∈ {0, 1}, the value of the following objective

function must be maximized.

n

Y

i=1

π(xi)yi(1− π(xi))1−yi

In the problem under analysis, each observation corresponds to a generic source document-category couple (d, c)∈ DS×C, the corresponding x value

is the cosine similarity of their representations and the y label is 1 if the two are related and 0 otherwise.

xd,c = cos(wd, w0c) yd,c= 1

if CS(d) = c

0 if CS(d)6= c

As an example, Figure 4.8 shows the function which is obtained through logistic regression on couples of the rec vs talk dataset, whose distribution of samples was reported in Figure 4.7. The function returns values above 0.99 for any cosine similarity of at least 0.2, reflecting that such similarity values are reached almost exclusively by related couples.

In practice, the base method is modified as follows. After computing representations for each document and each category from the source do- main and before starting the iterative phase, a univariate logistic regression

94 Chapter 4. Cross-Domain Text Categorization 0.5 0.6 0.7 0.8 0.5 0.6 0.7 0.8 0.9 1 Confidence threshold Accuracy comp vs sci 0.5 0.6 0.7 0.8 0.5 0.6 0.7 0.8 0.9 1 Confidence threshold Accuracy rec vs talk

cntf.idf (no LR) bin.idf logtf.idf ntf.idf cntf.idf

Figure 4.9 – Accuracy on two 20 Newsgrops two-categories splits with the logistic regression variant and different term weighting schemes as the confidence threshold ρ varies

model π is trained from observations extracted as specified above from all possible document-category couples (d, c) _{∈ D}S× C. Then, during the it-

erative phase, the model is applied to the result of cosine similarity when computing the absolute relatedness between any document d and any cat- egory c.

si(d, c) = cos(wd, wic) is replaced by si(d, c) = π(cos(wd, wic))

For what concerns computational complexity, to fit the logistic regres- sion model, the cosine similarity for NS =|DS| · |C| pairs must be computed

to acquire input data, which requires O(lc·NS) time; then the model can be

fit with one of various optimization methods which are generally linear in the number NS of data samples [82]. In practice, in experiments described

hereafter, the whole process roughly takes the time of one or two iterations of the refining phase, i.e. not more than 5 seconds.

Experiments on this variant have been carried on using the same pa- rameter values of those on the base algorithm. Plots in Figures 4.9 and 4.10 are the counterparts of those in Figures 4.5 and 4.6 with this variant applied: they show how accuracy varies with the weighting scheme and the

4.4. Iterative refining of category representations 95 0.3 0.4 0.5 0.6 0.7 0.8 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Confidence threshold Accuracy

comp vs sci vs talk

cntf.idf (no LR) bin.idf logtf.idf

ntf.idf cntf.idf

Figure 4.10 – Accuracy on a 20 Newsgrops three-categories split with the logistic regression variant and different term weighting schemes as the con- fidence threshold ρ varies

confidence threshold ρ, in all of them are also shown for comparison results for the base method with the default cntf.idf weighting. As can be seen, there is still a drop of accuracy for too high values of the threshold, but this now happens for higher values across all datasets and weighting schemes, as shown by the comparison with cntf.idf in the normal case, which is the most “stable” scheme together with ntf.idf.

In summary, this variant does not yield significant improvements of the results when parameters are optimally tuned, but guarantees optimal results for a wider range of parameters, derived from a more solid estimation of similarities between document and category representations.