INSTALACIONES 24

We propose to capture side constraints viaside constraint match featuresdefined between side constraints of a phrase pair𝑥, ̄̄ 𝑦and an of an input𝑥. With these additional features, the translation model score is computed as

𝑠_𝑤(𝑥, 𝑦) =max

ℎ ∑

{ ̄𝑥, ̄𝑦∈ℎ}

w⊤𝜙( ̄𝑥, ̄𝑦) +w⊤_{𝑚𝑎𝑡𝑐ℎ}𝜙_{𝑚𝑎𝑡𝑐ℎ}(𝑆( ̄𝑥, ̄𝑦), 𝑆(𝑥)) ,

Algorithm 5 Extraction of raw overlap features

1: At training time:

2: for all phrase pairs𝑥, ̄̄ 𝑦 do

3: 𝑆( ̄𝑥, ̄𝑦) ←Union(Lookup(𝑥)) for all training pairs 𝑥, 𝑦containing 𝑥, ̄̄ 𝑦 4: end for

5:

6: At test time: 7: Obtain test input𝑥 8: 𝑆(𝑥) ←Lookup(𝑥)

9: for all phrase pairs𝑥, ̄̄ 𝑦 such that 𝑥 ∈ 𝑥̄ do

10: 𝜙_{𝑚𝑎𝑡𝑐ℎ}(𝑆( ̄𝑥, ̄𝑦), 𝑆(𝑥)) ←Aggregate(Intersect(𝑆( ̄𝑥, ̄𝑦), 𝑆(𝑥))) 11: end for

where w𝑚𝑎𝑡𝑐ℎ are additional feature weights, which can be trained on the heldout set

along with the other feature weights.

Given a phrase pair𝑥, ̄̄ 𝑦, the phrase pair side constraints𝑆( ̄𝑥, ̄𝑦)are computed as the union over the side constraints of all training sentences that contain 𝑥, ̄̄ 𝑦. The side constraint match features are then computed by taking the intersection between the side constraints for test input𝑥,𝑆(𝑥), and the phrase pair side constraints𝑆( ̄𝑥, ̄𝑦). Algorithm 5 shows the feature extraction process. Lines 2-4 show the extraction of phrase pair side constraints at training time. The Lookup procedure maps the sentence identifier of an input 𝑥 to the side constraints extracted from the document-level metadata of the patent document containing𝑥. Lines 7-11 then contain the extraction of the side constraint match features at test time. Side constraints for input𝑥 are obtained viaLookup, and the intersection is computed for each phrase pair in 𝑥. The new features are dynamic features, as they depend on the input document side constraints, whose values are only obtained at test time. The final feature values are computed by intersecting𝑆(𝑥)with𝑆( ̄𝑥, ̄𝑦), and feeding the intersection to an aggregation procedure aggregate. Below, we motivate the need for feature aggregation and describe different aggregation schemes.

We propose three different ways of aggregating matching side constraints: no aggregation, aggregation by category and aggregation by sum. More aggregation produces fewer, more general features, while no aggregation allows the model to capture specific information, but can lead to sparsity issues. When doingno aggregation, each matching side constraint is treated as a separate binary feature. This method produces sparse, fine-grained features. However, the model will only learn weights for annotations it has seen during tuning. If the tuning set only contains few documents, few annotation features will be learned. The maximum number of features added by this method is equal to the number of different annotations present in the training data. We use aggregation by category to define more general features. This method aggregates all side constraints belonging to the same type

6.3 Side Constraints for SMT

– COMP, INV, IPC1, IPC2, IPC3 – in one single feature by summing over all side

constraint matches in this category. For example, if a test document has three inventors, and a phrase was found in documents from two of the same inventors, this phrase will be annotated with the featureINVMatch=2. Aggregating by category allows the model to differentiate between types of side constraints, while avoiding the sparsity issues arising without aggregation. This method of aggregation always adds 𝐹 features to the model, where𝐹 is the number of categories. Finally, aggregation by sum, is the most aggressive aggregation scheme. This method sums over all side constraint matches, adding only a single additional feature to the model. For example, if a phrase pair and an input document have two matching inventors and a matching IPC section, the phrase would be annotated withMatchSum=3.

The features we have defined so far capture whether a phrase pair shares the same side constraints as an unseen document. One weakness of this approach lies in its potential bias for more common phrases. Phrases that occur in more documents are more likely to have more matching side constraints. However, a phrase pair occuring frequently, but only within a single IPC class, for example, could be crucial for producing a correct translation for inputs from this class, but would only receive low aggregated match counts. We counter this weakness by introducing a weighting scheme inspired byterm frequency - inverse document frequency (TFIDF) weighting. Originally, TFIDF weighting has been designed to weight individual terms for information retrieval: Terms are given high weight if they occur frequently in few documents. In our case, the weighting scheme should assign more weight to a phrase pair with respect to a side constraint, if the phrase pair is strongly associated with this side constraint. Less weight should be assigned to a phrase pair with respect to a side constraint, if this phrase pair also occurs with many other side constraints. We therefore definedocuments by the side constraints, e.g., the patent class or the name of the inventor. The terms are phrase pairs occurring in patents with this side constraint. Note that as one patent can have multiple side constraints, the same phrase pair occurrence can be assigned to more than one “document”. In this regard our approach differs from classical TFIDF, where each term occurrence is assigned to exactly one document. The idea of using a TFIDF weighting scheme is inspired by the vector space adaptation model of B. Chen et al. (2013), who apply TFIDF weighting to measure the importance of phrase pairs with respect to different corpora for domain adaptation. In our setting, the term frequency TF of a phrase pair 𝑥, ̄̄ 𝑦 with respect to a given side constraint𝑠_𝑖 ∈ 𝑆( ̄𝑥, ̄𝑦)is computed as follows:

TF( ̄𝑥, ̄𝑦; 𝑠_𝑖) = count( ̄𝑥, ̄𝑦; 𝑠𝑖) max𝑥̄′_{, ̄𝑦}′(count( ̄𝑥′, ̄𝑦′; 𝑠_𝑖))

using maximum TF-normalization (Manning et al., 2008, Section 6). The inverse docu- ment frequency IDF of𝑥, ̄̄ 𝑦 is:

IDF( ̄𝑥, ̄𝑦) =log 𝑁

DF( ̄𝑥, ̄𝑦)+ 𝛽 ,

where the document frequencyDF( ̄𝑥, ̄𝑦)is the number of side constraints𝑥, ̄̄ 𝑦occurs with, and 𝑁 is the total number of side constraints. 𝛽 is a smoothing parameter to avoid zero-values ifDF( ̄𝑥, ̄𝑦) = 𝑁. The TFIDF weight of 𝑥, ̄̄ 𝑦 is then computed as

TFIDF( ̄𝑥, ̄𝑦; 𝑠_𝑖) =TF( ̄𝑥, ̄𝑦; 𝑠_𝑖) ⋅IDF( ̄𝑥, ̄𝑦) .

As an illustration, Figure 6.2 shows the 5 patent classes with the highest TFIDF weights for three translation options for “Speicher”. We can see that “storage” has highest weight in classes concerning physical storage, while “memory” ranks highest in classes on computing and electronics. Both options are relevant for class G11, “information storage”. For “reservoir”, the top-ranked classes concern heating, thermal insulation and liquids. Combining the variants described above, we conduct four sets of experiments:

(1) single: Sparse binary match features without aggregation or weighting.

(2) single weighted: Sparse match features with TFIDF weighting.

(3) aggregated: Binary match features from setup (1), either aggregated by side con- straint type, or by summing over all matches.

(4) aggregated weighted: Weighted match features from setup (2), either aggregated by side constraint type, or by summing over all matches.

6.3 Side Constraints for SMT

Top IPC classes for “storage”

F17,storing or distributing gases or liquids

E03,water supply; sewerage

F24,heating; ranges; ventilating

G11,information storage

E02,hydraulic engineering; foundations; soil-shifting

Top IPC classes “memory”

G06,computing; calculating; counting

H04,electric communication technique

G11,information storage

G05,controlling; regulating

G01,measuring; testing

Top IPC classes for “reservoir”

F24,heating; ranges; ventilating

F16, engineering elements or units; general measures for producing and maintaining effective functioning of machines or installations; thermal insulation in general

F03, machines or engines for liquids; wind, spring, or weight motors; producing mechanical power or a reactive propulsive thrust, not oth- erwise provided for

F02,combustion engines; hot-gas or combustion-product engine plants

E05,locks; keys; window or door fittings; safes