Models A to C introduce morpheme segmentation models with tunable levels of granularity. They help us feed neural models (NLMs or NMT engines) with multi- granular subword units, but they have a serious problem, namely their dependency on the external parameter (θ or m). Other similar models such as Byte-pair or
Morfessor also suffer from the same shortcoming. Their performance is highly in-
fluenced by the external parameter and they can only help us provide our desirable result when the external parameter is set to an optimal value. Finding such a value is usually performed in an empirical manner which can be quite costly (or even impossible) for (very) deep neural models. Therefore, apart from these parameter-
dependent techniques, we need to have a model which has no external parameter. Model D employs principles of dynamic programming and provides such a solution. The morpheme segmentation problem is an optimization problem, where a se- quence can be viewed as a container of character n-grams and the task of segmen- tation is to find the best boundaries between such n-grams. There could be several segmentation patterns for a single sequence, where the best segmentation is defined based on the evaluation metric we consider for the task. For example, to provide a linguistically correct segmentation we expect the model to separate stems from syntactic affixes (valid affixes which exist in the language), or in Byte-pair we try to set boundaries in a way which generates up to S unique symbols for the entire corpus (S is a predefine vocabulary size). Figure 4.3 illustrates an example of a Turkish sequence (word) with different segmentation boundaries to elaborate this issue. i n in i z ini ni iz niz iniz da nizda ki daki le r ler kiler
Figure 4.3: Different segmentation boundaries for the Turkish word ‘hakkinizdakiler’ meaning ‘things about you’. Different paths yield different segmentations for this word.
As Figure 4.3 shows, we can have several segmentation patterns for the Turk- ish word ‘hakkinizdakiler’ meaning ‘things about you’ such as ‘hakk.in.i.z.da.ki.ler’,
‘hakk.i.niz.da.ki.ler’, ‘hakk.i.nizda.ki.ler’ etc., as ‘i’, ‘in’, ‘ini’, ‘niz’, ‘nizda’, ‘da’, ‘daki’, ‘ki’, ‘kiler’, ‘le’, and ‘ler’ are all valid suffixes in Turkish (hakk is the stem).
Different segmentation models provide different patterns based on different criteria. If the model prioritizes linguistic aspects, probably the best segmentation would be
‘hakk.in.iz.daki.ler’, but if it considers the frequency feature together with linguistic
constraints the final segmentation would be ‘hakk.i.n.i.z.da.ki.le.r’. Therefore, the problem is an optimization problem and different results can be obtained based on different constraints.
As we do not have access to a precise morphological analyzer for all languages, we try to benefit from statistical techniques to find a suboptimal segmentation. It should also be noted that nothing forces us to find linguistically correct segmenta- tions. Existing (NLP) models are flexible/powerful enough to work with different segmentation schemes and the only prerequisite is that they work with a limited set of basic units. With this assumption, we try to discover such a set of basic units. In our model we consider two constraints, namely length and frequency. We wish to find (disjunctive) boundaries between sets of characters which maximize both con- straints at the same time. If we only consider the length constraint then all words (or sequences in general) would stay in their original form and the model would give the surface form as its output. Similarly, if we only take the frequency constraint into account we would not obtain the best result as the model will decompose words purely into characters. We try to make a balance between these two paradigms and obtain a better segmentation scheme. We mentioned that it could be challenging to assign an optimal value to these parameters, so we propose a model to set them automatically.
If a unit frequently occurs in a corpus, there is a chance for the unit to be a basic element of the language. However, we should define a prescriptive criterion for this situation. We know that all frequent units are not basic or they could be basic (such as characters) but not useful for our purposes. Since a frequency threshold labels a unit as frequent or infrequent, finding an appropriate threshold is a critical issue. A mechanism should be defined to recognize the threshold value but it can make the segmentation model fragile, as its performance would totally rely on a single parameter which is very context-dependent. A wrong threshold could lead to wrong segmentations. Another important issue about the threshold is that when it is set
to a particular value such as θ all units with frequencies higher than θ are considered as frequent and, all other units with lower frequencies even with very close values to θ (such as θ-1) are neglected. Accordingly, even in the presence of an optimal threshold it is a very strict constraint to treat units based on a single value.
As far as problems caused by frequency-based models are concerned, it is not an appropriate idea to set a global threshold value and we can decide to select (or skip) a unit based on its context and adjacent units. Furthermore, it seems that another complementary constraint should be included too, namely the length of units. Similar to the frequency threshold we cannot have a global value for this criterion. We do not know how many characters are enough to make an atomic block and we cannot set a predefined value. Any set of characters with any random length could be a basic block. So, apart from finding a dynamic and context-dependent frequency value for each subunit, the segmentation algorithm should also be able to find dynamic boundaries.
According to the discussion so far, it is not possible to separate subunits based on predefined global length and frequency constraints and the model should set dynamic values at each step. We propose a model in this regard. More formally, given two m-tuples of positive numbers ă v1, v2, ..., vm ą and ă l1, l2, ..., lm ą, and
L ą 0, we wish to determine the subset A Ď t1, 2, ..., mu that maximizes (4.9a),
subject to (4.9b): ÿ aPA va (4.9a) ÿ aPA la= L (4.9b)
An input sequence (word) with the length L can include up to M sub n-grams where
M = Lˆ(L+1)2 and 1 ď n ď L. The a subscript in Equation (4.9) indicates the index for each of these sub n-grams. The final goal is to select a subset of such n-grams (finding A) which satisfies all constraints. laindicates the length of the a-th n-gram
problem. The set of all possible sub n-grams extracted from the word ‘the’ is {t1, h2,
e3, th4, he5, the6} (L = 3 so there are 3ˆ42 sub n-grams). Our model tries to find a subset of these n-grams which maximizes the optimization constraints so that their combination gives us the original word ‘the’. Patterns such as ‘t.he’ (A = t1, 4u) or
‘the’ (A = t6u) could be possible alternatives as results of the segmentation process,
but a pattern such as ‘t.th.e’ (A = t1, 4, 3u) is wrong, as this violates the length constraint and does not generate the original surface form.
In order to solve the problem formulated by equation (4.9), we need to define
la and va. It is clear that what the length value la is for each n-gram. For va, a
simple estimation could be the frequency of each n-gram, as because it shows the importance (value) of the n-gram. However, with this simple estimation characters would have much higher values than others, which means the final result would be the same as (or very similar to) the character-level segmentation. This is not our desirable result so we approximate the value of each character n-gram vawith (4.10):
imp(n) ˆ f req(na) (4.10)
where f req(na) shows the frequency of the a-th n-gram in the entire training corpus
and imp(n) indicates the impact/importance of that n-gram category (to which na
belongs), i.e. imp(n) tries to make a balance between length and frequency. Clearly the frequency of a single character (unigram) is much higher than other n-grams, but we know that there is only a limited number of unigrams in the corpus. There could be up to 500 unique unigrams in a corpus but each of them could occur millions of times. Accordingly, unigrams are important in terms of frequency but the category to which they belong is not very popular/important in the corpus. Similarly the number of bigrams is considerably fewer than trigrams but the frequency of each bi-gram is much higher than trigrams. There should be a mechanism that shows how frequently a character n-gram occurs and how important the n-gram category is among others. To this end we define imp(n) which shows the portion of a specific
n-gram category among all other n-grams which might be estimated as in (4.11):
imp(n) = count(n)
řN
n1=1count(n1)
(4.11)
where count(n) enumerates how many n-grams exist in the corpus and N indicates the length of the longest n-gram possible in the corpus.
Our segmentation model is defined based on these criteria. In order to implement our model we propose a dynamic programming-based solution. To select the best subset of all items (character n-grams) for the input sequence, at each step we iteratively add a new item to an existing subset which is the optimal set up to that step. The new item is selected in a way where the combination of a new item and the existing set is another optimal set with one more member. We can compute the value of the obtained segmentation at the i-th step (the total value of the items selected up to the i-th step) with V (i, L) as in (4.12):
V (i, L) = $ ’ ’ ’ ’ ’ ’ & ’ ’ ’ ’ ’ ’ % 0 if i = 1 V (i ´ 1, L) if li ą L max(V (i ´ 1, L), vi+ V (i ´ 1, L ´ li) ) otherwise (4.12)
The model starts with an empty set and by the model explained in (4.12) adds a single item to the set. This procedure continues until the optimal set of subunits (A) is reached. We select items which satisfy our constraints and make a segmentation pattern for a given word. Figure 4.4 illustrates the python code for Equation (4.12). Word() is a function which takes two inputs and segments the word. The first input is a set of tuples which provides all possible character n-grams of the word (to be segmented) and their values, and the second one is the length of the word. For exam- ple, when we segment the word ‘the’, the set of all character n-grams and their values would be similar to items={(‘t’,vt),(‘h’,vh),(‘e’,ve),(‘th’,vth),(‘he’,vhe),(‘the’,vthe)} and
Figure 4.4: Morpheme segmentation with Model D.