Colcerámica S.A.S

In line with our initial assumption our analysis approach gains high precision results, higher than the analysis of CNSS. Also the type of false positives we recover is different and in our opinion less severe than those found with the rule-based approach. Where most of our false positives are reactions in the pathway that skip one or more steps between substrate and product, the false positives found by CNSS contain a large number of relationships that are not true, largely due to the extraction of terms that are neither enzyme nor metabolite. The occurrences that skip reaction intermediates still hold relevant information and in future attempts can be labelled explicitly as such. When processing large volumes of text, skipped reaction steps may become visible because the steps themselves are found in other passages.

The downside of a high precision approach is lower recall, which is also the case for our results. Yet the influence of lower recall of individual reactions from the text on the ability to recover the pathway in question is significantly lower than expected. Where we are only able to recover roughly half the amount of true positives compared to CNSS we manage to recover 75 – 100% of the pathway reactions also retrieved by CNSS. This is in line with the expectation that reactions of interest in

Evaluation of Metabolic Reaction Extraction the analysed literature are mentioned multiple times. We are able to recover at least one of each and therefore are able to recover that particular step in the pathway, whereas CNSS recovers most reactions more than once.

When applying any text mining approach to very large collections of text (e.g. Medline, BioMed Central) high precision of the results, like in our approach, is preferable. It is better to have a smaller result set of a very high quality than a large set of average quality, as also noted by Lee et al. (Lee et al. 2012). High precision limits the number of results to be studied and cleared of false positives by a human user. With regards to metabolic reactions the loss of recall associated with high precision may result in very rare reactions to be missed, but when a corpus is sufficiently large reactions will be mentioned more than once and thus have a higher probability to be picked up, as is borne out by this experiment. For our approach that increase in probability is not dependent on the number of occurrences of a given reaction in the text, but dependent on the fact that the more often a reaction is mentioned the higher the probability that the reaction is described in an explicit manner. This allows us to identify rare reactions and potential uncatalogued links in pathways.

Lack of recall when including enzymes

Studying the results of CNSS and our method, it becomes apparent that it is more difficult to recover a complete reaction including the enzyme than to recover a reaction with only the participating metabolites. We identified a number of reasons for this (see also Appendix Table A. 12 - Table A. 14).

Firstly, as part of the named entity recognition task, recovering a complete reaction involves the recognition of at least three named entities rather than two, either enzyme or metabolites, which might not be detected in the NER task. Secondly, as a special case of the former, it is perfectly normal to describe a chemical conversion of substrate to product without actually mentioning the active enzyme in the same sentence. This does occur in a number of sentences in all three investigated pathways. In this case extraction of a complete reaction is impossible. A third cause of loss of recall is on the level of relation extraction, and occurs even when all relevant entities in a sentence have been correctly identified. Our approach has specific difficulties in identifying the relationship between enzyme and metabolite as opposed to the relationship between metabolites. The reason for this lies in the way the sentences are constructed by authors: metabolites mostly occur closely together within the sentences, often linked in simple relations like ‘conversion of metabolite A to metabolite B ’. For these constructs the extraction of the relationship is simple and requires only linkage of two syntactic triples. The distance both in words and syntactic triples between the active enzyme and the metabolites within the sentences is greater, resulting in more syntactic triples that have to be linked in order to extract the relationship. Also the variety of syntactic triples and keywords necessary to link enzyme and metabolite is greater, making it more difficult to describe generic rules for extraction. One could argue that this is specific to our approach of using syntactic

Chapter 4

triples, but any other linguistic approach will suffer in a similar way as the linguistic degrees of freedom to describe an enzyme and its role in a reaction are much higher than the ones to describe the conversion of one metabolite into the other. A good example of this is the following sentence, which actually occurs in our corpus (with boldface highlighting relevant entities introduced by us):

”Hence, in addition to control of CoA synthesis on the level of pantothenate kinase, further modulation of flux through the pathway could occur at phosphopantetheine adenylyltransferase (PPAT), which catalyses the penultimate step in the pathway (Fig.1), the reversible adenylation of 4’’-phosphopantetheine to form 3’’-dephospho-CoA (dPCoA) and pyrophosphate (PPi).” (Geerlof et al. 1999)

Analysis of the experiments

The different experiments we performed to investigate the influence of the kind of input text on the performance of our approach turned out to have little effect on the results. One reason is the limited number of reactions and reaction pairs in the studied articles as well as the limited number of occurrences of each experiment’s targets. For example there are only seven Greek characters in the tetrahydrofolate pathway texts and none of them is part of a sentence with an interaction.

As for Experiment E, the pantothenate pathway texts do not contain a single anaphoric reference. The tetrahydrofolate pathway corpus shows a completely different picture, where there have been a total of 144 anaphor replacements of which 56 were true anaphora and 88 short-forms to long-form conversions. Only 20% (11) of the true anaphoric relations are found in sentences describing a reaction, and indeed we find an additional nine reaction pairs. Only three of them are true positives, the others skip steps in the pathway. While these still hold relevant information, they are strictly speaking false positives. The unbalance of anaphora between sentences describing reactions and those that do not can be just indicative for this corpus, or it can be an indication that authors prefer to give the enzyme name in a reaction sentence and use anaphora like ‘the enzyme’ when describing other content. In the fatty-acid β-oxidation pathway, generic terms are used in an anaphoric role to describe steps in the pathway (e.g. ‘fatty acyl CoA’), however these forms could not be resolved to more specific terms as the antecedents are not mentioned in the text at all.

As CNSS already note, the quality of results for the fatty acid β-oxidation pathway is much lower than for the other two pathways and is indicative of a different style of describing reactions.

The one experiment that clearly increases recall with little or no loss of precision is the conversion of chemical formula in the text to syntactic constructs that can be parsed. The results appear limited as there are only ten chemical reactions described across all three texts, yet the increase in true positives is seven for substrate – product interaction pairs, enzymes are not mentioned in the reactions.

Evaluation of Metabolic Reaction Extraction