Estructura escrita

Name Answer Finding Method Machine Learning Algorithm Scenario 1 Word-based (Baseline) -

Scenario 2 Phrase-based Support Vector Machine

Scenario 3 Phrase-based Maximum Entropy

Scenario 4 Phrase-based J48

Test is divided into four main scenarios with black box method. Details of each test scenario can be seen in Table 6. Test will be done using two testing tecniques:

1. Cross validation with 4-folds.

2. Data split with 75% training data and 25% testing data.

3. Experimental Result

Testing is performed to see the accuracy of answer finder component in classifying and sorting answers. The test results using cross validation technique can be seen in Table 7 (classification accuracy) and Table 8 (MRR). The test results using data split technique can be seen in Table 9 (classification accuracy) and Table 10 (MRR). Highest accuracy/MRR values are colored in blue while lowest accuracy MRR values are colored in red.

Table 7. Answer classification accuracy (Cross validation)

EAT SVM J48 MaxEnt

Yes No Yes No Yes No

Person 60.2 82.3 42.4 85.1 63.8 83.5 Organization 62.0 89.0 42.9 90.8 62.7 86.3 Location 74.4 61.8 50.6 80.6 75.6 64.8 Datetime 72.8 70.8 46.5 86.3 81.2 74.0 Quantity 78.1 67.1 66.8 78.3 76.8 70.0 All 66.8 75.8 47.2 84.7 69.1 76.9

Table 8. MRR value for answer finder (Cross validation)

EAT Baseline SVM J48 MaxEnt

Person 0.585 0.846 0.635 0.854 Organization 0.801 0.786 0.663 0.797 Location 0.696 0.813 0.614 0.716 Datetime 0.849 0.757 0.573 0.799 Quantity 0.680 0.811 0.601 0.755 All 0.723 0.802 0.617 0.784

Table 9. Answer classification accuracy (Data split)

EAT SVM J48 MaxEnt

Yes No Yes No Yes No

Person 58,82 89,16 47,06 81,53 62,75 88,55 Organization 65,00 78,04 35,00 81,84 62,50 79,44 Location 70,18 78,71 49,12 84,59 73,68 78,71 Datetime 88,89 85,54 61,11 89,76 94,44 87,35 Quantity 85,00 72,27 80,00 75,78 80,00 76,95 All 68,58 81,16 47,35 82,17 69,47 82,23

Table 10. MRR value for answer finder (Data split)

EAT Baseline SVM J48 MaxEnt

Person 0,589 1,000 0,683 1,000 Organization 0,769 0,821 0,655 0,833 Location 0,848 0,955 0,659 0,909 Datetime 0,914 0,909 0,727 1,000 Quantity 0,488 0,933 0,783 0,875 All 0,728 0,922 0,700 0,923 Question Analyzer Corpus User Interface Passage

Retriever Answer _Finder

Document Preprocessing Wikipedia Indonesia Dump Question Answer EAT Keywords Keywords Wikipedia Article Document Collection Relevant Paragraphs Answer Question Index File Document Indexing Document Collection Searching Index Searching Index Preprocessed Article EAT Classifier POS Tagger Phrase Chunker Stemmer Paragraph Searcher Document Searcher

Answer Classifier & Ranker POS Tagger Phrase Chunker Stemmer Reference Resolution NE Tagger

: Module from previous research : Refined module : New module Keyword

Extractor

WEKA

Lucene

Accuracy value for each algorithm with cross validation technique and data split technique are not much different. This indicates that the test results of built classification model are valid.

MRR value obtained in current research is higher than MRR value in previous research (baseline). This indicates that the phrase-based answer finding method is successful enough to answer factoid question correctly and address problems in previous research.

SVM and MaxEnt algorithms have better performance than J48 algorithm, both in classifying and sorting answers. SVM and MaxEnt algorithms are statistical-based learning algorithms that use features together to perform classification in single step. J48 algorithm uses features separately to perform classification in several steps. Statistical model is more suitable for text processing problems. In addition, J48 algorithm tends to have overfitting problems on training data and leads to poor performance on testing data.

From the test results, there are several questions that can not be answered by the system, especially case where the answer is not in the 1st _{rank. The causes are:}

1. Phrase distance value 2. Similarity value

3. Priority between NP and VP features

G. C

Answer finding method developed in current research is phrase-based answer finding method. This method is chosen to overcome problems in word-based method in previous research by Zulen and Purwarianti [7]. The basic process of this method is using phrase chunker to extract phrases from a sentence. Machine learning is used to classify and determine the ranking of candidate answers with combination of word- based calculation (previous research) and phrase-based calculation (current research) as features.

Better test results obtained indicate that phrase-based method developed in current research is successful enough to answer factoid questions correctly and address problems in previous research. But, there are several questions that can not be answered by the system. It is necessary to refine the module used in the system and consider using complete parse tree instead of phrase tree.

REFERENCES

[1] Jurafsky, D. and Martin, J.H., Speech and Language Processing, 2nd ed., USA: Pearson Education, 2009.

[2] Harabagiu, S.M., Pasca, M.A., and Maiorano, S., “Experiments with Open-Domain Textual Question Answering,” Proceedings of the 18th

International Conference on Computational Linguistics (COLING 2000),

Saarbrücken, Germany, 2000.

[3] M. A. Pasca and S. M. Harabagiu, “High Performance Question Answering.” Proceedings of the 24th Annual International ACM SIGIR

Conference on Research and Development on Information Retrieval,

New Orleans, USA, 2001.

[4] R. P. Nasution, D. H. Widyantoro and A. Purwarianti, A, “INAGP : Pengurai Kalimat Bahasa Indonesia Sebagai Alat Bantu Untuk Pengembangan Aplikasi PBA,” Proceedings of Seminar Nasional Ilmu

Komputer dan Aplikasinya (SNIKA 2009), Bandung, Indonesia, 2009.

[5] M. Kamayani and A. Purwarianti, “Dependency Parsing for Indonesian with GULP,” Proceedings of the 3rd International Conference on

Electrical Engineering and Informatics (ICEEI 2011), Bandung,

Indonesia, 2011.

[6] F. Ferdian and A. Purwarianti, “Implementation of Semantic Analyzer in Indonesian Text-Understanding Evaluation System,” Proceedings of

IEEE International Conference on Computational Intelligence and Cybernetics (CYBERNETICSCOM 2012), Denpasar, Indonesia, 2012.

[7] A. A. Zulen and A. Purwarianti, “Study and Implementation of Monolingual Approach on Indonesian Question Answering for Factoid and Non-Factoid Question,” Proceedings of the 25th Pacific Asia

Conference on. Language, Information and Computation (PACLIC 2011), Singapore, Singapore, 2011.

[8] S. Bird, E. Klein, and E. Loper, Natural Language Processing with

Python. USA: O’Reilly Media, Inc., 2009.

[9] I. Budi, S. Bressan and Nasrullah, “Co-Reference Resolution for Indonesian Language Using Association Rules,” Proceedings of the 8th

International Conference on Information Integration and Web-based Applications & Services (iiWAS 2006),Yogyakarta, Indonesia, 2006.

[10] S. Williams, M. Harvey and K. Preston, “Rule-based Reference Resolution for Unrestricted Text Using Part-of-Speech Tagging and Noun Phrase Parsing,” Proceedings of the 1st Discourse Anaphora and

Anaphor Resolution Colloquium (DAARC 1996), Lancaster, UK, 1996.

[11] M. McCandless, E. Hatcher and O. Gospodnetic, Lucene in Action, 2nd ed., USA: Manning Publication, 2010.

[12] V. N. Vapnik, The Nature of Statistical Learning Theory, 2nd ed., USA: Springer-Verlag, 2000.

[13] S. Guiasu and A. Shenitzer, “The Principle of Maximum Entropy,” The

Mathematical Intelligencer, vol. 7, 1985.

[14] J. R. Quinlan, C4.5 : Programs for Machine Learning, USA: Morgan Kaufmann Publishers, 1993.

Optimization Preparation and Characterization