The performance of the GA trained remote homology detection program named the DST was compared to leading methods using the ASTRAL30, ASTRAL20, and ASTRAL10 datasets. Four methods were tested against the DST. As noted in Section
0 500 1000 1500 2000 2500 0 5 10 15 20 25 30 35 Fi tn ess S um Generation
Initial Weight Variation
Random Weight Initial Weight of 1
110
3.1, BLAST and PSI-BLAST are leading sequence search programs. PSI-BLAST is well-known for its ability to detect remote homologs as well. Since this research extended the work from PropSearch, their algorithm was also tested. In addition, a support vector machine implementation of remote homology detection was implemented for ASTRAL10. Figure 5.10 illustrates the comparison between these techniques using the ASTRAL10 dataset. Figure 5.11 compares the techniques with the ASTRAL20 dataset, and Figure 5.12 compares all techniques with ASTRAL30.
111
112
Figure 5.12: Comparison of four remote homology programs using the ASTRAL30 dataset
Each of the Figure 5.10 through Figure 5.12 compares the number of false positives occurring at a percentage of true positives retrieved. SVMs were implemented for the ASTRAL10 database. The commercial SVM is a classifier with no ranking system. Because it is a binary classifier, it was unable to find all true positives. Figure 5.10 shows the SVM results, 65% of true positives were found using this method.
Precision and recall for the DST, PropSearch, PSI-BLAST, and BLAST were calculated. The formulas for precision and recall are defined by Equation 3.1 and Equation 3.2 described previously.
Table 16 illustrates the precision and recall for the four methods of remote homology search. From this table, DST has a 0.5% higher precision than PSI-BLAST,
113
BLAST, and PropSearch when sequence identity is less than 10%. Recall for the DST and PropSearch is always listed as 100% since the implementation of these programs is a full database scan. All true positives are always detected since the search techniques for both of these programs calculate distances between the query sequence and all database sequences. A threshold distance value could be applied to reduce time searching through the entire database in future searches.
Table 16: Precision and Recall for Four Methods of Remote Homology
At less than 20% sequence identity, the DST resulted in more accurate remote homology detection than BLAST, PSI-BLAST, and PropSearch with a .5% higher precision as can be seen in Table 16. The DST performs 4% better than PropSearch, 7% better than PSI-BLAST, and 8.1% better than BLAST when sequence identity is less than 20%.
To test the significance of these results, two-tailed t-tests of equal variance were performed. The t-tests were performed on the data from the ASTRAL10 dataset results only. The number of false positives gathered for each of the 207 queries was calculated for the DST, PropSearch, BLAST, and PSI-BLAST. The null hypothesis of the t-test was that there was no significant difference in the means between the number of false positives for the DST compared to PropSearch, BLAST, and PSI-BLAST. The results from the tests were significant. In the comparison between the DST and PropSearch, the p-value is 0.0143, in the comparison between the DST and BLAST, the p-value is
Precision Recall Precision Recall Precision Recall
DST 1.73% 100% 0.33% 100% 0.43% 100%
PSI-BLAST 1.01% 55% 1.25% 59% 72.68% 40%
BLAST 1.02% 58% 1.35% 60% 65.66% 66%
PROPSEARCH 1.21% 100% 0.36% 100% 0.30% 100%
114
5.06x10-15, and in the comparison between the DST and PSI-BLAST, the p-value is 7.65x10-15. Using a significance value of 0.05, the null hypothesis of the t-test is rejected and thus the results are significant.
5.7 Observations on Time Complexity
In remote homology search, speed is as vital as accuracy when searching through large databases. Thus, the time complexity was observed on the GA as well as the Database Search Technique. A benchmark analysis was performed against leading remote homology detection programs as well. For all analyses, the training dataset was split into smaller databases. As expected, the running time increased linearly with database size. Table 17 illustrates the time to complete both the GA and the DST using varying database sizes:
Number of Queries Database Size GA Time Database Search
Technique Time 52 704kB 10:01:25 00:04:43 26 188kB 00:34:26 00:01:04 13 74kB 00:04:15 00:00:14 7 36kB 00:00:37 00:00:08 4 20kB 00:00:11 00:00:04 2 4kB 00:00:02 00:00:01 1 381B 00:00:01 00:00:00
Table 17: Running time for various database sizes using the training dataset
In addition, times were collected for the various database sizes when using PropSearch, BLAST, and PSI-BLAST. These results are depicted in Figure 5.13.
115
Figure 5.13: Time complexity comparison between various remote homology detectors
The plot shows the DST running time increasing linearly with database size. While BLAST shows the fastest running time, the DST tracks linearly with PSI-BLAST but is 17% faster. PropSearch at 52 queries (largest database size) increased considerably in running time. Thus, while the DST results in slower time performance than BLAST, it is faster than both PropSearch and PSI-BLAST.
The GA running time also increases linearly with database size. Figure 5.14 illustrates the shape of the curve as the GA database size increases. As can be seen the GA running time increases significantly with database size.
0:00:00 0:00:43 0:01:26 0:02:10 0:02:53 0:03:36 0:04:19 0:05:02 0:05:46 0:06:29 0 100 200 300 400 500 600 700 800 Ti m e i n Se co nd s Database Size in kB
Time Complexity Comparison Between Various Remote Homology Detectors
DST BLAST PSI-BLAST PropSearch
116
Figure 5.14: Time complexity analysis for GA