Measuring the performance of a location-aware text prediction system

Abstract

In recent years, some works have discussed the conception of location-aware Augmentative and Alternative Communication (AAC) systems with very positive feedback from participants. However, in most cases, complementary quantitative evaluations have not been carried out to confirm those results. To contribute to clarifying the validity of these approaches, our study quantitatively evaluated the effect of using language models with location knowledge on the efficiency of a word and sentence prediction system. Using corpora collected for three different locations (classroom, school cafeteria, home), location-specific language models were trained with sentences from each location and compared with a traditional all-purpose language model, trained on all corpora. User tests showed a modest mean improvement of 2.4% and 1.3% for Words Per Minute (WPM) and Keystroke Saving Rate (KSR), respectively, but the differences were not statistically significant. Since our text prediction system relies on the concept of sentence reuse, we ran a set of simulations with language models having different sentence knowledge levels (0%, 25%, 50%, 75%, 100%). We also introduced in the comparison a second location-aware strategy that combines the location-specific approach with the all-purpose approach (mixed approach). The mixed language models performed better under low sentence-reuse conditions (0%, 25%, 50%) with 1.0%, 1.3%, and 1.2% KSR improvements, respectively. The location-specific language models performed better under high sentence-reuse conditions (75%, 100%) with 1.7% and 1.5% KSR improvements, respectively.