Idiom decomposability is measured in this rating study by native speakers of English. 2.2.3.1 Participants
136 native speakers of American English (average age= 33, SD= 11.67, male= 77) participated in the study. Participants were recruited online via Amazon’s MTurk website and were paid to take part in the study. L2 ratings on decomposability were not collected as previous research has shown that L2 participants use decomposition as a method for comprehension, even when idioms are not decomposable, and therefore do not provide accurate ratings for this measure (e.g., Abel, 2003b). 2.2.3.2 Materials
The 300 idioms were divided into 6 lists (50 idioms per list), and a minimum of 20 participants saw each list.
2.2.3.3 Procedure
After providing consent, participants were given detailed instructions and examples on how to rate decomposability. While definitions on this variable vary between authors (e.g., Gibbs & Nayak, 1989; Titone & Connine, 1994a), we adapted the definition from Titone and Connine (1994a) for easy comparison. Decomposability was defined as the extent to which the individual words contribute to the overall figurative meaning of the idiom. Participants were asked to decide whether the idiom is decomposable by selecting YES or NO, and then to indicate how difficult they found the decision on a scale from 1-7 (1 indicating very easy and 7 indicating very difficult). The indication of difficulty for the task was intended as a measure of certainty. The entire task took about 20 minutes. Screen shots from the study can be found in Appendix A.
2.2.3.4 Analysis and results
Participants’ YES/NO responses were compiled into the percentage of responses indicating that the idiom is decomposable. Based on the classifications from Abel (2003b), idioms with 85% of responses indicating either YES or NO were classified as decomposable (D) or non-decomposable (ND), respectively. If idioms did not reach this threshold, they were classified as neither (NA). Both the percentage of YES responses and the classification are listed in the database. Additionally, the ratings on the difficulty of the decision were averaged for each idiom, and this mean as well as the standard
31
error of the mean is provided in the database. While the difficulty of the decision cannot be directly compared to the earlier databases as it was not measured, it serves as an indication of certainty in the case that ratings differ either from earlier studies or experimenter expectations.
Table 2-3. Decomposability classifications and average responses per category
Decomposability n YES Responses Difficulty (sd)
decomposable 67 85% 2.79 (0.51)
non-decomposable 51 15% 3.04 (0.50)
neither 182 52% 3.40 (0.05)
Overall, 67 idioms were classified as decomposable, 51 as non-decomposable, and 182 did not meet the threshold for either. This distribution is displayed in Table 2-3 including also the average percentage of decomposable classifications (YES responses) and difficulty rating (from 1-5) for each category. As a whole, the idioms rated and classified as decomposable were easiest for participants, while the unclassified idioms were the most difficult. The distribution of mean difficulty ratings by the percentage of YES responses is displayed in Figure 2-1. While one may use the percentage of responses as scalar indication of how decomposable the idiom is, the variability of these responses may be an argument against such use. Additionally, as a majority of the idioms could be placed neither as clearly decomposable or non-decomposable, using only decomposability as a norm for studies of idiom processing may be difficult based on these great disagreements in classification.
32 2.2.4 Experiment 3: Predictability
Idiom predictability is measured in this study using cloze-probability responses in order to identify the placement of the idiomatic key of the idiom (Cacciari & Tabossi, 1988).
2.2.4.1 Participants
183 native speakers of American English (average age= 33, SD= 9.92, male= 80) participated in the online study. Participants were recruited online via Amazon’s MTurk website and were paid to take part in the study. Native speakers were used for this measure so as to ensure that predictability was more dependent on the predictive properties of the idiomatic phrase rather than differences in exposure via L2 learning of the idiom.
2.2.4.2 Materials
For this study, idioms were first divided into 4 lists, and final constituent words were removed from the idioms (e.g., throw money out the …[window]) as in Titone and Connine (1994a). Each list consisted of 160 items including fillers, all of which were incomplete phrases. A total of 100 filler items were developed for this purpose with differing lengths (e.g., do the…, ladies and…, and have a
problem with...). If idioms were correctly identified in one of the four lists by 70% of participants, they
were compiled and included in a new list in an additional round of testing with one additional word removed (e.g., throw money out …[the window]). Unlike in the previous study, which looked at idioms with only the final word removed, this process was repeated several times to allow for a more precise pinpointing of the idiomatic key. This process was repeated until idioms were no longer correctly identified. In total, 8 lists were presented to at least 20 participants each: 4 lists in Round 1, 3 lists in Round 2, and 1 list in Round 3.
2.2.4.3 Procedure
After giving consent, participants were informed that they would be doing a “complete the phrase” task. Participants were asked to complete the presented phrase naturally with the first word or words that come to mind which also complete the phrase in a meaningful manner. A screen shot with the examples and instructions can be found in Appendix A. The task took about 20 minutes to complete. 2.2.4.4 Analysis and results
Correct responses to idioms were analyzed after each individual round as correct or incorrect. In the case of items that were nearly correct, two native speaker judges agreed on a correct or incorrect judgement. A threshold of 70% of correct completions was set in order to determine the placement of the idiomatic key (Cacciari & Tabossi, 1988; Titone & Connine, 1994b). If the idiom was not completed correctly in the first round with a single constituent removed, the percentage of correct completions is listed in “Idiomatic Key, Predictability Percentage” in the database, and the idiomatic key is identified as the final constituent word. If the idiom was correctly identified, the idiomatic key
33
is marked with the symbol | following the key word under “Idiomatic Key” and the percentage of correct completions with the presentation of this word is listed in “Idiomatic Key, Predictability Percentage” in the database.
Table 2-4. Summary of total idioms and norming values from L1 and L2 participants
L1 Ratings L2 Ratings
Predictability n Familiarity Meaningfulness Familiarity Meaningfulness final word 233 5.80 (0.88) 5.62 (1.01) 4.81 (1.23) 4.09 (1.37) penultimate
word
39 6.25 (0.37) 6.20 (0.41) 4.83 (1.02) 3.96 (1.11) earlier 28 6.21 (0.41) 6.10 (0.51) 5.18 (1.27) 4.27 (1.42)
Note: n stands for the total number and the value in parenthesis is the standard deviation.
Of the 300 idioms, 233 were unpredictable before the final word, 39 were predictable at the penultimate word, and 28 were predictable at least 2 words before the end. See Table 2-4 for a summary of the results and average ratings for each of these groups. Comparisons between groups were not undertaken, as the group of unpredictable idioms is much greater than the other two groups. However, it appears that the most predictable idioms are also very familiar and meaningful. While these only represent L1 intuitions on predictability, this study set a high threshold of 70% predictability compared to 50% in earlier studies (e.g., Titone & Connine, 1994b). This higher threshold of 70% may better represents the intuitions of more language users, and hopefully include less-experienced language users, such as L2 or L1 learners.
2.3 C
ONCLUSIONThe current study provides an important addition to the available descriptive norms on American English idioms. In addition to updating existing databases with more current ratings (e.g., Titone & Connine, 1994a), it adds the novelty of L2 ratings for appropriate norms. Critically, some but not all of the measures correlated between L1 and L2 participants. Research that considers more than one speaker group must therefore also consider norms from the participant groups being tested rather than relying on L1 norms alone. Including L2 ratings allows researchers to more carefully select appropriate idioms for studies (e.g., high-familiarity for testing idiomatic processing or low-familiarity for testing acquisition and learning) and it opens up more possibilities for data analysis that may account for variability (i.e., including such norms in LMER models).
Finally, the online availability of the database as well as the raw data serves as an open source for future L1 and L2 (German L1) research in idiomatic processing and may also serve as an example to researchers looking to expand on L2 information about such idioms in other languages. While there are a number of studies dealing with L2 learner groups (e.g., Cieślicka, 2006; Siyanova-Chanturia,
34
Conklin, & Schmitt, 2011), many studies continue to utilize their own individual ratings in pre-studies because of a lack of availability of norming data on idioms for L2 learners or a lack of idioms appropriate for specific tasks with norms, even in a language as studied as English. The current data are also stored in the publicly available and searchable Tübingen CLARIN-D Repository (https://talar.sfb833.uni-tuebingen.de/about/), which specializes in web-based services for the Humanities and Social Sciences (see https://www.clarin-d.net/en/). By archiving the data here, the data will be available in a sustainable, long-term non-proprietary format, and can be easily found and accessed by future researchers studying idiomatic processing and beyond.