The main aim of the study was to determine whether the FAPI is an appropriate and reliable monitoring tool to describe the auditory performance of young children with ASD, regardless of the heterogeneity of the participants or their level of interaction. The aim was not to describe the instrument’s intervention monitoring properties but to determine reliability for use with children with ASD. The objectives were to describe the overall performance of 5-year-old children with ASD on the instrument and to determine the test–retest reliability and the inter-rater reliability of the FAPI. The study was exploratory in nature, as a relatively new and emerging subject was investigated (Fouché & De Vos 2011). The method was observational, using repeated measures (Leedy & Ormrod 2005). The design allowed comparison of the same participants’ performance, based on direct observation, over three sessions. The intention was not to trace or monitor development, but to determine the test–retest reliability of the FAPI as a tool for children with ASD.

The principals of three participating private preschools for learners with special needs gave permission to use their facilities for data collection. Parents of all participants gave informed consent. Participants were purposively selected according to the following inclusion criteria: 5-year-old children, any gender, formally diagnosed with ASD, with normal hearing, school attendance of longer than 6 months to exclude adjustment variables and with mothers who had a homogeneous education level. Because the language of learning and teaching was English in the participating schools and South African children are characterised by many different home languages, all participants had to be English as an additional language (EAL) learners. Participant characteristics are described in Table 1.

According to Table 1 all the participants were 5-year-old EAL learners, with four different home languages, all commonly spoken in the city where the research was conducted. According to parental report, 10 of the participants were non-verbal and communicated only by using gestures. The participants had been in their schools long enough to have acquired some basic interactive communication skills in English. They all displayed normal hearing based on a formal screening test. All had been formally diagnosed with ASD according to the DSM-IV (APA 2000), as they were diagnosed before the DSM-5 was used in South Africa. The participants’ mothers had a high education level.

Two speech-language therapists working in private practice in Pretoria were approached to participate in the research as raters of the video recordings. The two raters were qualified speech-language therapists with respectively 22 years (Rater 1) and 2 years (Rater 2) of clinical experience.

The FAPI, a progress monitoring tool of functional auditory skills developed by Stredler-Brown and Johnson, was formally revised in 2001, 2003 and 2004, translated into Spanish (Ferreira et al. 2011). The instrument is now mostly used in the Colorado Home Intervention Program in the USA for children with hearing impairment to compare their progress in intervention with themselves (Johnson & Stredler-Brown, pers. comm., 7 April 2016). The tool describes auditory performance based on observation of a child’s behaviour in seven auditory developmental areas:

A total of 33 skills are described within the seven areas. Because the aim of the FAPI is to monitor a child’s progress with him- or herself over time, no norms are used to interpret the data. A four-point Likert scale is used to score the child’s level of attainment of auditory performance, as follows: 1, ‘not present’ (with a score value of 0); 2, ‘emerging’ (with a score value of 1); 3, ‘in process’ (with a score value of 2); 4, ‘acquired’ (with a score value of 3), as indicated in Table 2.

For the present study pre-recorded sound stimuli were used to ensure a standard presentation format for each child, and across three data-collection sessions. A 20-minute recording was made inside an audiometric soundproof booth. An Olympus digital voice recorder (VN-5500PC) was used to record the stimuli. The stimuli followed the hierarchy of the monitoring tool, starting with basic sounds such as noisemakers (rattles and whistles); contemporary and classical music; environmental sounds, such as a fire engine driving by; and ending with speech prompts to elicit verbal responses from the child, such as imitation of sounds perceived or vocal reactions and a short story about a monkey collecting yellow bananas from the forest across the river. The speech sample was in South African English by a female EAL speaker and included typical supra-segmental characteristics (intensity, duration and pitch variations). The recorded voice named objects, gave directions and prompted the child to communicate. The intensity of the sound stimuli varied between 40 and 55 dB, which was regulated by means of the ‘Talk forward’ function of the audiometer. Material to keep the child busy but not overly engaged while the pre-recorded stimuli were presented included age-appropriate books, bubbles, puzzles and an electronic tablet. All sessions were recorded with a Canon video camera.

A pilot study offered the opportunity to test the positioning of the camera and the placement of a playback device. Three data-collection sessions with a pilot participant were conducted at school in a familiar therapy room over a 2-week period. The pre-recorded stimuli, placement of the table and two chairs, and tabletop activities to keep the child busy were tested. Changes to improve the sound presentation and the quality of the video recording were made after the pilot study. The intensity of the playback was changed to the average intensity of speech production (40–75 dB) and the playback device was placed at a higher level, but within 1 meter of the participant.

For the main study, data collection took place at the participants’ preschools. Prior to the formal recording and sound stimuli presentation, the researcher spent time with the participant in his or her classroom, where after the child willingly accompanied the researcher to a familiar therapy room, where data collection took place. The camera was already positioned on a table 2 metres away from the participant, with the participant’s chair facing the camera. The playback device was placed behind the participant, 1 meter away, but elevated to prevent the participants from reaching for the apparatus. The researcher explained to the participants that they were going to hear different sounds and a lady talking while playing with the researcher. After 5 minutes in the data collection room, the sound playback and video recording commenced. The researcher did not draw attention to the camera, but if a participant enquired about it, the researcher responded that they were going to make a movie. The researcher interacted quietly with the child by paging through a book, building puzzles and playing with an electronic tablet to mimic the natural environment in the school, while the pre-recorded stimuli were played back.

Each participant was recorded three times within a 2-week period. The dependent variable in this study was the functional auditory performance of the participant. The independent variables, which were kept consistent throughout the recordings, included the positioning of the furniture, video camera and the playback device, intensity of the pre-recorded stimuli, the duration of a session, the same facilitator and tabletop activities. The only variable that could not be controlled was environmental noise. Occasionally a child shouted outside or an ambulance siren could be heard, but it was found that the participants did not react to the sounds from the external environment.

Each participant’s data were collected over a 2-week period, with a time lapse of 3 days or more between recordings. The time lapse decreased the familiarity with the recorded stimuli, prohibiting the practice effect. In order to prevent interference with the pre-recorded stimuli, no verbal interaction took place between the researcher and participant during playback of the recorded stimuli.

The data of the main study were scored by two independent raters after they had received training on how to score the FAPI (Stredler-Brown & Johnson 2004). Data obtained from the pilot study were used to train the raters. The raters had to watch the complete video recordings several times before rating the participants’ auditory responses. The video recordings were provided on DVD. The recordings were arranged in random order so that raters could not predict the performance of a particular participant in subsequent ratings (Bless & Higson-Smith 2004; Leedy & Ormrod 2005). To enhance the reliability of the scoring further, the raters were blinded to the scores of the previous sessions by the use of the freeze frame function of Excel. This function ensured that a rater could not keep track of the actual chronological sequence of each participant’s scores.

Each rater captured the scores directly onto an Excel spread sheet. The duration of the analyses depended on the rater. Uninterrupted viewing of a single recording took approximately 20 minutes. Analysis and scoring of a single recording took a further 30–35 minutes. Each rater analysed a total of 36 recordings.

Data were analysed using Statistica version 12 to describe the overall performance of the participants for their auditory functioning and to determine the test–retest reliability and the inter-rater reliability. The test–retest reliability, in this case the intra-rater reliability, measures the degree of agreement when multiple repetitions of a tool are performed (Banach 2012). Nonparametric statistics were used owing to the small sample size of the study, the distribution of the data and the unpredictability and inconsistency of the participants’ behaviour. Average scores for each category were determined. The Friedman two-way analysis, which is the nonparametric equivalent of the parametric repeated measures ANOVA, was conducted to determine the test–retest reliability (Leedy & Ormrod 2005). The Wilcoxon signed-rank test was used to compare the scores obtained from the raters to determine the inter-rater reliability. The Kendall coefficient of concordance was used to determine the test–retest reliability of the raters’ performance.

Averages of the three recordings of each participant were calculated to obtain the mean group scores on the seven categories of the FAPI. The results are presented in Figure 1.

As shown in Figure 1, the results indicate that participants showed a decrease in responses to the pre-recorded auditory stimuli with an increase in the complexity of the stimuli. The participants showed limited awareness of the sound (24.8% for Category 1), but even poorer responses for discrimination, comprehension, memory and linguistic-auditory processing were noted. A sharp decline in reaction and participation was clearly noted from Category 4 onwards. An average of only 2.2% responses was observed in Category 7.

Category 1 represents the ability to identify the presence of sound and match the sound to its source in order to attach meaning to it. Typical responses of the participants were looking up or stopping an activity and, in some cases, pausing momentarily without looking up. Typical responses to Category 2 (auditory feedback and integration) were imitating the sounds. In some cases participants covered their ears in reaction to the sound; some were startled with the initial presentation of the sound. Another response included imitation of the sounds more loudly than those of the recording, possibly in an attempt to block out the incoming stimuli. The average score decreased from 24.8% for Category 1 to only 17.6% for Category 2. With localisation of the sound source (Category 3) the participants turned their heads or looked in the general direction of the sound source 16% of the time. Very few responses were observed for Categories 4–7, which include language and discriminating between sounds, words and instruction. Discrimination of sounds was observed when a participant imitated minimally paired words such as cat-hat. Observable comprehension of a word or utterance occurred only 2.1% of the time, such as when a participant identified a correct object. With short-term auditory memory (Category 6) a slight increase was observed with an average of 3.2%. Being a monitoring tool, the FAPI compares the scores obtained by an individual over time and does not rely on standard norms. It is, however, clear that the group of participants showed limited and unusual responses to the sound stimuli and that an obvious pattern of decline could be observed when stimuli became complex. The poorest responses were recorded for understanding of continuous speech, that is, Categories 5 and 7.

To determine the test–retest or intra-rater reliability, each rater’s three scores for the same participant were compared. The repeated measures for each of the seven categories of the FAPI for Raters 1 and 2 are indicated in Table 3 and 4, respectively.

According to Tables 3 and 4, the only category that presented with a significant difference in scoring was Category 1, scored by Rater 1. The statistically significant differences were evident in the ratings of Rater 1 with a median of 46.18% for Rating 1 and medians of 24.36 and 27.02% for Ratings 2 and 3. The median score allocated for Recording 1 showed an outlier. The differences between the three ratings were statistically significant with a p < 0.05 at 0.003. Table 4 indicates that the consistency of Rater 2’s scoring of Category 7 is within the 10% level of significance and therefore marginally inconsistent. The results of Rater 1, Category 1, are also depicted in Figure 2.

A possible explanation for the variation in scoring of the two raters could be that Rater 1 had 20 years more clinical experience and possibly paid closer attention while scoring the recordings. Subtle differences in participants’ behaviours across the three recordings would be noticed by an experienced rater. It could also be that the variations in scoring represent true variations in the participants’ responses over the three data-recording sessions. Tomchek and Dunn (2007) reported that sensory processing of children with ASD varies significantly from typically developing peers, especially when considering emotional reactivity, low endurance, inattention and poor registration. The variation leads to a lack of consistency among sensory studies. The test-retest reliability of the FAPI was therefore high, as only one test category showed a significant difference across the three recordings. Because the video recordings were placed in random order, the raters could not establish a routine following the chronological order of the data collections. The random order would have prevented a rater being more attentive during the first video recording than during subsequent recordings. Because there was only one category that presented with a significant difference, the conclusion is that the test–retest or intra-rater reliability for the FAPI for this specific sample was high.

Table 5 provides the standard deviation for each category of the FAPI, indicating whether there were similarities or differences between the two raters’ scores.

According to Table 5 there were significant differences between the scoring of the two raters for Categories 1 and 2 of the FÀPI. No significant differences were found between the scoring of the two raters for Categories 3–7. The data of the inter-rater reliability between the two raters for Category 1 are also depicted in Figure 3.

Figure 3 presents the average scores allocated for Category 1 by the two raters. The results clearly demonstrate the difference between the ratings of Rater 1 and Rater 2 for Category 1. Significant differences in scores were noted when comparing the weighted scores of Rater 1, scoring a maximum of 61%, with those of Rater 2, with a maximum score of 39% (p-value of 0.006). A possible reason for the difference is the subjectivity of the items in this category. Similar differences were found between the ratings of Category 2 as depicted in Figure 4.

In Figure 4 both raters’ lowest weighted scores converged at 0%. The score was given when no response could be observed with Category 2, Auditory feedback and integration. The more complex the auditory stimuli became, the fewer responses were observed in the participants. The maximum weighted scores of the two raters showed a 20% difference, with a p-value of 0.006, indicating a statistically significant difference.

The first two categories of the FAPI, Awareness and meaning of sounds and Auditory feedback and integration, rely on subjective scoring as there may be subtle variation in responses. An experienced rater may distinguish between a child pausing or ending an activity and looking up, or actually searching for the sound source. An experienced rater may therefore identify subtle differences, which can explain why the scores were more varied here than for the inexperienced rater. For both Categories 1 and 2, subjectivity may have influenced the scoring. Categories 3–7 include distinct responses and actions, such as repetition of a sequence of numbers where the participant either responds or fails to respond, which makes it easier to score. It may thus be concluded that, with thorough training in the scoring of this tool, consistent scoring may be achieved.