Reliability of the Functional Auditory Performance Indicators to monitor progress in 5-year-old children with autism spectrum disorder

The autism spectrum disorder (ASD) diagnostic criteria now divide the typical characteristics of the condition into two psychopathological domains (American Psychiatric Association [APA] 2013). The first domain includes deficits in social communication and social interaction across contexts, as well as deficits in nonverbal communicative behaviours. The second domain includes stereotyped motor movements, speech or use of objects; insistence on sameness; a focus on highly restricted, fixated interests; and hyperor hypo-reactivity to sensory input or activities. The description of variable reactivity to sensory input and activities, which includes ‘adverse ....’ response to specific sounds’ (APA 2013), places a new emphasis on an area of impairment in ASD that might have been underreported in the past. The auditory processing difficulties in this population are described as unique because of the heterogeneous patterns observed in response to sound and speech (Carpenter et al. 2014; Kargas & Lo 2015). Siegal and Blades (2003) state that difficulties with auditory processing in children with ASD may limit their participation in conversations, which may contribute to social isolation. In addition Paul et al. (2007) found that children with ASD show reduced responses to child-directed speech in comparison to neurotypical peers and that time spent listening to child-directed speech was related to the participants’ current and later receptive language development. In pursuit of evidence-based practice, it is therefore important that speech-language therapists participate in research relating to the auditory behaviour of children with ASD and remain informed about the heterogeneous group of individuals (American Speech-Language-Hearing Association [ASHA] 2007).


Introduction
The autism spectrum disorder (ASD) diagnostic criteria now divide the typical characteristics of the condition into two psychopathological domains (American Psychiatric Association [APA] 2013).The first domain includes deficits in social communication and social interaction across contexts, as well as deficits in nonverbal communicative behaviours.The second domain includes stereotyped motor movements, speech or use of objects; insistence on sameness; a focus on highly restricted, fixated interests; and hyper-or hypo-reactivity to sensory input or activities.The description of variable reactivity to sensory input and activities, which includes 'adverse ….' response to specific sounds' (APA 2013), places a new emphasis on an area of impairment in ASD that might have been underreported in the past.The auditory processing difficulties in this population are described as unique because of the heterogeneous patterns observed in response to sound and speech (Carpenter et al. 2014;Kargas & Lo 2015).Siegal and Blades (2003) state that difficulties with auditory processing in children with ASD may limit their participation in conversations, which may contribute to social isolation.In addition Paul et al. (2007) found that children with ASD show reduced responses to child-directed speech in comparison to neurotypical peers and that time spent listening to child-directed speech was related to the participants' current and later receptive language development.In pursuit of evidence-based practice, it is therefore important that speech-language therapists participate in research relating to the auditory behaviour of children with ASD and remain informed about the heterogeneous group of individuals (American Speech-Language-Hearing Association [ASHA] 2007).
A number of studies have been conducted regarding auditory performance in individuals with ASD (Alcantara et al. 2004;Azouz et al. 2014;Ferguson & Moore 2014).Most studies were conducted in the field of neurophysiology and were executed in controlled environments.The concept of unique listening skills, behaviours and difficulties in the population of children with ASD is supported by studies using neuroimaging and other experimental techniques (Bruneau et al. 2003;Ceponiene et al. 2003;Haesen, Boets & Wagemans 2011;Kuhl et al. 2005).Using mismatch negativity and event-related potentials, Kuhl et al. (2005) found that neurotypical children and There is a need to dynamically monitor progress of functional auditory performance in young children with autism spectrum disorder (ASD).The Functional Auditory Performance Indicators (FAPI) is a monitoring tool for children with hearing loss but has not yet been described in children with ASD.The aim was to describe the overall performance of 5-year-old children with ASD on the FAPI and to determine the test-retest reliability and inter-rater reliability of the tool.The study was exploratory with a descriptive within-subjects design incorporating repeated measures.Twelve participants with ASD were purposely selected.Prerecorded sound and speech stimuli were used to elicit responses from participants in their familiar therapy rooms.For test-retest reliability, three data collection sessions per participant were conducted over a 2-week period.Video recordings were analysed by two independent raters, who were blind to the order of data sets.With an increase in complexity of auditory stimuli, a marked decrease in response was observed.The test-retest reliability was good, with a single difference in one category.Inter-rater reliability indicated a significant difference in two of the seven categories.These categories may be the most subjective in the tool.Despite subjectivity the FAPI was reliable to plot functional auditory difficulties in the sample group.Because the instrument relies on direct observation with limited demands to participate with the rater, it has potential for use in children with ASD.Further research is required to determine the tool's performance using natural sound conditions to monitor children's progress against themselves during intervention.

Read online:
Scan this QR code with your smart phone or mobile device to read online.
those diagnosed with ASD presented with different neural and behavioural reactions to speech.The majority of participants with ASD in this study preferred non-speech analogue signals, characterised by continuous non-speech electronic input.Based on their listening preferences, the participants were divided into two groups.Participants who presented with a preference for speech also presented with less severe symptoms of ASD and were thus considered high functioning.Those who did not attend to speech were participants with severe ASD.The findings not only support the notion that autism represents a spectrum of impairments and cannot be described as a homogenous category but also reflect the children's difficulty with listening and attending to complex sound, that is, speech.Other characteristics of peculiar responses to sound in children with ASD include inconsistent response to their name, sound aversion, decreased awareness or recognition of a caregiver's voice, neglecting to pay attention to speech yet presenting with an awareness of environmental sounds, and a lack of interest or response to neutral statements (Johnson & Myers 2007).
There appear to be different interpretations of the atypical responses of children with ASD to speech.Paul et al. (2007).ascribe the problems that children with ASD experience with understanding sounds, especially speech directed at them, to pervasive difficulties in regulating responses, problems with paying attention to a range of stimuli and lack of motivation to participate in social interaction, rather than to auditory processing difficulties.It has become increasingly evident that neurophysiological studies ascribe atypical listening behaviour in children with ASD to auditory processing difficulties (Carpenter et al. 2014;Kuhl et al. 2005).However, neurophysiological tests are not accessible to practising speech-language therapists to understand and monitor the auditory skills of their young clients with ASD.Evidence of the auditory processing deficits in this population is therefore mostly based on neurophysiological studies, but limited research relating to the behavioural response to speech and sound in clinical contexts is available.The identification of a reliable clinical monitoring tool may thus assist in describing and understanding the functional auditory skills of children with ASD in a natural environment, allowing speech-language therapists to plan interventions and monitor progress.
The field of audiology offers investigative tools for children with hearing loss and their auditory skills (Johnson & Seaton 2012;Zhang et al. 2012)

Method Aims and design
The main aim of the study was to determine whether the FAPI is an appropriate and reliable monitoring tool to describe the auditory performance of young children with ASD, regardless of the heterogeneity of the participants or their level of interaction.The aim was not to describe the instrument's intervention monitoring properties but to determine reliability for use with children with ASD.The objectives were to describe the overall performance of 5-year-old children with ASD on the instrument and to determine the test-retest reliability and the inter-rater reliability of the FAPI.The study was exploratory in nature, as a relatively new and emerging subject was investigated (Fouché & De Vos 2011).The method was observational, using repeated measures (Leedy & Ormrod 2005).The design allowed comparison of the same participants' performance, based on direct observation, over three sessions.The intention was not to trace or monitor development, but to determine the test-retest reliability of the FAPI as a tool for children with ASD.

Participants
The principals of three participating private preschools for learners with special needs gave permission to use their facilities for data collection.Parents of all participants gave informed consent.Participants were purposively selected according to the following inclusion criteria: 5-year-old children, any gender, formally diagnosed with ASD, with normal hearing, school attendance of longer than 6 months to exclude adjustment variables and with mothers who had a homogeneous education level.Because the language of learning and teaching was English in the participating schools and South African children are characterised by many different home languages, all participants had to be English as an additional language (EAL) learners.Participant characteristics are described in Table 1.
According to Table 1 all the participants were 5-year-old EAL learners, with four different home languages, all commonly spoken in the city where the research was conducted.
According to parental report, 10 of the participants were non-verbal and communicated only by using gestures.
The participants had been in their schools long enough to have acquired some basic interactive communication skills in English.They all displayed normal hearing based on a formal screening test.All had been formally diagnosed with ASD according to the DSM-IV (APA 2000), as they were diagnosed before the DSM-5 was used in South Africa.The participants' mothers had a high education level.
Two speech-language therapists working in private practice in Pretoria were approached to participate in the research as raters of the video recordings.The two raters were qualified speech-language therapists with respectively 22 years (Rater 1) and 2 years (Rater 2) of clinical experience.

Material and apparatus
The FAPI, a progress monitoring tool of functional auditory skills developed by Stredler-Brown and Johnson, was formally revised in 2001, 2003and 2004, translated into Spanish (Ferreira et al. 2011).The instrument is now mostly used in the Colorado Home Intervention Program in the USA for children with hearing impairment to compare their progress in intervention with themselves (Johnson & Stredler-Brown, pers. comm., 7 April 2016).The tool describes auditory performance based on observation of a child's behaviour in seven auditory developmental areas: 1. Awareness and meaning of sounds: the child is aware of a specific sound and can associate a variety of sounds with a specific sound source.Stimuli that the child should respond to include environmental sounds, music, vocalisations and discourse.2. Auditory feedback and integration: the child reacts and adapts his or her own vocalisations based on auditory input received, such as responding to sounds with the production of own vocalisations.3. Localise sound sources: the child actively looks for the sound source.4. Auditory discrimination: the child distinguishes between characteristics such as intensity, pitch and duration of different sounds, vocally producing sounds in reaction on the same pitch or with the same variation in pitch. 5. Auditory comprehension: the child understands spoken language and can identify important ideas of the message.6.Short-term auditory memory: the child perceives, remembers, restates and recalls a sequence of numbers.7. Linguistic auditory processing: the child uses auditory information to process language, such as understanding an instruction and executing it correctly, or making use of syntactic language in response to the information received.
A total of 33 skills are described within the seven areas.
Because the aim of the FAPI is to monitor a child's progress with him-or herself over time, no norms are used to interpret the data.A four-point Likert scale is used to score the child's level of attainment of auditory performance, as follows: 1, 'not present' (with a score value of 0); 2, 'emerging' (with a score value of 1); 3, 'in process' (with a score value of 2); 4, 'acquired' (with a score value of 3), as indicated in Table 2.
For the present study pre-recorded sound stimuli were used to ensure a standard presentation format for each child, and across three data-collection sessions.A 20-minute recording was made inside an audiometric soundproof booth.An Olympus digital voice recorder (VN-5500PC) was used to record the stimuli.

Procedures
A pilot study offered the opportunity to test the positioning of the camera and the placement of a playback device.Three data-collection sessions with a pilot participant were conducted at school in a familiar therapy room over a 2-week period.The pre-recorded stimuli, placement of the table and two chairs, and tabletop activities to keep the child busy were tested.Changes to improve the sound presentation and the quality of the video recording were made after the pilot study.The intensity of the playback was changed to the average intensity of speech production (40-75 dB) and the playback device was placed at a higher level, but within 1 meter of the participant.
For the main study, data collection took place at the participants' preschools.Prior to the formal recording and sound stimuli presentation, the researcher spent time with the participant in his or her classroom, where after the child willingly accompanied the researcher to a familiar therapy room, where data collection took place.The camera was already positioned on a table 2 metres away from the participant, with the participant's chair facing the camera.The playback device was placed behind the participant, 1 meter away, but elevated to prevent the participants from reaching for the apparatus.The researcher explained to the participants that they were going to hear different sounds and a lady talking while playing with the researcher.After 5 minutes in the data collection room, the sound playback and video recording commenced.The researcher did not draw attention to the camera, but if a participant enquired about it, the researcher responded that they were going to make a movie.The researcher interacted quietly with the child by paging through a book, building puzzles and playing with an electronic tablet to mimic the natural environment in the school, while the pre-recorded stimuli were played back.
Each participant was recorded three times within a 2-week period.The dependent variable in this study was the functional auditory performance of the participant.The independent variables, which were kept consistent throughout the recordings, included the positioning of the furniture, video camera and the playback device, intensity of the prerecorded stimuli, the duration of a session, the same facilitator and tabletop activities.The only variable that could not be controlled was environmental noise.Occasionally a child shouted outside or an ambulance siren could be heard, but it was found that the participants did not react to the sounds from the external environment.
Each participant's data were collected over a 2-week period, with a time lapse of 3 days or more between recordings.The time lapse decreased the familiarity with the recorded stimuli, prohibiting the practice effect.In order to prevent interference with the pre-recorded stimuli, no verbal interaction took place between the researcher and participant during playback of the recorded stimuli.

Data analysis
The data of the main study were scored by two independent raters after they had received training on how to score the FAPI (Stredler-Brown & Johnson 2004).Data obtained from the pilot study were used to train the raters.The raters had to watch the complete video recordings several times before rating the participants' auditory responses.The video recordings were provided on DVD.The recordings were arranged in random order so that raters could not predict the performance of a particular participant in subsequent ratings (Bless & Higson-Smith 2004;Leedy & Ormrod 2005).To enhance the reliability of the scoring further, the raters were blinded to the scores of the previous sessions by the use of the freeze frame function of Excel.This function ensured that a rater could not keep track of the actual chronological sequence of each participant's scores.
Each rater captured the scores directly onto an Excel spread sheet.The duration of the analyses depended on the rater.Uninterrupted viewing of a single recording took approximately 20 minutes.Analysis and scoring of a single recording took a further 30-35 minutes.Each rater analysed a total of 36 recordings.
Data were analysed using Statistica version 12 to describe the overall performance of the participants for their auditory functioning and to determine the test-retest reliability and the inter-rater reliability.The test-retest reliability, in this case the intra-rater reliability, measures the degree of agreement when multiple repetitions of a tool are performed (Banach 2012).Nonparametric statistics were used owing to the small sample size of the study, the distribution of the data and the unpredictability and inconsistency of the participants' behaviour.Average scores for each category were determined.The Friedman two-way analysis, which is the nonparametric equivalent of the parametric repeated measures ANOVA, was conducted to determine the test-retest reliability (Leedy & Ormrod 2005).The Wilcoxon signed-rank test was used to compare the scores obtained from the raters to determine the inter-rater reliability.The Kendall coefficient of concordance was used to determine the test-retest reliability of the raters' performance.

Overall performance of the participants on the seven categories of the FAPI
Averages of the three recordings of each participant were calculated to obtain the mean group scores on the seven categories of the FAPI.The results are presented in Figure 1.
As shown in Figure 1, the results indicate that participants showed a decrease in responses to the pre-recorded auditory stimuli with an increase in the complexity of the stimuli.The participants showed limited awareness of the sound (24.8% for Category 1), but even poorer responses for discrimination, comprehension, memory and linguistic-auditory processing were noted.A sharp decline in reaction and participation was clearly noted from Category 4 onwards.An average of only 2.2% responses was observed in Category 7.
Category 1 represents the ability to identify the presence of sound and match the sound to its source in order to attach meaning to it.Typical responses of the participants were looking up or stopping an activity and, in some cases, pausing momentarily without looking up.Typical responses to Category 2 (auditory feedback and integration) were imitating the sounds.In some cases participants covered their ears in reaction to the sound; some were startled with the initial presentation of the sound.Another response included imitation of the sounds more loudly than those of the recording, possibly in an attempt to block out the incoming stimuli.The average score decreased from 24.8% for Category 1 to only 17.6% for Category 2. With localisation of the sound source (Category 3) the participants turned their heads or looked in the general direction of the sound source 16% of the time.Very few responses were observed for Categories 4-7, which include language and discriminating between sounds, words and instruction.Discrimination of sounds was observed when a participant imitated minimally paired words such as cat-hat.Observable comprehension of a word or utterance occurred only 2.1% of the time, such as when a participant identified a correct object.With short-term auditory memory (Category 6) a slight increase was observed with an average of 3.2%.Being a monitoring tool, the FAPI compares the scores obtained by an individual over time and does not rely on standard norms.It is, however, clear that the group of participants showed limited and unusual responses to the sound stimuli and that an obvious pattern of decline could be observed when stimuli became complex.The poorest responses were recorded for understanding of continuous speech, that is, Categories 5 and 7.

Test-retest (intra-rater) reliability of the FAPI
To determine the test-retest or intra-rater reliability, each rater's three scores for the same participant were compared.The repeated measures for each of the seven categories of the FAPI for Raters 1 and 2 are indicated in Table 3 and 4, respectively.
According to Tables 3 and 4, the only category that presented with a significant difference in scoring was Category 1, scored by Rater 1.The statistically significant differences were evident in the ratings of Rater 1 with a median of 46.18% for Rating 1 and medians of 24.36 and 27.02% for Ratings 2 and 3.The median score allocated for Recording 1 showed an outlier.The differences between the three ratings were statistically significant with a p < 0.05 at 0.003.Table 4 indicates that the consistency of Rater 2's scoring of Category 7 is within the 10% level of significance and therefore marginally inconsistent.The results of Rater 1, Category 1, are also depicted in Figure 2.
A possible explanation for the variation in scoring of the two raters could be that Rater 1 had 20 years more clinical experience and possibly paid closer attention while scoring the recordings.Subtle differences in participants' behaviours across the three recordings would be noticed by an experienced rater.It could also be that the variations in scoring represent true variations in the participants' responses over the three data-recording sessions.Tomchek and Dunn (2007) reported that sensory processing of children with ASD varies significantly from typically developing peers, especially when considering emotional reactivity, low endurance, inattention and poor registration.The variation leads to a lack of consistency among sensory studies.The testretest reliability of the FAPI was therefore high, as only one test category showed a significant difference across the three recordings.Because the video recordings were placed in random order, the raters could not establish a routine following the chronological order of the data collections.
The random order would have prevented a rater being more attentive during the first video recording than during subsequent recordings.Because there was only one category that presented with a significant difference, the conclusion is that the test-retest or intra-rater reliability for the FAPI for this specific sample was high.

Inter-rater reliability of the FAPI
Table 5 provides the standard deviation for each category of the FAPI, indicating whether there were similarities or differences between the two raters' scores.
According to Table 5 there were significant differences between the scoring of the two raters for Categories 1 and 2 of the FÀPI.No significant differences were found between the scoring of the two raters for Categories 3-7.The data of the inter-rater reliability between the two raters for Category 1 are also depicted in Figure 3.
Figure 3 presents the average scores allocated for Category 1 by the two raters.The results clearly demonstrate the difference between the ratings of Rater 1 and Rater 2 for Category 1. Significant differences in scores were noted when comparing the weighted scores of Rater 1, scoring a maximum of 61%, with those of Rater 2, with a maximum score of 39% (p-value of 0.006).A possible reason for the difference is the subjectivity of the items in this category.Similar differences were found between the ratings of Category 2 as depicted in Figure 4.
In Figure 4 both raters' lowest weighted scores converged at 0%.The score was given when no response could be observed with Category 2, Auditory feedback and integration.The more complex the auditory stimuli became, the fewer responses were observed in the participants.The maximum weighted scores of the two raters showed a 20% difference, with a p-value of 0.006, indicating a statistically significant difference.
The first two categories of the FAPI, Awareness and meaning of sounds and Auditory feedback and integration, rely on subjective scoring as there may be subtle variation in responses.An experienced rater may distinguish between a child pausing or ending an activity and looking up, or actually searching for the sound source.An experienced rater may therefore identify subtle differences, which can explain why the scores were more varied here than for the inexperienced rater.For both Categories 1 and 2, subjectivity may have influenced the scoring.Categories 3-7 include distinct responses and actions, such as repetition of a sequence of numbers where the participant either responds or fails to respond, which makes it easier to score.It may thus be concluded that, with thorough training in the scoring of this tool, consistent scoring may be achieved.

Discussion
The general observation from the results of the seven categories of the FAPI was that an increase in the complexity of the auditory stimuli showed a decrease in involvement and reaction in the participants.The poorest responses were seen in understanding of variations of speech, which included single words to continuous speech.There are different explanations for poor auditory responses in children with ASD in the literature.Boddaert et al. (2003) found evidence of abnormal cortical processing in a neurophysiological study of adults diagnosed with ASD.
According to O'Connor (2012), trends across studies suggest that auditory processing impairments in individuals with ASD are most likely to present during processing of complex auditory information and are more severe for speech than for non-speech stimuli.Kuhl et al. (2005) indicated that children with ASD, who lack social interest in communication, might be greatly disadvantaged in language learning, which typically depends on social factors.The findings of the current study are in agreement with Kuhl et al. (2005), who observed that non-verbal participants exhibited little to no concern about auditory stimuli that included vocalisations and language.The participants in the current study were mainly non-verbal, and those who had some verbal skills presented with increased observable responses to the sound stimuli.
Other possible explanations for the lack of interest in sound and speech or low response scores observed in participants, specifically in Categories 5 and 7, may be related to the unusual auditory preferences observed in children with ASD (Paul et al. 2007).According to Paul et al. (2007), reduced responses to sound may be associated with inattention, characterised by not being 'tuned in' to language or the sound patterns of language.Failure to tune in to language has a negative impact on auditory functioning, social interaction and language learning (Johnson & Myers 2007;Paul et al. 2007).Additionally, Ludlow et al. (2014) noted that children with ASD experience difficulty when automatic attention is required, presenting with an inability to automatically shift their attention to the variation in sounds that do not fall within their attentional focus.Abnormal automatic processing may also be related to multiple key characteristics associated with ASD, such as failing to notice important auditory information (Marco et al. 2011).The idea of automatic processing is specifically connected to language development (Ludlow et al. 2014).In children with ASD the inability to automatically process auditory information outside their attentional focus may result in the memorisation of specific facts, but poor arrangement of the semantic material results in weak understanding of the connection between concepts (Ludlow et al. 2014).O'Connor (2012) introduces another idea when stating that children with ASD are able to match the pitch contours of a sentence better than their typically developing peers but present with far less comprehension of the content.
The participants in the study were all EAL speakers, busy acquiring English for the past 7-22 months in their nursery schools (see Table 1).According to Shipley and McAfee (2016), basic interactive communication skills in a second language develop over a 2-year period when circumstances are ideal.The role of EAL acquisition in addition to language impairment associated with ASD in the participants was not investigated in the study.Further research, utilising children who are English first language learners, may show the effect of EAL in the present study.
In this study, variables such as room setup, intensity of auditory stimuli presented and duration of the session were kept consistent.Pre-recorded auditory stimuli were used and  generally consistent responses were found in the participants across three successive data collection sessions.When additional variables could not be controlled, such as a participant's behaviour, responses to sound were influenced.It happened that a participant had an emotional outburst in class prior to the recording session, which resulted in minimal to no reaction to the auditory stimuli presented.
With a clear pattern of responses emerging from the data of the present study, the FAPI was therefore a useful tool to plot and describe the functional auditory skills of a group of 5-year-old children with ASD.The inter-rater reliability of the FAPI was found to be acceptable in this study, despite variations in Categories 1 and 2. The areas of poorest performance in the participants were Categories 5 and 7, therefore not the categories where significant interrater differences were found.The poor performance in understanding of continuous speech and a story (Categories 5 and 7) in the participants appear to be a reliable finding.Differences in ratings by the two independent raters were not significant enough to describe the intra-rater reliability as not reliable (Leedy & Ormrod 2005).

Conclusion
Despite relying on subjective scoring for certain categories, heterogeneity in participants and variations in their levels of interaction, the FAPI (Stredler-Brown & Johnson 2004) was found to be a reliable instrument to plot and possibly monitor the functional auditory skills of 5-year-old children with ASD.Video recordings of participants' overt responses to recorded sound and speech stimuli and using an instrument that relies on direct observation of participant responses to sound made it possible to describe the participants' functional auditory performance.Even though most of the participants were non-verbal, responses could still be recorded.The hierarchical structure of listening skill development of the FAPI allowed a holistic view of the auditory difficulties of the participants.The results clearly showed that with an increase in complexity of the auditory stimuli, in particular speech, a decrease in behavioural responses was observed.Further research is required to determine the tool's performance using natural sound conditions to monitor children with ASD's progress against themselves during intervention.There is also a need to investigate the auditory performance of children with ASD over time.The participants who were non-verbal and suspected to have a severe form of ASD presented with lower scores in comparison with the verbal participants, who responded to some extent to complex language stimuli.The nature of functional auditory skills found in the participants clearly showed deficits that may be addressed in intervention.The current research positively indicates that the FAPI is a valuable and reliable instrument for plotting of the functional auditory performance of 5-year-old children diagnosed with ASD and for possibly monitoring their skill development.

TABLE 5 :
Inter-rater reliability for each category of the FAPI.