Examining oral reading fluency among grade 5 rural English Second Language (ESL) learners in South Africa: An analysis of NEEDU 2013

The ability to read for meaning and pleasure is arguably the most important skill children learn in primary school. One integral component of learning to read is Oral Reading Fluency (ORF), defined as the ability to read text quickly, accurately, and with meaningful expression. Although widely acknowledged in the literature as important, to date there have been no large-scale studies on ORF in English in South Africa, despite this being the language of learning and teaching for 90% of students from Grade 4 onwards. As part of the National Education and Evaluation Development Unit (NEEDU) of South Africa, we collected and here analyze data on 4667 grade 5 English Second Language (ESL) students from 214 schools across rural areas in South Africa. This included ORF and comprehension measures for a subset of 1772 students. We find that 41% of the sample were non-readers in English (<40WCPM) and only 6% achieved comprehension scores above 60%. By calibrating comprehension levels and WCPM rates we develop tentative benchmarks and argue that a range of 90-100 WCPM in English is acceptable for grade 5 ESL students in South Africa. In addition we outline policy priorities for remedying the reading crisis in the country.


1) Introduction and background
The ability to read for meaning and pleasure is arguably the most important skill children learn in primary school. Since almost all future learning will depend on this fundamental understanding of the relation between print and spoken language, it is unsurprising that literacy, built upon a firm foundation of basic reading, is used as one of the primary measures of school efficacy. Apart from the obvious cognitive importance of learning to read, children who become novice readers within the first three years of primary school also have higher levels of socio-emotional well-being stemming from improved self-expression and communication as well as the selfconfidence that comes from cracking this difficult code (Chapman, Tunmer & Prochnow, 2000). Sadly, the opportunity of learning to read with fluency, accuracy, prosody and comprehension is one not afforded to the majority of South African children. Whether children are tested in their home language or in English the conclusions are the same; the vast majority of South African children cannot read for meaning by the end of Grade 4 -even in their home language -and almost a third are still functionally illiterate in English by the end of Grade 6 (Spaull, 2013).
The aim of the present study is to add to our understanding of the reading crisis in South Africa by focusing on the oral reading fluency (ORF) of Grade 5 English Second Language (ESL) learners in rural South Africa. To date there have been no large-scale studies focusing on oral reading fluency in English, despite this being the language of learning and teaching for 90% of students from Grade 4 onwards. There are two principle research questions that animate this study: (1) What are the levels of oral reading fluency among grade 5 ESL students in rural areas in South Africa?
(2) Is it possible to identify tentative benchmarks or thresholds of oral reading fluency that correspond to acceptable levels of comprehension?
To answer these questions we assessed a large sample of students, collecting data on oral reading fluency and comprehension for 1772 grade 5 ESL students from 214 rural schools in South Africa. As will become clear, there is an ongoing reading crisis in South African rural primary schools which, if not resolved, becomes a binding constraint to future learning at higher grades.
After a brief overview of existing research on reading outcomes and large-scale reading interventions in South Africa, we turn to a discussion of the international literature on oral reading fluency. Thereafter we explain our methodology and provide background information on the sample and assessments that were used. Section 3 contains a descriptive analysis of the data, while the final two sections develop tentative benchmarks for oral reading fluency in English for ESL students in rural South African schools. Finally Section 5 provides some policy recommendations regarding reading and reading interventions in South Africa.

An overview of South African large-scale research on reading outcomes and large-scale reading interventions
South Africa is in the fortunate position of having considerable amounts of data on educational outcomes in different subjects and at different grades. By implementing local assessments and agreeing to participate in cross-national assessments, the Department of Basic Education has ensured that there exists a solid foundation of nationally-representative data on which to make evidence-based policy. The results of these assessments are stable, consistent, reliable and sobering. As far as reading outcomes in the primary grades are concerned the three most recent and reliable assessments are the pre-Progress in International Reading Literacy Study (prePIRLS Grade 4, 2011), the Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ, Grade 6, 2007) and the National School Effectiveness Study (NSES, Grades 3/4/5, 2007/8/9).
The NSES study assessed a nationally-representative sample of schools in South Africa (excluding Gauteng) and found that the average Grade 3 student scored 20% on a Grade 3 test conducted in English (Taylor & Taylor, 2013: p47). Given that the language of learning and teaching (LOLT) for most Grade 3 students in South Africa is still an African language (the switch to English is only in Grade 4), this is perhaps unsurprising. However, Spaull (2015: p.71) shows that the achievement of these students in their home language while better, is still extremely low. Given that some students wrote both the Systemic Evaluation 2007 Grade 3, which was conducted in the LOLT of the school, as well as the NSES 2007 Grade 3, which was the same test conducted in English one month later, Spaull shows that the matched sample scored 23% in the Systemic Evaluation in English, and scored 34% on the same assessment one month later when it was conducted in the LOLT of the school. While this shows that there is clearly a cost to writing the test in an unfamiliar language (particularly given that students had not yet switched to English), it also dispels the myth that students are performing acceptably in an African language before the switch to English in Grade 4.
The two cross-national assessments that focus on primary-school literacy provide complementary evidence given that prePIRLS was conducted primarily in African languages in Grade 4 (prePIRLS used the LOLT of the school in Grades 1-3), while SACMEQ assessed students in English and Afrikaans in Grade 6 after the language transition. Howie, Van Staden, Tshele, Dowse and Zimmerman (2012: p.47) show that 58% of grade 4 students did not achieve the Intermediate International Benchmark and 29% did not achieve the Low International Benchmark. That is to say that 58% of students could not interpret obvious reasons and causes and give simple explanations or retrieve and reproduce explicitly stated actions, events and feelings.
One can think of these students as those that cannot read for meaning in any true sense of the word. More disconcerting is the 29% of students that could not reach the most rudimentary level of reading; locating and retrieving an explicitly stated detail in a short and simple text. It would not be incorrect to classify these 29% of students as illiterate or non-readers in their home language 1 . The SACMEQ study of 2007 tested a nationally representative sample of learners in English and Afrikaans (the LOLTs in South Africa in Grade 6). It was found that 27% of learners were functionally illiterate in English or Afrikaans in the sense that they could not read a short and simple text and extract meaning (Spaull, 2013: p.439). Among the poorest 20% of schools this figure rises to 43% of learners that are functionally illiterate.

Large-scale reading interventions in South Africa
The crisis in basic literacy in South Africa has not gone unacknowledged by the Department of Basic Education. Since the early 2000s there have been a number of 1 For the majority of students the test was conducted in their home language. Only where a student's home language differed from the LOLT of the school in Grades 1-3 would this not be true. national policies, strategies, campaigns, and interventions in an attempt to address this. We provide a brief overview of the most pertinent endeavors.

Early Grade Reading Assessment (EGRA)
The first major intervention was the development of the Early Grade Reading Assessment (EGRA) for South Africa which began in 2006 (Hollingsworth, 2009).
EGRA aims to measure the early reading processes including recognizing letters of the alphabet, reading simple words, and understanding sentences and paragraphs. It was developed as an individual oral assessment of students' foundation reading skills, and has been successfully used in many developing countries (Bruns, Filmer, & Patrinos, 2011). In 2007, the EGRA instruments were field tested in South Africa by the Molteno Institute for Language and Learning. This included 315 learners from 18 schools in six South African languages. The results showed that learners were not able to read at their grade level and that learners performed worse than their counterparts in many other African countries. In 2012, NEEDU used the EGRA tests to assess the reading fluency of the best three Grade 2 students (selected by the teacher) in each of 215 urban and township classes. They found that that 72% of the three best learners in each class were reading at or below 70 WCPM and that 22% were reading at or below 20 WCPM. (NEEDU, 2013, p40). These results should be interpreted with some caution since the EGRA instruments were directly translated (rather than versioned) and they were not all piloted. Following the recommendation of NEEDU, the Department of Basic Education (DBE) resuscitated the EGRA project and in 2014-15, reading promotion was declared a Ministerial priority programme (Motshekga, 2014).

Systematic Method for Reading Success
In late 2008, the Systematic Method for Reading Success (SMRS) was developed and piloted. This is an early grade fast-track reading programme which uses a home languages approach to teaching initial reading (Piper, 2009). It is designed for teachers who do not know how to teach reading and can be seen as a scripted Teacher's Manual so that teachers with little preparation in reading instruction can teach it. SMRS is meant to be a supplementary introduction to a full literacy programme in learners' home languages. In the three provinces that participated in the pilot, the programme was deemed relatively successful (Piper, 2009).

Teaching Reading in the Early Grades -A teacher's handbook
As part of the FFL campaign, a teachers' handbook, Teaching Reading in the Early Grades (DBE, 2008b), was developed. The handbook was designed to help Foundation Phase teachers teach reading. It highlighted the core elements of teaching reading and writing including: shared reading and writing; guided reading and writing; independent reading and writing activities; word-level and sentence-level work. These materials form the foundation for the current national Curriculum and Assessment Policy Statements (CAPS).

The Integrated National Literacy and Numeracy Strategy
Also in 2008, the National Reading Strategy Grades R-12 (NRS) (Department of Basic Education, 2014) was developed as a strategy to address the growing concern over illiteracy, and to promote a nation of life-long readers and life-long learners. The NRS provides an outline of curriculum requirements, reading activities and resources needed, grade-by-grade in the Intermediate and Senior Phases for teachers and school managers, by grade level. It gives guidance to learners, teachers, school leaders, parents and systems managers. This strategy was closely followed by the Integrated National Literacy and Numeracy Strategy (INLNS) (Department of Basic Education, 2011b) which was the department's response to the need for urgency in addressing the low achievement levels of learners in literacy and numeracy as confirmed in the poor national Annual National Assessment (ANA) results and various international assessment results. In November 2011, the Council of Education Ministers (CEM) resolved that the INLNS should be implemented in 2012 as a national initiative. CEM further agreed that planning with provincial education departments and key stakeholders should begin in earnest, and that the strategy would target the classroom and teachers as key levers for change in learner performance and would be guided by the Department's 2012 education priorities (CAPS, ANAs and the workbooks).
The INLNS implementation plan is a high-level plan which aims to direct and integrate provincial initiatives, which in turn are expected to formulate detailed plans for districts and schools 'down to the classroom level'. The implementation plan elaborates the targets set in the DBE's Action Plan (Department of Basic Education, 2011a), prioritises areas requiring attention (teacher content knowledge, support material, quality Grade R, etc.) and lists the pre-conditions needed to implement the strategy (vacant posts filled, teacher time-on-task monitored, provisioning of districts, school nutrition, learner transport, etc.). But the INLNS stops short of recommending specific programmes for use at the classroom level, the choice of which is left to provincial departments.

2) Literature review
Reading is a highly complex task phenomenon, comprising many cognitive-linguistic skills (Pretorius, 2012). The importance of learning to read for meaning by the end of the third year of primary schooling is widely acknowledged and accepted throughout the local and international education literatures (Martin, Kennedy & Foy, 2007). This is both to ensure future academic success at school, but also because this creates independent learners. As Good, Simmons and Smith (1998) expound: "Professional educators and the public at large have long known that reading is an enabling process that spans academic disciplines and translates into meaningful personal, social, and economic outcomes for individuals. Reading is the fulcrum of academics, the pivotal process that stabilizes and leverages children's opportunities to success and become reflective, independent learners" (Good, Simmons & Smith, 1998: p45).
One of the essential components of competent reading is Oral Reading Fluency (ORF) which is the speed at which written text is reproduced into spoken language (Adams, 1990). In the literature ORF is generally regarded as the ability to read text quickly, accurately, and with meaningful expression (Valencia et al., 2010;Fuchs et al., 2001;Rasinski & Hoffman, 2003). This skill is believed to be critical to reading comprehension and the speed at which print is translated into spoken language has been identified as a major component of reading proficiency (NICHHD, 2000). When words cannot be read accurately and automatically, they must be analysed with conscious attention. If children use too much of their processing capacity trying to work out individual words, they are unlikely to successfully comprehend what they read (Hudson, Lane, and Pullen, 2005).
ORF can therefore be seen as a bridge between word recognition and reading comprehension. Problems in either oral fluency or reading comprehension will have a significant impact on a learner's ability to learn as they move through the phases of schooling. This has also been confirmed with longitudinal research which found high correlations between reading performance in early primary grades and reading skills later in school (Good et al., 1998;Juel, 1988). Reading fluency has also been found to be a significant variable in secondary students reading and overall academic achievement (Rasinski, et al., 2005).

ORF as a predictor of reading comprehension
At the most basic level, the Early Grade Reading Assessment (EGRA) is an oral reading assessment designed to measure the most basic foundation skills for literacy acquisition in the very early grades: recognizing letters of the alphabet, reading simple words, understanding simple sentences and paragraphs, and listening with comprehension. The EGRA tests, developed by RTI to orally assess basic literacy skills, have been used in over 40 countries (RTI International, 2008). For students in higher grades, ORF is generally measured by having an assessor ask a student to read a passage out loud for a period of time, typically one minute. A student's score is calculated with the number of words read per minute (WPM) and/or the number of words read correctly per minute (WCPM). In order to counter criticism that such an assessment does not validly measure comprehension, the passages are frequently accompanied by comprehension questions, as in the present study.
In their comprehensive review of numerous studies, Fuchs et al. (2001) provide converging evidence supporting ORF's validity as an indicator of reading comprehension. They conclude that: (1) ORF corresponds better with performance on commercial, standardized tests of reading comprehension than do more direct measures of reading comprehension; (2) text fluency (words read in context) compares positively to list fluency (words read in isolation) as an indicator of reading competence; and (3) oral reading fluency measured by reading aloud functions as a better correlate of reading comprehension than does silent reading fluency. In a recent study in South Africa (Pretorius, 2012), a strong correlation was found between three measures of decoding skill and reading comprehension with oral reading fluency emerging as a strong predictor of comprehension.
One explanation for the connection between fluency and comprehension comes from LaBerge and Samuels's (1974) theory of automaticity in reading (Rasinski, et al., 2005). According to this theory, readers who have not yet achieved reading fluency must consciously decode the words they have to read. This cognitive attention detracts from the more important task of comprehending the text. Poor reading fluency is thus directly linked to poor reading comprehension. As Fuchs et al (2001: p.42) explain: "Unfortunately as poor readers rely on the conscious-attention mechanism, they expend their capacity in prediction processes to aid word recognition. Little is left over for integrative comprehension processes, which happens for readers with strong word recognition skills, whereby new knowledge is constructed or new material is integrated into existing knowledge structures" For some languages, the practice of using WCPM as a predictor of comprehension has been criticized (Graham & van Ginkel, 2014). In a quantitative study of early grade reading in two European (English and Dutch) and two African languages (Sabaot and Pokomo) Graham & Van Ginkel (2014) analysed WCPM and comprehension scores of over 300 children in three countries and found that similar comprehension scores were associated with diverse WPM rates. This, they suggest, indicates that fluency measured as WCPM is not a reliable comparative measure of reading development since linguistic and orthographic features differ considerably between languages and are likely to influence the reading acquisition process.
Valencia & Buly's study (2004) raised concerns regarding the widespread use of WCPM measures and benchmarks to identify students at risk of reading difficulty. In their study, oral reading fluency data and standardized comprehension test scores were analyzed for students in grades 2, 4, and 6 in two Pacific Northwest school districts in America that had diverse student populations. One third of the student group spoke English as a second language. The results indicated that assessments designed to include multiple indicators of oral reading fluency provided a finergrained understanding of oral reading fluency and fluency assessment and a stronger predictor of general comprehension. Comparisons across grade levels also revealed developmental differences in the relation between oral reading fluency and comprehension, and in the relative contributions of oral fluency indicators to comprehension. When commonly used benchmarks were applied to WCPM scores to identify students at risk of reading difficulty, both false positives and false negatives were found. Valencia & Buly (2004) argue for a much more comprehensive assessment in order to understand the specific needs of different children. Their approach was to conduct individual reading assessments, working one-on-one with the children for approximately two hours over several days to gather information about their reading abilities. They administered a series of assessments that targeted key components of reading ability: word identification, meaning (comprehension and vocabulary), and fluency (rate and expression). Their research suggested that weak readers may not be weak in all three areas, and that there could be as many as six different profiles of readers, all needing different remedial attention. This approach may represent the 'gold standard' of reading assessment but the reality in most countries, and particularly in South Africa, is that this sort of assessment is unlikely to be realistic or practical.

Oral Reading Fluency among English Second Language Learners
The investigation of ORF for students reading in a second (or third) language is not as extensive as that for students reading in their first language. Notwithstanding the above, ORF studies on ESL students have been conducted in South Korea (Jeon, 2012), Kenya (Piper & Zuilkowski, 2015) and America (Al Otaiba et al., 2009;Jimerson et al., 2013). This does not include the numerous EGRA studies that have been conducted by RTI and USAID (Abadzi, 2011). For many second language readers, reading is a "suffocatingly slow process" (Anderson, 1999, p.1); yet developing rapid reading, an essential skill for all students, is often neglected in the classroom. Data from Segalowitz, Poulsen, and Komoda (1991) indicate that the English Second Language (ESL) reading rates of highly bilingual readers can be 30% or more slower than English First Language (EFL) reading rates. Readers who do not understand often slow down their reading rates and then do not enjoy reading because the process becomes laborious. As a result, they do not read extensively, perpetuating the cycle of weak reading (Nuttall, 1996, in Anderson, 1999. Conventional wisdom indicates that lack of oral English proficiency is the main impediment to successful literacy learning for young English Second Language learners, but recent evidence suggests that this may not be true. Conflicting data exist regarding the optimal or sufficient reading rate (Anderson, 1999). Some authorities suggest that 180 words per minute while reading silently "may be a threshold between immature and mature reading and that a speed below this is too slow for efficient comprehension or for the enjoyment of text" (Higgins and Wallace 1989 p 392, in Anderson, 1999). Others suggest that silent reading rates of ESL readers should approximate those of EFL readers (closer to 300 WPM), especially if the ESL is also the language of learning and teaching (LOLT), in order to come close to the reading rate and comprehension levels of EFL readers.
While research into reading in an ESL is not as extensive as its EFL counterpart, an increasing number of comparative EFL/ESL reading studies have been undertaken at different age levels. Pretorius (2012) argues that ESL reading theories tend to draw quite heavily on EFL reading theory, the assumption being that the underlying skills and processes involved in reading languages with similar writing systems are similar across languages. If these decoding processes are similar in alphabetic languages, then there is no reason why ESL reading rates should be so laborious. An area where differences between EFL and ESL LOLT readers may persistently occur will be vocabulary, but decoding per se should not be a stumbling block. Jimerson et al. (2013) tracked the ORF growth of 68 students from first through fourth grade in one Southern California school district in America, and used it to predict their achievement on a reading test. They found that both ESL students with low SES, and other students with low SES showed low performance in their initial first grade ORF, which later predicted fourth grade performance. The trajectory was the same for EFL students with low SES who performed poorly at the first grade level. The reading fluency trajectories (from the first grade) of the ESL and EFL students with low SES were not significantly different. Their study showed that initial pre-reading skills better explained fourth grade performance than either ESL with low SES or low SES alone.

Using ORF to set reading norms
ORF has been part of national assessments in the USA for decades and norms are well established, but the same cannot be said of most developing countries (Abadzi, 2011).
A search carried out in early 2010 showed that over 50 fluency studies have been conducted in various countries, but the studies often reported data in ways that were not easily comparable, and few had collected nationally representative data.
As early as 1992, researchers in the USA compiled norms for ORF in English based on reading data from eight geographically and demographically diverse school districts in the United States (Hasbrouck & Tindal, 2006). With the growing appreciation for the importance of reading fluency, new norms were developed in 2005 with greater detail, reporting percentiles from the 90 th through the 10 th percentile levels.
The use of norms in reading assessments can be categorised to match four different decision-making purposes (Kame'enui, 2002in Hasbrouck & Tindal, 2006 year when a more in-depth analysis of a student's strengths and needs is necessary to guide instructional decisions. 3. Progress-monitoring measures: Assessments conducted at a minimum of three times a year or on a routine basis (e.g., weekly, monthly, or quarterly) using comparable and multiple test forms to (a) estimate rates of reading improvement, (b) identify students who are not demonstrating adequate progress and may require additional or different forms of instruction, and (c) evaluate the effectiveness of different forms of instruction for struggling readers and provide direction for developing more effective instructional programs for those challenged learners.

Outcome measures:
Assessments for the purpose of determining whether students achieved grade-level performance or demonstrated improvement.
Such fluency-based assessments have been proven to be efficient, reliable, and valid indicators of reading proficiency when used as screening measures (Fuchs, Fuchs, Hosp, & Jenkins, 2001). This was also shown to be the case for ESL students, as shown by the work of Al Otaiba et al. (2009). They examined American Latino students' early ORF developmental trajectories to identify differences in proficiency levels and growth rates in ORF of Latino students who were (a) proficient in English, (b) not proficient and receiving ESL services, and (c) proficient enough to have exited from ESL services. They found that ORF scores reliably distinguished between students with learning disabilities and typically developing students within each group.
Setting ESL reading norms in the South African schooling context is a new and, as yet, largely unexplored terrain. One could argue that in the initial stages of ESL reading for LOLT (perhaps Grade 4 learners), reading at 70% the rate of EFL readers is not surprising or unexpected. However, as children go higher up the academic ladder (approaching the end of the Senior Phase), the gap between EFL and ESL reading for LOLT purposes should start narrowing, and by the end Grade 9, ESL norms should preferably start approximating EFL norms. One may also argue for a fluency continuum, with EFL and ESL LOLT reading norms divergent in the beginning stages of reading, but converging by high school. However, all of these suggestions are speculative in nature and are not based on empirical evidence, largely because such empirical evidence does not yet exist in South Africa. It is this gap in the South African literature to which this study hopes to contribute.

3) Methodology: Test Development and Sampling Information
To assess silent reading comprehension of Grade 5 ESL students in the written mode, an appropriate Grade 5 level passage was selected. This was followed by a range of literal and inferential questions in a mixed question format. In addition, Grade 4 and Grade 5 textbooks were used to select two reading passages appropriate to Grade 5 ESL students to assess ORF. Each of the two ORF tests was accompanied by five oral comprehension questions (All test instruments, questionnaires and administrator protocols are available in the Online Appendix 2 ).

Readability
Readability refers, broadly, to the ease or difficulty with which texts are read. Since the 1940s various readability formulae have been used to quantify aspects of texts that are deemed to play a role in determining the ease with which texts are read. These readability formulae invariably incorporate word length and sentence length in relation to overall text length, the assumption being that short words and short sentences are easier to read than longer words and sentences. Examples of readability formulae include the Flesch Reading Ease (RE), the Dale-Chall and the Grammatik formulae. Although the assumptions underlying the readability formulae have been criticised for oversimplifying the reading process, since there are several text-based and reader-based factors that affect reading ease, they continue to enjoy popularity as predictors of text difficulty (Klare, 1974).
The Flesch Reading Ease formula has been used in this analysis, primarily because it is easily available and in the educational context, serves as a useful guideline for establishing consistency across texts at specific grade levels. According to Hubbard The analysis also determines the number of passive constructions used in a text.
Passives are considered slightly more difficult to read than actives. The higher the number obtained from the computation, the easier the text is regarded as being while the lower the number, the more difficult the text. The scores have been measured in terms of readability categories, as shown in Table 1 below. Most academic/scientific texts and research articles fall into the last two categories of RE. One would expect Grade 4 and 5 textbooks to fall within the 90-70 range of scores. Using American textbooks as the data base, the Flesch-Kincaid formula was used to determine the reading ease of texts written for the different grades. These scores reflect the actual grade level, e.g. a score of 6 would indicate a text appropriate for Grade 6. This readability score does not reflect aspects such as the persuasiveness or credibility of a text or its interest level. It is to be expected that the RE score drops the more abstract and complex a topic is. The use of technical terms (e.g. pollution, precipitation) as well as general academic terms (e.g. operates, features) also affect RE.
A selection of Grade 4 and 5 textbooks across various subjects were obtained from primary schools in two townships near Tshwane, namely, Atteridgeville and Mamelodi respectively. From each textbook, four passages were selected, one from the beginning, two from the middle and one from the end. These passages were scanned and converted into MS Word text files; all the pictures and diagrams were removed and only running text used for the readability analysis. The results are given in Table 2 and Table 3 below. The RE range of the Grade 4 textbooks was between 82 -72, falling within the 'easy' to 'fairly easy' categories, while that of the Grade 5 textbooks was between 84 -68, falling between the 'easy' to 'standard' categories. As to be expected, there was a gradual decrease in RE scores from Grade 4 to Grade 5, with concomitant increases in the use of passives and more words per sentence, particularly in the content subjects. The latter textbooks also carry an increase in the use of specialist technical words as well as general academic words. It is interesting to note that across both grades the RE scores were higher (i.e. easier) in the English and Maths texts than in the other content subject texts.
The outcome of the readability analysis conducted served as a guideline for Steps 2 and 3, namely the selection of two passages appropriate to Grade 4 and 5 levels to assess oral reading fluency, and the selection of a passage appropriate to Grade 5 level to assess silent reading comprehension in the written mode.

The reading comprehension passage
Two passages were selected as the base for written reading comprehension test.
Eleven questions were asked, five based on the first passage, and six based on the second. The reliability scores of the combined comprehension passages, as well as the readability score of the questions are show in the Table 4 and 5 below while the question types are shown in Table 6. Based on the learner results, a Cronbach's alpha analysis was done on the written comprehension passage. Cronbach's alpha was 0.83 which indicates good reliability of the overall test.

The sample of schools and students
The data used in this study comes from a non-random sample of 4667 Grade 5 Very poor reading levels (poor letter and word recognition in the home language of learners) were identified in the first NEEDU evaluation cycle when Grade 2 learners were assessed using the EGRA instruments in 2012. Reading was thus identified as a critical factor inhibiting improvement in the sector. In the second NEEDU evaluation cycle, it assessed Grade 5 learners' reading in terms of their ORF and reading comprehension. It is these data that form the basis for this paper.
The labour-intensive nature of the approach to systemic evaluation adopted by NEEDU (NEEDU, 2013), meant that the number of schools selected for evaluation was limited and non-random. NEEDU aimed to assess one third of districts with the aim of covering all districts in three years. Within each district a district official was asked to select 8 schools for inclusion in the sample. This non-random selection clearly affects the generalizability of the sample, but if anything the results are positively biased (i.e. if better schools were put forward). The sample also seems to include more schools that were closer to amenities and fewer extremely remote schools. One further limitation is that the NEEDU school visits (and therefore the ORF assessments) were conducted throughout the year meaning that some schools were assessed earlier in the year and others later in the year. Analysis of the results by month and province shows no relation between the month of assessment and ORF or comprehension outcomes. Consequently we do not disaggregate the results by month but treat the sample as a grade 5 composite sample.
Notwithstanding the above, the sample of 214 schools is large by local and international standards and the number of students being assessed on oral reading fluency (1772)  Two NEEDU evaluators visited each school to conduct the NEEDU evaluation, and one of those evaluators was trained as a reading assessor. The learners selected for the ORF assessment read aloud to the reading assessor. The assessor recorded the number of words read correctly, and this together with the time taken to read the passage, calculated the total WCPM read by each learner assessed.
The assessment was discontinued for those learners who clearly could not read the first passage, and for those learners who read at such a slow pace that they failed to complete the first paragraph (56 words) in one minute. To test their comprehension of the text, learners were asked five simple questions relating to the passage. Learners who did not read beyond the first paragraph were only asked those questioned that were relevant to the sections read. Learners were allowed to refer to the passage to answer the comprehension questions. All learners that were able to read beyond the first paragraph in a minute were asked to read a second more difficult passage. 855 learners were in this group, and a similar process was followed for the second ORF passage. Table 9 below provides a range of descriptive statistics on each of the three tests (silent comprehension, ORF Test 1 and ORF Test 2), reporting the number of students who completed the test, as well as the mean, standard error of the mean, minimum, maximum and standard deviation for each measure and reported by province, gender, language of learning and teaching (LOLT) in grade 5, and grade-arrangement. It is worth re-emphasizing that the sample was not randomly selected and is therefore not nationally or provincially representative. That being said, the rank order of the provinces in the silent reading comprehension test is broadly the same as the rank order of provinces using the 2007 grade 6 SACMEQ reading test (Spaull, 2011;21) with the exception of the Northern Cape. In the SACMEQ test the Northern Cape scored lower than the Western Cape and Gauteng whereas here it is the province with the highest average reading comprehension score. Unsurprisingly, this provincial rank order is roughly the same for the ORF Test 1 and ORF Test 2. While we do not stress the provincial results in this sample, we would argue that there are enough boys (2357) and girls (2294) to interpret results by gender with some level of confidence.

4) Descriptive analysis of oral reading fluency and comprehension data
The same applies to reporting results by grade arrangement with 3701 students in monograde classes and 966 students in multigrade classes, and language of learning and teaching (LOLT 4 ) at grade 5 level with 623 students in Afrikaans-medium schools and 3867 students in English medium schools. 4 The astute reader will notice that the two categories "Afrikaans LOLT (Gr5)" and "English LOLT (Gr5)" do not sum to the total number of students. This is because there were 46 grade 5 students from one school in the Eastern Cape where the LOLT was recorded as isiXhosa. While this is unusual it is possible. The reason we do not include three categories for LOLT is that the results for isiXhosa would be based on one school rather than a large number of schools, as is the case with Afrikaans LOLT (45 schools) and English LOLT (161 schools). Apart from this, the remaining differences in any of the categories are due to missing information. were marginally lower (20.2%) than in multigrade classes (21.7%) however this difference is not statistically significant (Figure 1). The largest difference between the three groupings is seen between students learning in English (19.1%) and students learning in Afrikaans (30.2%). The fact that students learning in Afrikaans do better on an English comprehension test than students learning in English requires investigation.
Firstly, the vast majority (92%) of students learning in Afrikaans in grade 5 also spoke Afrikaans as their home language, and all of them had been learning in 5 The use of race as a form of classification and nomenclature in South Africa is still widespread in the academic literature with the four largest race groups being Black African, Indian, Coloured (mixed-race) and White. This serves a functional (rather than normative) purpose and any other attempt to refer to these population groups would be cumbersome, impractical or inaccurate. The gaps between the subgroups are smaller for ORF Test 2, as one might expect when there is a selection effect determining which students proceed to ORF Test 2.
Only students that could read at least the first paragraph of ORF test 1 proceeded to ORF test 2. While the first paragraph contained 56 words, and therefore the minimum WCPM scores here might seem strange, a student could have completed the first paragraph with many mistakes allowing them to proceed to ORF Test 2 while still having an extremely low WCPM score.  the average score (out of 5) was 1.3 with a standard deviation of 1.4. For the ORF 2

Correlations between oral reading fluency and comprehension
Comprehension the average score was 1.5 with a standard deviation of 1.2.    Figure 2 and 3 below show the scatterplots and respective histograms of silent reading comprehension and ORF Test 1 (Figure 2) and ORF Test 2 (Figure 3). These graphs show that the distributions of silent reading comprehension scores and words read correct was lower for the ORF 1 sample than the ORF 2 sample, as would be expected given that ORF Test 1 (n=1772) was representative of the schools, while ORF Test 2 (n=855) included only those students who could read at least one paragraph in ORF Test 1. Figure AB shows that a full 14% of the sample could only read 0-5 words correctly per minute.

Intra-class variation in oral reading fluency
While it is useful to understand average rates of WCPM, as well as overall standard deviations, it is also helpful to report the range of WCPM scores within a school. The ORF Test 1 results show large variation between the best perfomring learner and the worst performing learner within a school. If one looks at the distribution of the range (maximum WCPM -minimum WCPM), one can see that in 50% of schools this gap is more than 78 WCPM. In 25% of schools the gap is larger than 98 WCPM. The Two plausible explanations exist for the large intra-class gap: (1) the strong impact of home literacy practices where some students are exposed to text and encouraged to read more than others, and (2) teachers teaching to the best learner in the class such that the best learner(s) continue to improve while students performing at the bottom end of the spectrum stagnate.

The relationship between oral reading fluency and comprehension
While the aim of the current paper is not to estimate the nature of the relationship between oral reading fluency and comprehension, it is still helpful to illustrate the broad trends between these two measures. Before this discussion it is helpful to explain two decisions: firstly which measure of comprehension is used, and secondly which measure of oral reading fluency is used.
• Measure of comprehension: Of the three measures of comprehension, we believe that the most reliable measure of comprehension is the 40-minute silent reading comprehension test that consisted of 11 questions and totalled 20 marks. Although the ORF Test 1 and ORF Test 2 comprehension questions were based on the same text as the one used for the oral reading fluency measure, there were only five one-mark questions asked after each passage.
Hence this measure is less nuanced and has less variation. Consequently we use the silent reading comprehension measure for the remainder of the paper.
• Measure of oral reading fluency: Of the two measures of oral reading fluency (ORF 1 and ORF2) we use the ORF Test 1 measure since this included the full sample of those tested for oral reading fluency (n=1772).
Given that these students were selected from the top, middle and bottom of the class they are broadly representative of the classes from which they came. The same cannot be said of the ORF Test 2 results since only students that read past the first paragraph proceeded to ORF Test 2, making this a selective subsample of students in the class. Consequently we focus on ORF Test 1 as the measure of oral reading fluency. Figure 4 below shows the cumulative density functions (CDF) of words correct per minute on ORF Test 1 for three groups of students; (1) those achieving less than 30% on the silent reading comprehension test, (2) those achieving 30-59% and (3) those achieving 60%+ on the test. One can clearly see that the CDFs of the three groups differ substantially. If one looks at the 50 th percentile (y-axis) together with Table 11 one can see that in group 1 half of the 1220 students were reading at 37 WCPM or lower, in group 2 half of the 445 students were reading at 63 WCPM or lower, and in group 3 half of the 107 students were reading at 87 WCPM or lower.

Figure 4: Cumulative density function (CDF) of words correct per minute on
Oral reading Fluency Test 1 per category of performance on the silent reading comprehension test. If one looks at the oral reading fluency rates in Table 11 and compares these to  (typically English or French) in each country. This is obviously problematic since it is reasonable to expect oral reading fluency rates would differ based on text type and difficulty, whether it is in a student's home language or an additional language and whether the language is an agglutinating or fusional language. Notwithstanding the above, she recommends that as a broad rule of thumb children should be reading at 45 WCPM by the end of grade 2 and 90-120 WCPM by the end of primary school (Abadzi, 2011: p27). Given the lack of additional information on language, samplesize, grade etc. it is difficult to use these benchmarks in the South African context.
We follow Abadzi's (2011) approach and use our assessments of both oral reading fluency in English and comprehension in English (a second language to these students) and use these results to create tentative ORF benchmarks. If one specifies some minimum level of comprehension and then observes the distribution of words correct per minute associated with those students, it becomes possible to develop benchmarks that are specific to the South African rural context, and particularly the linguistic context where students are being assessed in a second language (English) and have only been learning in that language for 1-2 years.
Following this approach one can use Figure 4 and Table 11 to help identify logical thresholds of words correct per minute for South African ESL students. If students are performing below these thresholds teachers have reasonable cause for concern. Table  11 shows that of those 107 grade 5 students (from 61 schools) that are performing 'acceptably' (here defined as 60% or higher on the silent reading comprehension test), almost no student achieved lower than 50 WCPM and the majority (75%) scored above 68 WCPM. In contrast, of those students scoring less than 30% on the test, the majority (75%) scored less than 52 WCPM. English Speaker (C1). These are briefly described in Table 12 below.  Grades 1-5 in South Africa, at least until more data becomes available on oral reading fluency benchmarks in South Africa.
In order to develop accurate benchmarks and acceptable growth trajectories that are specific to South Africa one would need a large data set of panel data on student oral reading fluency scores at successive grades, or at the very least repeated cross-sections of large samples of students at successive grades. As yet this is not available and improvised schema -such as that of Broward County -may be of value in the interim.

5) Conclusions and policy discussion
While the reading crisis in South Africa is widely acknowledged (Fleisch, 2008;Spaull, 2013;Taylor Van der Berg & Mabogoane, 2013), almost no prior research exists on oral reading fluency in English, despite this being one of the major components of reading. The present study has begun to alleviate this paucity of information by analyzing a large data set of grade 5 rural ESL learners assessed in English. The four major findings emerging from the analysis are as follows: 1) The English oral reading fluency of grade 5 rural ESL learners in South Africa is exceedingly low. 41% of the sample read at cripplingly slow speeds of less than 40 words correct per minute with an average of only 17 WCPM and can therefore be considered non-readers. These students were reading so slowly that they could not understand what they were reading at even the most basic level. Almost all of these non-readers (88%) scored less than 20% on the comprehension test. A further 28% of the sample scored less than 30% on the comprehension test bringing the total to 69% of grade 5 students who could not score 30% on the comprehension test. A quarter scored between 30% and 60% and only 6% of the sample scored above 60% on the comprehension test.
2) The full sample of South African Grade 5 rural ESL students' ORF scores are approximately the same as the lowest category of Grade 2 ESL students in America (non-English Speaker: A1). These students cannot communicate meaning orally in English. Given that the language of learning and teaching from grade 4 is English for almost all of these students, this is of serious concern.
3) The correlations between various measures of ORF and comprehension were approximately 0.5. This is relatively low compared to most of the international literature, however, that literature reflects largely on home language speakers. More research on ESL learners is needed before concluding if these correlations are low or high in international context.

4) Setting Oral Reading Fluency benchmarks for South African ESL learners is a useful
endeavor, allowing teachers to identify and track struggling readers and to provide a yard-stick against which teachers can compare their students' progress or lack of progress. It is not possible to simply use the Hasbrouck & Tindal (1996) norms given that these were developed for the U.S. and primarily for EFL students. We argued that a benchmark of 90-100 WCPM in English in grade 5 for ESL students in South Africa is probably a good starting point until more data become available. Only 9% of the sample achieved these levels of oral reading fluency. We also highlighted the potential of using the Broward County ESL classification chart and following the 'Intermediate English (B1)' trajectory for South African ESL students.
From a policy perspective there are three main recommendations that we would put forward:

1) The majority of primary school teachers do not know how to teach reading in either
African languages or in English. This is evidenced by the cripplingly low ORF scores in grade 5. These students cannot engage with the curriculum (which is now in English in Grade 5) and hence fall further and further behind as the reading material and cognitive demands become more and more complex. There is a clear need to convene a group of literacy experts to develop a course to teach Foundation Phase teachers how to teach reading. This course should be piloted and evaluated and if it is of sufficient quality should become compulsory for all Foundation Phase teachers in schools where more than 50% of students do not learn to read fluently in the LOLT by the end of Grade 3.

2) The clear need for evidence-based interventions, evaluations and sustained support.
Much of the policy energy that has been expended in the last 10 years has been sporadic and haphazard. Successful programs (like the SMRS) are not pursued while new initiatives are funded (but not evaluated) without a clear understanding of how they improve on or learn from previous initiatives. Any new national literacy drive needs to be piloted, independently evaluated and only taken to scale if and when it is proved to be effective. This should be seen as a medium-to-long term goal rather than a short-term goal.
3) Reading as a unifying goal for early primary schooling. The single most important goal for the first half of primary school should be the solid acquisition of reading skills such that every child can read fluently in their home language by the end of Grade 3 and read fluently in English by the end of Grade 4. This goal is easily communicated and understood by parents, teachers and principals and is relatively easy to measure and monitor. The benefit of having a single unifying goal to focus attention, energy and resources should not be underestimated.

Research Foundation (NRF) Research Priority Area.
Given the scale of the reading crisis and the lack of research on African languages at South African universities (particularly relating to early literacy in African languages), the NRF should declare this to be a national priority and dedicate significant resources to those researchers and departments with the skills and expertise needed to understand more about how children learn to read in African languages and which interventions are most promising.