|
Effects of using instructional technology in elementary and secondary schools: What controlled evaluation studies say.
By James A. Kulik
SRI International
(2003)
Summary and Review | Profile and Features | Complete Study | Related Studies
 Print Version of Review
APA Reference:
Kulik, J. (2003). Effects of using instructional technology in elementary and secondary schools: What controlled evaluation studies say. Arlington, Virginia: SRI International. Retrieved October 3, 2003 from http://www.sri.com/policy/csted/reports/sandt/it/Kulik_ITinK-12_Main_Report.pdf
Summary:
In this paper, James A. Kulik uses measures of effect size to summarize findings from eight meta-analyses covering 335 studies published before 1990 and 61 controlled studies that were published after 1990. (Effect size is positive when the treatment group in a study outperforms the control group, and is negative when the control group in a study outperforms the treatment group).
The author identifies important factors that have influenced the change between studies done prior to 1990 and to those conducted subsequent to 1990. Findings of effectiveness of instructional technology for student learning show: - Dramatic changes in:
- computer to student ratio, from 125 students to every computer in 1984 to 6.3:1 in 1998.
- access to the Internet in schools; 98% of schools had access to the Internet in 2000, compared with 3% in 1989.
- ratio of multimedia computers to students, which improved to 9.8:1 in 1999
- ratio of students to Internet connected computers, improved to 13.6:1 in 1999
- In the case of some applications, teachers are better prepared than they were in the 1980s to integrate technology with classroom instruction.
- The digital divide remains wide. “Schools with minority and less affluent students have fewer computers and less Internet access than do other schools” (p.4). However, the inequities in schools are not as wide as those existing in homes. (National Telecommunications Information Administration--NTIA).
- A survey of teacher use of computers (Becker, Ravitz, Wong, 1999) shows that students today most frequently use computers as tools rather than as tutors, and the most frequent teacher objectives for student use are “to find out about ideas and information” in contrast to a decade ago when the most frequent objectives were for “basic skills training and computer literacy” (p.2). The Becker, Ravitz and Wong results are similar to findings of Coley, Cradler, and Engel (1997) that 71% of students reported using computers for word processing in schools (p. 3).
Summary of Kulik’s findings from studies published since 1990:
- Integrated Learning Systems (ILS)
Students do as well with ILS as do students who do not use ILS. In every study, reading performance of ILS students was as good as reading performance of control group students, and ILS did not have a negative effect on student performance, although the studies did not show that ILS improved performance in reading. The studies suggest that Integrated Learning System’s effects would be greater if students spent more time on the ILS instruction and if students worked in pairs on the ILS instruction, rather than individually.
The overall effect size of .28 for all the ILS studies is a composite of effect sizes for reading and for mathematics. Evaluation studies of ILS where mathematics instruction was alone and not combined with reading instruction showed greater effect sizes for mathematics.
- Writing to Read (WTR)
The overall effect of WTR in studies published since 1990 was positive. The median effect in five studies published since 1990 was an increase in reading scores of 0.25, equivalent to a boost from the 50th to the 60th percentile (p.32).
Reasons for the difference in findings between an earlier review of WTR by Slavin and the more recent review of evaluations of WTR include:
- teachers in the 1990s had more experience using the Writing To Read (WTR) approach
- WTR are better designed now than they were earlier
- more recent evaluation studies of WTR may be better designed and analyzed.
- Reading Management
There have been only three controlled studies of the reading management program that guides and tracks student reading. (Accelerated Reader was most often mentioned by respondents in the Becker et al. 1999 survey of teachers.) More evaluation studies are needed to determine the overall effect of the program for reading, math and science.
- Word Processing and Computer Enrichment
According to a 1999 survey, word processing software is the type of software used most often in elementary, middle and high schools (Becker et al., 1999).
In Bangert-Drowns’ 1993 review of 32 studies of effectiveness of word processing programs for writing skills, five studies demonstrated the positive effect of word processing on composition length. Four recent reviews conducted after 1993 found that in 7 out of 10 studies of word processing in K-12 schools there were positive effect sizes of 0.28.
- Effects of Writing Prompts
Recent research suggests that the effectiveness of writing prompts is greater when the computer provides the prompts independent of student requests for help. (p.42).
- Effects of Computer Enrichment
During the 1990s, evaluation of computer enrichment focused on student writing in contrast to less structured evaluations in the prior two decades. Five of six studies in the 1990s reported positive effects of computer enrichment on the quality of student writing. The six studies had a median effect size of 0.34, equivalent to an increase from the 50th to the 63rd percentile (p. viii). The results of the six studies suggest that the enrichment programs did help students improve their writing skills.
- Microcomputer-based Laboratories
In the typical study reviewed, students learned just as much without MBL as with MBL.
- Tutorials and Simulations
A 1994 review of 58 studies of computer tutoring demonstrated positive effects for technology-based teaching approaches. The median effect size was 0.36, meaning that computer-tutored students would perform at the 64th percentile on relevant achievement tests, whereas conventionally taught students would perform at the 50th percentile.
Recent studies suggest that computer tutoring can be a very effective aid when used to teach concepts in elementary and high schools. The median effect size for findings from six studies was 0.59, meaning that students who received computer tutorials in these studies would perform at the 72nd percentile on their tests in contrast to students receiving conventional instruction, who would perform at the 50th percentile. The median effect was 1.10 for attitudinal outcomes, which means the computer-based tutoring contributed to the development to favorable attitudes toward instruction.
Results from studies of simulation programs are weaker and less consistent than are results from studies of tutorials. A review of six studies of simulations demonstrated a median effect size of –0.06, which means that students learning with and without simulations scored at nearly the same levels on the relevant tests of science learning (p. ix).
Overall conclusions:
- Recent studies suggest that instructional technology is thriving today as teachers and students are more computer literate and computers are faster, friendlier, and more accessible in schools.
- Most studies carried out during the 1990s found that enrichment programs have positive effects on student writing skills and that instructional technology often improves teaching programs in mathematics and in the natural and social sciences. In fact, “simply giving students greater access to computers and Internet resources often results in gains in writing skill” (p.x).
- Integrated Learning Systems (ILS) have been producing positive results in mathematics programs for decades, and computer tutorials in natural and social science classes have had an almost uniformly positive record of effectiveness over the last three decades.
Major implications for educators/decision makers:
Findings from Kulik’s review of studies show that:- Professional development for teachers and easy access to Internet connected computers for teachers and students enhance the learning effectiveness of instructional technology.
- Computer enrichment programs have positive effects on students’ writing, mathematics, and performance in the natural and social sciences. In fact, “simply giving students greater access to computers and Internet resources often results in gains in writing skill” (p.x).
- The effects of using Integrated Learning Systems can be increased by providing more time for students to spend on the ILS instruction and by enabling students to work in pairs on the ILS instruction, rather than individually.
- Student familiarity with and knowledge of computers influences effectiveness of technology-based instruction.
- The effectiveness of simulation programs for helping students to acquire higher order thinking skills can be increased with additional hands-on activities, and when the simulations are used as preparation for further instruction.
- Writing skills programs that provide prompts independent of student requests are most effective.
|
Major implications for educational researchers/evaluators:
To paraphrase the author, with the focus on fully describing the studies, this review incorporates features of both narrative and meta-analytic reviews (p.14). An important implication is that in order to make meaningful comparisons, researchers need to report at least at the level of detail described below under "Source of evidence" and "Replicable strategies".
Major intervention(s) or variables studied:
The subject areas covered in this review of research on effects of technology on student learning pertained to: - Integrated Learning Systems (16)
- Writing programs for learning to read (12)
- Reading management (3)
- Word processing and Internet resources (10)
- Microcomputer-based laboratories (8)
- Science tutoring and simulations (12).
The reviewed studies primarily measured student learning outcomes indicated by an achievement test given at the end of the intervention or program of instruction.
Major questions addressed:
Sources of evidence identified:
This review used the findings from evaluation studies that met the following criteria: - Actual field implementations in which students received for-credit instruction
- Quantitative results on outcome variables measured in the same way for experimental and comparison groups
- Free from “crippling” methodological flaws such as substantive differences in pretests, unfair teaching of the criterion test to one of the comparison groups, differential rates of subject attrition from the groups being compared
Effect sizes were calculated directly from the measurement provided in the original reports for studies that reported means and standard deviations for both experimental and control groups.
Replicable strategies, practices, and/or products:
Replicable strategies that may be useful for future reviews of research on the effects of technology on student learning include: - Criteria for selecting the studies to review
- Full description of the technology applications studies
- Consideration of the changing uses and presence of technology in schools and homes and increasing sophistication of technology and student and teacher technology skills
Strengths and limitations of the study:
A strength of the review is the careful selection of studies based on sound methodological criteria. A limitation of the study is how deeply embedded the primary findings are within the text of the report. As with any meta-analysis, effect sizes are averaged over multiple studies. How well they will predict the effects of other approaches will depend on the extent to which the new interventions are similar to the analyzed studies in terms of subjects, settings, and implementation.
Suggested related studies or resources to consider:
Bangert-Drowns, R. L. (1993). The word processor as an instructional tool: A meta-analysis of word processing in writing instruction. Review of Educational Research, 63(1), 69–93. ERIC: EJ463379.
* Blok, H., Oostdam, R., Otter, M., & Overmaat, M. (2002). Computer-assisted instruction in support of beginning reading instruction: A review. Review of Educational Research, 72(1), 101-130.
[go to CARET review]
* Kulik, J. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker, E. L. and O'Neil, H. F. Jr. (Eds.), Technology assessment in education and training. (pp. 9-33) Hillsdale, NJ: Lawrence Erlbaum.
[go to CARET review]
Comments:
See the reference list below for additional meta-analyses reviewed or cited within CARET.
* = Reviewed in CARET
|
