| 1. |
General Articles and Expert Positions: These are articles
that describe processes, strategies, approaches, theoretical models, policies,
curriculum or technology using standards, that may or may not allude to
full-scale evaluation or empirical studies. Often such articles are based
on experience, observations, and ideas proposed by the author(s). Examples
include the ISTE standards, National or State technology plans, technology
use planning guidelines and templates, testimony to commissions and legislative
bodies, professional association platforms, etc. Type A information resources
will be rated and designated a "level 1, 2, or 3" according to
the following rubric: |
| |
|
Level A: The report is clearly documented by research or credible
evaluations, provides clearly stated recommendations that can be easily
applied in a wide variety of educational settings and would help inform
educational technology planning and/or policy decisions to a wide audience. |
| |
|
Level B: The report is based on some type of consensus, discusses
examples and shows awareness of available methods, research or evaluation
studies on the topic, and would help inform educational decisions. |
| |
|
Level C: The report is basically one person's/one group'sopinion
/ experience (e.g., outlining a theory or describing a methodology or analysis
technique), may or may not be supported by documentation or research, is
primarily of use in program implementation decisions. |
| 2. |
Descriptive and Survey Studies: These studies are descriptive
research and typically use data derived from surveys, case studies, or more
qualitative methods for gathering the information to inform the conclusions
and recommendations of the study or report. Type B information is often
data collection to yield new information, generate clearer questions or
needs identification, and focus on specific projects or settings--the primary
intent is to synthesize emerging patterns. Examples range from single project
case studies to national studies such as the CEO Forum report, NCES Surveys
on technology access in schools, meta-analysis reports of studies such as
the recent SIIA report on technology effectiveness, state-wide or large
scale needs assessments, Quality Educational Data (QED) and Market Data
Retrieval (MDR) surveys, and many others. |
| |
|
Level A: The data collection methods are clearly explained and
credible with multiple measures, are current, could include strong mixed
methods research, with adequate sampling and the data collected can easily
justify the conclusions drawn from the study. |
| |
|
Level B: The data collection methods are credible with validated
surveys and/or other methods are clearly described such as observational
strategies, case study methodologies, the data or information is current. |
| |
|
Level C: The data collection -is focused on a limited sample of
the population (e.g., single project case study) - and has minimal application
in terms of generalizing the findings. |
| 3. |
Formal Evaluation Studies: These are evaluation studies
that assess the extent of implementation and impact of a specific program
or project and usually emphasize needs assessment and/or formative evaluation
methods designed to provide ongoing feedback to program or project managers.
Impact or summative data is used to estimate the extent of change in the
targeted population based on particular project or program interventions.
Evaluation studies tend to focus on specific contexts of particular grants
or projects with no attempt to generalize beyond the cases at hand. They
often use a range of qualitative and quantitative tools but rarely use control
or comparison groups and statistical methods to test specific hypotheses.
Examples of such studies include evaluations of Technology Innovation Challenge
Grants (TICG), Star Schools, Preparing Tomorrows Teachers for Technology
(PT3), National Science Foundation (NSF) projects and programs, as well
as a variety of state and privately supported programs, initiatives, and/or
products. |
| |
|
Level A: The evaluation includes and justifies valid and relevant
measures and clearly describes the specific project interventions that contribute
to the program or project outcomes and then provides sufficient contextual
information for someone to generalize and possibly replicate the results
by adopting the program in another location. |
| |
|
Level B: The evaluation clearly describes design, the measures
used and how they were developed or selected and analyzed to provide answers
to specific evaluation questions that are directly related to the goals
and objectives of the project. |
| |
|
Level C: The evaluation is not thoroughly described in terms of
theory base, design, methods, findings, and summary/recommendations. Rather,
one or more of these components is emphasized in sufficient detail for replication
or decision making. |
| 4. |
Formal Research Studies: These studies are quantitative
research studies, which incorporate a formal research design to test a hypothesis
with validated measures. These studies should document that this research
can be replicated or generalizable to other settings. The criteria used
for reviewers to judge the level or extent of validity of studies is adapted
from the Fouts (1999) model for rating educational research according to
level of depth and empirical validity. An example would include a study
of a specific computer delivered curriculum in the area of primary level
reading in comparison with a textbook delivered curriculum with variables
such as curriculum content and student characteristics are held constant.
The design allows for a statistical test of the specific differences found
between the two groups on a validated measure that is relevant to the skills
and knowledge intended to be developed. |
| |
|
Level A: As with Level B, the study must incorporate a formal research
design to test a hypothesis with validated quantitative measures. The study
must clearly document that the results found in the initial evaluation or
research have been, or have a high probability of being replicated in other
settings, locations, or contexts. |
| |
|
Level B: A level B study must incorporate a formal research design
to test a hypothesis with validated quantitative measures and can be backed
with qualitative measures to provide more descriptive information in describing
the findings. |
| |
|
Level C: This level of study would probably be described as research
but upon closer inspection relies on surveys and measures that are not triangulated.
This study tends to lack a clear relationship between the intervention studied
and the outcomes attained due to inadequate research design. |
| After careful consideration of the features, study types,
and levels, please use your best and most objective judgement to categorize
the study as follows by placing a check in the appropriate box that corresponds
with the Categories and Level Rating Criteria described above. |
| Study Type |
Level A
|
Level B
|
Level C
|
| Type 1 |
|
|
|
| Type 2 |
|
|
|
| Type 3 |
|
|
|
| Type 4 |
|
|
|
|