EMAN-quality-in-wbde

Quality Standards and Quality Assurance in
Web-Based Distance Education

[WWW document]. URL http://www.sbg.ac.at/erz/eman/review.htm. 2000

Hermann Astleitner
University of Salzburg

Abstract
Web-based distance education which uses the World-Wide-Web (WWW) for educational purposes will be one of the most promising ways of flexible learning in the future. But this often heard prediction will only come true, when new emerging forms of distance education environments are of high quality. In the long run high quality can be achieved when quality standards are identified and quality assurance systems based on these standards are established. It is the purpose of this paper to comprehensively review and integrate on- and off-line information resources concerning types of quality standards and ways of quality assurance in web-based distance education. The paper tries to answer the following questions: what and who is controlled?, which modules of standards are used?, who controls quality and what functions serve quality control?, and how is quality controlled and assured? Also theoretical assumptions and empirical evidence in relation to the significance and the problems of the identified quality factors are discussed.

Keywords: Online learning, distance education, virtual classroom, quality management, guidelines, criteria

Lifelong learning is seen as essential to professional development in the 21st century. For many educational experts "web-based distance education" offers the kind of solutions that our societies need for ensuring access to a lifelong learning process (Bourdeau & Bates, 1997). "Distance education" is based on two constituent elements: a pre-produced course and a non-continuos communication between students and the supporting organization with its teachers and tutors. "Web-based distance education" takes place within a hypermedia instructional program which utilizes the World-Wide-Web (WWW) to create a supportive distance education environment (Willis & Dickinson, 1997). Web-based distance education can be integrated in traditional instruction or can substitute it by covering some or all parts of a lesson, some or all parts of a course, and some or all parts of a study program. Web-based distance education has to be discriminated from "web-based instruction" which represents an unsystematic and incomplete way of using the web for educational purposes in traditional instruction (e.g., showing the students in regular classroom the capacities of the WWW or illustrating contents with examples accidentally found on the WWW) and which does not cover a fully developed distance education lesson, course, or study program. Fully developed web-based distance education can take over nearly all teaching functions of traditional instruction and deliver, for example, teaching materials, tutor assistance, "virtual" office hours, learner-to-learner interaction, web-based testing and performance evaluation, enhanced inter-institutional collaboration, and program administration and marketing (Collis, 1996). Good examples of fully developed web-based distance education courses can be found at the "Globewide Network Academy" (URL http://uu-nna.mit.edu:8001/uu-gna/index.html).

Using the WWW for distance education has several advantages: (a) high time and place flexibility for students, (b) potential to reach a global audience, (c) no concern about the compatibility of the computer equipment, (d) quick development time compared to videos and CD-ROMs, (e) easy updating of content, and (f) usually lower development and operating costs compared, for example, to satellite broadcasting. Web-based distance education is usually cheaper, faster, and more efficient than other distance learning environments, but not necessarily more effective, because the flexible access to information does not automatically expand the knowledge and skills of students.

At present, limited capacities of communication hard- and software hamper the delivery of sound or video in the WWW. Most web-based distance education environments lack adaptive features and therefore rely mainly on learner initiative, which can be a drawback for those who prefer more guidance and structure. The success of learners also depend on the ability to cope with technical difficulties which are often connected with information overload resulting from simultaneously dealing with different emails, databases, or web-sites what require sophisticated information management skills which people often do not dispose of. Other problems in web-based distance education are drop-outs, social isolation, or the lack of nonverbal cues. Web-based distance education also puts educators and instructional designers in a global competition with tough labor conditions. There are new challenges to be faced by distant teachers, for example, looking at a course in a new way, shifting from the role of content provider to content facilitator, using technology as a primary student-teacher link, or understanding distant students` lifestyles.

In order to meet the requirements of students, web-based distance education has to be improved continuously, especially when one takes into account the high general untrustworthiness of the WWW and the tremendous speed of WWW expansion. A major basis for quality improvement is "evaluative information". It provides timely feedback and constructive critiques to the decision makers (school boards, legislators) and to the developers of the web-based distance education environment. Evaluative information represents the central basis of a quality management system which has to address the following questions: (a) what is controlled in web-based distance education (e.g., inputs, processes, or outputs)?, (b) what are the standards of quality usable in web-based distance education? (e.g., usability, reliability, or readability), (c) who is controlled and which functions serve quality control (e.g., educational systems, institutions, teams, or individuals for design guidance, legitimization, marketing, or staff controlling)?, and (d) how is quality controlled and assured (e.g., testing, self-evaluations, or quality management)? (Posch & Altrichter, 1997). The questions (a), (b), and (c) are addressing types of quality standards, the question (d) is dealing with methods and procedures of quality assurance.
It is the objective of this paper to answer these questions reflecting current developments in web-based distance education. The basis for answering the questions is a comprehensive review of contents of books, of journal articles, and of web-sites dealing with quality standards and quality assurance in web-based distance education. The identified quality standards are collected, categorized and briefly discussed based on empirical research evidence and on the degree of integration into existing theories from the fields of education and psychology. Ways of quality assurance will be discussed in reference to their practicability and efficiency.

Types of Quality Standards

What is Controlled in Web-Based Distance Education?

In respect to types of quality standards, the first issue is the question of what is or should be controlled in a web-based distance education environment. The answers on this question are "inputs", "processes", and "outputs".

"Inputs" are independent conditions which influence the web-based distance education environment as a whole. Some parts of these inputs are, as a rule, under the control of politicians or other not local decision makers (i.e., legal regulations and budget), some are controlled by local decision makers (i.e., time prescriptions, technical equipment, staff and staff development, and audience), and some are directly influenced by the distance education institution (i.e., teaching materials and instructional context).

"Processes" are influenced by inputs and are themselves influencing "outputs". "Qutputs" are quality factors which are influenced both from inputs and from processes. Is has to be mentioned that in the long run outputs may also influence inputs and processes. For example, the actual success of web-based distance education, measured in completion- and drop-out-rates, often represents an important factor in future budget allocations which are primarily seen as inputs in Figure 1. Processes as quality factors in web-based distance education concern political negotiations and decisions, the management of the distance education institution, the information flow (interaction, participation, and knowledge transfer) within the institution, the production process (design, development of learning materials, and quality management), and the teaching-learning situation in the virtual classroom.

Important outputs of web-based distance education are courses involving different subject areas, instructional methods, and technical platforms. Also, outcomes more closely related to the students have to be considered in quality control (i.e., achievement, contentment, or completion- and drop-out-rate). Further quality factors concerning outputs are the reputation of the distance education institution, the staff contentment with working conditions, and the costs resulting from operating the whole distance education institution (in which the cost savings or the rise in costs in comparison with other distance education institutions or traditional forms of instruction should be included).

A closer look at the inputs, processes, and outputs of web-based distance education and their evaluations in practice shows that in most cases of evaluations hierarchically lower quality factors (e.g., the teaching-learning-processes) are more often controlled than hierarchically upper quality factors (e.g., policy or management), what correspond with evaluations in other educational fields (Specht & Thonhauser, 1996).
It is not always clear whether certain quality factors are inputs or outputs. Teaching materials are, for example, important inputs for the teaching-learning-process, but they are also the outputs of the design- and development-process. It is the timing and the objective of the evaluation which helps to decide whether the input or the output aspect of a quality factor is in the foreground of evaluation. Most of evaluations are undertaken relatively late when implementing web-based distance education, therefore most of evaluation activities are output-related. Also, many institutions do not allow evaluations to disturb in some respect the development- or the design-process, because they are afraid of too much discussions or conflicts of interests which can result in inefficiency. In this case, the objective of efficiency is also stressing the output perspective.
Here it was tried to depict all the quality factors in a hierarchical order showing factors that influence many others on the top and showing factors that are influenced by many others at the bottom of each quality category. Of course this ranking is not a causal-like or a fully consistent representation of quality factors` relationships, but it reflects some highly probable or plausible preconditions and consequences of each of the quality factors.

Who is Controlled in Web-Based Distance Education?

Referring to the question of who is controlled in web-based distance education Thach and Murphy (1995) described several professional roles reflecting the necessary competencies for successful distance education all of which should be in the focus of evaluation (see Figure 1). Politicians and other decision makers in educational systems are responsible for the information-infrastructure in a society. They have to formulate educational targets, to anchor them in corresponding laws, or to create and coordinate financial resources (Astleitner, 1998). The duties of managers in web-based distance education are, for example, to supervise staff and operations, to balance the budget, to advocate and market the distance education program, to ensure evaluational processes, or to plan future technology use. The instructional designer develops courses from a pedagogical point of view, revises existing courses, and ensures that the course design works with technology and other parts of the distance education program. The technology expert or site facilitator has to assure the smooth operation of technology, to advise in the selection of technology, to work cooperatively with instructional designers, to ensure that the remote site is properly prepared, or to technically distribute and collect materials. The tutor (or teacher) facilitates information presentation, gives learning support to students, plans and prepares class sessions, monitors and evaluates student performance, initiates and maintains interactive discussions or peer learning, and others. The technician keeps the technical equipment in running condition, responds to users` questions and problems, or provides technical trainings for users when needed. The administrator has to promote and organize teamwork, to inform students of available courses, to conduct registration and scheduling, or to coordinate support service issues. The duties of the editor or web-designer are to proofread and edit the instructors` work and to arrange and program the textual and graphical layout and all interactive features for presentation with web-based software tools (e.g., HTML). The librarian provides library assistant to the distance education students, offers library or information skills trainings, assists students with library searches or research, and delivers library materials (books, cdroms, etc.) to the students. The evaluation specialist provides tools and evaluation instruments, monitors program successes and problems, and consults with the tutor, instructional designer, and others on results of the evaluations. The students are the persons for which all the given duties are undertaken. Students have to apply for the courses, to work with the distance education environment, to cooperate with tutors and peers, to utilize the technical equipment, to prepare and take exams, and so on.
Considering all the different professional roles in web-based distance education, it has to be mentioned that students are the most frequently used sources of evaluational information. If students discover problems or shortcomings in a distance education environment, then further evaluational processes are started including other professional roles.

Which Standards are Used in Web-Based Distance Education?

Modules of standards do not represent specific standards, but define different categories of standards, criteria, or guidelines. These categories were identified based on a comprehensive review of literature and of web-sites. First, a list of found standards was set up, then the items on the list were clustered and finally cross-checked (Miles & Huberman, 1984).

As a rule, the first contact of a student with a distance education environment lies in the usage of student services which are usually implemented in the WWW and/or are accessible via telephone or by office attendance. Good quality student services help students in registration, payment, and scheduling duties. They also offer or manage access to admission standards, to library resources, to possible jobs, to study advice, and to financial aids. Student services which do not meet these quality requirements are in the need of improvement. But having all the mentioned components of good student services does not guarantee high quality. High quality in student services is above all accomplished when their components are easy to use, not bureaucratic, and freely and continuously disposable.

Modules of quality control concerning technical functioning refer to the (good) working conditions of the technical equipment used within the distance education environment, to the site accessibility regardless of time and place (means access free of time and place restrictions), to offers in (efficient) technical support, and to the availability of back-up-systems (in cases of severe technical problems). In general, the technical equipment achieves high quality performances when their components are regularly checked and updated, easy to handle, and highly compatible with different user profiles.

Quality factors in web-design concern information structure and design, navigation within the documents, and aesthetic and affective aspects (Wilkinson, Bennett, & Oliver, 1997). Information structure and design deal with how the web-document and its contents are organized and whether the document is based on accepted instructional design standards. These standards are, for example, stating the purpose of the document, describing its scope, implementing interactivity, or providing different media to support individual learning preferences (e.g., visual, aural, numerical, verbal, etc.). Navigation within the document concerns questions about how easily documents can be explored and how efficiently organizational structures, menu designs, indexes, tables of contents, search functions, or online help can be used. Aesthetic and affective aspects deal with how the document is designed in respect to graphics, readability, and the use of fun, motivating, or creative elements.
Ethical and ethnical concerns prescribe that the students and the staff of a distance education institution should have a right of co-determination or should be heard from decision makers in questions of general interest. The distance education environment needs to describe the goals of its institution, and the social, political etc. backgrounds. It should create a climate of cultural sensitivity and anti-discrimination, within an environment that is safe and based on trust and equality. Special attention in information design and access should also be given to disabled persons suffering, for example, from low vision, hard hearing, or physical disabilities in fine-motor skills. All electronic communication needs to be seen as private, that is not to be forwarded, shared, or used unless express permission is given. Distance education institutions should also have an obligation to inform students about psychological, social, etc. risks (e.g., social isolation) that may result from being a distant student. Of course documents containing violence, pornography, etc. should be avoided (e.g., Schrum, 1995).
Legal matters in quality control refer to resource and author identification, questions of intellectual property and copyright, data security, and also credits compatibility and course equivalency. Resource and author identification deals with such information as the address of the document and a brief description of its purpose, its content, or its intended audiences. Also contained as quality feature should be descriptive information about the author, such as name, position, education, or contact information. Other individuals or organizations who sponsor or are involved in the production of the distance education environment must also be identified. As a further quality characteristic, good distance education institutions dispose of the exclusive right to their teaching materials presented on the web. Quality control in respect to copyright laws for distance education clarifies the access permission, the character of the use and the nature of the teaching materials to be copied, the usage of materials from archives or special collections, the incorporation of texts or images to new materials, and so on. It addresses questions like, for example, what does copyright protect, when does it begin and end, what does it mean to owners and users, or what is "fair use" (Harper, 1997). Data security as an issue in quality control deals with the necessity to send, receive, save, and use data from the students or other persons in a way that nobody outside the distance education environment can have illegal access to it. Data from students are, for example, address, family background, education, access patterns in using the distance education environment, problems in completing a course, or grades on examinations. Also, credits compatibility and course equivalency are important legal matters in distance education quality control. Students when registering for courses consider information about how many credits a particular course gets in comparison with courses from other schools, universities, or distance education institutions. Students also pay attention on whether an offered course matches other courses in comparable study programs. Both, high credits and high equivalency increase the flexibility of the distance education offer and allow students to choose course offers according to their individual needs. Both of these two features of distance education quality have to be explicitly fixed based on legal provisions with other educational institutions.

One of the most important factors in quality control of web-based distance education represents the content of the teaching materials. A first criterion is that the contents are based on a given curriculum. A second criterion, the relevance and scope of the content, deals with questions whether the content is related to the users` needs, whether the content provides new information on the topic, whether there are obvious gaps in the coverage of the topic, or whether the document can be integrated within a broader field of knowledge. The validity of the content deals with the confidence one can place in the information contained in the teaching materials, for example, whether the document was subjected to a review process, whether the content contradicts or confirms other sources of information, or whether the author provides a bibliography in order to prove the origins of the contents. The criterion of accuracy and balance of the content concerns questions like, for example, whether all sides of a controversial issue are presented, whether possible sponsors have a commercial interest in the topic of the document, or whether there are indications of careless preparation, such as spelling errors. Most of distance education web-sites show links to other web-sites containing teaching materials, guidelines, etc. The criterion quality of links deals with how useful links are, how clearly they are marked, how the users are informed of the type of information they are linking to, whether the given links are evaluated prior to inclusion, or how reliable the links are. A further criterion in relation to the content indicates, how understandable and explicit the contents of the documents are. Understandability applies to simplicity, structural arrangements, pithiness, and motivational features within the texts of the document. Explicitness refers, for example, to the degree of noticing hidden assumptions, of showing analogous or other relations between different parts of the content, or of using advance and post organizers in the document. All the presented contents have to be feasible for the target population in respect to the task difficulty of exercises and exams, to the quantity of the subject-matter, and to the given time for completing the lessons. Finally, a last content-related criterion means that content presentation, given exercises, and feedback to the students should be based on detailed performance standards. They describe what specific type of behavior students should show after having completed the lesson or important parts of a lesson.
From a pedagogical point of view the successful support in learning is a further essential characteristic of a high quality web-based distance education. Learning support should consist of tutorial support and collaboration between students in combination with teaching strategies and instructional principles focussing on special instructional problems found in distance education. Tutorial support and collaboration have to be realized with synchronous (e.g., via videoconferencing) and asynchronous (e.g., via email) communication. Both forms of communication are necessary in order to flexibly cope with different types of learning problems. For example, problems in geometry might not be effectively dealt with via email, because most of such problems need textual or graphical representation which can more suitably be delivered by videoconferencing. Web-based distance education takes place in a technical environment in which direct human relationships are minimized and only technically mediated. Tutorial support and measures for establishing collaboration have to create a friendly, social environment in which social interactions and group cohesiveness are fastened. Small group discussions, debates, polling activities, dyadic learning partnership exchanges, or one-to-one message exchanges are necessary activities in order to encourage participation. Also opportunities for subject-matter-related and private conversations in traditional (e.g., classroom) environments have to be designed and regularly held. Tutors should also draw attention to opposing perspectives or conflicting opinions that lead to highly interactive debates or peer critiques (Berge, 1996). The major task of tutorial support and collaboration between students is to assist students in problem solving concerning learning, social, organizational, and technical problems. The tutor and to some respect other students have to present or to make accessible the subject-matter, to give constructive feedback to students` questions and responses, to advise students in successful learning strategies, to facilitate interactivity, to reinforce and model good social behavior, to provide information about activities for registration, admissions, student counseling, or other administrative duties which are unanticipated by the novice distant student, to give or organize technical support for emailed or telephoned inquires, and many more.
The second component of successful learning support and therefore for quality control in web-based distance education relates to the application of traditional and distance education teaching strategies and instructional principles. Successful teaching is motivating, highly goal-orientated, reminds past knowledge, provides guidance and feedback (e.g., with guide questions), uses self-checking exercises, and ensures transfer and critical thinking. In respect to instructional principles students` errors and problems have to be anticipated and addressed through adequate teaching materials and tutorial support. Multi-view presentations concerning the structure of the presented subject-matter, the use of text-, audio-, or video-formats, the difficulty of the exercises and others are a major source for adapting instruction to students` individual needs. In order to facilitate comprehensive learning experiences also a sufficient number and variety of learning activities and of online testing opportunities have to be implemented in web-based distance education. Finally, generative learning (which requires higher-order cognition and defending answers) and contextual engagement (with focus on real world problems) have to be ensured (see principles and examples in detail in: Hall, 1997; Khan, 1997; Oliver & Reeves, 1996).
Miscellaneous aspects in quality control in web-based distance education consist of the adequacy and (long-term) stability of financial resources, of regularly accompanying measures in faculty education, and of the establishing and the maintaining of a quality assurance program (see next section).
It is obvious that certain modules of standards are related to certain inputs, processes, outputs, or professional roles in web-based distance education. For example, standards in respect to technical functioning will be used for evaluating the work of the technology expert and/or the technician. Standards concerning content and learning support are, for example, connected to teaching materials (as inputs), the teaching-learning-processes (as processes), and students` achievements (as outputs), but also to the work of the instructional designer and the tutor. In this article, it was not intended to assign each module of standards to a particular input, process, output, or role, because in the practice of distance education the duties of involved persons are not always clear or cannot always be delimitated from each other. The modules of standards presented in this article must be seen as pools of standards, criteria, or guidelines which can flexibly be selected and combined according to individual requirements in the evaluation of the distance education environment.

Ways of Quality Assurance

Ways of quality assurance in web-based distance education concern the questions of who controls quality and which functions serve quality control, and how quality is controlled and assured.

Who Controls Quality and Which Functions Serve Quality Control?

Quality control in web-based distance education is usually done by experts in the fields (e.g., university professors in educational technology), by staff members from inside and from outside the distance education institution (e.g., the work of technicians can be evaluated by technicians from other distance education institutions or by colleagues from the own institution), and by users or clients of the distance education institution (e.g., students and/or their parents).

Quality control helps to distinguish between good and bad distance education institutions or parts of them. It provides information for the students` selections of suitable institutions. It also gives educational decision makers a basis for deciding which institutions should be forced or promoted and which institutions should be improved or shut down. At a national and at an international level quality control makes it possible to compare distance education institutions with each other and with commonly accepted standards. Quality control tells the managers of a distance education institution which parts or services function well and which parts need improvement in order to be successful in the main objective, the education of students. Quality control can also serve as a way of self-control to all staff members. It is especially helpful, when new technical, pedagogical, etc. approaches are introduced or when some unspecified pedagogical or other problems arise.

How is Quality Controlled and Assured?

Methods and instruments. Quality control in web-based distance education uses methods and instruments from empirical social research: surveys, interviews, observations, and tests in experimental and non-experimental settings which are often based on given checklists and on competence lists or tests, sometimes on general guidelines, and seldom on course evaluation systems. Methods and instruments are applied in traditional ways (e.g., using paper questionnaires) or through using the WWW (e.g., presenting interactive questionnaires which results are emailed to the distance education institution). Results discovered with methods and instruments are reported to the relevant persons which use the information to improve the quality of the web-based distance education environment. Regular repetitions of this process and accompanying measures (see below) should achieve quality assurance.
A checklist of 125 indicators of resource quality or questions to be asked about a distance education web-site was presented by Wilkinson, Bennett, and Oliver (1997). The major criteria of this checklist, which are closely related to some of the quality factors presented in this paper, are: site access and usability, resource identification and documentation, authority of author, information structure and design, relevance and scope of content, validity of content, accuracy and balance of content, navigation within the document, and aesthetic and affective aspects. This checklist might be used for evaluation, but can also function as guideline for distance education web resources design. A further more practicable checklist of 68 items with other priorities which should be used when considering the purchase of a web-based distance education course and when developing such courses is presented by Brill-Pisik (1997). The items are dealing with content and instruction, activities of learners, potential for job transfer, design and packaging, and operation characteristics. Porters´ (1997) checklist for evaluating online courses is focussing on course content, degree programs, time frame, costs, and technical requirements. Schrock (1996) developed a critical evaluation survey for web resources on different school levels. It addresses questions like: how does it look?, what did you learn?, what is on the page and who put it there?, and gives also opportunity to a narrative evaluation. In respect to narrative evaluation a comprehensive list of open questions for online course evaluation can be found, for example, at http://pstcc7.pstcc.cc.tn.us/ost/formeval.html, other rating scales can be found at http://www.cyberbee.com/guide1.html, http://newark.rutgers.edu/~rps/nsurvey.html, or http://www.microweb.com/pepsite/. In web-based distance education also other forms of instructional technology can be integrated. For example, one part of a web-based distance education course might consist of traditional computer-assisted instruction using drills, tutorials, or simulations for instructional purposes. In this case checklists for (traditional) educational software can be used to evaluate the quality of instruction (see such checklists, for example, in: Gärber, Harbeck, & Lauterbach, 1992; Wazel, 1998). In this respect, it has to be mentioned that also general measures for the evaluation of traditional instruction can be used in the field of distance education. Gagné, Briggs, and Wager (1988) showed a list of many possible assessments including: need (i.e., the necessity of the product in the field of education), market (i.e., the possibility for selling the product), evidence of good performance in field trials, consumer performance, performance comparison (with other competitive products), long-term effects of the product, side effects of the product, performance process (i.e., indication that the processes of instruction are as proposed in the product), statistical and practical significance of performance, costs and cost-effectiveness, and extended support (e.g., continued monitoring and updating of the product).
Schnepf, Mashayekhi, Riedl, and Du (1994) defined pre-, in-, and post-class competencies for distance education instructors (tutors/teachers) which can be used as a list of competencies for evaluational purposes. Pre-class competencies concern the search, the capture, the creation, the storage, and the integration of learning material. In-class competencies are the presentation of the content or parts of it, the establishing and terminating of sessions, the organization of workspace, the breaking of students into small groups, the giving of exams and quizzes, and others. Post-class capabilities are related to reviews of class sessions, to the providing of tutorial services, or the submission, the grading, and the recording assignments. Thach and Murphy (1995) presented a detailed list of competencies for all types of distance education professionals which can function as an action-orientated way of evaluating individual duties in a distance education environment. These competencies are related to different professional roles and concern planning skills, instructional design skills, skills in distance education teaching, collaboration skills, technology repair skills, managerial skills, editing skills, library research skills, or data analysis skills. McKenzie (1997) developed a rating scale reflecting an important competence in distance education, namely "information skills" which fall into questioning (i.e., recognizing decisions, issues, and problems when looking at a topic), planning (i.e., identifying sources of information likely to build understanding), gathering (i.e., collecting and storing information for later consideration), sorting (i.e., reorganizing information to support understanding), synthesizing (i.e., recombining information to develop decisions and solutions), evaluating (i.e., determining whether the gathered information is sufficient to support a conclusion), and reporting (i.e., translating findings into usable products). A very important form of the evaluation of individual competencies concerns the assessment of achievements. The assessment of staff and students` progress has also the function to give them feedback so that they know how they succeed, and, in all cases where diplomas are required, to provide the basis for marks. The assessment of staff and students should be based on reliable and valid tests which are applied in a repeated-test-design and based on the definition of behavioral objectives which express what a person is expected to be able to do in his job and/or after having completed a course or a course unit (Holmberg, 1995). Commonly used dependent variables in assessments are: number of errors made, positions where the errors were made, time taken to complete a given task, time taken to recover from an error, quality of the resultant task output, preference ratings, attitude scores, or latency time before initiating a response. These variables can be measured by tests complemented by observation, protocol analysis, or online monitoring.
General guidelines for distance education as a further universal basis for evaluations are formulated by Willis (1993). They contain prescriptions about meeting students needs (of both content and preferred learning styles), about course planning and organization, about teaching strategies, and about interaction and feedback. The guidelines of Wagner and McCombs (1995) are closely related to the "learner centered psychological principles" of the APA (American Psychological Association) and are dealing with 12 important factors influencing learning in distance education courses (i.e., the nature of the learning process, goals of the learning process, the construction of knowledge, higher-order thinking, the depth and breath of processed information, negative cognitions and emotions, curiosity and creativity, individual developments, social learning, caring relationships, individual differences in learning, and cognitive filters).
Course evaluation systems are based on systematic instructional design and on empirical research demonstrating effective instructional strategies. They are a diagnostic and prescriptive method for evaluating the quality of educational measures and for suggesting remedial measures. Within course evaluation systems instructional products are evaluated according to the appropriateness of the objectives, test items, and presentations for the overall course goal, according to the consistency of objectives and test items (do the test items match the objectives?), according to the adequacy of the test items (are the test items properly constructed?), according to the consistency of the instructional presentation and the course objectives, according to the adequacy of the instructional presentation, and according to the instructional effectiveness (Ellis, Knirk, Taylor, & McDonald, 1993).

Procedures of quality control and assurance. Quality control in web-based distance education is implemented by certain procedures, like standardized evaluational processes, QM (Quality Management), special awards or competitions, or WWW repair approaches. For example, Collis (1993) presented an evaluational model for distance education which consists of several stages and which describes a highly standardized evaluational process. In a first stage, information about the assumptions and intentions of a distance education project is gathered, then, in a second stage, the contingencies among project intentions and assumptions are assessed. In a third stage, the implementation of the project and the actual distance education context are observed. Finally, congruities and incongruities between what was intended and what is occurring are identified. Each of the stages is based on assumptions about the context, general intentions, plans for executions, and short- and long-term success indicators. According to Dick and Carey (1985) a "formative evaluation", i.e. the collection of data and information during all phases of distance education, should be used in addition to "summative evaluations" which are undertaken, when a final version of instruction or learning materials has already been produced and which represent the most common form of evaluation in distance education. For example, formative evaluations in respect to instructional design consist of observing different steps: identifying instructional goals through conducting instructional analysis (task analysis) and through identifying entry behaviors or characteristics, writing performance objectives, developing criterion-referenced test items, developing instructional strategies, and developing and selecting instructional materials. These observations are, for example, undertaken in a one-to-one evaluation. The purpose of the one-to-one evaluation is to identify and remove obvious errors in the instruction and to obtain first reactions to the content from learners. This is accomplished by direct interaction between the instructional designer and students who are representative of the target population. Dringus (1995) presented an iterative procedure for the formative evaluation of web-based distance education courses, particularly their interface presentation, in which different methods of other approaches are combined and integrated.
A quality management approach in web-based distance education can be found, for example, at the Murdoch University (Brown, 1998). A first important part of this approach consists of standards for the production and delivery of web-based distance education courses which are available to all staff members. A second part concerns permanently available practical advice and assistance in planning (e.g., the instructional design), delivery (e.g., class administration), and in the creation of web documents (e.g., technical help for staff). A third part consists of standardized study guides for students which represent a way of organizing the content of a course. All contents are built from an introduction and a structured sequence of topics with a repeat structure for all topics. Each topic contains learning objectives, key concepts, topic notes, readings/materials, and learning activities (e.g., learning tasks).
Special awards and competitions are a way in quality control that make distance education institutions to present their products to a national and international audience. Participating in such contests is motivating for staff members. Showing a good contest performance can be used for marketing and advertisement purposes. Outstanding products (web-based distance education courses) also represent good models for the design of distance education environments and inform institutions about actual and future developments around the world.
WWW repair approaches are measures to improve many aspects of web resources over the long run and should be considered when starting new or when revising existing web-based distance education courses (Ciolek, 1996). There are six repair approaches: programming, procedural, structuring, bibliographical, evaluative, and organizational ones. Within the programming approach it is the idea to use (continuously improving) software to organize, channel, and guide publishing and communication activities on the WWW what should reduce the scope for errors and blunders on the web. The procedural approach is focussing on the documentation of design, production, and maintenance procedures and on promoting web publication guidelines and templates. Structuring approaches deal with data-location tags in web documents, with data-filtering, or with multiple ways of viewing and analyzing information in collections of documents. Bibliographical approaches try to elaborate citations schemes and to establish them as minimal conventions. Evaluational approaches use ratings concerning quality factors mentioned in this paper to judge the usability, instructional quality, etc. of the web document. Within organizational approaches it is assumed that cooperation and coordination between producers of web learning materials should improve quality in web documents.

All mentioned procedures of quality assurance have to be effective during the whole process of implementing and using web-based distance education environments. Quality assurance means that evaluation has to be done repeatedly and has to include all parts or all staff members of the distance education institution, that the outcomes of the evaluation have to be presented to all parts or staff members of the institution, and that there is assistance (advice and education) for staff members for permanently improving their achievements (Limacher, 1996). Quality assurance should become an integrated part of the staff members` daily work. This can be realized when all staff members fix their own quality development and continued education plans and activities in respect to their individual duties, in arrangement with their superiors and in consideration of the evaluational standards, processes, and outcomes. Such individualized quality development and continued education plans are the most important key to successful quality assurance in the long run.

Discussion: The Significance of Quality Factors and Quality Assurance

Evaluation methods strongly vary in their sophistication and their applicability. The least sophisticated approaches require little preparation and are the least time-consuming but are the least diagnostic with respect to the problems at the web-based distance education environment. It is the task of the evaluator in cooperation with the distance education institution to find out what way to go, because the more sophisticated evaluation methods are often time consuming and require knowledge of complex data collecting and data computing methods (Johnson, 1992).
The selection of evaluation methods and instruments also depends on the used technology. None of the given instruments can be used to evaluate all possible distance education environments. Some parts of the available instruments can be usable in all environments, but other parts of an instrument for evaluation have to be adapted to the given distance education environment. In this respect it is also not clear, what kind of distance education technologies should be used for what kind of teaching objectives and learning activities (Astleitner, Sams, & Thonhauser, 1998). Text-based WWW-sites might be suitable for storing learning materials and for acquiring basic knowledge, facts, and deep understanding. Multi-media-WWW-sites (or -CD-ROMs) motivate students, entertain them and give opportunities for browsing. Videoconferencing might be necessary when audio and visual demonstrations, simulations, or real-world-interactions are part of the teaching process, when highly complex problems have to be solved, or when immediate feedback is fundamental for learning. Email and Internet Relay Chat (IRC) help in discussing issues which need no audio or visual representation (Gay, 1997).
Many checklists for quality control, that can be found on the WWW, are not comprehensive, they are only dealing with some aspects of quality. For educational practice it is acceptable that different modules of different instruments are combined in order to evaluate a particular distance education institution. This procedure means to eclectically gather and select modules of quality factors and apply them, not for testing theoretically deduced hypothesis, but for getting comprehensively in touch with different aspects of the distance education institutions` reality and their efforts for improvement. Of course evaluations have to be based on adequate theories, but primarily they have to cope with practical problems.
Despite the acceptance of theoretical foundations of evaluations in the field of distance education, many evaluational approaches do not reflect theories of distance education or, at least, do not mention them explicitly (e.g., Clearinghouse, 1998). There are theories which were formulated taking into account the conditions and consequences of distance education, but many theories, like theories of independence and autonomy, theories of industrialization of teaching, or theories of interaction and communication are highly general, so that applying these theories in the practical field of distance education needs a lot of work in defining more concrete concepts or in developing research instruments (Astleitner & Leutner, 1998). Other theories come from the field of traditional instructional theories and are applied in the field of distance education (e.g., cognitive apprenticeship, anchored instruction, or reciprocal teaching) without reflecting particular conditions in distance education. Some theories are adapted to typical learning conditions in distance education environments (e.g., open learning theory, social learning, or theories of self-organized learning), but only few theories are developed in the distance education context and for explaining important aspects of distance education (e.g., Holmberg`s (1995) "theory of learning and teaching in distance education", or Carroll`s (1990) "minimalist instruction"). Future theories of distance education need to take into account particularly the situation of the distant learner, the technical settings, and specific (interactive) instructional methods which are primarily developed for distance education like self-checking exercises, student progress tracking, or competency-based mastery learning (Brusilovsky, Kobsa, & Vassileva, 1998). Such theories will be more considered in web-based distance education evaluations` than existing theories and therefore decrease the theoretical vacuum in evaluations of web-based distance education (see theoretical approaches in: Astleitner, 1997; Astleitner & Keller, 1995; Astleitner & Leutner, 1995, 1996).
It is also often not possible to figure out where the criteria in evaluational instruments and procedures come from. Some of the checklists are based on individual experiences, or some on a poll of a small group of experts in information quality which expertise is hardly to prove (see, for example, the criteria list from: Khan & Vega, 1997). For many guidelines and checklists it is not clear which theoretical and empirical bases they have, most of them are not validated (Fricke, 1995). Some instruments, like the DESP (Distance Education Student Progress) - questionnaire (Kember, 1989) need updating in respect to the technical development or were primarily designed to develop and test a highly specific model of student progress in distance education. Others are so trivial and general that they cannot help in solving complex problems in the design or evaluation of web-based distance education (e.g., Oughton, 1998). Many instruments and procedures in the field of distance education evaluations do not reflect new methods of distance education like "learning circles" (geographically divers classrooms with a shared curriculum focus joined together), "teleapprenticeships" (students serve a variety of roles in interaction with practicing adults), or "telementoring" (distant support for on-the-job learning) (Levin, 1995).
It has also to be mentioned that criteria contained in the stated instruments tend to be highly general. Criteria have to be applied to web-based distance education courses in a very general way, as if all courses used the same technology, were written in the same language, had the same complexity, the same structure, or served the same objectives. Also comprehensive testing of the intra- and inter-evaluator consistency of the frequently used rating procedures is, as a rule, not yet given in an acceptable quality and quantity. Many phenomena in distance education have to be measured more exactly, especially in respect to their practical significance. For example, what does it mean, that the loading of a web-page takes too long? How many seconds are too long and under what circumstances longer or shorter loading times are acceptable for learning?
Despite some empirical evidence it is not clear, which quality factors are the most important ones. It is also not evident, which web-based distance education environments can function as "best practices" and as good models for future developments. Often variables found as important in research are not included in given checklists. Kember, Lai, Murphy, Siaw, and Yuen (1992) found in their synthesis of evaluations of distance education courses several important variables for student success: tutorial attendance, face-to-face meetings with tutors and other students, colloquia and workshops as integral parts of a course, content-based selection of way of tutorial communication (telephone, fax, email, etc.), or text packages instead of video and television broadcasts. Moore and Kearsley (1996, p. 76) identified in their review further variables that determine the effectiveness of distance education courses: number of students at learning site, duration of courses, reasons for student taking class, educational background of student, instructional strategies used (lecture, discussion, problem-solving), kind of learning involved (concepts, skills, attitudes), type of pacing (student determined, teacher defined, completion dates), amount and type of interaction/feedback, course involvement of tutors, preparation and experience of instructors and administrators, and extent of learner support.
At the moment there are no or only few institutions dealing with quality control and quality assurance of web-based distance education in a professional way. Most quality control and assurance activities are semi-professional conducted, mostly by more or less trained staff members with some expert advice. The presented paper should help in looking at quality of web-based distance education in a more comprehensive way and should lead to a higher professionalism in quality control, which represent an important way for promoting new information technologies and their capabilities for lifelong learning.

References

Astleitner, H. (1997). Effects of external learning aids on learning with ill-structured hypertext. Journal of Educational Computing Research, 17, 1-18.
Astleitner, H. (1998). Die Einführung neuer Informationstechnologien ins Bildungswesen. Neun Schritte zu einer nationalen Informations-Infrastruktur. Erziehung und Unterricht, 148, 241-249.
Astleitner, H. & Keller, J. M. (1995). A model for motivationally adaptive computer-assisted instruction. Journal of Research on Computing in Education, 27, 270-280.
Astleitner, H. & Leutner, D. (1995). Learning strategies for unstructured hypermedia. A framework for theory, research, and practice. Journal of Educational Computing Research, 13, 387-400.
Astleitner, H. & Leutner, D. (1996). Applying standard network analysis to hypermedia systems. Implications for learning. Journal of Educational Computing Research, 14, 285-303.
Astleitner, H. & Leutner, D. (1998). Fernunterricht und neue Informationstechnologien. Aktuelle Entwicklungen. Zeitschrift für Pädagogik, 44, 105-123.
Astleitner, H., Sams, J., & Thonhauser, J. (1998). Womit lernen wir in der Zukunft? Schulbuch und CDROM - Ein kritischer Vergleich. Wien: Bundesverlag.
Berge, Z. L. (1996). The role of the online instructor/facilitator [WWW document]. URL http://star.ucc.nau.edu/~mauri/moderate/teach_online.html.
Bourdeau, J. & Bates, A. (1997). Instructional design for distance education. In S. Dijkstra, N. M. Seel, F. Schott, and R. D. Tennyson (Eds.), Instructional design. International perspectives (Vol. 2, pp. 369-397). Mahway, NJ: Erlbaum.
Brill-Pisik, G. (1997). Is this course instructionally sound? A guide to evaluating online training courses. Educational Technology, 37, 50-59.
Brown, A. (1998). Murdoch online. Quality assurance guidelines [WWW document]. URL http://cleo.murdoch.edu.au/teach/online/q-a.html.
Brusilovsky, P., Kobsa, A., & Vassileva, J. (Eds.).(1998). Adaptive hypertext and hypermedia. Dordrecht: Kluwer.
Carroll, J. M. (1990). The nurnberg funnel. Designing minimalist instruction for practical computer skills. Cambridge: MIT Press.
Ciolek, M. T. (1996). The six quests for the electronic grail. Current approaches to information quality in www resources [WWW document]. URL http://www.ciolek.com/papers/quest/questmain.html.
Clearinghouse, California Instructional Technolgy (1998). Guidelines for the evaluation of instructional technology resources. Eugene, OR: International Society for Technology in Education.
Collis, B. (1993). Evaluating instructional applications of telecommunications in distance education. Educational and Training Technology International, 30, 266-274.
Collis, B. (1996). Tele-learning in digital world. The future of distance learning. London: Thomson.
Dick, W. & Carey, L. (1985). The systematic design of instruction. Glenview, IL: Scott, Foresman and Company.
Dringus, L. P. (1995). An iterative usability evaluation procedure for interactive online courses. Journal of Interactive Instruction Development, 7, 10-14.
Ellis, J. A., Knirk, F. G., Taylor, B. E., & McDonald, B. A. (1993). The course evaluation system. Instructional Science, 21, 313-334.
Fricke, R. (1995). Evaluation von Multimedia. In L. J. Issing & P. Klimsa (Hrsg.), Information und Lernen mit Multimedia (pp. 401-413). Weinheim: Psychologie Verlags Union.
Gagné, R. M., Briggs, L. J., and Wager, W. W. (1988). Principles of instructional design (3rd ed.). New York: Holt, Rinehard, and Winston.
Gärber, W., Harbeck, G., & Lauterbach, R. (1992). EPASoft. Instrument zur Bewertung Pädagogischer Software. Kiel, Germany: Kiel University, Institut für die Pädagogik der Naturwissenschaften.
Gay, G. (1997). Using research to design effective distance education [WWW document]. URL http://www.oise.utoronto.ca/~ggay/distance.htm.
Hall, B. (1997). Web-based training cookbook. New York: Wiley & Sons.
Harper, C. (1997). Copyright law for distance learning [WWW document]. URL http://www.utsystem.edu/ogc/intellectualproperty/distance.htm.
Holmberg, B. (1995). Theory and practice of distance education. London: Routledge.
Johnson, P. (1992). Human computer interaction. Psychology, task analysis, and software engineering. London: McGraw-Hill.
Kember, D. (1989). A linear process model of drop-out from distance education. Journal of Higher Education, 60, 278-301.
Kember, D., Lai, T., Murphy, D., Siaw, I., & Yuen, K. S. (1992). A synthesis of evaluations of distance education courses. British Journal of Educational Technology, 23, 122-135.
Khan, B. H. (Ed.).(1997). Web-based instruction. Englewood Cliffs, NJ: Educational Technology Publications.
Khan, B.H. & Vega, R. (1997). Factors to consider when evaluating a web-based instruction course. A survey. In B. H. Khan (Ed.), Web-based instruction (pp. 375-378). Englewood Cliffs, NJ: Educational Technology Publications.
Levin, J. A. (1995). Organizing educational network interactions. Steps towards a theory of network-based learning environments [WWW document]. URL http://lrs.ed.uiuc.edu/guidelines/levin-aera-18ap95.html.
Limacher, J. J. (1996). Die 2Q-Methode als Grundlage für ein Qualitätsmanagementsystem in Bildungseinrichtungen. Bildung und Erziehung, 49, 155-181.
McKenzie, J. (1997). A measure of student success. Assessing information problem-solving skills [WWW document]. URL http://www.sol.com.sg/mirrors/fromnowon/oakharbor.html.
Miles, M. B. & Huberman, A. M. (1984). Qualitative data analysis. A sourcebook of new methods. Beverly Hills, CA: Sage.
Moore, M. G. & Kearsley, G. (1996). Distance education. A systems view. Belmont, CA: Wadsworth.
Oliver, R. & Reeves, T. C. (1996). Dimensions of effective interactive learning with telematics for distance education. Educational Technology, Research, and Development, 44, 45-56.
Oughton, J. (1998). Design criteria for distance education [WWW document]. URL http://www.oise.on.ca/~joughton/distance.html.
Porter, L. R. (1997). Creating the virtual classroom. Distance learning with the internet. New York: Wiley & Sons.
Posch, P. & Altrichter, H. (1997). Möglichkeiten und Grenzen der Qualitätsevaluation und Qualitätsentwicklung im Schulwesen. Innsbruck, Wien: Studienverlag.
Schnepf, J., Mashayekhi, V., Riedl, J., & Du, D. (1994). Closing the gap in distance learning. Computer-supported, participative, media-rich education. Educational Technology Review, 3/4, 19-25.
Schrock, K. (1996). Critical evaluation survey [WWW document]. URL http://www.capecod.net/schrockguide/evalelem.htm.
Schrum, L. (1995). Framing the debate. Ethical research in the information age. Qualitative Inquiry, 1, 311-326.
Specht, W. & Thonhauser, J. (Hrsg.).(1996). Schulqualität. Entwicklungen, Befunde, Perspektiven. Innsbruck: Studienverlag.
Thach, E. C. & Murphy, K. L. (1995). Competencies for distance education professionals. Educational Technology, Research, and Development, 43, 57-79.
Wagner, E. D. & McCombs, B. L. (1995). Learner centered psychological principles in practice. Designs for distance education. Educational Technology, 35, 32-35.
Wazel, G. (1998). Evaluationskriterien für sprachliche Multimedia-Software [WWW document]. URL http://www.iik.com/eval.htm.
Wilkinson, G. L., Bennett, L. T., & Oliver, K. M. (1997). Evaluation criteria and indicators of quality for internet resources. Educational Technology, 37, 52-59.
Willis, B. (1993). Distance education. A practical guide. Englewood Cliffs, NJ: Educational Technology Publications.
Willis, B. & Dickinson, J. (1997). Distance education and the world wide web. In B. H. Khan (Ed.), Web-based instruction (pp. 81-84). Englewood Cliffs, NJ: Educational Technology Publications.

Author
Dr. HERMANN ASTLEITNER is Associate Professor, Institute of Educational Research, University of Salzburg, Akademiestrasse 26, A-5020 Salzburg, Austria. He specializes in educational technology, education in the information age, and research synthesis. Homepage: http://www.sbg.ac.at/erz/people/astl.htm. Email: Hermann.Astleitner@sbg.ac.at.