JALN Volume 1, Issue 2 - August 1997
Jef Moonen
Faculty of Educational Science and Technology, University of Twente
Enschede, The Netherlands
ABSTRACT
Telelearning relates to "making connections among
people and resources, via communication technologies, for learning-related purposes".
Telelearning as a general term includes asynchronous learning activities; in all cases
efficiency is an important construct. Efficiency is defined as the relation between costs
and effects/quality. An educational system is said to be "efficient" when an
optimum balance is found between minimizing the costs and maximizing the effects/quality.
This balance is often difficult to obtain, for an example when the opportunity for direct
contact between tutor and tutee through telecommunications facilities can lead not only to
beneficial learning and social effects but also to a large amount of messages and a huge
time investment by tutors/tutees, leading to an uncontrollable rise in costs. Measuring
the effect/quality of telelearning has its own methodological problems and therefore it is
difficult to come up with viable and reliable data. How to deal with the efficiency
problem is illustrated in the article based upon literature research and data from a
recent project executed in the context of the telematics research program of the European
Union.
KEYWORDS
efficiency
telelearning
costs
cost-effectiveness
distance education
quality
Improving efficiency in education is important. Efficiency and effectiveness are often mentioned as reasons and criteria for solutions of educational and training problems. There is a general "feeling" that introducing communication and information technologies (CIT) in education and training improves efficiency. Very often however, "feelings" are not based on facts. Basing attitudes on "feelings" rather than facts is common in society. Recently a prominent member of the Dutch Auditor General's Office said that many policies are not based on facts but on "general opinions" [1]. Do we have to accept such an approach? With respect to this paper we will discuss this issue further. Therefore we need a more precise description about what is meant by efficiency and how efficiency can be measured.
Efficiency indicates a relationship between input and output. There are a number of techniques to analyze such a relationship.
Consider an educational/training system as a production activity. Within a production activity the following four main parts can be distinguished: input, process, output and outcome.
Input refers to the resources used in the production activity, to be divided into general categories such as: teacher/trainers, school and training facilities, educational/training materials, and hard- and software infrastructure/equipment.
The process stage refers to the means by which educational inputs are transformed into educational outputs. The instructional/training organization, the use of specific educational technologies (lecture, use of computers, use of telecommunication facilities, etc.), and teacher/trainer time as well as the time of the students/trainees are relevant in this part of the production activity.
The interaction of the inputs and the process determines the costs of the system. Those costs can be split in many ways. One way is to distinguish costs per activity phase: (a) development phase, (b) delivery, operation & maintenance phase. One can also distinguish costs for (a) the course providers (the suppliers) and (b) the costs for the users. Fixed costs (independent from the number of students) versus variable costs (dependent on the number of students using a system) is another way to define cost categories. Measuring costs can be carried out following the "ingredients" method in which all the ingredients that are necessary for the realization and use of a product are specified according to their market values [2]. Therefore a distinction can be made among cost ingredients in categories such as: (a) personnel costs, (b) equipment costs, (c) facilities costs, (d) material costs, and (e) others.
The effects of the production activity involve both outputs and outcomes. Outputs are the direct and immediate effects of the educational process, such as cognitive achievement, manual skill development, attitudinal changes, and behavioral changes.
Outcomes are the less direct and less immediate results of the interaction of educational outputs with the social environment, such as employment, earnings, status, job- and life-satisfaction.
To determine the efficiency of an educational/training system the costs, which are based upon the inputs and the process, have to be related to the effects, which are based upon the outputs and the outcomes. An educational/training system is said to be "efficient" when an optimum is found between minimizing the costs, and maximizing the effects. Efficiency can be used in an absolute or in a comparative way. A system is said to be "efficient" when the comparison between costs and effects exceeds a certain preset criterion. Very often however, such a criterion is not made explicit. A statement that "a system or an approach is efficient" is therefore based upon an implicit comparison with another system or approach.
Problems arise when operationalizing the efficiency concept because of difficulties with: (a) the identification of those categories within the four main aspects of the production activity that are felt to be relevant, and (b) the specification of how those categories will be measured.
Although the measurement of the costs seems to be very straightforward, in practice many problems are encountered because: (a) there is disagreement about what kind of costs should be taken into account, (b) no reliable data about costs are available because they have not been gathered in a systematic way, (c) the costs are not stable and are evolving very fast, which in particular is the case when new communication and information technologies are being introduced, or (d) data are confidential and may not be used in a public document.
The measurement of educational effects creates even more problems, in particular when outputs are related to quality issues [3].There are questions about how to incorporate cognitive as well as affective and psycho-motoric aspects in a measurement instrument. Moreover there is the unsolved problem about how to measure higher levels of cognitive and meta-cognitive achievement and skills. Effects used when determining the efficiency of a system are often limited to very "down-to-earth" aspects such as the number of graduates in a course or the average grades in a course. Measurement of long-term effects or outcomes are almost never taken into account. As outcomes can only be observed in the future, longitudinal studies within institutions should be set up, but apparently they are not.
Thus, the operationalisation of efficiency creates many substantial and conceptual problems, in particular with respect to the measurement of the constituting items.
Relative to the way costs and effects are measured, many kinds of efficiency analyses can be distinguished. Two approaches are used regularly: (a) cost-benefit analysis, and (b) cost-effectiveness analysis.
Cost-benefit analysis can be applied when both costs and effects can be measured in monetary terms. However, as it is almost impossible to assess the effects of an educational process in a reliable monetary amount, the cost-benefit analysis is practically not applicable in an educational context. In a training situation it is more likely that the benefit of a training program can be valued in monetary terms and therefore cost-benefit analysis could be appropriate in such situations.
Cost-effectiveness analysis is applied when the costs are expressed in monetary terms and the effects are measured in non-monetary terms (but still quantifiable, for instance in the number or percentage of graduates in a course). In order to be able to compare two alternative educational systems, either the cost or the effects of both the alternatives have to be fixed. When the costs are the same for both alternatives, the system with the largest effects is the most efficient. When the effects are the same for both alternatives, the system with the smallest costs is the most efficient.
Often the exact measurement of the effects of an educational system is very difficult. A practical solution then is to assume that the effects of suggested different educational approaches are the same, although not quantifiable. The analysis can then concentrate on the cost issue.
What is telelearning? According to Collis, telelearning is "making connections among people and resources, via communication technologies, for learning-related purposes" [4]. There are many ways to organize telelearning. Collis distinguishes eight scenarios of which traditional distance education is only one. Bates focuses on media instead of usage scenarios and divides media to be used in distance education into one-way media (print, radio, audio cassette, educational broadcast TV, pre-recorded TV, video cassette, computer-based learning, multimedia) and two-way media (audio conferencing, live interactive TV, video conferencing, computer-mediated communication) [5]. The Ministry of Education of Denmark distinguishes three implementation scenarios which are called: (a) the time-independent learning model, (b) the simultaneously-distributed learning model, and (c) the independent study model. In the time-independent learning model elements from classroom teaching are combined with studies at home or at work, from which a continual dialogue with the teacher and with the other students can be organized [6]. Typically this model employs a computer conference system. In the simultaneously-distributed learning model students can see and hear their teacher and they can carry on a dialogue with a teacher. Typically this model uses satellite, cable television or video telephony. In the time-independent study model the student works alone with the subject matter in an organized sequence of modules. Use of CD-ROM techniques can provide high-quality material. Often this model has to be supplemented with the option of counselor support at a study center. The "time-independent learning model" is a good example of a kind of telelearning that is becoming very popular due to the growth of the Internet and the facilities provided by the WWW.
In order to investigate the efficiency of telelearning the effects and the costs have to be examined in each of the various scenarios of telelearning that are taken as examples.
Much research has been done, in particular using meta-analysis techniques, to relate the use of media-based learning to achievement, the most commonly used output variable. Depending upon the kind of the media-based intervention, the effects vary from positive to undetermined [7],[8],[9],[10],[11]. Recently Russell mentions 218 reports about the "no-difference" effect of using media on learning, and questions why telelearning should make a difference [12]. Collis overviewed more than 400 articles on the use of telelearning in education and concluded that "there is strong expectation of substantial educational impact from telecommunications use" [13], p.5. However no specific data about such impact are yet available. As a general conclusion it seems reasonable to assume that student achievement will, at least, remain equal when using media to support education and training, and there is no reason why such a conclusion should not be formulated with respect to telelearning.
The research on output effects reflects the context and goals of traditional educational systems. Many claim however that specific approaches integrating multimedia and telelearning lead and are particularly focused towards new kinds of outputs and outcomes, in particular in the affective domain (feeling "good" and "self-confident" about new technologies), the psycho-motoric domain (being able to handle new technologies with confidence), and in particular with respect to higher cognitive and meta-cognitive skills (having insight in global and complex problems, being able to locate and use available information, being able to solve problems). Often traditional measurement methodologies are less appropriate to capture such outputs. Therefore, other measurement methods will have to be developed and applied.
There is certainly an expectation that telelearning will increase the effects of the teaching-learning process. However, capturing such outputs and outcomes is not yet straightforward.
The other side of the efficiency coin is the cost of telelearning. Recently Curran [14], European Association of Distance Teaching Universities [15], Bates [5], Taylor and White [16], Jones & Simonson [17] and the Danish Ministry of Education [6] have published studies concerning the costs of distance education. In these reports many alternative distance education modes have been described. There is a high fluctuation in costs due to continual changes in the costs of the hardware and in telecommunication costs. Moreover there are many conceptual problems to be taken into account. For instance: how to compare costs over longer periods of time, taking into account different inflation rates and changes in currency exchange rates? A common comparison unit is the "costs/student/study hour." However how is a "study hour" defined? Is it the time a student is in direct contact with the material provided by the course (for instance, all the time the student uses to learn with respect to the content of the course), or is it the time a student is in direct contact with a specific activity provided by the course (for instance, watching a television broadcast, a direct contact with a tutor, or the time spent on typing, reading and reacting to e-mail)? There are huge differences in total study time, depending on how "study hour" is defined. Bates mentions 1.68 to 322.50 US$ /student/study hour [5]. Moonen give cost figures from 7.8 to 392 US$ /student/study hour when a study hour is defined as direct-contact related activity, or 4.32 to 84 US$ /student/study hour when a study hour is related to the total time a student spent on the course [18].
Therefore it is more interesting to look at costs from a more generic point of view. In that respect Bates suggests that his "book attempts to provide a methodology by which decision-makers can reach their own conclusions" [5], p.1. Wetterling reaches a comparable conclusion about his own approach [19].
One way to look at the costs is by calculating the cost ratios using different telelearning scenarios. Based upon figures provided above, Bates has found ratios for different distance teaching modes varying from 1 to 150 (with 50 students per annum) or 1 to 17 (N = 1250), while Moonen has found cost ratios of 1 to 50 or 1 to 19 depending upon the definition of "study hour." Whatever the definition, there is a wide variety in costs.
Another way to look at costs is to look at different cost items. Costs of telelearning depend on personnel costs and costs of materials, equipment and operations. Telelearning requires the input of many people. Keskinen mentions the following actors in that respect: (a) learning material providers, (b) other learners, (c) tutors, (d) mentors/coaches, and (e) technical support staff [20]. Bates reports that for telelearning delivered as computer-mediated communication (CMC) 50 to 60% of the costs have to be allocated to personnel costs (instructors' fees, training, technical support) [5]. Also the Danish report mentions that in the time-independent learning model and because of its "relatively labor-intensive study mode, such courses will have a built-in cost buoyancy" [6], p.11. Given the fact that the costs of hardware and of telecommunications, are continually decreasing, the personnel costs as a category in the cost picture of telelearning will gradually become even more dominant.
In all modes of telelearning a critical dimension is the amount of contact that is allowed between the individual student and other persons (the teacher, the instruction team, the tutor, other students) in the course. The same is true in all other forms of instruction. For instance, in a face-to-face situation an instructor can allow students to ask many questions and stimulate considerable interaction within the class, but another teacher may deliver the instruction without any interaction with the students. In a traditional teaching situation the total amount of contact time is limited to the lesson-time (unless the teacher makes himself available outside of class hours). In the case of telelearning however, the opportunity for direct contact through telecommunications facilities can lead to an enormous stream of messages, which will demand a huge time investment of teachers or tutors and therefore to an uncontrollable rise in costs. Active members in electronic networks such as local area networks and Internet, receive an average of 30-100 messages per day nowadays. In a telelearning class with 60-120 students when each student sends an instructor only one message per week, and the response takes only 5 minutes/message, the extra time investment per telelearning course is on the order of 5 to 10 hours/week. This kind of extra time investment very soon becomes unmanageable, and therefore special measures have to be taken with respect to limiting the number of electronic communications--for instance by organizing small groups of students who interact with each other, moderated by one of the group who gets in electronic communication with the tutor--or by hiring extra personnel.
A discussion about rising costs in relation to the expected quality (the effects) of a telelearning course is therefore unavoidable. This brings us back to the question about the efficiency of telelearning. The optimization of the efficiency can be brought forward by lowering the costs or raising the quality. In each separate case and situation specific circumstances will determine in which way to proceed. When lowering the costs is the major target, controlling the electronic communication between students and teachers/tutors is a major issue. When improving the quality of the course is the major target, finding measurement methods in order to capture a rise in quality must be a main goal. Of course, in practice both aspects will be on the list of points to consider.
The following sections describe some examples of the use of telelearning in which costs and effects considerations are addressed.
In addition to existing institutions for distance education such as the Open Universities, many traditional universities are becoming interested in distance learning activities and in particular telelearning. In the USA for instance, almost all of the traditional universities and colleges now have WWW-pages and are offering kinds of distance education. The reasons are very straightforward: many universities need more students in order to balance their budgets. Universities are moving into a situation which is called a "mixed-mode" or "dual-mode" teaching situation. The implicit assumption is that adding telelearning to the traditional instruction will make the university activities more efficient and productive. Is this the case, or what should be done in order to make it that way?
It was argued earlier that when talking about efficiency in education, the focus is often on the cost issue, and in particular with the intention of bringing down the costs. In Section VIII we indicated that a significant proportion of the costs are determined by personnel costs. In addition, one should realise that adding telelearning to a traditional mode of teaching needs different people with different skills. In other words, for a traditional university to become a mixed-mode university, it is necessary to hire new people, or retrain available personnel. Taylor & White have thoroughly investigated such a situation and state that "mixed-mode teaching is not readily available on a cost-effective basis for non-distance education institutions" [16], p. 37. On that basis it can be concluded that adding telelearning to traditional higher education may attract more students, but, because of the need for hiring new personnel, it is unlikely that these extra students will solve the budgetary problems of a university.
The added value of telelearning therefore should be found in the other variable of the efficiency equation: effectiveness and quality. For traditional universities telelearning should be introduced in circumstances where it provides an added value to the existing situation. The implicit assumption thereby is that the quality, and thus the effectiveness, of the courses will increase.
In addition to the suggestions made above, there is another kind of activity that can contribute to the efficiency of higher education. A variable often neglected in media research is the time involvement of the student. Research has indicated that computer-based learning motivates students to invest more time in a subject-area (time-on-task), in particular when the student can work according to his own pace and time schedule, and when the system creates extra possibilities for the student to communicate with other students (through e-mail, bulletin boards and computer conferencing) [21]. The enthusiasm of students working with the WWW is a clear illustration of these research findings. Research has also shown that using computer-based learning can reduce the necessary learning time of students to 2/3 of the time needed in a conventional course [8],[22]. The major reason for this result is the more effective structure of the learning materials and the better organization of the instructional process used in a computer-based context [23], p.21. When telelearning is well-structured and well-organized, a positive result can be expected from telelearning as well. It therefore can be assumed that a better-structured and better-organized situation using telelearning will contribute to a better time investment of the students (they will do the same in less time, or do more in the same amount of time).
In a recent report Jensen concluded that using educational technology will only become efficient if a system can transfer more of the work to the learner [24]. Using telelearning can be the way to materialize that approach.
In summary, telelearning in higher education should be used to provide an added value and quality to the existing modes of teaching, thereby optimizing the use of available student time. In that way telelearning can contribute to the efficiency of the teaching-learning process.
Costs of public education are provided by the user (student) and by the institution (government). As public education is considered of national interest, the registration fees of students are not self-supporting. This is not the case for private education nor in corporate training situations. In those circumstances being self-supporting is essential. Costs of training have to be covered by users' fees or by measurable (financial) benefits to the corporation. Issues of costs and effectiveness and thus efficiency will therefore be of much crucial interest when considering telelearning.
When in a traditional situation an employee is sent to a training course, there are three aspects that constitute the major parts of the costs:
a. the registration fee for the training course,
b. travel and accommodation costs,
c. costs related to the fact that the employee is away from work and therefore is not productive.
The loss of productive time for the employer could be dealt with when an employee is willing to attend the training in his/her own "free" time. Telelearning is an excellent approach to make this possible and is becoming widespread because of this benefit. Curran even concludes that "cost savings arising from distance and flexible learning is due not to a reduction in the cost of training time but rather in a transfer of the cost from the company to the employee" [14], p. 34.
Telelearning however can also be used to bring down the costs relating to points (a) and (b). In principle, the registration costs can be brought down when a course can be delivered to a large audience. In this respect economies of scale through telelearning should do the work of cost reduction. Travel and accommodation costs can be lowered when the training course can be delivered at the working place or the house of the employee. An obvious solution is of course telelearning.
To illustrate the concepts discussed above, the Telescopia Project is taken as an example [25]. The Telescopia (TransEuropean Learning System for Crossborder Open and Interactive Applications) project was a one year project funded by the Commission of the European Community. Investigating cost-effective solutions was one of the main objectives of the Telescopia project. The target of the project was to realize a telematics instrastructure comprising interactive TV, interactive audio-video connections using ISDN, and a variety of facilities, such as computer conferencing, not requiring an ISDN connection.
Six course providers participated in the project offering a diversity of courses to students and trainees. Each course provider offered a variety of telematics facilities. In very global terms each course provider offered on of the following (for further details see the final report of the Telescopia project [25]):
a. WWW-facilities and videoconferencing;
b. lectures distributed through an internal TV-distribution system, with e-mail, telephone and a microphone connection between lecture and students;
c. interactive TV and videoconferencing via ISDN;
d. TV-broadcasting via satellite, with regular feedback via regular mail, fax, telephone, e-mail;
e. computer conferencing, interactive TV, telephone, e-mail, and fax;
f. videoconferencing via ISDN connections and application sharing with the Pro-Share system.
During the project, and while gathering the necessary data, it became very clear that, due to the experimental approach of the project, and given the limited number of participants in the courses offered by the course providers, trying to find a valid and reliable measure of effectiveness was not feasible. Therefore, the focus of the investigation was rerouted to the cost issue. It was expected that gathering cost data would not be too difficult. However, during the data collection the conclusion powerfully emerged that in many cases reliable cost data were not available either. Nevertheless, enough data were collected to get an interesting insight and maybe even an very interesting conclusion [26].
The most interesting figure is the total cost/contact hour. "Contact time" is defined as the time a trainee spends on contact-related activities provided by the course provider (and therefore constitutes a cost for the course provider). The costs mentioned are the costs for the course providers. Although the course providers each have a substantially different delivery approach and telematics platform, the total cost/contact hour (taking into account the projected total lifetime of the courses which is different for each course) is within a very small range, around 7,000 ECU (= 8,400 US$; currency exchange rate: 1 ECU = 1.2 US$).
When an average and acceptable estimate for a course fee/contact hour and the projected number of course repetitions without alterations (the lifetime of course) are taken, a calculation can be made about the minimum numbers of participants needed in one course-session to reach a cost break-even point. It can be argued that in a commercial context 72 US$/contact hour is a reasonable fee (in Western Europe). In that case and taking 8,400 US$ as an average total cost/contact hour, the number of participants per course-session needed to reach the breakeven point of a course delivery is approximately 120. Although 120 participants seems to be a rather small number when considering a European market, one has to realize that language barriers can act as a limiting factor, or when a language barrier is not an issue because the potential trainees are all fluent in for instance English, then it could well be that the potential target group is probably highly specialized and therefore limited in numbers. If courses aim at a broader public and are provided for a much lower course fee (for instance 6 US$/contact hour), the minimal number of students to breakeven is around 1400 per course-session.
From these kinds of cost calculations it becomes clear that when course fees are kept to a minimum a rather high number or trainees or learners is needed in order to reach a break-even point. Striving for economies of scale is a clear answer in this respect. However, one has to realize that because of the popularity of communication between students and tutors in telelearning situations, extra costs to hire extra personnel (tutors) will be added to the cost equation when larger numbers of students participate in courses. These circumstances could create serious limits to the idea of economies of scale, especially when the opportunity costs of the students are taken into account.
In Section I it was stated that very often policies are based upon "general opinions." We hoped that following a more exact analysis, "general opinions" could be turned into facts. It has become clear however that in many cases situations are too complex to deliver straightforward answers. Nevertheless some conclusions can be formulated: