Exploration on the Graduate Course Objective Examination

¹Ge Wang, ²Yuanrui Wei

¹Department of Electrical Engineering, ²Department of Computer Science, Graduate School of Academia Sinica, Beijing, China (Received by the Editorial Office on January 13, 1987; translated from Chinese into English by the first author in April 1999)

Abstract: A method for the objective examination design is developed for teaching of the graduate course "Digital Picture Processing". In this method, questions are made to cover the course material in a balanced manner, and the number of the questions are determined according to an entropy-based criterion - the uncertainty reducibility. It is concluded that the objective examination method is accurate, reliable, cost-effective, and widely applicable for teaching at the graduate level.

Key words: Examination design, objective examination, graduate teaching, entropy

1. INTRODUCTION

Examination is an important aspect of graduate teaching. The examination method has a substantial impact on the motivation, the learning pattern, and even the research capability of graduate students. Recently, the objective examination approach was introduced to China, which is playing a positive role in updating our traditional examination paradigm.

In reference to the syllabus of advanced graduate courses by universities in Taiwan, the first author taught a graduate course "Digital Picture Processing" using the classic works by Rosenfeld [1] in the second semester of the 1985-1986 academic year. An objective examination method was developed and applied to evaluate the teaching/learning outcome of this important course, which summarizes a large collection of the digital image processing literature up to 1980s. This examination reform was well received and highly remarked in a following-up survey. In this paper, the design principles is described for the first part (true-false type) of the examination (see Appendix). The design ideas for the other two parts (multiple-choice and blank filling types respectively) of the examination are similar, hence omitted for brevity.

2. METHOD

The first part of the test contains true-false questions. An effort was made to make the questions representative, that is, to cover all major aspects of the textbook. In the same time, the difficulty levels of the questions were made comparable for convenience of quantification. The analysis can be similarly performed even the difficulty levels differ significantly. The grading policy is as follows: one score for a correct answer, minus one score for a wrong answer, and zero score for no answer. The students were advised not to randomly guess, in which case there would be no any advantage for them. The rationale behind this grading policy is to measure a student's understanding, mis-conception, and no knowledge of the material. For this course, it is very informative to make such distinction.

How many questions are needed to adequately survey a student's capability in answering the true-false questions? According to information theory [2,3], we will define the concept of the uncertainty reducibility below, and quantify the relationship between the number of questions and the amount of information extracted from answers to the questions.

Let the probabilities of understanding, misconception, and no knowledge be denoted as P₊, P_-, and P₀ respectively, which characterizes the capability/proficiency of a student. Theoretically, the domains of P₊, P_-, and P₀ are [0, 1]. However, P₊, P_-, and P₀ are discretized for numerical reasons. Given the common practice that four bands, (A, B, C, D), are used to grade our graduate students, we approximate the domains of P₊, P_-, and P₀ as PD={0, 0.2, 0.4, 0.6, 0.8, 1}. Suppose we have no information about a student's capability before examination, it is reasonable to assume that the entropy of the probability distribution of a student's capability reaches the maximum. In other words, we have

where , and .

Let N denote the number of the questions, the conditional probability can be expressed as

,

where N₊ and N_- are the numbers of correct and incorrect answers respectively, , , . The posteriori probability

,

where , , and .

The uncertainty reducibility is defined to measure the degree to which the uncertainty on a student's capability is eliminated. Mathematically, we have the student's capability entropy E₁,

,

the conditional entropy E₂,

,

and the uncertainty reducibility a,

.

3. RESULTS

The numerical evaluation of the uncertainty reducibility was done relative to the number of the questions on an IBM-PC/XT using the BASIC programming language. The relationship between the number of the questions (N) and the uncertainty reducibility (a) was tabulated in Table 1. According to the data in Table 1, we constructed 20 questions for the first part of the examination. From these question, about 80% information on a student's capability can be derived.

N	1	2	5	10	15	18
a	0.16	0.26	0.47	0.65	0.75	0.79

Table 1. Relationship between the number of the questions (N) and the uncertainty reducibility (a).

4. DISCUSSION AND CONCLUSION

Because our modeling and analysis are approximate, the method should not be used without cautions. First of all, the questions should be as representative as possible. If the questions are not well balanced, blind areas of the evaluation would exist. We point out that the first part of the examination is lack of representation of image reconstruction, which is an important aspect of the course. To compensate for this flaw, a question on image reconstruction was given in the second part of the examination (see Appendix). In fact, there are totally 61 questions in the three-part examination, which we consider does not miss any significant aspects of the course.

It is recognized that the objective examination approach has its limitations, just as the subjective examination approach does. Roughly speaking, an objective method suits better for evaluation of understanding of ideas and concepts, while a subjective method may be more effective for assessment of logic and analytic skills. To objectively judge logic and analytic skills of the students, we designed 20 blank-filling questions in the third part of the examination (see the Appendix). The answers must be derived in a number of logic and analytic steps, which are technically not very complex. To thoroughly appraise these skills, we believe a subjective examination method should be used. Because the graduate students have been well trained in these skills before admission to our Graduate School, it is the objective examination method that reduces the test redundancy and focuses on the knowledge acquisition in terms of broadness and depth. The objective examination is knowledge-oriented, instead of creativity-oriented, which seems its primary weakness. Psychological findings have indicated that there is no strong correlation between knowledge and creativity. How to develop the objective examination approach so that the creativity can be measured and encouraged is an open topic.

The objective examination designed using our method is accurate and reliable, which is consistent to the practice of other objective tests such as TOFLE and GRE in USA. On the other hand, the objective examination is cost-effective. The amount of computation and marking is relatively limited, hence a wide coverage is possible given a test duration, which is desirable for intensive teaching of graduate students. The grading for an objective examination is formalized, which avoids subjective variability and facilitates computer-aided education. Furthermore, the objective examination method is quite versatile. It is generally feasible to decompose any area into a number of small facets. Questions for an objective examination can be naturally constructed from these facets and their combinations. Advanced graduate courses are usually not mature, featured by different schools, multiple methods, and rapid evolution, which makes it quite easy to generate questions for an objective examination. In conclusion, the objective examination method is accurate, reliable, cost-effective, widely applicable for teaching at the graduate teaching.

REFERENCES

1. A. Rosenfeld, A. C. Kak, Digital Picture Processing, vols. I & II. Academic Press, New York, 1982
   
2. L. D. Wu et al., Probability Theory, vol. I. People's Education Press, Beijing, 1979 (in Chinese)
   
3. Z. Y. Fu, Information Theory. Electronic Industry Press, Beijing, 1986 (in Chinese)

APPENDIX. Final Examination Book for the Graduate Course "Digital Picture Processing" (June 24, 1986)

Designed by Ge Wang, Department of Electrical and Computer Engineering Graduate School of Academia Sinica, Beijing, P. R. China (to be translated)