In this issue:
Peter Bryant
College of Business
University of Colorado at Denver
Denver, CO 80217-3364
pbryant@castle.cudenver.edu
303-556-5833
Plans for CSNA '97 in Washington are well underway. I've sent in my registration form, including my registration for the new one- day short course on nonparametric regression. The regular one-day course introducing classification and clustering, is being offered,too -- it's a great introduction for those new to the area. I urge you to sign up, and look forward to seeing you there.
In another couple of months, the nominating committee will begin to put together a slate of nominees for Directors and for President-elect for our election this fall. The Society depends on its volunteer officers and directors, not just for day-to-day operations, but also for its continued awareness of and relevance to the wide variety of application areas in which classification and related techniques are used. Are you interested in playing a more active role? Do you have ideas about new directions in which we should be looking? If so, let some officer or board member know, so we can make the nominating committee aware of your interest.
Stanley L. Sclove
Department of Information and Decision Science
College of Business Administration
University of Illinois at Chicago
601 S. Morgan Street
Chicago, IL 60607-7124
slsclove@uic.edu
www.uic.edu/~slsclove
I'm happy to report that we've received a number of applications from prospective members. These will be officially acted upon at the annual meeting of our Board of Directors, on 12-June at CSNA/97 in D.C. I'm also happy to see all those renewals from "old" members!
Membership fees are still $65 for regular members, $50 for retired members, $35 for student members, and $50 for affiliates (who receive the Journal of Classification but do not vote). Encourage your colleagues to join!
Previous newsletters are posted and accessible at the CSNA homepage at the URL http://www.pitt.edu/~csna/ . Check the page from time to time for updates and new issues. In addition, the CSNA bibliographic reference Service is also posted there. The Service, as well as the newsletters, are now therefore free and accessible anytime.
See you at CSNA/97!
F.R. McMorris
Department of Mathematics
University of Louisville
Louisville, KY 40292
frmcmo01@homer.louisville.edu
(502)852-6826
This issue we again have two Forum articles. The first is our regular feature from David Banks and the second is a response from Phil Sutcliffe to an earlier Banks Forum piece.
SOME GEOMETRY FOR LOCATION ESTIMATORS
David Banks, Dept. of Statistics
Carnegie Mellon University, Pittsburgh, PA 15123
banks@stat.cmu.edu
This note has little to do with clustering or classification, but it describes a perspective for the comparison of location estimators that I find useful (albeit not mathematically deep). The approach leads to some interesting geometry, and can be motivated in two ways.
The first motivation results from slight a modification of Pitman's closeness criterion. Pitman (1937) considered the following situation:
Suppose one has a sample x= (x_1, ..., x_n) with joint distribution F(x), and two possible estimators, say a_1(x) and a_2(x), of the parameter of interest theta. Then Pitman argues that one should prefer a_1 to a_2 if
P[ | a_1 - theta | < | a_2 - theta | ] > .5
or, in words, we prefer a_1 if it is more likely to be close to theta in absolute value than a_2.
The small modification I propose is, for some context-specific value epsilon, to prefer a_1 to a_2 if one has
P[ | a_1 - theta | < epsilon ] > P [ |a_2 - theta | < epsilon].
In words, we prefer a_1 if it has greater probability of being within epsilon of theta than a_2.
The second motivation is directly tied to classical 0-1 loss. Suppose one's loss function is
L(theta, a(x)) = 0 if | theta - a(x) | < epsilon, and
L(theta, a(x)) = 1 otherwise.
Then standard decision theory leads to comparisons of the kind that are discussed in this note. The loss function is reasonable in many practical situations; if one is estimating the breaking strength of a roller bearing in a helicopter shaft, it makes little sense to weigh which of two disastrously wrong answers is less bad, as happens under squared error loss, but it makes good sense to ask which of the two estimators has the greater probability of being close to the correct answer (and then to calculate that probability).
Clearly, comparisons of this kind can depend upon F, theta, and the epsilon which seems reasonable for the particular problem. To fix ideas, consider the case of estimation of the population mean using a sample with joint distribution F. Without loss of generality, assume the mean is zero. One natural estimator, say a_1, to consider is the sample mean. Then a_1 will be exactly zero if the data lie upon the hyperplane in R^n which is normal to the vector (1, ..., 1) and which passes through 0. And the estimator will be within epsilon of zero if the perpendicular distance of the data vector to the hyperplane is less than epsilon. Thus the locus in which the data will give a good estimate is a thick hyperplane, and one must integrate the joint density f(x) over this region to find the desired probability. This integral may be hard if F is awkward, but the geometry of this thick hyperplane is so simple that the computational burden in this kind of calculation is typically much less than is encountered with the Pitman criterion.
As an alternative to the sample mean, many statisticians use the median of the data to estimate location. In the situation described above, the geometry of the region over which one must integrate changes according to the sample size. For example, if the sample size is 2, then the median and the mean agree and the regions are identical. But if the sample size is 3, then the region is entirely different.
Specifically, the median is exactly 0 whenever either
x_1 = 0, x_2 > 0, x_3 < 0, or
x_1 = 0, x_2 < 0, x_3 > 0, or
x_1 < 0, x_2 = 0, x_3 > 0, or
x_1 > 0, x_2 = 0, x_3 < 0, or
x_1 < 0, x_2 > 0, x_3 = 0, or
x_1 > 0, x_2 < 0, x_3 = 0
which defines six "quarter-planes" in R^3. The region of integration is obtained by thickening each of these quarter-planes to include all points which lie within distance epsilon (clearly, points near 0 may lie in more than one thickened quarter-plane). As before, the shape of this region adds nothing to the computational burden, provided one thinks just hard enough to avoid the overlap near 0. Also, the cases of larger values of n are easy to work out---the geometry of the regions looks very different according to whether n is even or odd, but pattern is quickly apparent.
Why is this perspective interesting? Well, several facts about the mean and median can be read off immediately from the geometry. For example:
1. If the data are negatively associated, then the mean is great, but if the data are positively associated, then the mean is horrible. This follows from the fact that if the association is negative, then contours of the joint density concentrate on the region of integration, but if the association is positive, the contours are orthogonal to that region.
2. The median is more "robust" than the mean, in the sense that the volume of the region of integration for the median is "larger" than that of the mean. To see this, note that in the case with n=3, the mean's region is one hyperplane (=four quarter-planes) whereas the median's region is six quarter-planes. Of course, both estimators have regions with infinite volume, so it would be more accurate to say that within any fixed sphere, the proportion of the volume corresponding to the region of integration for the median is larger than that for the mean. This suggests that the median should give the right answer more often than the mean, under some technical assumptions that express the right kind of agnosticism about the kinds of F one is likely to see.
3. Samples for which the mean is correct can be very different from samples in which the median is correct. This insight, so carefully drilled in classes with examples using outliers, is immediately apparent from the fact that the regions of integration for the two estimators are entirely different.
There are other things one can learn from this geometric representation of the estimators, but I think the value of the perspective is shown.
Obviously, one can pursue this in different ways. The winsorized mean gives a different kind of geometry, as does the shorth. (Recall that the winsorized mean throws out the largest and smallest values in the sample, replacing them by largest and least values among those that remain, and that the shorth, which stands for "shortest half", is the average of the 50% of the sample which has the smallest range.) Working out the geometry is fun, and leads to insights that are similar to those described above.
Another way to extend this is to examine the kinds of F that work well with specific estimators. Such treatment is concerned with independence versus various kinds of dependence in the sample, and multimodality, and heavy tails. This route can find that the choice of the best estimator is sensitive to the value of epsilon, and this leads to interesting and useful advice for practitioners.
Reference
Pitman, E.J.G. (1937). The closest estimates of statistical parameters. _Proceedings of the Cambridge Philosophical Society_, 33, 212-222.
ON GRADE CATEGORIES AND MISGRADING AND THE NEED TO MAKE EXPLICIT THE INTENSIONS OF THE RELEVANT CONCEPTS.
J. P. Sutcliffe
Department of Psychology
University of Sydney, Sydney, NSW 2006, Australia
jps@psychvax.psych.su.oz.au
Dr. Banks (CSNA Newsletter, Issue #47,November 1996) has drawn attention to the issue of misclassification in the context of examining, a matter of considerable importance bearing on students' course progression and vocational prospects. Although his title was "Who will measure the measurers? "his discussion focussed more on other (albeit related) important matters, viz. WHEN and HOW to evaluate WHAT classificatory procedure? His concerns were: WHAT implies the assigning of educational grades; WHEN implies in a climate of accountability; and HOW implies via the categorization of a test score range. Dr. Banks first set out the procedure which he uses for grading his students, and then he gave a particular evaluation of it, moving to a general conclusion (of "how bad we are"), and an expression of concern (at "how undesirable it would be for students to discover this").
My note draws attention to some additional causes for concern with the kind of approach to educational examining espoused by Dr. Banks, and argues that, for the resolution of all such problems, one must first make explicit the intensions of the basic concepts entailed. Only when it has been made quite clear what is the nature of the class (grade category) in question can one understand the nature and import of any misclassification made with respect to it, and justify openly and objectively one's examination practices.
In essence, Dr. Banks' grading procedure for a given course comprises: [i] finding three "suitably" distinct sets each of k test items; [ii] administering the test of 3k ( e.g. 3xk = 3x32 = 96) items to the students on the course; [iii] assigning item-j scores according to answers given (1=correct; 0=otherwise) ; [iv] finding for each student-i the total score Ti (sum of item-ij scores for particular i and for j=1(1)3k); and [v] defining grade status according to the following categorizations of the possible range of Ti: A :- more than 3/4 of the 3k items correct; B:- from more than 1/2 up to 3/4 of the 3k items correct; C:- from more than 1/4 up to 1/2 of the 3k items correct; D:- 1/4 or less of the 3k items correct.
One can (and doubtless many "educators" do) follow such a procedure without any reference to the "Rasch model" or any other variant of "item response theory". In such cases - of atheoretical operational procedure - the only misclassification which may be conceived of relative to the grade categories as defined are those arising from administrative errors such as: incorrect scoring of items; incorrect calculation of some Ti; incorrect attribution of T to student-i; incorrect application of the grade definitions, etc. There is no generally useful information available concerning the relative incidence of such errors, and Dr. Banks makes no mention of them for his particular case. Thus for his case, stripped down to atheoretical operational procedure, the misclassification rates with respect to the given grade-category definitions are unknown. On the other hand, with automation of the various administrative steps, probability of misclassification could be reduced effectively to zero. How does this jibe with Dr. Banks being able to state precise numerical non-zero probabilities of misgrading?
The answer is that "misclassification" is always relative to the "class" specified, and different classes are involved in the two cases. Put more precisely, misclassification is relative to the intension of that class-concept of which the class specified is the extension (see Sutcliffe, 1995a), and different intensions are involved in the atheoretical and the theory (Rasch model) guided cases.
For the atheoretical operationalist, the grade-category intensions are precisely those given above. For example. what is required for a student to get an "A" is to give a correct answer to each of more than 3/4 of the items set for the examination. If a student actually gives Ti correct answers as required, for example, for an "A", but it is recorded that that student gave T*i correct answers, then either the student will be graded "A" if T*i exceeds 3/4 of the items in the examination (regardless of whether or not T*i = Ti), or there will be misgrading as "B", or as "C", or as "D', according to the interval into which T*i happens to have been administratively placed.
For Dr. Banks, however, the calculation of misclassification rates presumes intensions for his grade categories which are not the above given categorizations of the T-range. The intensions needed for his calculations are not made fully explicit, but they are specifiable by reference to subintervals of the "ability" continuum assumed for application of item response theory. That is, he is using a classification of his students with respect to "ability" as estimated from the Rasch model. What then it is for a student to get an "A" is to have "ability" in such degree that one's probability of passing item-i is that given by the item response model assumed. The errors of misclassification in this case arise out of slippage in the estimation of the (unknown) alpha-i (latent ability) from the (know) Ti (observed test score).
Note here some further relativities. Dr. Banks has assumed (without demonstration) that the appropriate "law" relating probability of correct response to "ability" is that expressed in the 1-parameter Rasch model. There are good a priori arguments for the use instead of a 2-parameter or a 3-parameter model (Lord & Novick, 1968; Lord, 1980), and indeed for use of a 4-parameter model (Sutcliffe, 1995b). See also Grayson (1988). It has not been shown, for a specified population of examinees and a specified selection of items, that misclassification rates are invariant across such models. Thus Dr. Banks was not entitled to generalize from the case of the 1-parameter model. Furthermore, all of the foregoing assume that probability of a correct response is a function of a single (latent) ability. Plausible a priori arguments can be made for a function of two or more latent abilities. In principle one could still have a full order for the A-B-C-D gradings with respect to any one of those latent variables conditioned on the remainder, but the practicalities of such a scheme are beyond realization in existing administrative contexts. Consequently one would either have to give up the notion of fully ordered grades under item response theory, or one would have to redefine (state new intensions for) grade categories relative to a non-trivial vector of latent abilities. What this shows, in the absence of a demonstration by Dr. Banks that the Rasch Model is exclusively applicable to his case, is that his arguments for and details of misclassification rates are arbitrary.
Administratively this would seem to be a let-off, since for example, one could equally arbitrarily shift from his (embarrassing) ten-percent-misclassification-rate-Rasch-model rationale to the (seemingly innocuous) zero-percent- misclassification-rate-atheoretical-operational rationale when explaining examination practices to any student enquirer. Note, however, that the astute student will then seek justification for the operationist's adopted grade categorization of the range of observed test scores T, asking, for example, "if Ti is worthy of an 'A' grade, why is not (Ti-1)?" and so on. Repeatedly one will be forced back to the question "What is it to be graded an "A"? (or a "B"?, or whatever?). More specifically the question is: what are the respective intensions for the concept of "A", the concept of "B", . ? or equivalently, what are the necessary and sufficient conditions (on performance in an educational context) for any given grade categorization? To answer that question one must first provide an explicit rationale for what one is doing when "educating" and "examining" students.
The proper function of university education is to cultivate in students an objective interest in what is the case. A university teacher's proper task is to foster independently thinking auto- didactes through discovery in students of any existing curiosity, and through subsequent development of such curiosity into a continuing mode of critical enquiry - via encouragement, example, shaping, instruction, and discipline. Students are to be prepared to pose critical questions, to formulate conjectures by way of answer, and to attempt in any given case to distinguish facts (what is the case) from opinions, prejudices, wishful thinking, etc. (such as have only rhetorical rather than evidential or logical support, and may be false). To gauge progress with this process one must observe periodically how a student is applying herself/himself to any task of intellectual enquiry. If administratively required, by reference to different stages in this advancing educational process one can define graded categories of performance. In this context, the symbols A, B, C, and D, or suchlike are simply convenient labels for the relevant performance-defined grade categories (classes).
Unfortunately, subject to a complex of social, political, and economic forces, education is more widely taken to be something other than the nurturing of critical enquiry. Education is otherwise commonly conceived of instead as an imparting of information which is to be put to some other purpose (training for a job), an inculcation of skills to be applied somewhere else (training for performance), and suchlike. The job, the performance, the thing trained for is conceived of as something external to the educational context of training; and that training (as distinct from the continuous development of critical enquiry) is thought of as having a discrete beginning, a relatively short duration, and a definite end. For the imparting and the inculcation there tends to be an authoritarian relationship between teachers and taught, with passive (and asymmetric non- interactive) transmission of "information" from teacher to pupil. Critical questioning of received opinion is not welcomed, and correspondingly there is an emphasis on "right" answers, "right" ways of doing things, "consensual" techniques, "routine" procedures, and suchlike. There is also demand for credentialism, and, for its implementation, examinations and the awarding of grades are emphasized. In particular, for administrative reasons, primacy is given to "objective tests" (particularly the multiple- choice) rather than to the setting of those tasks, such as problem solving and spoken or written exposition, performance of which are needed to provide evidence of capability for independent critical enquiry.
Dr. Banks is misguided when claiming that . "the natural place to start in assessing the accuracy of our educational assessments is to frame the problem in terms of item response curves". The objection to that is that it leads to attempting "to estimate ability" rather than to attempting "to assess educational achievement". As its name indicates, the aim with the Scholastic Aptitude Test (to which Lord has applied his item response theory), is to measure aptitude for study. Given instead that one is dealing with students who have already demonstrated via their SAT performance that they are fit for university study, the task then is, not to estimate further aptitudes or abilities but, rather to gauge progress in that university study which has been legitimized by the prior evidence of aptitude for it. One gauges educational progress by [i] making explicit the syllabus for a course; [ii] preparing (a priori) an examination based upon the syllabus and with the intention of making explicit what kinds of performance will be expected of students by the end of the course; [iii] assuming for the students an initial state of ignorance; [iv] administering the examination to check the initial state of knowledge; [v] taking the students through the course; [vi] assuming a final state of heightened knowledge; [vii] re- administering the examination given at the beginning of the course; and [viii] comparing the initial and final performances of each student on the syllabus-based examination to assess any educational achievement.
For space reasons, and because it appears to bear more directly upon Dr. Bank's case, I will now concentrate on the spelling out of intensions for grade categories where the examining has been done via multiple choice objective tests. Operational psychometric considerations, such as excessive ease or excessive difficulty of items, are not to be ignored, but they are of secondary importance. The choice of the items for the course examination will be determined, not by reference to item response theory considerations, but primarily by reference to item content relative to the relevant syllabus and the teacher's related educational objectives. (See, for example, Glaser (1963), Popham & Husek (1969) and Ebel (1972) for the contrast of "criterion referenced" and "norm referenced" examining.) The grade categories are then defined by reference to different degrees of progress from initial to final state of knowledge. Failure to make minimal progress (as evidenced by failure at the final examination to answer correctly a pre-specified number of questions each of prerequite content) would be the condition for assignment of a "failing" grade. Evidence of minimal progress would be the condition for assignment of a minimally "passing" grade. Evidence of progress in varying degrees in excess of a minimally "passing" performance would be the conditions respectively for the various more distinguished grades than "pass". The explication of minimality conditions will be entailed by the educational objectives of the teacher relative to the specified course syllabus and have to be spelled out particularly for each course.
Conventionally priority is given to the nominal aspects of grading; and in administrative practice an "A", for example, is treated as a constant ("meaning the same thing") across different items of work for a given student, from student to student on a given course, from course to course within a given degree structure, from subject to subject within and between degree structures, from one university to another, and so on. The matters of primary importance, however, are the intensions of the various grade categories used in those various contexts, not their names; and if in fact the intensions are not constant from one context to another, one is not justified in treating their common name as if they were. The priority must be reversed from the nominal to the intensional. Just what labels - letters, numerals, or other symbols - are to be used to identify the grades then defined is conceptually arbitrary, and so may be determined by other administrative or conventional considerations. Under this understanding, with respect to achieving the reality as distinct from the appearance of inter-student parity with respect to quality of performance, it is only via explication of intensions that one may discover whether or not an "A" is an "A" is an "A". from course to course, from subject to subject, etc.. Only by making explicit the intensions of one's concepts, can one give a non-arbitrary account of grading and misclassification rates, properly assess the reliability of one's examining procedures, and avoid dissimulation and be free of trepidation when explaining one's examining procedures to students.
References:
Ebel, R.L. (1972) Essentials of educational measurement. Englewood
Cliffs, NJ: Prentice Hall.
Glaser, R. (1963) Instructional technology and the measurement of
learning outcomes. American Psychologist, 18, 519-521.
Lord, F.M. (1980) Applications of item response theory to
practical testing problems. Hillsdale, NJ: Lawrence Erlbaum
Associates.
Grayson, D.A. (1988) Two-group classification and latent trait
theory: scores with monotone likelihood ratio. Psychometrika, 53,
383-392.
Lord, F.M & Novick, M.R. (1968) Statistical theory of mental test
scores. Reading, Mass.: Addison-Wesley.
Popham, W.J. & Husek, T.R. (1969) Implications of criterion
referenced measurement. Journal of Educational Research. 6, 1-
9.
Sutcliffe, J.P. (1995a) Logical machinery for deciding what is or
is not classification. The Newsletter of the Classification
Society of North America. 41, September, 2-6.
Sutcliffe, J.P. (1995b) Under binary scoring of multiple choice
items, plausible psychological theory implies four parameters for
the item response function. Conference paper presented at the 9th
European Meeting of the Psychometric Society. Leiden, Holland,
July 4-7.
FINAL REMINDER: 1997 ANNUAL MEETING OF THE CLASSIFICATION SOCIETY
OF NORTH AMERICA,
June 12-15,1997, to be held at the Mary Graydon Center/Butler
Pavilion
American University, Washington, DC, USA
Please see previous issues of the Newsletter for the details or contact:
Olga I. Cordero Brana
Department of Mathematics and Statistics
American University
Washington DC, 20016-8050
Telephone: 202 885 3130
Fax: 202 885 3155
e-mail: olgacb@american.edu
* MAY 29-30, 1997: DIMACS workshop on Exploring large data sets using classification, consensus, and pattern recognition techniques. Organizers: F.R. McMorris, Department of Mathematics, University of Louisville (frmcmo01@homer.louisville.edu) and Ilya Muchnik, DIMACS, Rutgers University (muchnik@lunar.rutgers.edu)
* AUGUST 11-15, 1997: Third International Conference on Statistical Data Analysis Based on the L1-Norm and Related Methods, Neuchatel, Switzerland. Organizer: Prof. Yadolah Dodge, email: yadolah.dodge@seco.unine.ch
* JULY 21-24, 1998: 6th Conference of the International Federation of Classification Societies, Rome, Italy. Put it on your calendars - - - more information will be appearing later.
Rian van Blokland-Vogelesang
SWOV: Institute for Road Safety Research
P.O. Box 170
2260 AD Leidschendam
The Netherlands
E-mail: Blokland@SWOV.nl
The following recent and forthcoming books might be of interest to CSNA-members. (Prices are approximate; '#'represents English pounds).
P. Armitage and H.A. David, Advances in Biometry, New York: Wiley, Series in Probability and Statistics: Applied Section, 1997, pp. 473, #50.00. ISBN 0471-16018-0.
H.-H. Bock and W. Polasek, Data Analysis and Information Systems: Statistical and Conceptual Approaches, New York: Springer, Studies in Classification, Data Analysis, and Knowledge Organization, 1996, pp. 372, #63. ISBN 3-540-61081-2.
A. Ferligoj and A. Kramsberger (Eds.), Contributions to Methodology and Statistics, Proc. Int. Conf. Statistics and Methodology, Bled, Slovenia, 1993. Ljubljana, FDV, 1996, pp. 290. ISBN 86-80227-42-0.
B. Chauvin, S. Cohen, A. Rouault, Trees, Boston: Birkhauser, 1996, pp. 548, #35.91. ISBN 3-7643-5453-4.
E. Diday, Y. Lechevallier, and O. Opitz (Eds.), Ordinal and Symbolic Data Analysis, New York: Springer, Studies in Classification, Data Analysis, and Knowledge Organization, 1996, pp. 372, #75. ISBN 3-540-60774-9.
E. Diday, Y. Lechevallier, M. Schader, P. Bertrand, and B. Burtschy (Eds.), New Approaches in Classification and Data Analysis, New York: Springer, Studies in Classification, Data Analysis, and Knowledge Organization, 1994, pp. 700. ISBN 3-540- 58425-0.
L.D. Fisher and G. van Belle, Biostatistics: A Methodology for the Health Sciences, New York: Wiley, Science Paperback Series, 1997, pp. 1024, #35.00. ISBN 0471-16609-X.
W. Gaul and D. Pfeifer (Eds.), From Data to Knowledge: Theoretical and Practical Aspects of Classification, Data Analysis, and Knowledge Organization, New York: Springer, 1996, pp. 372, #69. ISBN 3-540-60354-9.
D.J. Hand, Construction and Assessment of Classification Rules, New York: Wiley, Series in Probability and Mathematical Statistics, 1997, pp. 300, #35.00. ISBN 0471-96583-9.
G.K. Kanji and M. Asher, 100 Methods for Total Quality Management, London: Sage, 1996, pp. 256, #14.95. ISBN 0-8039-7747-6 (pbk).
C.R. Latimer and J. Michell (Eds.), At once Scientific and Philosophic: A Festschrift for John Philip Sutcliffe, Boombana Publications, Brisbane, Australia, 1996. ISBN 0-9586685-0-7.
W.B.G. Liebrand and D.M. Messick, Frontiers in Social Dilemmas Research, New York: Springer, 1996, pp. 438, #83. ISBN 3-540- 61299-8 (includes computer simulations).
T.P. Ryan, Modern Regression Methods, New York: Wiley, Series in Probability - Applied, 1997, pp. 536, #50.00. ISBN 0471-52912-5.
D.G. Saari, Basic Geometry of Voting, New York: Springer, 1995, pp. 300, #23.50. ISBN 3-540-60064-7 (includes controversies of Condorcet and Borda).
J.R. Schermerhorn Jr., J.G. Hunt, and R.N. Osborn, Managing Organizational Behavior (6th ed.), New York: Wiley, 1997, pp. 608, #19.99. ISBN 0471-15416-4.
J. Stoyanov, Counterexamples in Probability (2nd ed.), New York: Wiley, Series in Probability and Statistics, 1997, pp. 370, #50.00. ISBN 0471-96538-3.