"For doctoral programs, class grades are a smaller part of the picture. Test data on what a college student remembers cannot predict how the student will use new information to think in creative and original patterns, manage complex research tasks, and explore the unknown. Completing a doctoral dissertation requires financial resources, social skills, aggressiveness, creativity, persistence, resilience, managerial skills, motivation, ability to work independently, family stability, health, and luck. There is a need for qualitative measures that predict student–graduate program compatibility".
Read the debate in Science about whether test scores can predict good PhD performace
4 comments:
"Completing a doctoral dissertation requires financial resources, social skills, aggressiveness, creativity, persistence, resilience, managerial skills, motivation, ability to work independently, family stability, health, and luck. There is a need for qualitative measures that predict student–graduate program compatibility"
There are a couple of issues here
(i) we are not at all sure in the literature what combination of "non-cognitive skills" determine outcomes and it will be at least a decade before proper structure gets put on this. I could think of at least 200 more names for qualities that one would need to finish a PhD (indeed see some of the other posts).
(ii) in terms of using such measures to assess graduate student compatability rather than just testing scientific theories, this needs to be informed at least partly by fairness considerations. Are they suggesting that if someone does not have the right combination of family background and health that they should be denied admission to a programme despite having the other entry criteria? Some colleges at the moment very aggressively screen applicants to graduate studies to ensure that they will be able to finance themselves. I dont know whether this is ethical or not but we do need some criteria to decide what can and cannot be considered in a graduate student hiring process. Companies routinely administer detailed psychometric batteries to assess compatability. I wonder are people suggesting that colleges should do so to either replace or augment standard ability measures such as grades, GRE's etc.,
In regard to augmenting current measures- The literature on the objectives of systems of admissions to medical courses (which I would take as a leading faculty in the field of college admissions) focuses on using procedures which are consistent, transparent and equitable in order to assess the cognitive and non-cognitive aptitudes of applicants. This is mainly due to the reason that aptitude and being a "good doctor" is accepted as a broader concept which incorporates non-cognitive characteristics such as personality traits and motivation as well as intelligence.
Non-cognitive measures of motivation, communication, interpersonal characteristics, emotional intelligence seem only to be of use when they link into a cognitive component. The medical admissions test producers are putting their eggs in the "interpersonal reasoning" basket and should have some evidence regarding the predictive validity of such measures in the coming year (UMAT and MSAT tests from what I recall). Measures which look at pure non-cognitive characteristics can't control for social desirability effects and therefore are only useful for use outside the admissions process to measure the validity of combination (cog & non-cog) type tests.
Interviews aren't much better than pure non-cognitive measures and even a carefully structured interview designed to distinguish amongst aspects of students’ non-cognitive performance on the medical course (e.g. listening skills, respect for others), failed to do so (Streyffeler, Altmaier, Kuperman, & Patrick, 2005). Pen and paper tests of interpersonal reasoning or even better realistic situational behavior tests with clearly specified criteria seem to be the first step forward from what I've read.
interesting, one wonders what incentives that tests such as this create. We by and large know what happens with things like the Leaving Cert, SAT's and GRE's. People study for them like the clappers and their final result may or may not be reflective of what the exercise was initially designed to measures. The practical environment tests sound interesting. Particularly interesting about this is that they produce incentives for applicants to do precisely what you would like to them to do i.e. get better at their potential career rather than spending lots of socially wasteful hours cramming for aptitude tests or paying for books or lessons or whatever on how to jump arbitrary hurdles. In general, a good principle for a screening mechanism is that any effort that can be expended by the applicant to increase their score should not be arbitrary or wasteful effort.
This is true and I was most impressed by a simplified version of the highly successful Objective Structured Clinical Examination (OSCE) being used for admissions to Canadian medical schools. This test has been proposed to assess both cognitive and non-cognitive qualities of applicants in an applied context through a series of planned encounters engaged in by the student and observed by a trained assessor who evaluates the student in relation to critical areas of professional performance such as ethical reasoning, communication and problem exploration. Preparation for such tests is certainly not a waste of time but raises the question as to what extent you should select students based on non-cognitive characteristics which the course also aims to instill in the students.
Post a Comment