A question for my readers: if you have ever sat on a Ph.D. program admissions committee or similar (e.g. recruited Ph.D. students directly into your lab), what are the major criteria that help you pare down the applications into a short list for interviews? Here are some examples:
high GPA
prestige of undergrad institution
good GRE scores
experience working in a research lab
personal statement
great letters
Now, here’s the second question: are there any of these possible criteria that do not probably select for some kind of privilege? Obviously there are plenty of Cinderella stories of overcoming hardships (and truly, this is who we’d want in our lab, is it not?), but the reality is that broadly speaking, excelling in academia is more easily achieved when you’re financially comfortable. Therefore, applicants with the time, money, and resources for internships, test prep, and independent study are generally going to be more attractive.
So the question for admissions committees and PIs is: how do we correct for this bias while still identifying applicants for whom a career in science is a the right choice? Obviously it would be great if we could just accept everyone and let things shake out, but that’s not how Ph.D. programs work. It’s a commitment for both the student and the mentor, and our goal in admissions decisions is to make an educated guess about who’s got the best chance of coming out the other end ready to put that Ph.D. to good use. In biomedical fields, that student is often being paid and/or conducting research with taxpayer dollars, so we also have at least some responsibility to ensure that the student is productive in terms of scientific output. Once again, is it possible to tip the scales to these outcomes without privilege bias?
In my program, we don’t have rotations. Students are accepted directly into labs, and so it’s up to me to choose 3-4 students to interview out of the dozens of applicants who expressed interest in my lab. Since my lab is tiny and my time in the lab is limited, I would love a student who already knows how to work with animals, make solutions, and use a microscope. Plenty of applicants have these boxes ticked, either from undergraduate research or a year or two post-bac working as a lab tech. But many don’t. It would be easy to just select from the former, but that’s biased, right? So among the latter, how do we find those who are like little scientist butterflies who just haven’t been given the opportunity to bust out of their cocoons yet? Is there any way within standard application materials to identify students with amazing potential but poor resources, or does the whole system need an overhaul?
Comment away!
you’re back, back, back!
I know, it’s been a while! Hope to get back in the groove (slowly).
Our application has an optional adversity personal statement. Many take advantage of this opportunity. That being said, previous research experience is essentially a requirement for admission into our program.
I was in a PhD class with a guy who transferred from a community college after 2yrs to a state uni with scholarship $. He worked through college and had little time for lab. Nothing stunning on paper, but the educational trajectory might be a good flag?
Got into top PhD program in the field because he told that story to people he interviewed with, in a class of 9 where everyone else had worked 2+ yrs in labs. Grad in 4.5yrs, productive and ultimately left science, Mckinsey got him.
We look for students who will excel in the lab and who can pass their courses (i.e. get at least a B). We take all sorts of factors into account, including students whose less than stellar grades might have resulted from having to work full time (one of my current students fits that description). GREs count a distant fourth after lab experience, letters and grades. If everything is stellar, great, but great lab experience and letters trumps great grades. Great grades/GREs with no relevant lab experience won’t even get them an interview. We look for diamonds in the rough but have learned the hard way not to take students who we don’t think can pass the required courses. We bend over backwards to identify students from underprivileged backgrounds who seem promising, but we can’t make up for a complete lack of lab experience.
My advice is based on inclusive hiring practices which I have found be effective for hiring faculty in the sciences.
Ideally application materials should line up with what you are looking for. Most likely that isn’t the case here. That makes it harder to use evidence to be deliberate in the process. Being deliberate and methodical using evidence and questioning yourself is important for minimizing bias.
I would look at the list of shortcuts you have listed and ask what sort of information do they provide that is useful for you. Those are your criteria you will be using in a rubric. Are there other criteria that are missing from your list? What about teamwork? What about diversifying your group?
Once you have the criteria you are looking for, what would be evidence for meeting the criteria? Yes some of those shortcuts might fit but what else? You mention solution prep and using a microscope. Students in my lab courses have these skill sets so research experience isn’t the only place to look for the evidence. Look at the resumes/CVs, courses taken, letters, personal statements, etc. to find that evidence.
Do ask yourself, how important are those skills especially for anything that is expensive. Is there a way for a student to demonstrate potential to do well if they didn’t have access to expensive resources?
Go through multiple rounds of looking at the applications. What is the minimum set of criteria? Use that to create your first pool of candidates. Be inclusive. Use a rubric then to look at this pool more closely. Look holistically and at individual criteria. Ask why do certain candidates rise to the top on some criteria but not others. Use that analysis to develop your next pool. Keep iterating and at each stage stop to look at your pool. Give yourself time. Bring in others. They may see things you don’t and vice versa.
There will be bias. Especially if recruitment and application materials aren’t aligned with goals of being inclusive. For example, I mentor my advisees to include relevant lab skills/techniques/instrumentation on resumes/CVs and to point that out in personal statements when discussing research groups interested in joining. I am at a small liberal arts college. I have the time to do that and to look at those resumes to provide feedback. Not all undergrads get that type of advising. If that is important information for deciding who to accept then put that in your materials requested. Don’t assume students will know what to include.
The unwritten rules are a big obstacle. Identify in the process what those are and make them written criteria in the future. Applicants should know what you are looking for and how to give e you that information. Be as transparent as possible as early as possible. Ideally this goes with outreach efforts. Also don’t rely on letters to provide.
Going through the applications now is going to be long and painful especially if you want to minimize bias. The effort you make now will help improve the process in future years for both you and the applicants.
And yes your goals in getting your lab to be productive in the short term are not aligned with being inclusive as you note. That is a systematic issue.
Report from the Council of Graduate Schools on “Holistic Review in Graduate Admissions”:
http://cgsnet.org/ckfinder/userfiles/files/CGS_HolisticReview_final_web.pdf
I think passion is arguably the most important quality in a successful graduate student. This is really hard to see on paper. I didn’t come from a particularly disadvantaged background but I did have to work at least one full-time job throughout college, but I still made time to work in a lab because it was something I was passionate about (I didn’t even know about graduate school or that research experience was something I would need to get there). Again, this is really hard to see on paper but if you’re looking for it, which it sounds like you are, it will stand out.
How to make that a standard for others to look for in application materials? If you (the committee) have a system of quantifying, ranking, or systematically discussing elements of applications, make ‘passion’ or ‘perseverance’ or ‘je ne sais quoi’ one of the categories in addition to the standard materials.
‘je ne sais quoi’ — I’m not sure you really meant to say this. This is precisely where implicit bias plays a *huge* role. It’s tempting to rely on that gut feeling — he has “it”; she doesn’t, etc. — but if the goal is to minimize the impact of bias, these unexamined places of our psyche are exactly what we should be questioning and trying to make more objective.
I would wager that it’s a lot easier to recognize “passion” in those who are most similar to us. You’re already doing it in your comment: “I did have to work at least one full-time job throughout college, but I still made time to work in a lab….” I know several students for whom making a certain amount of money while in college was a requirement and who therefore chose jobs that paid better than lab work just to make ends meet.
Finally, I’m not trying to pick on you. It’s just really easy to fall into this trap.
I want to second anon’s response. The ‘je ne sais quos’ is exactly what we should be looking to remove from the evaluation process.
Thank you for pointing this out! I was coming from the opposite perspective – if standard quantitative metrics have bias, then what kind of qualitative assessment can be made that looks past those? – but I completely see what you are saying. It is a difficult problem, the point of this post.
For the record, I meant that I ‘worked’ in a lab by volunteering my time to do something I loved, outside of my unrelated paid employment – I am that student who worked whatever jobs I could just to make ends meet.
The REAL answer is that we need to make it possible (1) to try being a graduate student and then switch and do other things, so that we are not trying to predict based on inadequate information, and (2) to come back from failure. The real problem with trying to find the diamond in the rough is that it’s rough around the edges and almost certainly has stumbled along the way. We need to provide better safety nets so that students can stumble and recover.
Of course, I don’t know how to do that in our current you’re-on-your-own era of lack of funding resources for the unprivileged.
Completely agree. Right now it’s seen as such a shameful thing to fail out of grad school or not find a lab that suits or just plain realize that research isn’t for you, that I think we end up pressuring students to stay who should in all honesty go find a better fit for them.
But as qaz says, the money is spent so no one wants to do that.
I just don’t consider previous experience to be a marker of privilege like some do I guess. Even mediocre schools often provide opportunities for placements, internships etc. To me it’s more a marker of an interest in science beyond the classroom and an interest in the minutiae of lab work and a career in research. Having said that I don’t think it should be a requirement at all. It’s a case by case thing for me.
seriously? You don’t think the ability to work/do an internship (in my field these are always unpaid- actually, the student pays tuition for the experience!) isn’t a sign of privilege? I’m a primatologist in the US, and these days, it’s practically a requirement that applicants have spent time in the field (meaning international travel, etc) before applying. Obviously, it shows you can handle these tough, often necessary parts of the work, but wow…who has the ability to travel abroad at cost, and work for free for 6 months to a year? It is rare to pay these people, even their flight costs! Those who can afford to play, get in to grad school, those who can’t, don’t
Ugh, OK. Believe it or not the science world is bigger than just the US, and some countries actually provide opportunity to very unprivileged people to get an education. That includes undergtaduate degrees and grad school.
Standardized test scores are always the place to start. Modify opinion up or down from there depending on relevant circumstances.
When you are choosing from 3-4 folks for direct entry to your lab, I think talking on the PhD with every reference has to be a primary goal.
If you are talking with references have prepared common questions to get at the information you want. Obviously follow-up questions will vary but goal remains the same. Get the evidence to assess the candidates with the criteria you have established.
As for GRE
http://www.nature.com/naturejobs/science/articles/10.1038/nj7504-303a
http://www.theatlantic.com/education/archive/2016/03/the-problem-with-the-gre/471633/
You need to know how to read applications to modify.
I think it’s not possible to eliminate the effect of privilege but you can try to minimize things. Looking at applications holistically and being flexible on things such as GRE scores or grades is useful. I readily admit that I had middling grades which kept me out of consideration some places despite good GREs and lab experience.
I’m guessing some variation in how people due things have to do with the how their grad programs are structured and what your applicant sets look like. We don’t have rotations, so the applicant is essentially applying directly to a lab. Me personally I don’t pay too much attention to GRE or grades. I check to make sure they aren’t awful because it can be an issue with the Graduate School. I wouldn’t take a 4.0 over a 3.3 just because of that. I’m more concerned with prior experience and interest.
To provide a non US perspective (following up on @Dave), my impression is that the system here in France, whilst FAR from perfect, manages to remove a number of your cited biases.
1. Students will have roughly equivalent months of research lab experience as part of two separate 1-year Masters courses. Extra lab placements (e.g. summer) have to be paid (as in, it’s the law), so we take plenty of folk who wouldn’t be able to do an unpaid placements. In any case, simple volume of lab time is rarely a factor.
2. In France there is an odd concept that education is a right and should be free to all. So for undergraduate courses there is no entry, you just turn up. This strongly diminishes the ‘argument through prestigious institution’ part. [I should add there are a very small number of “elite” institutions. I’ll save you my rant on those]
3. No standardized tests
4. No letters. If you want a reference from a previous lab, you phone them up.
5. The whole process is far less codified (e.g. personal statements). My (of course probably biased) view is that helps to get a measure of the applicant.
my ranking of these factors + comments
-experience working in a research lab
This is probably the most important, but has a big privilege component in the US and many other places. Virtually all undergraduate science programs have independent study programs (for course credit). It has to be read against opportunity. requiring long-term research experience in a lab is too exclusionary.
-personal statement. important because students need to address issues related to opportunity or hardship here. I don’t read much into self-professed “passion for science” or other cheap career center boiler plate. (I am not sure my enthusiasm for science rises to what I consider passionate levels, and being surrounded by people in the throes of such feeling sounds exhausting.) It shouldn’t be boring, it can be quirky, flawed, idiosyncratic. it should sound like it was written by an individual, not a PR team.
-great letters. have to talk on the phone with short list refs. another place where opportunity/hardship can be understood.
-high GPA
GPA should be reasonable and/or show improvement. without looking at each transcript or with transcripts that don’t give you comparative data, GPA is a useless number. The GPA from many private US universities is useless. They know what they’re selling.
-good GRE scores
we don’t ask for them, so only get high ones. uninformative.
-prestige of undergrad institution
counts for nothing.
This is an anecdote, but I identified a great student that did not have the sparkly GPA or GRE (or even great letters) through email correspondence. It seems weird but the level of discourse I had with this student through email really wasn’t matching up with the standard metrics. The other thing that stood out to me was the student had successfully applied for programs to increased diversity in science and a small grant. Writing skills and initiative go a long way. Sadly, our fellowship formulas (GPA + GRE) provide no way to account for this so it’s really up to us look for the needles in the haystack.
1-Grades are always a starting point, but there should be more than that to base selection for a graduate program.
(Because grades are an indicator that the individual can handle well the existing education situation at the local and national/international level, and how disciplined they are at competing under those current circumstances. I should think that in the cases of scarce money and access to other resources they at least are a marker for interest and abilities)
2-Letters of recommendation/references can be useful for more than networking with other investigators, if they have a more accurate perspective of a student capacity and attitude, they will be a an ok marker.
3-Application statement can be genuine or just well coached, like a CV presentation, so ideally a personal interview shows more but above all helps the applicant to decide if the program objectives are fitting, if it is well explained.
4-A diamond in the rough will somewhat show up, for either mostly theory or lab, when people go though at least part of the programs at that particular school.
So: overall background tells a story, including leadership ability and latent talents, and if the particular program and school can provide the applicable resources for polishing the ‘rough’ part you will uncover the needed diamond. Hopefully then, it will be appreciated.
Personally, when I am in the process of selecting/evaluating URM applicants who come from socio-economically disadvantaged backgrounds, I ask to see their transcripts, so that I can make some judgement regarding scholarly progress and depth of science experience/immersion. I also ask for references, who I either call or email (if they worked non-science jobs, I speak to those references). I care about job performance, in general, from an actual paying job over an internship in a lab.
If the student has no prior lab experience, I still consider them. From what I’ve witnessed, most kids seem to overstate their lab experiences, anyway. If they worked in a lab, of course I also speak to those references. References from class professors count less, but I take those into account, as well.
GRE math scores matter to me, as those scores are really based on high school-level math, anyway, and those are the basic math skills that will be required for work in my lab. There is really no way around that, at least in my subfield. GRE verbal and “analytical” scores don’t really mean as much to me. If they took writing-intensive courses, I care more about those grades than GRE verbal/analytical scores.
I’ve been sitting on the admissions committee for four years, R1 institute, state uni, medical school umbrella PhD program. The rank order for us is generally experience, transcripts (not just GPA), recommendations, and then personal statement. No one cares about the GRE, we’ve compiled these numbers for years, and have never seen a correlation between success and GRE score.
Having come from a “disadvantaged background” I applaud your efforts. While I think the NIH and NSF has made strides in promoting diversity, they have gotten a lot of things wrong when it comes to identifying and helping disadvantaged students. For example, they tend to stop helping financially disadvantaged students once they hit grad school, as though the special issues they face suddenly end when they get into a program.
That being said, you cannot discount experience as being a top predictor of success (which leads to better references as well). Once you have identified those candidates I think it is fine to do another sweep and try to find diamonds in the rough, and a thorough analyses of transcripts will go a long way. We have a saying, if you’re going to take a chance, take a chance on a 3.9. But generally you need to go deeper. Did they maintain good grades by taking easy classes? Or did they challenge themselves? Did they come back from a bad semester? Did they suddenly mature when they realized what they really wanted to do?
What are your measures of success?
It should go without saying that an actual interview with the applicant should always occur. If a student enters my lab (even just rotating or doing a short internship/paid stint as tech), I *always* interview. In person interviews are a 1000 times more preferable to phone or skype, as I’ve heard horror stories about recruits (on all levels, even post-docs) who misled PI’s over the phone or skype (paid other people to pretend to be them).
I honestly don’t understand what “privilege-free” would mean. Everyone has some privileges (a prerequisite for being able to apply for a graduate program). Do you mean “uninfluenced by applicant privilege”? I don’t believe it’s remotely possible to disentangle achievement from privilege. We can apply affirmative action to ensure we get the best candidates regardless of race/gender. But I don’t think it’s possible to truly level any playing fields.
For admissions:
Our primary criteria are GPA and prior research experience. ProdigalU has a minimum GPA required for grad admission, so we need to make a case for an exception for students with CGPA below the cutoff. We consider everyone above the cut-off, though. From prior experience, lower GPA students are more likely to struggle to pass our classes (or not pass them at all–grad students must get a B or better for the class to count). We are more likely to admit lower GPA students who have some extenuating circumstance that they explain in their application (family obligations, health problems, working off campus while a full-time student, etc), since getting things done in non-perfect situations counts for a lot. I’ve never seen an application from someone who wasn’t a full time student for at least the last two years. We generally only accept students who have research experience, which is pretty easy to get in my field at schools that have some form of research on campus (research for credit as at least an elective is pretty common). Students who attend schools that don’t have on campus research and who cannot/did not do a summer research experience somewhere are out of luck in my department, but we are upfront about this before students pay the application fee.
We look at letters, but mostly for recommendations at either extreme (bad or good). A great letter can get a diamond in the rough considered. Similarly, a letter with red flags about behavior will sink a 4.0 student with publications. We don’t want to import headaches. In my experience, most students can get 2-3 people to write them fine letters.
For my lab:
I interview all candidates before they join my group to make sure I can work with them. I assume if they are admitted, they can pass our classes. I don’t look much at GPA, mostly at motivation and enthusiasm for my projects. Fortunately, in my department, students are admitted to the department and join a group after arrival, which makes interviewing easy.
For those interested, this analysis of predictors of papers in grad school just came out: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0169121
Looks like recommendation letters are the most reliable.
With the biases in the writer and the reader, are the letters indicators of bias students are going to encounter in graduate school? Maybe measuring something else here than what we think. Publication rates do vary by gender and there is bias in letter.
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000127