Posts by Beth:
- If there is a type of career you are particularly interested in, let us know and we’ll see what we can dig up.
- If you are a PhD who hasn’t followed the tenure track and would like to share your experiences with our readers in this series, please let us know. We’d love more guest posters!
- Michael Patton’s Utilization-focused Evaluation2
- Kellogg Foundation Logic Model Development Guide
- Consulting and evaluation with nonprofit and community-based organizations by Judah J. Viola, Susan Dvorak McMahon.
- if such a thing appeals to you, like it does for me. [↩]
- Patton has also recently published a book called Developmental Evaluation – I haven’t read it yet, but it’s probably worth checking out [↩]
- Claremont Graduate University in California seems to be one of the most prominent places to get a degree in evaluation, though more places are starting (or planning) to offer such a thing… but that’s the subject of another posting! [↩]
- e.g., my formal education is in Nutrition, and other evaluators I know have degrees in a wide variety of fields, including Social Work, Psychology, Kinesiology, Criminology, and History, just to name a few [↩]
- ;-) [↩]
- full disclosure: my Master’s program was a 12-month “coursework and a project” Master’s, not a thesis-based Master’s [↩]
- I never did figure out where one gets undergraduate minions… they would have made my life so much easier during my PhD! [↩]
- It doesn’t taken long to whip up a blog posting, but when you need a short break from your work, it’s a more productive way to spend 5-10 minutes between doing other work (as opposed to, say, surfing the ‘net). How is it productive? you ask… read on!
- Blogging allows you to be part of a community. Thanks to the web, it’s easy to find someone who is going through (or has just gone through or is about to go through) the same thing as you. So even if you are in a small department with no other people at the same stage of career as you, you can find the support/advice/motivation you need. Whether you are a PhD student considering whether or not to go the academic route, a newbie on the tenure-track, someone who has left academics, etc., you will be able to find others like you in the blogosphere – and you will be able to help out others in the same way.
- Blogging can get energized about your work. Sometimes you need a forum to vent or to ruminate. Blogging about problems with your research (or your career track) can sometimes help you figure out the next steps to take.
- Blogging can help you learning about other people’s perspectives. Because writing a blog doesn’t stop at writing a blog – you’ll get comments from readers that really make you think. Then you’ll follow the links of those readers to their blogs and learn all sorts of new stuff there.
- raising awareness about your own line of research
- communicating with the public about science (and all the benefits that go along with that)
- Alternative Scientist - a group written blog by people who didn’t follow the academic path (it hasn’t been very active lately, but there are some good postings in the archives)
- FrogHeart - “Commentary about nanotech, science policy and communication, society, and the arts”
- Nature.com blogs – a whole slew of scientists blog here
- Nerd Girl’s blog on the Globe & Mail site
- Prof-Like Substance - “Because being early tenure track faculty is even stranger than you think”
- UBC Faculty Association blog (this one is focused on the Faculty Association’s current round of bargaining with the university as a way of keeping the faculty members informed)
- If you are interested, I suggest you check out his series “How Blogging Helps Me”: parts 1, 2, 3, 4 and 5. [↩]
- feel free to recommend other blogs in the comments section too! [↩]
- age 18-22: paying high tuition fees at an undergraduate college
- age 22-30: graduate school, possibly with a bit of work, living on a stipend of $1800 per month
- age 30-35: working as a post-doc for $30,000 to $35,000 per year
- age 36-43: professor at a good, but not great, university for $65,000 per year
- age 44: with (if lucky) young children at home, fired by the university (“denied tenure” is the more polite term for the folks that universities discard), begins searching for a job in a market where employers primarily wish to hire folks in their early 30s
- Maybe in the comments section those of you who have chosen to stay on the academic career path can shed some light onto why you think being an academic scientist is a good job? [↩]
- I don’t have any data on this, but I’m willing to bet that few, if any, universities provide maternity/paternity top-ups as a benefit to their postdocs [↩]
- people start to get on wait lists for daycare during their pregnancy, even if they aren’t planning to send the kid to daycare until after a full year of maternity leave [↩]
- Keynote: The Power of One: Simon Jackson (one of Time Magazine’s 60 Heroes of the Planet)
- Keynote: Using Olympic Principles for Success: Jennifer Walinga (Double Olympic Gold Medalist)
- Panel: Los Tres Amigos – Global Excellence in Monitoring and Evaluation – Robert Lahey (representing Canada), John Pfeiffer (USA), and Agustin Caso (Mexico)
- Keynote Panel: Summarizing and Celebrating the Conference – Four Evaluation Association Presidents – Nancy Porteous, Hallie Preskill, Florence Etta, Linda Lee.
- Green – Evaluation in the context of managing positive environmental change.
- Gold – Evaluation of major events (e.g., Olympic Games) and striving for excellence in evaluation.
- Global – current trends and developments in international evaluation.
- as there doesn’t seem to be a Creative Commons license on there, so I couldn’t put the diagram into this posting [↩]
- unless you want to publish the findings in an academic journal that requires an ethical review to have been conducted. Or if you have an external funder that requires it [↩]
- though we don’t like calling any job a PhD holds other than a professorship an “alternative career,” which suggests that the professorship is the “right” job, we haven’t actually thought of a good name for this. Any suggestions? [↩]
- -talking science to non-scientists
- -talking to the media
- -what’s in it for you
- -make your message Simple (but not oversimplified) – when you are trying to explain what DNA is to grade 4s, you quickly learn how to strip concepts down to just the essentials. As a scientist, I find that I revel in the nuances, the eccentricities, the many variables in play in a given situation. But when trying to get scientific ideas across to someone who isn’t familiar with all those nuances, it makes a lot more sense to figure out “what’s the core message I’m trying to communicate?” I’m not suggesting oversimplification, but I am suggesting that you don’t need to overcomplicate! Can you explain your concept with a simple analogy?
- -say something Unexpected – people love a good mystery! And what is an experiment if not a detective job to solve a mystery? Setting up a mystery at the start of your class/lecture motivates the audience to listen – they want to know the answer to the mystery!
- -be Concrete – nothing makes science more concrete than doing a hands-on demonstration or experiment. Another aspect of concreteness is talking about how science comes into play in our everyday lives. The movements of electrons are pretty abstract, but become a lot more real when you talk about the electricity that powers our iPods.
- -show that the information is Credible – again, experiments really come into play here. If you can demonstrate that a light object and a heavy object accelerate at the same rate when dropped, it becomes a lot more believable than if someone just tells you that it is so.
- -tap into Emotions – this includes things like “self-identity” (e.g., when you refer to the students in the class as “scientists,” it really seems to change their outlook on science in a positive way, it makes them care about doing science. As well, it pays to think about the “WIIFY” – “what’s in it for you?” – talking with students about how science plays a role in their everyday life helps provide inspiration for learning about science. Last year I did a series of activities on “structures” with the grade 3-5 class that I was volunteering in and now none of us, myself included, travels over a bridge without thinking about its design and the forces acting on that bridge!
- -tell Stories – telling stories is something I’ve always done rather instinctively when I teach (though that may be just because I like to tell stories!) Stories are a great way to demonstrate concrete ideas and show why science matters.
- -Volunteer for a science outreach program in your area. This could involve things like doing experiments in classrooms, teaching science at summer camps/Guide Guide or Boy Scout troops, mentoring high school students in your lab or judging at science fairs. Check out this posting for a list of science outreach organizations (and please tell us of any others you know about so we can add them to our list!)
- -Sign up with your university’s “expert” database – the one that journalists will go to when they are looking for a scientific expert to comment on a news story. With all your newfound knowledge about effectively communicating science to non-scientists, you will be the perfect expert for this job!
- -Write a blog – Blog about cool things that are going on in research – your own or other stuff in the published literature
- -Write a letter to the editor – Read a news article that provides incomplete – or even incorrect – information? Write a letter to the editor to set the record straight!
- for the record, I have no affiliation with this book or its authors. I don’t get any money if you buy this book and I didn’t get a review copy or anything – I just picked it up from the library and I thought it was a really good book! [↩]
- well, I guess technically those days aren’t behind me, as I still volunteer as a science fair judge and I write for CurioCity magazine [↩]
- the not so good was framed as “OK, we aren’t saying that journalists are out to get you, but here are some tips from keeping the big bad journalists from getting ya!” [↩]
- As an aside, I think that this is a place where PhDs who decide academia isn’t for them can put their expertise to good use. I feel like I can play the role of “interpreter” who translates the culture of practice to academics and vice versa [↩]
So, you want to be a Program Evaluator when you grow up…
July 22nd, 2010Since Dave has started us off on the “So you want to be a blank when you grow up” series, I’m going to take the lowest of the low hanging fruit and tell you all about what I’ve learned since becoming a program evaluator! But first I’d like to mention two things:
— Now, onto the exciting world of program evaluation! So, you want to be a Program Evaluator when you grow up… Evaluation is a great career for a PhD, because it allows you to use the skills you developed in your training – research skills, problem solving, critical thinking, project management, and more. Though I’d not heard of evaluation as a field per se during my education, it actually is quite established as a profession. If you want to get a good sense of what evaluation is all about, I’d suggest checking out:
Each of these societies’ websites have a wealth of information about evaluation, including things like evaluation standards of practice, literature, conferences and workshops, and, importantly, job postings and requests for applications for contracts. Which brings us to the question:
In-House Evaluator or Independent Consultant?
There are pros and cons to both, of course. Working in house (like I do) offers things that are appealing both personally (like a stable income and benefits) and professionally (like the ability to have a thorough understanding of the programs you evaluate, the opportunity to work with programs over the long-term and see them flourish, and a chance to build evaluation capacity in your sector1. Working as an independent consultant offers benefits like flexibility of when and where and on what you work and the chance to make a lot more money than you will in-house. Apparently there is a tendency for evaluators to flip between in house and independent throughout their careers (as one colleague described it to me, you get sick of the bureaucracy of working in-house, so you go out on your own for about 5 years, then you get lonely from working on your own for so long, so join an organization. And repeat). You can, of course, combine the two – for example, having an in-house job and do small evaluation projects on contract on the side). And working for an evaluation firm – where you have some security, benefits, etc., but some of the flexibility of working on your own – is a possibility as well.
Training in Program Evaluation
There are a number of training opportunities to learn more about program evaluation and to build particular skills. The above-named societies offer a variety of training workshops, as does the Evaluators’ Institute. As well, if you are interested in doing more schooling, you can get graduate-level training in program evaluation at:
As well, the CES has recently launched a credentialing program, and though it’s still early days, I think it will be well worth working towards obtaining that credential, especially if you plan to work as an independent consultant. And a few resources worth checking out: In addition to the wealth of resources on the CES, AEA, EES websites, here are a few things that I would recommend reading:
Final words: Like Dave said about science writing, it’s worth starting to build your portfolio of evaluation projects early. You can certainly use aspects of your graduate research as a demonstration of your skills, but picking up some small evaluation projects while you are in school – perhaps even working under a more established evaluator as a mentor – would be good way to start off your portfolio.
Federal Government Scrapping The Census Long-Form: What Will It Mean for Evidence-Based Policy?
July 8th, 2010A friend of mine just sent me a link to this news story and it’s gotten me quite livid:
Tories scrap mandatory long-form census
StatsCan says quality of data will suffer
Every five years, Canada conducts a census, with the next one scheduled for 2011. In the past, every household received the short census form, which contains just a few questions (like number of people in the home and their age and sex) and 1 in every 5 households received a mandatory long form. The long form contained questions about a variety of things, like income, education, and ethnicity, and provided a lot of really important information about the population of Canada. For example, I work in health care and we use census data all the time. Knowing the makeup of our population allows us to make informed decisions about providing health care to meet the needs of the people living in our region. All levels of governments (municipal, provincial and federal), community agencies, and other organizations use the data from the census long form to develop evidence-informed policy.
But the ability to do that now at risk, as the federal government has, apparently without consulting anyone, decided to scrap the mandatory long form, citing that the long form represented “what most Canadians felt was an intrusion into their personal privacy in terms of answering the longer form” (Source) – though I haven’t seen from anything I’ve read thus far how they determined that this is how “most” Canadians feel. Instead, they are replacing it with a “voluntary household survey” that will be sent to 1 in 3 households, the members of which can choose to complete – or not to complete – the survey. This raises very serious concerns about the quality of the data – the people who choose not to respond to the long survey may be different from those who choose to respond to it, which will result in skewed information. Which means we won’t have the data we need to make policies and provide appropriate services.
“Senior statisticians at Statistics Canada have conceded the change will make it more difficult to obtain reliable, detailed information.” (Source)
All of this brings up questions about politicians’ understanding of the importance of data and evidence-informed practice, not to mention their ignoring the scientific experts on the matter – in this case, the statisticians at Stats Canada. As David Eaves noted in his article, “Why you should care about the sudden demise of the mandatory long census form”:
“This is a direct attack on the ability of government to make smart decisions. It is an attack on evidence-based public policy. Moreover, it was a political decision – it came from the minister’s office and does not appear to reflect what Statistics Canada either wants or recommends. Of course, some governments prefer not to have information; all that data and evidence gets in the way of legislation and policies that are ineffective, costly and that reward vested interests (I’m looking at you, tough-on-crime agenda). ” [emphasis mine; Source]
In another news article on the topic I read:
Liberal MP Marlene Jennings “argued that Clement has shown in postings to the social media site Twitter that he does not understand how the mandatory nature of the long-form census allows Statistics Canada to properly weight the short form data. Clement debated sample size and data weighting with other posters, including an economist.
“(That’s) something Mr. Clement seemed not to understand when he was tweeting yesterday, so maybe he should take a stats course,” she said” (Source)
This lead me to check out Clement’s Twitter stream, where he referred to having a mandatory (as opposed to voluntary) long form as “state coercion” – I guess it’s fine to have the state “coerce” you to complete the short form – or, you know, obey any of our other laws – but they draw the line at the long form?
Now, as Dave mentioned in his most recent posting, we here at The Black Hole don’t want to just complain about problems, but rather want to focus on what we can do about addressing those problems. To this end, I’d like to note that the Liberals are demanding the Conservatives reverse this decision to scrap the mandatory long form and, if they don’t, to introduce legislation to protect it themselves. Personally, I’ll be writing to my own MP and Industry Minister Tony Clement to inform them about why the long form is so vital. And I’ll be writing to some Liberals to suggest that they stick to their guns on this one and introduce legislation to protect the mandatory long form.
Tip: Don’t know who your MP is? You can search here to find out!
Update: There is now a petition on this issue. Check out: http://www.gopetition.com/petitions/keep-the-canadian-census-long-form.html
What Does A Graduate Degree Mean?
June 22nd, 2010I was at the recent Canadian Evaluation Society conference where they revealed the process for obtaining their new “credentialed evaluator (C.E.)” designation. In order to become a “C.E.,” one must demonstrate through education and experience that they are qualified as an evaluator. Since there are few formal educational programs in “evaluation”1, most evaluators do not have a degree in “evaluation,” but rather are educated in some other field2. Thus, the formal education requirement for the C.E. is that you hold a graduate level degree or certificate. The reason given for this requirement was that by earning a graduate-level education, one has demonstrated that they are capable of critical thinking, analysis and research. Despite my advanced skills in research3, I’ve been unable to locate any formal source that states what a graduate degree is meant to signifies, so I’m falling back on Wikipedia to give you this:
“A master’s degree is an academic degree granted to individuals who have undergone study demonstrating a mastery or high-order overview of a specific field of study or area of professional practice. Within the area studied, graduates possess advanced knowledge of a specialized body of theoretical and applied topics; high order skills in analysis, critical evaluation and/or professional application; and the ability to solve complex problems and think rigorously and independently.” (Source. Emphasis mine)
This got me thinking, yet again, about countless discussions I’ve had with other grad students about what it really should take to get a graduate-level degree. There seems to be a lot of variation in what it takes to get that “MSc” or “PhD” behind your name. I’ve seen some Masters students who conducted original research from the initial research idea through all the stages of study design, implementation, data analysis and writing; but I’ve also seen Masters students walk right into an already designed study and really only needed to implement a study protocol that their supervisor created 4. Or those who sent their samples off to another lab for analysis (or have undergraduate minions who do it for them5 and never really learned the ins and outs of the techniques they were supposed to be an “expert” in.
So, I’m curious, what has been your experience? And if you were hiring someone with a Master’s degree or a PhD, what would you expected of them, based on their holding that degree?
Footnotes:
Why Do Scientists Blog?
June 9th, 2010One of the ideas I’ve been tossing around for a blog posting here is: Why do scientists blog? I mean, I’ve been a personal blogger for nearly five years and have been blogging here at The Black Hole since October, surely I have some insights on why people do this, right? Yet I’d been struggling to come up with a coherent explanation…. and then one of the science bloggers that I read went and wrote a five-part series on exactly this1. Some of the reasons given there (both by the blogger and by the blog readers in the comments section):
Now, those ideas come from a blog focused on the process of being a tenure track prof, rather than a blog about the science itself. Plenty of scientists use a blog to talk about their work and for them it’s easy to see benefits like:
So now I put the question to you – do you blog? If so, why?
Some interesting blogs you may want to check out2
Why does anyone think science is a good job?
May 24th, 2010“A science researcher at Harvard now earns an annual salary that is only 1/50th the price of a family-sized house in Cambridge, a fact that may not be lost on an intelligent female Harvard undergraduate choosing a career.” (Source)
The announcement of the new Canada Excellence Research Chairs (CERC) – 19 scientists granted $10 million over 7 years to establish research program in Canada – has raised some eyebrows for its notable lack of women among the recipients. We’ve talked a bit about this issue here before, notably about the mat leave issue and the disadvantage one faces in a career in academics should they choose to take time off to have kids. But while reading some of the articles and blog postings about the CERCs, I came across this interesting article, which poses the question: “Why does anyone think science is a good job?”
From the article:
The average trajectory for a successful scientist is the following:
Not to mention that academics who make the above salaries aren’t just putting in a 40-hour work week with four weeks paid vacation and stat holidays that aren’t spent working on the next grant application, as they would be in another, higher paying, career. Put like that, it’s not hard to see why I – and many people like me – have gotten off the academic career track, regardless of whether or not we have/plan to have kids. In fact, when I see the fierce competition that recent PhD grads and post-docs face trying to land a tenure-track position, it makes me wonder why so many people are willing to fight so hard for what, in many ways, is a thankless job1.
Now, don’t get me wrong. I’m not saying that no one should ever become an academic researcher. In fact, I think that academic research is essential – we can’t leave science solely in the hands of industry, for example. But I think we need to give some serious thought to the compensation and work-life of our academic scientists. So while we talk about the ways to advocate for appropriate compensation and benefits for postdocs to help keep good scientists in the game – a very important issue, to be sure – we also have to be thinking about what comes after that for those postdocs who are lucky enough (or is that “unlucky” enough?) to land a tenure-track gig.
Making Little Scientists: The Plight of the Postdoc Who Wants To Have a Family
April 22nd, 2010A comment on one of Dave’s recent postings got me thinking about an “issue affecting trainees” that we haven’t yet talked about here on The Black Hole: babies. Specifically, the having of babies and where to keep your babies while you are in the lab.
Having Babies
PhDs take a long time and by the time you are done that and have started a postdoc or two, you might start thinking you are getting to an age where you’d like to have a family! So, the discussion about whether postdocs are “employees” or “trainees” has implications beyond the “taxable or not” question. What kinds of benefits trainees get is important because maternity/paternity/parental leave is funded primarily by Employment Insurance (EI) and, if you are lucky, a top-up from your employer. If your pay isn’t counted towards EI insurable hours, you won’t be eligible for EI should you decide to have a kid and take some time off for maternity or paternity leave. And even if you are eligible for EI, it only covers a portion of your income, and given that postdoc salaries are already low, it may not be a livable amount. If your employer doesn’t provide a top-up to this amount1, it can be a big problem!
Where To Put Your Baby Once You Have It
If you manage to survive your maternity/paternity/parental leave on your meager EI (should you be lucky enough to get EI) and head back to the lab, you face something that all new parents who work outside the home face: daycare. Daycare spots are hard to come by2 and the ones that are available are expensive – I saw a poster recently that pointed out that a year of daycare costs more than a year of university!. For example, at UBC ChildCare Services, a month of daycare for a child between the ages of 4 and 18 months will cost you $1,105 – that’s $13,260 per year! And if you are taking home $35,000 a year with an NSERC/CIHR scholarship, that’s 38% of your take home income!
Beyond the Money
Even beyond all the financial implications of having a kid, there are implications to taking time off that are particularly important for academics.
I remember being in my Masters program and hearing one of my profs lament that so many females who could have gone on to be excellent scientists left academics to pursue other careers because they wanted to have a family. In his experience, a lot of females who did their Masters degrees went on to do a health profession degree, such as medicine or occupational or physical therapy, instead of doing a PhD because in those other fields, “if you take a year off for maternity leave, you can come back to your job and start up where you left off. In science, if you take a year off, you are left in the dust.”
Fast forward to my doctoral program: I remember having a conversation with a prof while we were putting together a grant application and her CV noted her two six-month leaves of absence for maternity leaves for her two kids. She said, “Things are better now than they used to be. Now you can put a maternity leave on your CV as a “reason for gaps in your productivity” and they have to accept it. You are still disadvantaged because if it comes down to you and someone else who didn’t take a leave, they have more papers and they’ll get the grant (or the job), but at least now you can write in your mat leaves.”
Academics is fast-paced and if you take time off during your postdoc to have a kid or two, you’ll have gaps in your publication and grant record and you’ll be behind on the latest advances in your field. Waiting until you’ve attained a faculty position – when you’ll at least have a more reasonable salary and probably even a maternity/paternity/parental leave top up – isn’t ideal either, as a new faculty member needs to be focused on getting tenure, so again gaps in your publication/grant record and getting behind in your field will be a problem. Wait until you have tenure, especially if you’ve done several years as a postdoc or research associate before landing that tenure-track job, and the fertility train may have passed you by!
Of course, all of this is just my thoughts on the prospect of having a baby while being an academic, based on discussions I’ve had with colleagues (some of whom have done it, some of whom have thought about it). I’m curious to hear what others’ thoughts/experiences are with this?
More About Communicating Science to NonScientists
April 8th, 2010I recently attended a Stats Can workshop and it struck me that a few of the things I learned about there would be useful here in our conversations about communicating science to nonscientists.
One was “20 Questions A Journalist Should Ask About Poll Results” from the National Council on Public Polls. While this list is geared towards polls specifically, a lot of the questions are relevant to scientific study in general. Things like “who did the research?” and “who paid for the research?” and “how were study subjects selected?” are all good questions for a reader to ask about a scientific study. So, if you are interested in educating journalists (or other nonscientists) in the types of questions to ask when they hear about a study, passing on that list may be a useful start.
Second, I learned about something called the fog index. Often academics get so used to their academic style of writing, they forget that lay audiences won’t know all the giant, specialized words they use. I’ve often joked that I specialize in “academic-to-English” translation for colleagues who are trying to write for a lay audience but can’t seem to see that, despite their best efforts, what they’ve written about their research is impenetrable to someone with fewer than two university degrees in their specialty. The fog index is a quick and dirty way to assess how readable your writing is. You calculate the fog index by:
“1. Tak[ing] a full passage that is around 100 words (do not omit any sentences).
2. Find[ing] the average sentence length (divide the number of words by the number of sentences).
3. Count[ing] words with three or more syllables (complex words), not including proper nouns (for example, Djibouti), familiar jargon or compound words, or common suffixes such as -es, -ed, or -ing as a syllable.
4. Add[ing] the average sentence length and the percentage of complex words”
5. Multiply[ing] the result by 0.4(Source)
As a formula, it looks like this:

The result of this calculation gives you an approximation of how many years of schooling you would need to understand the passage. For example, this blog posting has a fog index of 11, so someone would need at least a grade 11 education to be able to read it (by contrast, the fog index of the abstract of my thesis is 23 – meaning you’d need a PhD to understand it!)). It’s not a perfect measure, of course - not all words with three or more syllables are hard to understand and not all short words are easy to understand – but it does give you a sense of whether the news release or letter to your MP that you just wrote about your research would require a postgraduate education to decipher!
Evaluation vs. Research
March 27th, 2010I think I’ve mentioned on here before that I now work in the world of evaluation. One thing I’ve noticed since I’ve been in this world is the tendency of some to view “evaluation” and “research” as different things. I get a lot of confused “how did you get into evaluation after doing a PhD?” when I first tell people I left the academic world for that of health care. Which has always puzzled me, as I’ve always seen evaluation as just one form of research.
When you read about (or talk to) those who conceptualize research and evaluation as very different things, you basically here something along these lines: “research” is seen as something you do for the purpose of generating knowledge for knowledge’s sake (or “to learn more about how the world works”) while evaluation is done to assess the process and/or outcomes of a program (e.g., “do people learn what they are supposed to learn from this educational program?” or “does implementing this health promotion program result in a lower prevalence of childhood obesity?” or “how can we more efficiently implement this service?”). To my mind, the difference isn’t as clear as some might make it out to be. Certainly we can garner things from conducting evaluations that might be useful “knowledge for knowledge’s sake.” We use a lot of the same concepts and techniques in evaluation as one would in research. As I was thinking about writing this blog posting, a blog posting from the American Evaluation Society blog popped into my Google Reader: John LeVelle on Describing Evaluation. I thought the diagram there was an interesting way of looking at this (follow the link to check it out1) – though I would quibble with the notions that research is always “researcher-focused” (community-based research and participatory action research, for example, aren’t) and that all research starts with a “hypothesis” (this reflects a quantitative research bias – qualitative research doesn’t start with a “hypothesis”). The key thing I note from this diagram is that both research and evaluation use the same methods and data analysis techniques.
One of the implications of the perceived difference between research and evaluation is in the realm of institutional ethics review. Generally, if something is “research,” you have to go through an ethics review, whereas if something is “evaluation,” you don’t2. Now, I understand that if you want to review your files to check out how efficient your program is or make some process improvements, it would be unnecessary to go through a research ethics review. But in some cases an evaluation uses the exact same techniques (e.g., a survey, interview or focus group) as a research project does (and thus, one would expect the level of “risk” to the participants would be the same), and so it’s not really clear to me why whether or not you are publishing the results is the deciding factor as to whether an ethical review would be needed. At UBC the ethics review board is very clear that if you think there is even a possibility that you might want to publish the results of your evaluation work, you need to apply for ethics approval before you start, as they won’t accept ethics applications after the work has been conducted (e.g., you can’t say “we were doing an evaluation but it turned out that we have publishable findings, so now we want retroactive ethics approval”) – so at the very least, evaluation can’t be used a back door route to avoid the ethics process for research.
As per usual, I don’t have any solutions here – really just putting my preliminary thoughts on this out there. I welcome your thoughts/ideas/feedback!
And, on a related note, for anyone interested in an “alternative career”3 in evaluation may want to check out the upcoming Canadian Evaluation Society conference:
Canadian Evaluation Society 2010 Annual Conference
May 2 – 5; Victoria, British Columbia
Going Green, Gold, and Global: New Horizons For Evaluation
Version:1.0 StartHTML:0000000167 EndHTML:0000002743 StartFragment:0000000552 EndFragment:0000002727
Come to beautiful Victoria on the west coast of British Columbia for the 31st CES Annual Conference. Canada is a world leader in evaluation, and we invite you to come and hear what is happening in evaluation; and on our credential. Meet professionals who can provide insight for your evaluations and discuss the challenges we all face. Listen to our dynamic speakers and panels which include:
Our themes are:
We are expecting an inspiring conference. Online registration is now open and the early bird dead line is March 17! Special prices for students, seniors and members of CES, AEA and AES. Check us out at http://c2010.evaluationcanada.ca. See you here.
Talking Science to Non-Scientists and What’s In It For You?
March 4th, 2010Piggybacking off of Dave’s last post, which detailed ways that scientists can “help further the public knowledge of, and excitement about, scientific research,” with a specific focus on disseminating academic research and communicating with government, my posting today is going to take a look ways to increase science literacy and getting the general public excited about science. A while back I wrote about why science literacy is important and listed some of the science outreach organizations that are working to increase science literacy, so I’m not going to rehash that here. What I am going to talk about is:
Talking Science to Non-Scientists
I recently read a book called “Made to Stick” by Chip & Dan Heath1 that was all about how to get people to remember the things you say. They used examples from a variety of areas – from advertisers who try to get people to remember their products to teachers who want their students to remember how mitosis works. As I read the book, a lot of what they said resonated with me from my days as a science outreach volunteer2. They provide a framework in the form of the acronym “SUCCES,” which contains their keys to getting your message to “stick.” Using their framework, I have a few thoughts to share on “talking science to non-scientists”:
Talking To The Media
One specific group of people that I think scientists should really be talking to more is the media. The general public gets a lot of their scientific information from the media and journalists may or may not (and I think its more often the “not”) have any training in science. If you have the chance, it’s really worth getting to know journalists and helping to make the science they report on more accurate. I’ve had some media training (some good, some not so good3) and I think that more scientists should learn a bit about journalism and talk to the media when they can. And it should be a collaborative relationship between journalists and scientists, not one where each thinks the other is out to get them!
Things you can do
So now that you are all excited about promoting science literacy and getting people excited about science, what specific things can you do? Here are a few ideas:
Now it’s your turn. What things have you done to promote science literacy and/or get people excited about science? What tips do you have to share?
Knowledge Translation and Collaboration with Community
February 18th, 2010“Knowledge translation” is one of those buzz words you seem to hear a lot these days. To put it briefly, it basically means getting research results to be used in “the real world.” Traditionally, academics have disseminated their research results through publishing in academic journals and presenting at academic conferences, but of course the people who would actually “translate” research results into practice (e.g., in health care, this might be health care providers or administrators) don’t tend to be up on academic journals and conferences. “Knowledge translation” requires that one get their research results out to the people who can actually use them (let’s call them “decision makers” for the sake of simplicity) , in a way that is accessible to, and understandable by, those decision makers. Further, effective “knowledge translation” really should involve decision makers in the entire research process, from the inception of the research question, through the development of methods to the application of results. This, however, is a pretty tall order for a number of reasons. Those who work in health care, for example, often don’t have training in research methods. Nor do they have a lot of time (and we all know that research takes time!). On the flip side, academics don’t tend to have a good understanding of the day-to-day, on the ground realities of providing health care and sometimes their ideas just aren’t feasible given the constraints of time, finances, and human resources1.
A lot of research grant applications require the commitment of “community partners” or “decision maker applicants” – with the idea being that their involvement will help to ensure research results will be “translated” to practice. But I have to wonder how often are community partners truly engaged in collaboration with researchers and how often are they just asked to sign a letter saying that they are on board with the research merely to meet granting agency requirements for “letters of support”? I think it’s worth exploring ways in which we can make collaborative research truly collaborative.
I’m not claiming to have any answers to this – but I’d definitely be interesting in hearing about your thoughts on it!
