Name: Beth, aka "Beth"

Email:

Web Site: http://scienceadvocacy.org

Posts by Beth:

Not Linear

August 23rd, 2010

I had to create a presentation for a meeting at work and decided to try out Prezi, a new online presentation software that helps you to think about giving presentations differently than you would with PowerPoint. Unlike PowerPoint, which is a deck of slides presented in a linear way, Prezi is a giant canvas on which you can place words and images of various sizes, zooming in and out and around to present.  While I was using it, I came across this presentation called “Math is not linear” that I thought would be worth sharing here (Note: click the right-pointing arrow at the bottom of the presentation to advance through the slides):

Math is not linear on Prezi
I thought this was worth sharing here because it speaks to a lot of the issues that I’ve talked with my grad student and other teaching colleagues about teaching science in general, like:
• show students how the material relates to their lives
• think big picture
• present students with problems first (which then motivates students to learn, because they want to solve the problem at hand), rather than basing on memorizing formulae from the textbook and then asking them to solve a problem/answer a question about it. This is, after all, how the scientific method works.

I really wish more of my undergraduate education had been taught this way.  Because, honestly, I can still remember lots of things from the classes that I had that were taught this way (and we are talking a decade later), whereas for the classes I had that were taught in the traditional lecture format and tested via regurgitation-type exams – I don’t seem to be able to remember a thing!

I’d love to hear your thoughts on how we approach the teaching of science in the comments section!

Note: We have no association with Prezi.com, other than that I am a user of the service. We aren’t making any cash or anything for blogging about them – I just think it’s a pretty cool product!

3 Comments "

So, you want to be a Project Manager when you grow up…

August 8th, 2010

My first introduction to the world of project management as a field was in my previous job, when I took an intro to project management workshop offered.  After taking the workshop, I really wished I’d learn about this field during my PhD instead of after it, because the concepts and methods of project management are super useful!  But I’m getting ahead of myself.  First, some definitions:

“Project management is the discipline of planning, organizing, and managing resources to bring about the successful completion of specific project goals and objectives. [...] The primary challenge of project management is to achieve all of the project goals and objectives while honoring the preconceived project constraints (e.g., scope, time, budget)”

and

“A project is a temporary endeavor, having a defined beginning and end (usually constrained by date, but can be by funding or deliverables), undertaken to meet unique goals and objectives, usually to bring about beneficial change or added value. The temporary nature of projects stands in contrast to business as usual (or operations), which are repetitive, permanent or semi-permanent functional work to produce products or services.”¹.

Project managers, then, are responsible for “planning, organizing, and managing resources” of “temporary endeavors” with specific goals (like, say, a research project). I’m definitely no expert on the topic, but it seems that project managers traditionally worked in areas like construction and engineering, but are starting to become more common in science.  I mean, I certainly was the “project manager” for my own thesis, but I managed things based on my best guesses/following along with what I saw scientists doing, not based on any “project management” methods.  Project managers in research institutes, as I understand it, separate out the planning, monitoring and execution of tasks towards the deliverables, from the science per se.  They are the ones making sure you have access to the people and materials you need, when you need them, to conduct your research; making sure you stay within the intended scope of the project; and making sure you get the job done, to the necessary level of quality by the needed date (or identifying at the start that it won’t be possible to do what you want to do by the time you want to do it!).  Which frees up the scientist to do their science. So if you don’t want to run your own lab, but still enjoy working in the research field, a job as a project manager might be up your alley.

If you are interested in finding out more about project management, I’d suggest checking out the Project Management Institute. They are the people who produce  “A Guide to the Project Management Body of Knowledge” (the bible of PM) and offer a slew of credentials in PM.  You may also want to check out this open source Project Management handbook, because, you know, it’s open source, so you can check it out without having to buy it.

Like I said, I’m not an expert on this topic, so if there are any project managers in the audience, please let us know your thoughts or tips!

Image credit: From Wikimedia Commons, in the public domain.

1. The source of both these quotations is the Wikipedia page on project management [↩]

No Comments "

So, you want to be a Program Evaluator when you grow up…

July 22nd, 2010

Since Dave has started us off on the “So you want to be a blank when you grow up” series, I’m going to take the lowest of the low hanging fruit and tell you all about what I’ve learned since becoming a program evaluator!  But first I’d like to mention two things:

• If there is a type of career you are particularly interested in, let us know and we’ll see what we can dig up.
• If you are a PhD who hasn’t followed the tenure track and would like to share your experiences with our readers in this series, please let us know. We’d love more guest posters!

— Now, onto the exciting world of program evaluation! So, you want to be a Program Evaluator when you grow up… Evaluation is a great career for a PhD, because it allows you to use the skills you developed in your training – research skills, problem solving, critical thinking, project management, and more. Though I’d not heard of evaluation as a field per se during my education, it actually is quite established as a profession.  If you want to get a good sense of what evaluation is all about, I’d suggest checking out:

• Canadian Evaluation Society
• American Evaluation Association
• European Evaluation Society

Each of these societies’ websites have a wealth of information about evaluation, including things like evaluation standards of practice, literature, conferences and workshops, and, importantly, job postings and requests for applications for contracts.  Which brings us to the question:

In-House Evaluator or Independent Consultant?

There are pros and cons to both, of course.  Working in house (like I do) offers things that are appealing both personally (like a stable income and benefits) and professionally (like the ability to have a thorough understanding of the programs you evaluate, the opportunity to work with programs over the long-term and see them flourish, and a chance to build evaluation capacity in your sector¹.  Working as an independent consultant offers benefits like flexibility of when and where and on what you work and the chance to make a lot more money than you will in-house.  Apparently there is a tendency for evaluators to flip between in house and independent throughout their careers (as one colleague described it to me, you get sick of the bureaucracy of working in-house, so you go out on your own for about 5 years, then you get lonely from working on your own for so long, so join an organization. And repeat).  You can, of course, combine the two – for example, having an in-house job and do small evaluation projects on contract on the side).  And working for an evaluation firm – where you have some security, benefits, etc., but some of the flexibility of working on your own – is a possibility as well.

Training in Program Evaluation

There are a number of training opportunities to learn more about program evaluation and to build particular skills.  The above-named societies offer a variety of training workshops, as does the Evaluators’ Institute. As well, if you are interested in doing more schooling, you can get graduate-level training in program evaluation at:

• University of Victoria
• University of the Fraser Valley
• Claremont Graduate University

As well, the CES has recently launched a credentialing program, and though it’s still early days, I think it will be well worth working towards obtaining that credential, especially if you plan to work as an independent consultant. And a few resources worth checking out: In addition to the wealth of resources on the CES, AEA, EES websites, here are a few things that I would recommend reading:

• Michael Patton’s Utilization-focused Evaluation²
• Kellogg Foundation Logic Model Development Guide
• Consulting and evaluation with nonprofit and community-based organizations by Judah J. Viola, Susan Dvorak McMahon.

Final words: Like Dave said about science writing, it’s worth starting to build your portfolio of evaluation projects early. You can certainly use aspects of your graduate research as a demonstration of your skills, but picking up some small evaluation projects while you are in school – perhaps even working under a more established evaluator as a mentor – would be good way to start off your portfolio.

1. if such a thing appeals to you, like it does for me. [↩]
2. Patton has also recently published a book called Developmental Evaluation – I haven’t read it yet, but it’s probably worth checking out [↩]

2 Comments "

Federal Government Scrapping The Census Long-Form: What Will It Mean for Evidence-Based Policy?

July 8th, 2010

A friend of mine just sent me a link to this news story and it’s gotten me quite livid:

Tories scrap mandatory long-form census
StatsCan says quality of data will suffer

Every five years, Canada conducts a census, with the next one scheduled for 2011.  In the past, every household received the short census form, which contains just a few questions (like number of people in the home and their age and sex) and 1 in every 5 households received a mandatory long form.  The long form contained questions about a variety of things, like income, education, and ethnicity, and provided a lot of really important information about the population of Canada.  For example, I work in health care and we use census data all the time.  Knowing the makeup of our population allows us to make informed decisions about providing health care to meet the needs of the people living in our region. All levels of governments (municipal, provincial and federal), community agencies, and other organizations use the data from the census long form to develop evidence-informed policy.

But the ability to do that now at risk, as the federal government has, apparently without consulting anyone, decided to scrap the mandatory long form, citing that the long form represented “what most Canadians felt was an intrusion into their personal privacy in terms of answering the longer form” (Source) – though I haven’t seen from anything I’ve read thus far how they determined that this is how “most” Canadians feel.  Instead, they are replacing it with a “voluntary household survey” that will be sent to 1 in 3 households, the members of which can choose to complete – or not to complete – the survey.  This raises very serious concerns about the quality of the data – the people who choose not to respond to the long survey may be different from those who choose to respond to it, which will result in skewed information.  Which means we won’t have the data we need to make policies and provide appropriate services.

“Senior statisticians at Statistics Canada have conceded the change will make it more difficult to obtain reliable, detailed information.” (Source)

All of this brings up questions about politicians’ understanding of the importance of data and evidence-informed practice, not to mention their ignoring the scientific experts on the matter – in this case, the statisticians at Stats Canada. As David Eaves noted in his article, “Why you should care about the sudden demise of the mandatory long census form”:

“This is a direct attack on the ability of government to make smart decisions. It is an attack on evidence-based public policy. Moreover, it was a political decision – it came from the minister’s office and does not appear to reflect what Statistics Canada either wants or recommends. Of course, some governments prefer not to have information; all that data and evidence gets in the way of legislation and policies that are ineffective, costly and that reward vested interests (I’m looking at you, tough-on-crime agenda). ” [emphasis mine; Source]

In another news article on the topic I read:

Liberal MP Marlene Jennings “argued that Clement has shown in postings to the social media site Twitter that he does not understand how the mandatory nature of the long-form census allows Statistics Canada to properly weight the short form data. Clement debated sample size and data weighting with other posters, including an economist.

“(That’s) something Mr. Clement seemed not to understand when he was tweeting yesterday, so maybe he should take a stats course,” she said” (Source)

This lead me to check out Clement’s Twitter stream, where he referred to having a mandatory (as opposed to voluntary) long form as “state coercion” – I guess it’s fine to have the state “coerce” you to complete the short form – or, you know, obey any of our other laws – but they draw the line at the long form?

Now, as Dave mentioned in his most recent posting, we here at The Black Hole don’t want to just complain about problems, but rather want to focus on what we can do about addressing those problems.  To this end, I’d like to note that the Liberals are demanding the Conservatives reverse this decision to scrap the mandatory long form and, if they don’t, to introduce legislation to protect it themselves.  Personally, I’ll be writing to my own MP and Industry Minister Tony Clement to inform them about why the long form is so vital.  And I’ll be writing to some Liberals to suggest that they stick to their guns on this one and introduce legislation to protect the mandatory long form.

Tip:  Don’t know who your MP is?  You can search here to find out!

Update: There is now a petition on this issue. Check out: http://www.gopetition.com/petitions/keep-the-canadian-census-long-form.html

10 Comments "

What Does A Graduate Degree Mean?

June 22nd, 2010

I was at the recent Canadian Evaluation Society conference where they revealed the process for obtaining their new “credentialed evaluator (C.E.)” designation. In order to become a “C.E.,” one must demonstrate through education and experience that they are qualified as an evaluator.  Since there are few formal educational programs in “evaluation”¹, most evaluators do not have a degree in “evaluation,” but rather are educated in some other field².  Thus, the formal education requirement for the C.E. is that you hold a graduate level degree or certificate.  The reason given for this requirement was that by earning a graduate-level education, one has demonstrated that they are capable of critical thinking, analysis and research.  Despite my advanced skills in research³, I’ve been unable to locate any formal source that states what a graduate degree is meant to signifies, so I’m falling back on Wikipedia to give you this:

“A master’s degree is an academic degree granted to individuals who have undergone study demonstrating a mastery or high-order overview of a specific field of study or area of professional practice. Within the area studied, graduates possess advanced knowledge of a specialized body of theoretical and applied topics; high order skills in analysis, critical evaluation and/or professional application; and the ability to solve complex problems and think rigorously and independently.” (Source. Emphasis mine)

This got me thinking, yet again, about countless discussions I’ve had with other grad students about what it really should take to get a graduate-level degree.  There seems to be a lot of variation in what it takes to get that “MSc” or “PhD” behind your name.  I’ve seen some Masters students who conducted original research from the initial research idea through all the stages of study design, implementation, data analysis and writing; but I’ve also seen Masters students walk right into an already designed study and really only needed to implement a study protocol that their supervisor created ⁴.  Or those who sent their samples off to another lab for analysis (or have undergraduate minions who do it for them⁵ and never really learned the ins and outs of the techniques they were supposed to be an “expert” in.

So, I’m curious, what has been your experience?  And if you were hiring someone with a Master’s degree or a PhD, what would you expected of them, based on their holding that degree?

Footnotes:

1. Claremont Graduate University in California seems to be one of the most prominent places to get a degree in evaluation, though more places are starting (or planning) to offer such a thing… but that’s the subject of another posting! [↩]
2. e.g., my formal education is in Nutrition, and other evaluators I know have degrees in a wide variety of fields, including Social Work, Psychology, Kinesiology, Criminology, and History, just to name a few [↩]
3. ;-) [↩]
4. full disclosure: my Master’s program was a 12-month “coursework and a project” Master’s, not a thesis-based Master’s [↩]
5. I never did figure out where one gets undergraduate minions… they would have made my life so much easier during my PhD! [↩]

No Comments "

Why Do Scientists Blog?

June 9th, 2010

One of the ideas I’ve been tossing around for a blog posting here is: Why do scientists blog?  I mean, I’ve been a personal blogger for nearly five years and have been blogging here at The Black Hole since October, surely I have some insights on why people do this, right? Yet I’d been struggling to come up with a coherent explanation…. and then one of the science bloggers that I read went and wrote a five-part series on exactly this¹.  Some of the reasons given there (both by the blogger and by the blog readers in the comments section):

• It doesn’t taken long to whip up a blog posting, but when you need a short break from your work,  it’s a more productive way to spend 5-10 minutes between doing other work (as opposed to, say, surfing the ‘net).  How is it productive? you ask… read on!
• Blogging allows you to be part of a community. Thanks to the web, it’s easy to find someone who is going through (or has just gone through or is about to go through) the same thing as you.  So even if you are in a small department with no other people at the same stage of career as you, you can find the support/advice/motivation you need.  Whether you are a PhD student considering whether or not to go the academic route, a newbie on the tenure-track, someone who has left academics, etc., you will be able to find others like you in the blogosphere – and you will be able to help out others in the same way.
• Blogging can get energized about your work.  Sometimes you need a forum to vent or to ruminate.  Blogging about problems with your research (or your career track) can sometimes help you figure out the next steps to take.
• Blogging can help you learning about other people’s perspectives.  Because writing a blog doesn’t stop at writing a blog – you’ll get comments from readers that really make you think. Then you’ll follow the links of those readers to their blogs and learn all sorts of new stuff there.

Now, those ideas come from a blog focused on the process of being a tenure track prof, rather than a blog about the science itself.  Plenty of scientists use a blog to talk about their work and for them it’s easy to see benefits like:

• raising awareness about your own line of research
• communicating with the public about science (and all the benefits that go along with that)

So now I put the question to you – do you blog?  If so, why?

Some interesting blogs you may want to check out²

• Alternative Scientist - a group written blog by people who didn’t follow the academic path (it hasn’t been very active lately, but there are some good postings in the archives)
• FrogHeart - “Commentary about nanotech, science policy and communication, society, and the arts”
• Nature.com blogs – a whole slew of scientists blog here
• Nerd Girl’s blog on the Globe & Mail site
• Prof-Like Substance - “Because being early tenure track faculty is even stranger than you think”
• UBC Faculty Association blog (this one is focused on the Faculty Association’s current round of bargaining with the university as a way of keeping the faculty members informed)

1. If you are interested, I suggest you check out his series “How Blogging Helps Me”: parts 1, 2, 3, 4 and 5. [↩]
2. feel free to recommend other blogs in the comments section too! [↩]

1 Comment "

Why does anyone think science is a good job?

May 24th, 2010

“A science researcher at Harvard now earns an annual salary that is only 1/50th the price of a family-sized house in Cambridge, a fact that may not be lost on an intelligent female Harvard undergraduate choosing a career.” (Source)

The announcement of the new Canada Excellence Research Chairs (CERC) – 19 scientists granted $10 million over 7 years to establish research program in Canada – has raised some eyebrows for its notable lack of women among the recipients.  We’ve talked a bit about this issue here before, notably about the mat leave issue and the disadvantage one faces in a career in academics should they choose to take time off to have kids.  But while reading some of the articles and blog postings about the CERCs, I came across this interesting article, which poses the question: “Why does anyone think science is a good job?”

From the article:

The average trajectory for a successful scientist is the following:

1. age 18-22: paying high tuition fees at an undergraduate college
2. age 22-30: graduate school, possibly with a bit of work, living on a stipend of $1800 per month
3. age 30-35: working as a post-doc for $30,000 to $35,000 per year
4. age 36-43: professor at a good, but not great, university for $65,000 per year
5. age 44: with (if lucky) young children at home, fired by the university (“denied tenure” is the more polite term for the folks that universities discard), begins searching for a job in a market where employers primarily wish to hire folks in their early 30s

Not to mention that academics who make the above salaries aren’t just putting in a 40-hour work week with four weeks paid vacation and stat holidays that aren’t spent working on the next grant application, as they would be in another, higher paying, career. Put like that, it’s not hard to see why I – and many people like me – have gotten off the academic career track, regardless of whether or not we have/plan to have kids.  In fact, when I see the fierce competition that recent PhD grads and post-docs face trying to land a tenure-track position, it makes me wonder why so many people are willing to fight so hard for what, in many ways, is a thankless job¹.

Now, don’t get me wrong. I’m not saying that no one should ever become an academic researcher.  In fact, I think that academic research is essential – we can’t leave science solely in the hands of industry, for example.  But I think we need to give some serious thought to the compensation and work-life of our academic scientists. So while we talk about the ways to advocate for appropriate compensation and benefits for postdocs to help keep good scientists in the game – a very important issue, to be sure – we also have to be thinking about what comes after that for those postdocs who are lucky enough (or is that “unlucky” enough?) to land a tenure-track gig.

1. Maybe in the comments section those of you who have chosen to stay on the academic career path can shed some light onto why you think being an academic scientist is a good job? [↩]

1 Comment "

Making Little Scientists: The Plight of the Postdoc Who Wants To Have a Family

April 22nd, 2010

A comment on one of Dave’s recent postings got me thinking about an “issue affecting trainees” that we haven’t yet talked about here on The Black Hole: babies.  Specifically, the having of babies and where to keep your babies while you are in the lab.

Having Babies

PhDs take a long time and by the time you are done that and have started a postdoc or two, you might start thinking you are getting to an age where you’d like to have a family!  So, the discussion about whether postdocs are “employees” or “trainees” has implications beyond the “taxable or not” question.  What kinds of benefits trainees get is important because maternity/paternity/parental leave is funded primarily by Employment Insurance (EI) and, if you are lucky, a top-up from your employer.  If your pay isn’t counted towards EI insurable hours, you won’t be eligible for EI should you decide to have a kid and take some time off for maternity or paternity leave.  And even if you are eligible for EI, it only covers a portion of your income, and given that postdoc salaries are already low, it may not be a livable amount.  If your employer doesn’t provide a top-up to this amount¹, it can be a big problem!

Where To Put Your Baby Once You Have It

If you manage to survive your maternity/paternity/parental leave on your meager EI (should you be lucky enough to get EI) and head back to the lab, you face something that all new parents who work outside the home face: daycare.  Daycare spots are hard to come by² and the ones that are available are expensive – I saw a poster recently that pointed out that a year of daycare costs more than a year of university!.  For example, at UBC ChildCare Services, a month of daycare for a child between the ages of 4 and 18 months will cost you $1,105 – that’s $13,260 per year!  And if you are taking home $35,000 a year with an NSERC/CIHR scholarship, that’s 38% of your take home income!

Beyond the Money

Even beyond all the financial implications of having a kid, there are implications to taking time off that are particularly important for academics.

I remember being in my Masters program and hearing one of my profs lament that so many females who could have gone on to be excellent scientists left academics to pursue other careers because they wanted to have a family.  In his experience, a lot of females who did their Masters degrees went on to do a health profession degree, such as medicine or occupational or physical therapy, instead of doing a PhD because in those other fields, “if you take a year off for maternity leave, you can come back to your job and start up where you left off. In science, if you take a year off, you are left in the dust.”

Fast forward to my doctoral program: I remember having a conversation with a prof while we were putting together a grant application and her CV noted her two six-month leaves of absence for maternity leaves for her two kids.  She said, “Things are better now than they used to be.  Now you can put a maternity leave on your CV as a “reason for gaps in your productivity” and they have to accept it. You are still disadvantaged because if it comes down to you and someone else who didn’t take a leave, they have more papers and they’ll get the grant (or the job), but at least now you can write in your mat leaves.”

Academics is fast-paced and if you take time off during your postdoc to have a kid or two, you’ll have gaps in your publication and grant record and you’ll be behind on the latest advances in your field.  Waiting until you’ve attained a faculty position – when you’ll at least have a more reasonable salary and probably even a maternity/paternity/parental leave top up – isn’t ideal either, as a new faculty member needs to be focused on getting tenure, so again gaps in your publication/grant record and getting behind in your field will be a problem.  Wait until you have tenure, especially if you’ve done several years as a postdoc or research associate before landing that tenure-track job, and the fertility train may have passed you by!

Of course, all of this is just my thoughts on the prospect of having a baby while being an academic, based on discussions I’ve had with colleagues (some of whom have done it, some of whom have thought about it).  I’m curious to hear what others’ thoughts/experiences are with this?

1. I don’t have any data on this, but I’m willing to bet that few, if any, universities provide maternity/paternity top-ups as a benefit to their postdocs [↩]
2. people start to get on wait lists for daycare during their pregnancy, even if they aren’t planning to send the kid to daycare until after a full year of maternity leave [↩]

7 Comments "

More About Communicating Science to NonScientists

April 8th, 2010

I recently attended a Stats Can workshop and it struck me that a few of the things I learned about there would be useful here in our conversations about communicating science to nonscientists.

One was “20 Questions A Journalist Should Ask About Poll Results” from the National Council on Public Polls.  While this list is geared towards polls specifically, a lot of the questions are relevant to scientific study in general.   Things like “who did the research?” and “who paid for the research?” and “how were study subjects selected?” are all good questions for a reader to ask about a scientific study.  So, if you are interested in educating journalists (or other nonscientists) in the types of questions to ask when they hear about a study, passing on that list may be a useful start.

Second, I learned about something called the fog index. Often academics get so used to their academic style of writing, they forget that lay audiences won’t know all the giant, specialized words they use.  I’ve often joked that I specialize in “academic-to-English” translation for colleagues who are trying to write for a lay audience but can’t seem to see that, despite their best efforts, what they’ve written about their research is impenetrable to someone with fewer than two university degrees in their specialty.  The fog index is a quick and dirty way to assess how readable your writing is.  You calculate the fog index by:

“1. Tak[ing] a full passage that is around 100 words (do not omit any sentences).
2. Find[ing] the average sentence length (divide the number of words by the number of sentences).
3. Count[ing] words with three or more syllables (complex words), not including proper nouns (for example, Djibouti), familiar jargon or compound words, or common suffixes such as -es, -ed, or -ing as a syllable.
4. Add[ing] the average sentence length and the percentage of complex words”
5. Multiply[ing] the result by 0.4(Source)

As a formula, it looks like this:


The result of this calculation gives you an approximation of how many years of schooling you would need to understand the passage. For example, this blog posting has a fog index of 11, so someone would need at least a grade 11 education to be able to read it (by contrast, the fog index of the abstract of my thesis is 23 – meaning you’d need a PhD to understand it!)).  It’s not a perfect measure, of course -  not all words with three or more syllables are hard to understand and not all short words are easy to understand – but it does give you a sense of whether the news release or letter to your MP that you just wrote about your research would require a postgraduate education to decipher!

1 Comment "

Evaluation vs. Research

March 27th, 2010

I think I’ve mentioned on here before that I now work in the world of evaluation.  One thing I’ve noticed since I’ve been in this world is the tendency of some to view “evaluation” and “research” as different things.  I get a lot of confused “how did you get into evaluation after doing a PhD?” when I first tell people I left the academic world for that of health care.  Which has always puzzled me, as I’ve always seen evaluation as just one form of research.

When you read about (or talk to) those who conceptualize research and evaluation as very different things, you basically here something along these lines:  “research” is seen as something you do for the purpose of generating knowledge for knowledge’s sake (or “to learn more about how the world works”) while evaluation is done to assess the process and/or outcomes of a program (e.g., “do people learn what they are supposed to learn from this educational program?” or “does implementing this health promotion program result in a lower prevalence of childhood obesity?” or “how can we more efficiently implement this service?”).  To my mind, the difference isn’t as clear as some might make it out to be.  Certainly we can garner things from conducting evaluations that might be useful “knowledge for knowledge’s sake.”  We use a lot of the same concepts and techniques in evaluation as one would in research. As I was thinking about writing this blog posting, a blog posting from the American Evaluation Society blog popped into my Google Reader: John LeVelle on Describing Evaluation.  I thought the diagram there was an interesting way of looking at this (follow the link to check it out¹) – though I would quibble with the notions that research is always “researcher-focused” (community-based research and participatory action research, for example, aren’t) and that all research starts with a “hypothesis” (this reflects a quantitative research bias – qualitative research doesn’t start with a “hypothesis”). The key thing I note from this diagram is that both research and evaluation use the same methods and data analysis techniques.

One of the implications of the perceived difference between research and evaluation is in the realm of institutional ethics review.  Generally, if something is “research,” you have to go through an ethics review, whereas if something is “evaluation,” you don’t². Now, I understand that if you want to review your files to check out how efficient your program is or make some process improvements, it would be unnecessary to go through a research ethics review. But in some cases an evaluation uses the exact same techniques (e.g., a survey, interview or focus group) as a research project does (and thus, one would expect the level of “risk” to the participants would be the same), and so it’s not really clear to me why whether or not you are publishing the results is the deciding factor as to whether an ethical review would be needed.  At UBC the ethics review board is very clear that if you think there is even a possibility that you might want to publish the results of your evaluation work, you need to apply for ethics approval before you start, as they won’t accept ethics applications after the work has been conducted (e.g., you can’t say “we were doing an evaluation but it turned out that we have publishable findings, so now we want retroactive ethics approval”) – so at the very least, evaluation can’t be used a back door route to avoid the ethics process for research.

As per usual, I don’t have any solutions here – really just putting my preliminary thoughts on this out there.  I welcome your thoughts/ideas/feedback!

---

And, on a related note, for anyone interested in an “alternative career”³ in evaluation may want to check out the upcoming Canadian Evaluation Society conference:

Canadian Evaluation Society 2010 Annual Conference

May 2 – 5; Victoria, British Columbia

Going Green, Gold, and Global: New Horizons For Evaluation

Version:1.0 StartHTML:0000000167 EndHTML:0000002743 StartFragment:0000000552 EndFragment:0000002727

Come to beautiful Victoria on the west coast of British Columbia for the 31^st CES Annual Conference. Canada is a world leader in evaluation, and we invite you to come and hear what is happening in evaluation; and on our credential. Meet professionals who can provide insight for your evaluations and discuss the challenges we all face. Listen to our dynamic speakers and panels which include:

• Keynote: The Power of One: Simon Jackson (one of Time Magazine’s 60 Heroes of the Planet)
• Keynote: Using Olympic Principles for Success: Jennifer Walinga (Double Olympic Gold Medalist)
• Panel: Los Tres Amigos – Global Excellence in Monitoring and Evaluation – Robert Lahey (representing Canada), John Pfeiffer (USA), and Agustin Caso (Mexico)
• Keynote Panel: Summarizing and Celebrating the Conference – Four Evaluation Association Presidents – Nancy Porteous, Hallie Preskill, Florence Etta, Linda Lee.

Our themes are:

• Green – Evaluation in the context of managing positive environmental change.
• Gold – Evaluation of major events (e.g., Olympic Games) and striving for excellence in evaluation.
• Global – current trends and developments in international evaluation.

We are expecting an inspiring conference. Online registration is now open and the early bird dead line is March 17! Special prices for students, seniors and members of CES, AEA and AES. Check us out at http://c2010.evaluationcanada.ca. See you here.

1. as there doesn’t seem to be a Creative Commons license on there, so I couldn’t put the diagram into this posting [↩]
2. unless you want to publish the findings in an academic journal that requires an ethical review to have been conducted. Or if you have an external funder that requires it [↩]
3. though we don’t like calling any job a PhD holds other than a professorship an “alternative career,” which suggests that the professorship is the “right” job, we haven’t actually thought of a good name for this.  Any suggestions? [↩]

No Comments "