Canadian Politics Today

He's calling for a more standardized approach.

This doesn't work in research because data sources, research questions, and even research resources vary. As long as methods are clearly explained, someone educated in the field can understand what the limitations are.

It looks to me like the main thrust of his argument is that they very wildly, which suggests that the regulatory bodies that evaluate these studies don't appear to have clear standards in deciding what's good or not good.

One doesn't necessarily follow from the other. Varying methods doesn't undermine the findings. Varying methods just makes interpretation more work. Just because different studies use different methodologies, that doesn't mean their conclusions are not sound.

Maybe they just put the report on a desk and measure how thick it is.

Don't be facetious.

I don't think he's saying that any particular methodology is crap, just that there doesn't seem to be any way of telling which reports are crap and which are not.

They might all be good. It doesn't mean that some are crap and some aren't because they use different methods. It's just a matter of understanding what they're reporting and how they've come to those conclusions. It involves actually reading, understanding, and analyzing the material. All things the public doesn't have the time nor attention span to do.

There might not be settled science on these issues yet. And maybe going forward one can look at all these reports with these different methodologies and find which ones were better than others.

 -k

That's certainly what needs to be done. All I'm saying is that a wide range of methodologies is not necessarily a problem. If there is a problem with some of the methodologies then they should be pointed out. It's not enough to just observe that many different methods were used. In fact, using a wide range of methods and coming up with largely the same results would imply that the findings are accurate. If there's a wide range of results and just as much of a range of methods, then we need to take the time to analyze how the methods influence the results.