A survey created by Metro and sent to their nearly 10,000 member Opt In Panel last month has drawn the ire of non-profits, local business owners, and citizen advocates — 26 of whom have signed onto a letter outlining their concerns that was sent to Metro President Tom Hughes and members of the Metro Council today.
When the survey was released at the end of December, we shared the negative reactions it received from many in the local transportation advocacy world. Many who took the survey, myself included, cringed at the content and framing of several of the questions.
The letter, coordinated by Coalition for a Livable Future (CLF) urges Metro to “disavow” the “flawed” survey. Adding strength to the concerns are 18 businesses and nonprofit organizations and 8 individuals who signed onto the letter (all of whom are usually Metro allies). The signatories include: Ron Carley, CLF’s Executive Director; Mary Kyle McCurdy, Policy Director, 1000 Friends of Oregon; Jim Labbe, Urban Conservationist Audubon Society of Portland; Gerik Kransky, Advocacy Director, Bicycle Transportation Alliance; Caitlin Baggott, Executive Director, The Bus Project; Chris Hagerbaumer, Deputy Director, Oregon Environmental Council; Mike Houck, Director Urban Greenspaces Institute; Steph Routh, Executive Director, Willamette Pedestrian Coalition; economist Joe Cortright, and others.
“Misleading and skewed questions perpetuate the false dichotomy that the region must choose either environmental protection or economic development as its priority for the future, a dichotomy that Metro and the region’s residents have rejected many times.”
— From the letter
Here are some excerpts from the letter (available as PDF here):
“Misleading and skewed questions perpetuate the false dichotomy that the region must choose either environmental protection or economic development as its priority for the future, a dichotomy that Metro and the region’s residents have rejected many times…
Several questions in the survey directly pit a strong economy against a healthy environment. This is a specious proposition; these two goals have never been in opposition. The survey offers no option for residents who believe that a high quality environment is essential to a strong economy. Experience and evidence demonstrate that productive businesses and talented people choose to live in communities that have a healthy environment, and that protecting environmental quality creates good-paying, stable jobs for the region’s residents while saving taxpayers’ money from being wasted on unnecessary, sprawling infrastructure.
…
The questions on regional infrastructure funding priorities and the Columbia River Crossing also require numerous false choices and read like a push poll. A question on regional transportation funding priorities forces readers to indicate which transportation mode they do not support “at all,” creating an unnecessary “us vs. them” dynamic between transportation modes. This “choice” eliminates the possibility for residents to indicate they believe the region should pursue a multimodal, balanced approach to meeting its mobility needs…”
…
By implying –with no basis in fact–that jobs and environmental quality are mutually exclusive goals, the Metro Opt-In poll transforms what could be useful opinion research into research that yields meaningless and misleading results.
If any metropolitan region can demonstrate to the nation and the world that economic development, environmental protection, and equity are mutually accomplishable goals, the Portland region can, and should.
The letter calls on Metro to not only “disavow this flawed Opt In survey” but also to “affirm its commitment” to a healthy environment and vibrant economy.
When asked about the survey last month, Metro’s Director of Communications, Jim Middaugh acknowledged they were “attempting to provoke a bit” and said the forced-choice style of questioning was an attempt to “get at underlying values.”
Download a PDF of the letter here.
Thanks for reading.
BikePortland has served this community with independent community journalism since 2005. We rely on subscriptions from readers like you to survive. Your financial support is vital in keeping this valuable resource alive and well.
Please subscribe today to strengthen and expand our work.
BTA didn’t sign?
Nevermind, I guess my speed-reading ain’t what it used to be.
This survey put me off so much that I do not have much desire to engage in any of their other surveys again.
Hi folks. Thanks for the interest in Opt In and for all the posts. Here’s some more information for your consideration…sorry for the length but there’s lots to cover. For those who are interested solely in my reaction to CLF’s letter please jump to the end.
Here’s some background on the recent Opt In survey and methodological context for the questions:
The survey is part of Metro’s ongoing effort to recruit Republicans and residents of Clackamas and Washington counties, areas in which the current panel membership is lacking. Survey topics were intended to attract these groups to join the panel.
Every page of the survey had a comment box where respondents could add context to their answers, or give feedback about the questions. This is a standard practice for all Opt In surveys and the comments are used for added qualitative perspective on the results.
Rating: Primary Focus and Not a Focus
It is common practice in public policy, customer satisfaction, and product surveys to ask people what is most important to them from a list of options, and what is not important to them in that list. These questions are designed to get a sense of priority and avoid a situation in which each item is listed as “very” important (as could happen using a traditional “very important to not at all important” scale). In addition to the comment box, a “don’t know” option was given for both these questions.
Ranking: Most Important to Least Important
Forced ranking is commonly employed to determine people’s priorities for budgets and infrastructure plans because it avoids a situation in which all items are given a similar rating. While all items may have equal importance to respondents, in planning and budget realities, it is rare that all items can be given equal focus. Therefore, determining priorities is essential. In analysis, the item given the highest “least” important percentage is not considered unimportant, but instead the lowest priority within a list of important items. Again, when we used this technique we also gave respondents a “don’t know” option as well as a comment box.
Forced Choice: Environment vs. Economy
Forced choice statements are also common practice in surveys, and are appropriate when other, non-forced choices are included to add context. The purpose of forced choice is not to get at hard and fast beliefs, but to provide an understanding of how respondents lean when only offered two choices, again to get a sense of where their priorities fall when facing difficult decisions.
Focusing on the environment vs. focusing on the economy were paired against each other. Respondents were also given a “don’t know” option. This question was complimented by a rating question later on in the survey in which people were asked what the focus should be on a scale of 1 (economic growth) to 7 (environment), or 4 (both equally). An example of forced choice can be found on page 2 of this Pew survey: http://www.people-press.org/files/legacy-questionnaires/11-10-11%20Energy%20topline.pdf
In terms of a comment on CLF’s letter:
First of all, I’m solely responsible for the decision to field the survey. The Metro Council didn’t see it or approve it before it was sent out. If people are critical of the survey their criticism rightfully should be aimed at me.
Notwithstanding the fact that many surveys use similar techniques I think CLF makes some good points and I’m grateful to them for sending the letter.
Opt In is a new and experimental approach to public involvement and Metro is still learning how best to use it. The fact that we’re having a debate, in my opinion, is an improvement over other methods Metro has used that generated little awareness or dialogue.
I heard from a lot of people, including a number of my family members, friends and Metro colleagues, who agree with CLF that forced choices and forced rankings don’t do justice to the complexity of the challenges facing our region or the intelligence of the people who live here. I’ve definitely learned from that reaction and as a result I think our surveys will be better going forward.
It’s exciting to me that we have nearly 10,000 people (9,628 as of 2:50 p.m.) signed up to participate in Opt In. It’s also exciting that for the first time Metro is publishing demographic information about the people it hears from so anyone who is interested can see the populations we are missing.
And, we are clearly missing some, particularly suburban Republicans, young people, people with high school educations or less, and minorities. All of us at Metro are working hard to address those gaps. You can learn about the demographics of the panel in real time at the “who’s joined” tap at optinpanel.org.
I believe Opt In represents an honest attempt to improve transparency and accountability at Metro. The jury is still out on whether or not we can make it work over time.
Did I just understand you right, Jim? Did you just say that you wrote biased and inflammatory questions for the Opt-In survey for the purposes of getting Republicans to join the panel? That seems like a bizarre and deeply troubling use of a regional survey instrument.
They wanted to include the people living in the urban area who don’t want to live in an urban area. The sort of people that hate it here and spout phrases like “Peoples Republic of Portland”.
I support their freedom to not to live in urbanized areas and their right to believe and speak what they want to about well organized urban living conditions.
I also support that the consequences of that belief should apply as well: no schools, no police, no fire fighters, no EMS, slow internet access, neglected road conditions.
In essence it is this: modern life in America rests upon an infrastructure of technology, machines and construction. If you don’t like that go live in the woods like Les Stroud.
“And, we are clearly missing some, particularly suburban Republicans…”
So, you wanted more suburban republicans to join in so you you hired an Oregonian headline writer to write the questions? Brilliant! But seriously, changing the wording of questions to attract one demographic is seriously wrong. The questions should be worded neutrally and not enflame people’s preconceived prejudices. I took this survey and I left angry comments on many of the questions because they were worded so unfairly. Thanks for taking the blame, I hope you learn something from the response you’re getting. Maybe go to PSU and take a course in public polling.
They want to “get at underlying values”.
Here’s an old fashioned value that I think applies:
“The first thing we do, let’s kill all the lawyers!”
“…If people are critical of the survey their criticism rightfully should be aimed at me….”
Metro wants more participation from low information constituents and republicans who are eager to hear misinformation about the conflict between the environment and jobs which will agree with their preconceived notions. That sounds great, with enough influence from an ignorant and regressive anti-environment, anti-government (anti-metro) republicans we should be able to turn back the clock to 1955.
Upon his prompting I extend criticism to Mr. Middaugh.
Forced choice #7y
Have a kid or bike to work?!
As I said in response to the first discussion of this survey here:
A survey, any survey can be designed well, or not. If the respondents feel they are being asked to make a false choice, are second-guessing the questioner, feel compelled to add lots of explanatory notes in the margins, then these are all very familiar signs that the folks designing the survey did not know what they were doing. Yes it happens all the time but no it doesn’t have to.
Thank you Jessica for your comments. I took this survey and shared with my friends online how saddened I was by the poor layout of this survey. Republicans and suburbanites have an EVEN HIGHER NEED to be educated about the mutual dependence of economy and ecology. Secondly if Jim created and sent this survey out without the input and support of the council than this reflects poorly on his judgment and ability to work with a group.
I’m impressed that Mr. Middaugh would take credit for the survey. I wonder if he was concerned about how it would affect their credibility? If he’s comfortable with his agency alienating the rest of us to throw raw meat to Republicans perhaps he should reconsider. Using the “everyone is doing it” defense doesn’t work for my 6 year old, why should it work with him?
There was an awful transit survey from Portland Community College. Now PCC is using bond money to build parking spots at $25,000 each. We need a museum for the survey authors and the surveys.
Bad leaders come up with bad ideas then desperately try to find some people to prop up their bad ideas. Bush was able to torture by going to lawyer after lawyer until he found one who wrote a memo supporting torture.
Does OptIn ever report the comments when reporting their results? Looking at a couple of reports from their web site (link below) I didn’t see any that included the user comments.
http://www.optinpanel.org/index.php/work
If comments are not reported then there is less incentive to provide them.