Connections

Experience research that’s designed to gauge the needs and desires of consumers who use technology is important but if wide groups of consumers are being excluded from that research – which is what we have seen – then that research is flawed, according to Kellie Hodge, principal design researcher at Sutherland Labs.

Exclusion can become embedded in research practices and there are steps that companies, including those in the M&E sector, can take to make their research more inclusive, she said during the session “Who are we Designing for?” that was part of the virtual SoCal Women’s Leadership Summit on Nov. 10.

She started the session with a story, reflecting back on the time she was attempting, on a Friday evening, to enroll a delivery driver in Kansas working an evening shift in a research study she was running. He had a very weak signal and needed to have cellular data running to start the study but it seemed to be disabled, she recalled. Earlier that day, she had a set of problems with a student who was using her brother’s Amazon Prime account to complete the study. The owner of the account was needed to change a setting.

“Eventually, she gave up and just dropped out of the study; it was just too hard to get her set up,” she remembered. The main issue was she had not anticipated dealing with issues more complicated than her research design, she conceded. She had made some assumptions about study participants, assuming they could stop what they were doing and focus on the screens they had to do the study, that they had high-speed Internet access at all times, and that they controlled the software accounts they were using, she pointed out. Those assumptions turned out to be inaccurate in at least some cases.

“In other words, we were assuming that they were like desk-bound knowledge workers like us,” she pointed out, noting: “This discussion is really about asking the question: What happens when, despite best intentions, the accumulation of practice creates a culture of exclusion and what can we do about it?”

There’s also an “elephant in the room,” she conceded: “Me; I’m obviously a white researcher living in a comfortable place in the Bay Area, so I’m talking for a very, very privileged position. I want to acknowledge that but I also want to say that we need to, as an industry, examine the privilege of our position.”

As a result of this inquiry, the question being asked, she noted, is: “Is experience research like a camera? Does it reflect reality by capturing the needs and desires of all the different people who use the products and services that we have designed? Or is it more like an engine that is reproducing a fixed idea of who the people who use technology really are?” (She credited sociologist Donald MacKenzie for this concept.)

She told viewers to assume that’s true and noted it is important to figure out what can we do about it. She admitted to not having all the answers but said: “I think we’re hopefully at a point where we can ask better questions.”

When planning research, “how does exclusion creep in at these early stages?” she asked and then answered her own question: “We think that it starts right at the moment we define the research sample.”

As one example, she said: “Most research studies define an income range for the participants in a study.” She pointed to a listing for an Oakland, California home priced at $599,000 and guessed that, “to afford this house, you probably need a household income in the range of about $200,000 a year.” On the other hand, to afford a similar home in Mississippi that’s listed at $99,000, you probably only need a household income of about $35,000 a year, she said.

It is important to note that most sampling criteria for research “sets an income range at a national level for the United States and the floor is often $50[,000] to $70,000 a year,” she said.

As a result, what is inadvertently happening over time is that “we are cutting out a huge slice of middle and working class America from our studies, simply because of a number” that is being used for each study, she explained.

Economists are used who know how to adjust for regional variations in the cost of living, she noted. She conceded that can’t be done perfectly but said: “We can improve a lot…. A takeaway is just avoid using national averages in a country as regionally and economically diverse as the U.S. and that applies to most other countries as well.”

It is also often assumed that homes are occupied by single-family households, so it is important to also include more household types to “make sure everyone’s going to be included in the research,” she said. It’s important because one in five Americans live in a multigenerational household and most of them are Asian, Black or Latinx, she pointed out.

“The challenge for us, as our industry, is to deliver unbiased but efficient sampling,” she said. To help accomplish that, there are three steps that can be taken, she told viewers: audit recruiting criteria for inclusivity and diversity; become familiar with population statistics, where available; and consider sampling based on experience, not demographics. The latter “can cause complications later,” however, she conceded.

Recruiting services are often brought in to recruit people for studies, she went on to say. And there is one type of person who tends to be “way too overrepresented” in research, she said: people recruited online, use digital devices and don’t need technical assistance, have a valid ID, work in a location where they can do a survey without interruption, live in a major city, have a great Internet connection, have a computer with a webcam, are the service account holders, and are articulate in English.

Who are often left out of research, on the other hand, are first-generation immigrants who won’t typically respond to an invitation from a recruiting firm, she noted.

“The irony is that it’s undeniably faster and cheaper to speak to a highly-paid software engineer in San Francisco than it is to speak to a middle-aged man who lives in an RV somewhere out in the Midwest,” she said. The result is that research tends to not factor in a diverse set of people and experiences, she explained.

“There is overwhelming support for more inclusive recruiting but the demand needs to be there,” she said. In the meantime, we can expect higher recruiting costs for more inclusive research and it will take longer to get it, she noting, adding: “We need to consider if that’s a cost we can bear.”

Also important is to make sure that people who are using older devices and communicating only via text message can participate in research the same way that people with the latest smartphones can, she said.

To create more inclusive research, there are several steps we can take, she said: let participants define their identities in their own terms; create accessible research experiences that are inclusive of people using older devices or with limited connectivity; and optimize for the convenience of participants, not stakeholders.

The last area to consider is how a product team can estimate the size of a market opportunity based on research findings, she said. Even if you are successful in reaching out to include a more diverse population, stakeholders may still find it hard to justify actions based on one’s findings, she added.

However, based on findings that have been seen from more inclusive research so far, the “opportunities for service are very, very exciting and I think we need to find, as an industry, a way to manage these risks,” she said.

She then posed a final challenge: Recognize the opportunities by examining innovative edge cases surfaced from underserved groups and partner with quant to advocate for “fuzzy” populations, especially if quantitative evidence is required by product leadership.

Hodge was introduced by Susana Sheil, VP of CM&E, technology at Sutherland.