Standardizingthe research and design process for NYC's first municipal service-design studio

client: The Service Design Studio at NYC Opportunity (formerly the Mayor's Office for Economic Opportunity)
role: UX Researcher, Service Designer
duration: June 2018- September 2018
advisor: Emily Herrick, Designer @NYC Opportunity Civic Service Design Studio
1.1- TLDR

The challenge

The Service Design Studio at NYC Opportunity deals with unnecessary hurdles and inefficiencies when conducting user-research interviews. I was honored to collaborate with the Studio to see how we could standardize an efficient process for obtaining, sorting, and synthesizing key insights gathered from user-interviews while helping the Studio fulfill its mission of making public services more accessible to New Yorkers.

The solution

We came up with a participatory-designed framework for standardizing the process that the Studio can now use for researching, ideating, and designing solutions in the civic-space.

A special thank you to Service Designer Emily Herrick who provided guidance throughout the project.

1.2- The Who

The Mayor's Office for Economic Opportunity (now named NYC Opportunity) was mandated by Mayor Bill deBlasio in May 2017 to use evidence-based principles to combat poverty and increase equality in NYC.

The Civic Service Design Studio is a unit within NYC Opportunity that uses principles in UX and Service Design to make public services more "effective, accessible and simple for all New Yorkers."

__The Process
Understanding the project
1.3- The Process

Which problems are we going to solve?

01/03

Roadblocks throughout the research process

Duplications cause inefficiencies. The Studio's approach to user-research was filled with roadblocks that made collecting and managing user-data tedious and duplicative. For example, a common workflow after conducting a user-interview would be to tag key insights on sticky notes, record them on a cloud-based service (i.e. Google Docs), and then dispose of them in the bin afterwards. That's a lot of wasted post-its.

02/03

So many UX tools to choose from

With so many moving parts throughout the research process, the Studio needed a tool to centralize all quotes, insights, and discoveries in one repository and share it with the rest of the team. However, there were too many software vendors to choose from and we needed to pick the right one that would solve for our specific needs.

In order to choose the most appropriate software tool for our team, we needed to fully understand what our design process was like in the first place, and identify any pain-points to solve for with that tool.

03/03

Not many actionable takeaways from weekly meetings

Every week, the Studio would meet with key stakeholders on a project to discuss findings gathered during user-interviews. While it was important to have everyone at the table share quotes and insights, the format of the meetings was structured around passive listening and note-taking.

The meetings were in need of a revamped framework that would enable our Studio team and stakeholders to do the following:

  • take actionable notes
  • ask informed questions
  • generate hypotheses
  • ideate solutions
The following points outlined the vision for the Studio's success for creating a design process:
Collect and organize the data so that it can be manageable
Identify key opportunities
Sort and cluster the data
Make share-out meetings with stakeholders actionable

Here's how we would go about solving for them:

01/05
Set the Stage

Before embarking on a design-journey, it's important to know which solutions are currently addressing the issue(s). More specifically, it's helpful to understand what works and what could be improved about those current solutions.

02/05
Talk with people

From whom do we want to learn? Who are the key stakeholders involved in the service we're designing for that can provide first-hand perspectives on their pain-points, past attempts to solve for them, and what an ideal solution looks like in the future?

03/05
Connect the dots

After talking and learning from various stakeholders, we need to make sense of their valuable insights. In this phase- we sort data, recognize patterns and themes, and see trends into what everyone else considers to be important.

04/05
Try things out

After making sense of our stakeholders' painpoints, we are in a better position to brainstorm solutions with their needs in mind. This phase also involves prototyping ideas and testing them with our users in a low-risk way before being heavily-invested in the idea.

05/05
Focus on impact

How do our prototypes and solutions measure up in the "real world?" Were our ideas validated by users and key stakeholders? This is where we see our solutions at work, based on the metrics and evaluations we set initially.

__The Process
Set the stage
1.3- The Process

What solutions are currently out there that work?

Before embarking on this adventure, I scanned the "digital landscape" to see which solutions were out there that could help achieve the Studio's goals for creating a streamlined design process. I came across some powerful, all-in-one UX tools that did amazing things and jotted them down in my handy notebook- but the question remained: "were these even the right tools that could help enhance the Studio's UX process in the first place?"

In order to answer this question, I had to understand what our process was like to begin with.

After digging through the Studio's list of tools, I learned that we use the following for recording and digitizing interview notes:

Excel
  • To manage data
  • mine for emerging patterns and insights
  • sort through data
Google Docs
  • To transcribe interviews
  • include key insights and additional notes
  • share notes with the team
Post-its
  • To visualize key-quotes
  • generate insights
  • synthesize and visualize emerging patterns
__The Process
Talk with people
1.3- The Process

Getting the Studio team involved

Recruiting participants for research interviews

I reached out to four of the Studio's designers: two of whom worked full-time and two who were on a short-term fellowship. I liked that there was variation in how long they’ve been at the Studio. In my mind, the recent fellows would most likely have a different take on how things are done as opposed to the more established designers who may have grown accustomed to the same way of doing things.

Fresh minds beget fresh perspectives.

What did I want to understand?

In order to understand which UX Tools would best facilitate the Studio’s research process, I had to understand how the Studio did research in the first place. All of my research efforts were focused on understanding three components: People, Processes, and Tools, and how they all affect each other.

I wanted to center my research efforts on understanding my colleagues better: to know what their design process is like, which tools they use, and which possibilities they'd like to explore for solving their pain-points and accomplishing their goals.

__The Process
Connect the dots
1.3- The Process

Making sense of the data

After conducting the first round of user interviews, I noticed recurring pain-points that everyone in the team was having, as well as opportunities for solving them.

Here's what the Studio team expressed regarding their shared pain-points:
Painpoints
Lack of consistency
“We need more structure on how to formalize our tools and processes. A lot is missing because a lot of people are involved on the research side. How we defined insights were varied.”
-Studio_team_member-TR.
"Lack of consistency is what's most difficult about managing data"
-Studio team member-TR.
"It wasn't consistent. The consistency wasn't there, feeling that the synthesis process was different for the next round"
-Studio_team_member-MK
"It would have been good to have some type of reporting structure for our synthesis days. For process that would have been good- I don't know how that would look like"
-Studio_team_member-LS.
Content overload and burnout
"A lot of data, and really overwhelming"
-Studio_team_member-LS
"Lets say we limited to 5 insights and opportunities, could we have really captured that instead of overwhelming ourselves with 40-50 per person? Could we have done with with 5-10 insights, I don't know. Maybe not. It was definitely overwhelming."
-Studio_team_member-MK
"After the synthesis and capturing that synthesis, we were really exhausted because it took 4-6 hours a day. How do you take that and take a snapshot of it and have it synthesized for you?"
-Studio_team_member-MK
"There's a big amount of raw data in everyone's head that no one else knows about"
-Studio_team_member-EH
Many moving parts
"That’s when most of the research was being conducted. Since I’m the project lead, I was leading some of the synthesis sessions, but when I was gone, the team was using different frameworks every time. And they were also conducting research while they were doing synthesis, and it wasn’t paced very well."
-Studio_team_member-EH
"One thing that was challenging for us- it was time-consuming to transcribe into note-taking form and then bringing it into post-its, that whole process was cumbersome"
-Studio_team_member-LS
"I really wanted to organize this, but just couldn't for little technical reasons like your stickies would start falling out if you stacked too many on top of one another. We don't have our own room so we acn't have stuff up for an extended amount of time. So you're constantly moving paper with paper, so stuff starts falling down, you're losing it"
-Studio_team_member-LS
" We had these 6 hour long days, and we were really tired and all this stuff would be up and hanging and we'd be like, 'oh we'll return to it tomorrow', but we'd have 800 other things so it would stay up and wait till the next week that'd we'd block and then we'd forget. We'd have to start over and we'd ask 'what is this again?' "
-Studio_team_member-LS

Wow, a lot of fatigue sifting through piles of sticky notes for hours at a time. This was definitely a painpoint for our Studio designers, but beneath the difficulties there were opportunities worth uncovering to solve for those problems.

Opportunities
On what an ideal workflow looks like
"Having a designated space for your research- the space was such a restraint. This was the only space that allowed for everyone to use to do synthesis"
-Studio_team_member-LS
"Worflow wise, collaboration is super key- having another at least one or two people in the team working with you so that you can bounce ideas off, having that collaboration so that you're not doing everyting by yourself, even in the synthesis aspect."
-Studio_team_member-MK
"When possible, streamlining your notes and typing directly into a computer and then having time blocked off right after an interview to digitize and then that becomes a norm or if that's something you do during debrief with your partner, which is to type your notes right away during the debrief?"
-Studio_team_member-EH
"Have set synthesis days during a sprint, so you know that every week you have a 4 hour block for synthesis that you know you're coming to the session so you can prepare accordingly"
-Studio_team_member-EH
__The Process
Try things out
1.3- The Process

Bringing all minds together

We hosted a brainstorming workshop to address our pain-points and ideate solutions

After sorting the key-findings into painpoints and opportunities, I wanted to bring the Studio together for a workshop in which we could all offer input to solve for those pain-points and capitalize on opportunities.

We began with a round of ice-breakers in which we “aired a grievance” from the past day or week. This helped position our minds to think about negative situations and imagine ways to improve them. Our minds were now in "process-improvement" mode.

Validating our painpoints as a team

Next I presented the team's pain-points and issues. I wanted to see how often they came up in their work and how they felt about them.

I asked the following two questions:

  1. How painful are these issues?
  2. How often do you encounter them?

We came together as a team and placed our pain-points on a four-quadrant matrix along the lines of "intensity versus frequency". The vertical axis labelled the two extremes of frequency: “Rarely” and “Always”- and the horizontal one labelled the extremes of intensity: “not a pain” and “ouch, big pain.”

The most pressing and frequent issues were the following:

"Lack of consistent standards, processes, and defined outputs"
"Onboarding new stakeholders and partners in the research process"
"Physical exhaustion when sorting sticky-notes for hours when synthesizing data"
"Rotating teams and positions within teams"
The least pressing problem that rarely happens is insight overload. That's a good thing- because as a data-driven team, we can never get enough of data and insights.

What should we "stop, start, and continue" in our design processes?

The team was then invited to deliberate on how to map out an ideal workflow that would ensure consistency and solve for the above pain-points by asking the following questions:

  1. What do we want to stop doing?
  2. What do we want to continue doing?
  3. What do we want to start doing?

Here’s how the team envisioned a standardized process going forward:

Before we conduct a user-interview, we should...
  • Have a diverse team so that stakeholders can get involved
  • Clarify and align on understanding between interview-lead and note-taker
When we finish conducting an interview we should...
  • Surface initial and most revelatory insights and quotes
  • Generate fifteen immediate insights on sticky notes
  • Take five minutes to debrief
  • Interviewer sends a "thank you" note and follow-up information to keep stakeholders invested
When we sort through data we should...
  • Have a near-transcript record of conversation
  • Tag major themes and insights
  • Organize info into research themes
When we discuss our findings as a team we should...
  • Create discussion-frameworks so that workshops and meetings are actionable, engaging, and not passive.
  • Incorporate discussion points on patterns, insights, open questions, and hypotheses/ideas

In order to make our shareout sessions more actionable and less reliant on passive listening, we ideated the following framework for taking notes:

Each person in the meeting is given a ‘Notes Template’ to fill out as they are listening to the interviews being read aloud. The template will be divided into the following sections:
Patterns

What data has been validated?

Insights

Which quotes did you find novel/ interesting? why?

Open ?'s

What do you still not know?

Hypotheses and Ideas

What might work? Write your "fixes" down to get them out of your head!

Deciding on outcomes and output

One key takeaway from our brainstorm session was that there was a need to track and share data that was emerging from interviews with stakeholders. Together, we decided that we needed to create a repository of emerging data and insights that would be "open-sourced" with key stakeholders and participants.

This gave way to the "Field Logs" which the Studio now uses to journal each important milestone during the design-process for any given project.

__The Process
Focus on impact
1.3- The Process

Putting our new design framework to the test with the "Pathways to Prevention" initiative.

The "Pathways to Prevention" initiative is a year-long collaboration between the Administration for Children's Services (ACS) and the Studio at NYC Opportunity to deliver an approach that would allow families to use their "voice and choice" to opt for social-services and thereby prevent their children from entering the foster-care system.

The Studio noticed the following improvements:

01/03

The Studio began to take more actionable notes called "Field Logs"

After coming to terms with the fact that we needed to make our share-out meetings more actionable and less dependent on passive-listening, the Studio began to create note templates for all participants in order to:

  • ask insightful questions
  • identify emerging patterns and insights/trends
  • generate hypothesis and ideas

The team was able to come up with more insightful questions to ask families in future interviews

The quality of follow-up questions was higher- helping the team explore solutions that addressed the needs of family stakeholders in a more meaningful way.

Here are some of the questions the team thought of before prototyping solutions:

Family "voice and choice"
How might we provide parents with opportunities to make informed choices about key parts of their prevention experience?
Trust and safety for families
How do we ensure families safe enough to share their preferences for prevention services?
Personalizing prevention-services for families
I wonder how we might design materials that meet each family’s unique situation?
On what families care about most when opting in for services
what content is important to families when signing up for Prevention services?

It became easier to visualize trends and insights during the synthesis phase

The Studio was able to better identify emerging trends from interviews with families. For example, the team learned that families needed to feel safe knowing that any interaction with ACS would not lead to the separation of their child

The team also learned that ACS' communication with families should be trauma-informed and easy to understand.

How to communicate prevention services to families
In our research, we heard that the information available to families about the different types of programs and providers could feel unclear, inconsistent, and difficult to access.
Using a strengths-based approach for interacting with families
We (the Studio) should be including strengths-based phrasing whenever asking parents to identify things they may be struggling with to provide to their families.
On tailoring prevention services to the needs of families
Prevention services are not one-size fits all, and we should tailor how families learn about them to meet their diverse needs.
Shifting the way ACS is viewed by families
These types of tools would help case-workers effectively move from the role of investigator to supporter.

The team was better informed when testing out prototypes

By focusing on key emerging-trends gathered during interviews, the team was able to quickly prototype ideas that spoke to the needs of their stakeholders and receive validation from case-workers at ACS.

Friendly look-and-feel to written materials
The DCP staff welcomed the idea of well-designed and colorful materials with standardized, family-focused language.
Provide families with the right info to select a program that works best for them
This step involves presenting key information about Prevention providers so families can make informed choices.
Making supporting materials more human
We learned that something as simple as showing dad a photo of their new case planner could go a long way in demystifying what Prevention can do for his family.
02/03

Diverse groups of teams and stakeholders were incorporated in the research process

We took our findings from our initial brainstorming session and decided to include diverse stakeholders throughout the research process.

For the Pathways to Prevention project, the Studio invited families, ACS case-workers, and members from the Community Based Strategies team at ACS for more enriched perspectives.

Each Thursday in February and March the Pathways team will meet with families, prevention provider staff, and Child Protection Specialists at ACS to test a lot of prototypes designed to answer the project's guiding question: How might we provide parents with opportunities to make informed choices about key parts of their prevention experience?
03/03

A centralized and digitized tool for tracking, sorting, and mining data

Now with a revamped design process in place, we needed a software tool that could help fulfill our goals and address our existing pain-points.

We narrowed our search to the following tools based on a decision-matrix:

RealTimeBoard
Pros:
  • An infinite canvas
  • Ability to create "frames" or sections that can contain any piece of data
  • Easy to import photos of sticky-notes and include within your canvas
  • Collaborate with others
  • Easy to group notes into categories
Cons:
  • Not much in terms of statistical analysis
Trello
Pros:
  • Freedom to manipulate trello boards
  • Create “sprints”, along with interview notes
  • Easy to import photos of sticky-notes and include within your canvas
  • Tagging notes as you envision it best
  • Free service
Cons:
  • Not much you can do with data analysis
  • Not designed to be a UX research tool
  • You’ll need to get creative and “hack” it to your benefit
Reframer
Pros:
  • Useful for data-analysis
  • Tagging feature allows for data-mining and clustering into similar categories
  • Visualizations to help make sense of repeat tags
  • Collaborate with others
Cons:
  • Not easy to import existing notes from a Google Doc or Word Doc
  • Learning curve for getting used to their tagging system
Dovetail App
Pros:
  • Easy to import existing docs for digitizing
  • Easy to tag and create tagged-groups
  • Visualizations to help make sense of repeat tags
  • Collaborate with others
Cons:
  • We have to manually create categories and insights
The Studio decided to go with RealTime Board as a UX tool to centralize and digitize all of our findings going forward.
__Lessons Learned
Key takeaways
1.4- Lessons Learned

What did I learn?

Don't underestimate the design process

When I was first asked to research a set of tools that could facilitate the Studio's design process, I thought it wouldn't require anything beyond a mere hunt for online tools.

To my excitement, it turned out to be much more than a Google search. After hearing my colleagues express their collective pain-points (i.e. working with limited office space, missing sticky-notes, fatigue from spending days sorting post-its)- I realized there were underlying issues that needed to be solved in a user-centered way.

I took this as an invitation to learn about our Studio, my colleagues' design process, and what a collective vision looked like for standardizing our process in the future.

The lesson for me was to never underestimate the need for thoughtful design in our everyday lives.

Ask "why" to get to the root cause

I learned that before engaging in any design work, it's essential to get to the root cause and not be misled into designing a "band-aid" solution for a problem that's deeper than what initially meets the eye.

Consider the following dialogue:

"We need a software tool to enhance the way we manage data."

"Why?"

"Because there's a lot of needless duplication and inefficiencies in the way we gather data.

"Why?"

"Because the way we in which we obtain data isn't consistent across the projects that we engage with."

"Why?"

"Because our design process isn't really defined."

This method of questioning helped me uncover the Studio's true needs: it wasn't about software tools as much as it was about process-improvement and creating shared, best-practices for the entire team.