Overview and Guiding Design Framework
A participatory design approach engaged users and an expert panel in a mixed methods (qualitative and quantitative) iterative user-centered design process starting from discovery to design and testing (Figure 1). We utilized common design principles throughout this process that focused on: 1) being person centered; 2) communicating visually and inclusively; 3) collaborating and co-creating; and 4) iterating.(23, 24) We conducted five 90-minute design sessions conducted over Zoom from September 2022 to February 2023 (Table 1). All design sessions were conducted using Mural(25) or Figma(26), collaborative platforms that allowed users to contribute written artifacts (i.e., responses) during the session. FigJam, a tool in Figma, was used to draw collaboratively during design sessions using an online whiteboard. Following design sessions, qualitative data were summarized and merged with written artifacts (e.g., a sticky note contributed on the mural board) from design sessions or survey results. Weekly team meetings were used to reach consensus on design impacts (i.e., how we would incorporate user feedback into design features of the D4DS planner).
Once a working-version prototype tool was created, usability testing was conducted with a separate group of users who were not familiar with the tool and expert panel members using a combined think-aloud and survey-based approach. The prototype was revised and usability was re-assessed by the same users. Details on the methods for data collection, analysis and results for each design session and usability testing are presented below in the Design Process and Results section. The project was approved as an exempt study by the Washington University in St. Louis Institutional Review Board (#202207165).
Participants and Recruitment
Users included researchers with experience conducting a health-related project, including academics (e.g., researchers, students, project managers) and clinical and public health practitioners (e.g., clinicians, health department employees). Using purposive and snowball sampling, we recruited 14 users (71% academics and 29% practitioners) to participate as co-designers. Email invitations sent out to users included a brief description of the project goals and D4DS principles. Each co-designer was asked to participate in at least two design sessions. Co-designers reported varying levels of D&I knowledge prior to starting the design sessions, with 21% (n=3) reporting below average, 43% (n=6) reporting average, 36% reported above average knowledge.
The expert panel consisted of academic faculty members (n=6) who are leaders in D&I science, including the developers of the F2C Framework, across three academic institutions. Collectively, the expert panel focuses on public health research in both clinical and community settings and has expertise in the development, implementation, and evaluation of digital health tools. The expert panel members were engaged separately from users in a subset of the design exercises and usability testing.
Design Process and Results:
Design Session 1: Discover
This first design session engaged 6 co-designers to identify the need and demand for key issues this tool should address and explore potential features that will address these needs. We started this design session with a brief overview of D4DS principles and then conducted five design exercises (DE). Co-designers started the session by adding sticky notes to free list: 1) how the tool could benefit and empower users (DE 1.1) and 2) who might benefit from using this tool (DE 1.2). Next, we conducted a persona creation and value proposition generation exercise (DE 1.3). Personas are archetypes of different users who could use the tool and has been used in design to ensure diverse perspectives are accounted for in product design. Therefore, in our sessions, we asked co-designers to consider the perspectives of other potential users (listed in DE 1.2) using a template value proposition stated (i.e., I’m a (user type), who uses the D4DS web tool to (use case) to define (impact)). We then asked co-designer to free list potential features (i.e., ways a user may interact with the content and experience learning in the tool (DE 1.4). Free listing features is a fast way to generate a lot of ideas in a short period of time.(27) Lastly, co-designers placed the listed features into a prioritization bullseye and discussed their ideas by giving 2 minute presentations on their favorite ideas (DE 1.5).
Results: Design Session 1
The overall design impact of the first design session was that the tool should increase the transparency of D4DS and accessibility to multiple audiences, and foster collaboration. The tool is intended to help users plan their research and have D4DS as a key principle in their research from the start. To provide education, users suggested the tool contain a repository of key references and resources for individuals engaging in D4DS work. Users felt the tool should provide methods and strategies for engaging community partners in D4DS work. When asked who would benefit from the tool (DE 1.2), there was an emphasis on providing resources not only for people with D&I background, but also the people they partner with, whether that’s a site champion, an individual implementing an intervention, or a software developer designing materials for dissemination. While co-designers felt the primary users were researchers (with various level of D&I knowledge/background), they felt strongly that the tool would facilitate conversation with other partners. Other users may include community members and organizations, practitioners (e.g., clinicians, public health), policy makers, commercial partners, and funders. The main impact of the tool (DE 1.3) was described as: 1) enhancing co-design processes, as well as dissemination and implementation research; 2) increasing understanding of D4DS for multiple audiences; and 3) improving the design, effectiveness and sustainability of research products. The users free-listed (DE 1.4) 35 features that were grouped into 5 categories: grant proposal resources, methods/tools, dissemination resources, educational materials, and general format ideas. The research team combined similar features to generate a final list of 25 features (Table 2), which was used as input for the next design session to prioritize features.
Design Session 2: Ideate
The second design session focused on further exploring and prioritizing features. Using information from design session 1, we utilized the Kano Model of Customer Satisfaction exercise for prioritization for the 25 features (Table 2).(28, 29) The Kano Model classifies features as Must-have (i.e., I expect it and would be dissatisfied without it), One-dimensional (i.e., I expect it), Attractive (i.e., I like it), and Indifferent (i.e., I’m neutral). Co-designers and expert panel members completed the Kano as an electronic (REDCap) quantitative survey during or outside the session (DE 2.1).(28) After completing the Kano survey, the 3 zoom participants performed an exercise to free list any other feature ideas and identify their favorite feature from the Kano survey (DE 2.2). Co-designers then grouped features into content areas for learning (DE 2.3) and prioritized the most important content areas to identify which ideas are most important to users (DE 2.4).
Results: Design Session 2
A total of 17 participants completed the Kano survey. Table 2 lists the features in order of prioritization according to the Kano survey responses. We identified 1 Must-have feature, 17 Attractive features, and 7 Indifferent features. The Must-have feature was URL links to other content that falls within the discipline/field of D4DS, especially dissemination and sustainability. The number of Attractive features was a promising finding give the many potential opportunities to delight users (a key goal of design), with a relative low risk of dissatisfaction if the feature was not integrated at all. As for the Indifferent features, these features would not yield either satisfaction or dissatisfaction for our users. As a result, the Indifferent features occupied the bottom of our development prioritization, and ultimately were not incorporated into version 1.0 of the web tool.
Design Sessions 3-5: Wireframing
The next 3 design sessions focused on iteratively envisioning and creating wireframes of the features that were prioritized in the previous design session. Wireframes are illustrations of a product that are not yet built and typically lack functionality but represent the interface and its intended features and functionalities. The iterative design process allowed users to draw and give feedback on design options (e.g., wireframes) in successive versions. During design session 3, 4 co-designers ideated and drew 1) account creation, 2) a questionnaire that would allow users to set-up a project and guide them on how best to use the tool, 3) a roadmap or visual process that walks the users through the D4DS process, 4) and the landing page. In design session 4, 4 co-designers provided feedback on two versions of the roadmap and sketched a final version of the roadmap. We also discussed which features co-designers wanted to be present on each page that derives from the roadmap and sketched the layout of a single page. In design session 5, co-designers provided comments on a refined version of the single page sketch, sketched the layout of the education hub and spent time discussing the name of the tool and each feature.
Results: Design Sessions 3-5
Co-designers felt that users should have the option to use the tool as a guest or with an account, making it clear to users the benefits of having a login (e.g., saving data, returning later to update work). Discussion identified that the login should be simple but make users feel that the information they add into the tool is secure. A major focus of our prototyping sessions was brainstorming and drawing the feature of a roadmap that would help users walk through a D4DS process. The wireframe of this iterated in each session (Figure 2), with co-designers realizing they did not want this to feel like a linear process. Co-designers were critical in the process of developing easy to understand, action-oriented language to communicate different features of the tool (e.g., D4DS Planner, action planner, action item, cue to equity).
Usability Testing: D4DS Planner Prototype
Following all design sessions, our team delivered low-fidelity wireframes and features to a software development company (HICAPPS, https://hicapps.com) that has expertise in developing health-related tools. Our team collaborated with HICAPPS on an iterative build of a prototype of the D4DS Planner to ensure that all feedback from design sessions was incorporated.
In the usability phase, test users were asked to use the D4DS web tool on a laptop computer in a private space in-person or via zoom. The test users were given a general description of the web tool but were not given explicit instructions on how the web tool operated or how it was designed to be used for developing a dissemination and sustainability plan. Test users were asked to carry out two out of a possible three heuristic tasks, including: 1) familiarize yourself with the web tool and figure out the purpose and function of the web tool, 2) create an account, and 3) set up a project to create a D4DS action plan.
To assess the heuristic usability data, an affinity grouping (30, 31) exercise was conducted in Mural, an online collaboration software app, by four team members (MMK, BM, TS, AL) who were integrated into the design and development of the web tool. The research team analyzed internal notes taken during the usability sessions and re-watched recordings of users navigating the tool. Summary phrases were extracted and typed on Mural’s “sticky notes” and mapped together based on similarities in relation to four categories and the collaborator type, e.g., researchers, practitioners, and leadership. The four categories were as follows: (1) What works well in the D4DS web tool? (2) What is not working well, or what are the significant pain points, in the D4DS web tool? (3) What changes do users want to see in the D4DS web tool? (4) Other comments that pertain to the usability of the D4DS web tool. At the end of the usability sessions, test users completed the ten item System Usability Scale (SUS) to quantitatively measure perceived usability of the tool.(32) Usability results were used to refine the prototype tool. Usability was retested on the refined version one of the D4DS planner among the same users who participated in the first round of usability testing. During this round of testing, five questions (reported on a 5-point Likert scale) about the usefulness and appropriateness of the tool were added to the survey.