Project Background

Eppo is an experimentation and feature management platform. By providing advanced statistical analysis models within an intuitive interface, it enables both technical and non-technical teams to run experiments at a high velocity without sacrificing rigor.

For many customers, there can be dozens of experiments running at the same time. We created an experiment progress bar to help users track experiments. Despite the simple mission, the initial progress bar had created more confusion than clarity.

A Bigger Problem Than It Seemed

The failure of the progress bar not only hurt the company's business goals but also significantly reduced the trust our customers had in the product, which is central to Eppo's value proposition. This led us to the redesign of the progress bar.

Uncovering User Pains Through Research

We know that our users, especially the PM persona, are frustrated with the current design. However, we need to understand the reasons behind this frustration before we jump to solutions. We had several assumptions that needed to be validated, and I am working with our statistical engineer, who is leading the technical design of the project, to dig deeper into the issue.

Research Methodology

User interviews

Qualitative research
Zoom sessions
45min each
Led by: designer (me) and stats engineer

Interview participants

3 PMs from customer companies
2 Customer Data Scientists from Eppo

How Is the Progress Bar Confusing?

Our users are confused about a lot of things. To our surprise, none of our interview participants understood what the bar is actually tracking or how progress is measured.

The current progress bar actually measures statistical power. When setting up an experiment, users need to set an desired effect or lift for the primary metric. Once experiment starts running, the bar shows how close or far it is from detecting the desired effect, if it would have existed.

It is also unclear to users when they should review an experiment or when to take action. The experiment status label, the color of the bar, and the number displayed on the bar were intended to work together to help decision-making, but instead, they have caused more confusion.

Old progress bar states based on experiment status and interview notes from Miro, click to view full image

Why Did the Progress Bar Fail to Help Users?

To gain a more comprehensive understanding of real customer pains, we had an in-depth conversation with our customer data scientists. They are specialists that help customers set up the data infrastructure and they work closely with both the product team and data team within a customer's organization. This conversation shed light on why our users are experiencing confusion and dissatisfaction with the current progress bar design

PMs struggle to navigate the statistical complexity

Statistical power

Most users do not understand power and how it determines experiment progress. This led them to look for other, non-primary indicators like sample size for decision making.

MDE & CDE

Minimum detectable effect and current detectable effect are concepts that represent confidence. Users need a lot more explanation to interpret them correctly.

Non-linear progress

Experiment progress is not linear and should be interpreted based on specific statistical analysis model. Users often misunderstand the correlation between run time and progress.

Data scientists struggle with unrealistic expectations created by the progress bar

Inaccurate power calculation

Progress bar relies on Eppo's power calculator which only uses current experiment data, leading to inaccurate estimations for experiments using conversion metrics.

Limitation by sample size

Companies with a smaller sample size are stuck with underpowered experiments. The bar could only show unfavorable results, undermining data scientists' credibility.

Misrepresentation of power

The expectation of a progress bar is to reach 100%, but this is unrealistic from a power analysis perspective. This creates unrealistic expectations and raises many questions.

Problem Statement

We realized that the issue goes beyond just visual representation. Progress needs to be easily interpreted by PMs to make informed decisions. At the same time, the statistical reasoning behind the calculations must be present to back up any decisions. The team agreed that to achieve our goal of rebuilding trust, we need to address the pain points of both technical and non-technical users.

Solution Discovery

The discussion with our customer data scientists sparked a lot of good ideas. The statistical engineer and I had a couple of brainstorming sessions afterward where we explored a wide range of solutions.

We were given autonomy to discard the old design and start fresh. As a designer, the creative freedom is both exciting and challenging.

Solution brainstorming whiteboard after group discussions and voting. Click to view full image.

Anything but the Bar

There is a strong consensus within the team that the linear progress bar needs to be retired. I began exploring alternative visual representations, sketching out meters and graphical displays to reimagine how we can communicate experiment progress in a non-linear way.

I presented three design concepts to the team. While each concept had its pros and cons, we believed we were heading in the right direction. Concept 2 with the dial meter design was the simplest and had most buy-in, so I did more explorations around it.

Early design concepts, three non-bar visual concepts. Click to view full image.

I spent the following week exploring the feasibility of the dial design while addressing the following key challenges:

Space constraint: The component needed to fit within the existing experiment card
Affordance: Users should intuitively understand what the new “progress bar” is measuring
Clarity: Clear labeling to draw user attention to the "ready for review" experiments

After developing more detailed designs for concept 2, I felt that the dial meter might overwhelm users with additional labels and text. The old design's strength, despite its flaws, was its simplicity. Users were able to scan the bar and obtain information very quickly. I revisited the bar component and added a new bar design to the mix.

Further explorations of the dial meter design and a new concept referencing the old design. Click to view full image.

A New Direction

I presented all three explorations, including the bar, during design review. The team liked the clarity in the dial meter design, but ultimately preferred the bar for its readability and minimal visual noise.

For the new bar to succeed, its intuitiveness and clarity for action still needed improvements. I created four variations based on the feedback and presented them at a follow-up design review.

Version 2.2 was the winner. Here are what the team liked about it:

New "Ready" label. It's a clear indicator for users to take action
Hiding "MDE" or "CDE" in the UI. This reduced cognitive load
"Precision" as a progress measurement. This concept seemed more intuitive and accurate

Key Decisions

Measure progress by precision instead of jargon like "power" and "MDE"
Combine progress bar with clear labels to prompt appropriate actions
Hide supporting data from the at-a-glance view to reduce cognitive load
Surface primary metric lift to help inform decisions

After a discussion with stakeholders, the redesigned bar UI was approved and the team aligned on the scope of the project.

MVP Scope

Project Goals

An intuitive progress bar UI to reduce cognitive load
Transparent data and logic display to increase confidence in decision

Design Milestones

New bar component
Hover peeks
Updated status labels
Experiment setup page

Design Principles

Respect the real estate constraints

The new progress bar must fit within the existing experiment card's limited space

Visuals are used as an aid to reduce users’ cognitive load

The new bar needs to accurately represent how progress is measured instead of creating confusion like the old design
The bar visual element should complement the text but not solely rely on the text to make sense

Mindful and limited use of colors

Avoid conflicting with Eppo's established color meanings, such as red for negative metric lifts
Use color sparingly to maintain its highlighting power

Accessibility

Do not rely solely on color to convey critical information
Combine text, icons, and color to maximize readability

The anatomy of the new progress bar in an experiment card. Click to view full image.

Making Progress Actionable

If the bar component was the biggest visual challenge, the underlying decision-making logic was the biggest technical design challenge.

The progress bar uses precision as an indicator of when to review an experiment. However, precision is not the single decision-making factor. The experiment outcome depends on all three of the following:

Precision: how certain users can be about the experiment result
Confidence interval method (CI method) : or statistical analysis model, including sequential, fixed sample, and Bayesian
Statistical significance: whether the lift on the primary metric is statistically significant (stat-sig)

Precision, CI method, and lift are monitored together to move an experiment from "Running" state to "Ready" state, but the different combinations of these factors lead to different outcomes.

To better illustrate the decision-making logic, I collaborated with our statistical experts and created an experiment decision flow chart. This chart helped guide the rest of the design.

How experiment moves from one state to another and what decisions can be made in each state. Click to view full image.

Updated Labels and Dynamic Peeks

To help users navigate the decision-making flow, we combined progress bar visuals with intuitive and actionable labels.

Ready for Review indicates a decision point where evaluation is required
Ready for Review 🎉 indicates a positive ship decision can be made
Confetti is a symbol of celebration and it brings delight to the product

In addition to the labels, we wanted easy access to supporting data to help users make a decision. This is where the hover-triggered peeks come in handy.

The peeks provide data and recommendations based on the configuration of each experiment. It only appears when users want more information, keeping the overall interface clean and simple.

The End-to-End Experience

The progress bar component might seem small, but the changes we made to the mechanism of progress tracking required a redesign to the experiment creation flow and updates to experiment details page.

The following configuration steps were added to inform the progress bar and the peek content:

Select statistical analysis model (CI method)
Customize precision target

Once an experiment is created, it will stay in draft stage or start running depending on the start date.

The new progress bar component was also added to the the experiment details page.

Launched & Loved

Implementing the new progress bar required a lot of engineering effort because we had to refactor the experiment progress tracking logic. We rolled it out in two phases:

Phase 1: Ship the new progress bar component

Build progress bar component with precision tracking
Replace MDE with precision everywhere
Update experiment card and experiment detail page header to match Figma

Phase 2: Add minimum requirements

Show min. requirements on the progress bar
Allow companies to set min. requirements defaults in Admin Settings

After several rounds internal QA, we finally launched the progress bar at the end of Q1 2023!

Impact of New Progress Bar

This was a highly-anticipated launch and we received an overwhelming amount of customer love.

The progress bar has achieved:

Reduced number of related support tickets
Increased customer satisfaction
Strengthened trust between Eppo and data expert types of users

Designers are resilient in facing unfamiliar territory

As designers, we can develop strong understanding of complex domains outside our expertise. In this project, I was able to grasp and simplify statistical concepts I had never encountered before.

The perfect solution does not exist

While iterative design is natural to designers, we should actively guide cross-functional teams toward shippable solutions. Working with our statistical engineer taught me to take ownership in aligning the team on MVP solutions that move projects forward.

Good process leads to efficiency

Design reviews, stakeholder check-ins, and decision logs may seem like overhead during tight timelines. However, these practices align team goals and clarify next steps, ultimately creating efficiency and saving time.

Experiment Progress Bar Redesign