Finding high-value customers with AI
AI-powered Adaptive UI Proposal + A/B testing
How might we modify Doordash’s interface to increase orders from high-paying users while minimizing dropoff?
This was a brief assignment to design an adaptive user interface. In this project, I learned about creating effective value propositions using AI. Professor Zimmerman encouraged me and my peers to look for “low-hanging fruit”– applications of AI that are simple to build with an existing dataset.
This was a group project. I worked on creating user personas, identifying value propositions, and creating the user interface.
Contents
1. Why this is valuable ︎︎︎
2. How I made a low-risk interface ︎︎︎
3. A/B testing ︎︎︎
This was a group project. I worked on creating user personas, identifying value propositions, and creating the user interface.
Contents
1. Why this is valuable ︎︎︎
2. How I made a low-risk interface ︎︎︎
3. A/B testing ︎︎︎
Keywords
UI design, product thinking, value proposition, AI design, A/B testing
Class
Design of AI Products and Services (05-317)
Time
October 2022
UI design, product thinking, value proposition, AI design, A/B testing
Class
Design of AI Products and Services (05-317)
Time
October 2022
Why is this valuable for Doordash?
By designing for the most valuable customers, we can make a financially significant impact on revenue (see the 20-80 rule, where 80% of consequences come from 20% of causes). We identified Doordash’s most value users as families and businesses that want mid-size catering. These customers share the characteristics of having large and predictable orders.
With an existing dataset and a relatively simple prediction task, this would be a low-risk and high-return investment.* Additionally, by suggesting pairings of orders, we target users who make larger orders ($50+ per order) that can help offset fixed costs of delivery and increase per-order profit.
Persona
One of our users, David, has a family of four. As a working professional, he doesn’t have time to think about what to get for food. David is already a Doordash user, but only orders when he thinks about it.
With an existing dataset and a relatively simple prediction task, this would be a low-risk and high-return investment.* Additionally, by suggesting pairings of orders, we target users who make larger orders ($50+ per order) that can help offset fixed costs of delivery and increase per-order profit.
Persona
One of our users, David, has a family of four. As a working professional, he doesn’t have time to think about what to get for food. David is already a Doordash user, but only orders when he thinks about it.
* There are still risks such as a user perception of a violation of privacy. Additionally, notifications pose a risk of being too overbearing on certain users if used incorrectly. This should be tested with A/B testing, which I talk about below.
1. Using a predictive model informed by personal ordering habits, Doordash predicts when David is likely to order.
2. A push notification shows up on David’s phone, reminding him to order when he otherwise might not.
3. David orders a large meal, as suggested by the AUI.
2. A push notification shows up on David’s phone, reminding him to order when he otherwise might not.
3. David orders a large meal, as suggested by the AUI.
Wireframe
1. Notification lockscreen // iOS long press notification
2. Recommendations (adaptive user interface), with a back button or card
3. Default doordash home screen
2. Recommendations (adaptive user interface), with a back button or card
3. Default doordash home screen
How to design for failure cases of AI
This design would only send out push notifications when a user is most likely to want to re-order something. Because we’re actively pushing this recommendation system, we can turn high-value users into higher-value users by building habits on Doordash. However,
The suggestion list card below displays a ranking existing orders from the user by how likely they are to order them. This reduces cognitive load for choosing food, and encourages users to order multiple items. By taking the form of a card, this page is also low-risk to the users: If the user taps into the notification and doesn’t want to proceed, they can easily swipe down with an existing iOS design pattern to the Doordash main page. The “change or add items” button ensures that users have a readily-available option for changing their order if it’s close to what they want to have.
The suggestion list card below displays a ranking existing orders from the user by how likely they are to order them. This reduces cognitive load for choosing food, and encourages users to order multiple items. By taking the form of a card, this page is also low-risk to the users: If the user taps into the notification and doesn’t want to proceed, they can easily swipe down with an existing iOS design pattern to the Doordash main page. The “change or add items” button ensures that users have a readily-available option for changing their order if it’s close to what they want to have.
Model type
Regression
Model input
Model output
Regression
Model input
- Time of day
- Order history
- Location
Model output
- Likelihood of ordering
- List of suggested orders by confidence
Measuring success + A/B testing
I would conduct A/B testing to verify the effectiveness of this proposal. I would measure these variables across both the control group and the test group:
Because notifications and active interventions with users have their own set of risks, I would measure the following additionally in the test group:
If this proposal was successful, I would see an increase in both order rates and order sizes from a moderately-sized population while reducing abandonment/dropoff.
- Rates of ordering
-
Average order size
(dollar value and number of items)
- Total user retention/dropoff
Because notifications and active interventions with users have their own set of risks, I would measure the following additionally in the test group:
- Notification clicks
- Notification dismissal rates (both singular notifications and app-wide)
- Abandonment rates within the AUI flow
If this proposal was successful, I would see an increase in both order rates and order sizes from a moderately-sized population while reducing abandonment/dropoff.