Before proceeding with the redesign, I conducted a deep dive into the existing review flow. This analysis revealed a significant drop-off point where many users abandoned the review process mid-flow. Insights emerged from both quantitative data and user feedback, which showed:
User Hesitation: Users were reluctant to install apps with limited reviews and low ratings, as reviews were identified as a top deciding factor.
High Drop-Off Rates: The step in the review flow that required users to provide lengthy, free-form feedback was the highest drop-off point. Of those who started a review, only 29% completed it, and 67% abandoned the flow at this step.
Negative Reviews Dominated: The users most motivated to leave a review tended to do so only after negative experiences, which contributed to the overall low ratings in the Marketplace.
The combination of low review density, poor ratings, and significant drop-offs in the review flow made it clear that a foundational redesign of the review experience was needed to improve user engagement and, by extension, the App Marketplace’s performance.
.png)
Once the problems were identified, I dove into redesign and experimentation. The main goals were to: simplify the flow and reduce friction, particularly at the ‘Share’ step (where freeform feedback was required).
Key Focus Areas:
Shorten the Review Flow: Streamline the review process to make it easier for users to leave feedback with minimal effort, encouraging more reviews.
Make the ‘Share’ Step Optional: For users leaving a review with three stars or more, the ‘Share’ step (where free-form feedback was required) was made optional, aiming to encourage positive reviews and reduce abandonment.
Rather than releasing the redesign untested, I advocated for an A/B testing approach to validated the change before full implementation.
I advocated for and led experimentation around the newly optional Share step. I worked with data tools including Amplitude to determine the impact the experiment would need to have to achieve statistical significance. Partnering closely with engineering, I ran an A/B test for 15 days with a 50/50 split between the newly redesigned flow and the existing experience (control).
The results were significant:
Our primary metric was the conversion rate for users to move past the ‘Share’ step.
For this metric, we saw an 81% higher completion rate when comparing the variant to the control.
Our secondary metric was the total conversion rate for users completing the review flow.
For this metric, we saw a 50% higher completion rate.
Additionally, we saw significant impact in positive reviews: The number of 5-star reviews saw a massive uptick, with 39.2% of reviews in the variant group being 5 stars, compared to just 16.3% in the control group. (A 140% Increase)
The massive success of the experiment was clear and led to immediate rollout.
Following the redesign’s release, the review count steadily increased, with projections showing a 53% increase in reviews by the end of 2024, compared to a 45% increase in 2023. Additionally, the monthly number of 5-star reviews grew by 28%, contributing significantly to the higher overall ratings in the marketplace.
Building on the success of the redesign, I organized a workshop with the Marketplace team, using the experiment’s findings to pinpoint further areas of improvement within the review process. These insights helped influence the team’s ongoing roadmap for the App Marketplace, ensuring the continuous optimization of the review experience.
Workshop
The redesign of the App Marketplace review flow proved to be a game-changer, leading to a notable increase in the quantity and quality of reviews.
The project not only achieved its goal of improving the app ratings but also had a lasting impact on the overall success of the App Marketplace. This initiative demonstrates how even small changes can drive major impact and contribute to the company’s wider strategic goals.
