I worked on this project for a recruitment process I was involved in in Spring 2020. The task was to analyse the results of an AB experiment which tested two different versions of a fictional Free-to-Play mobile game. The goal of the analysis was to provide a recommendation for product managers on which version provides the better experience to users.
- Players in group A are significantly more engaged in terms of game rounds played than players from group B (this finding is robust to outliers)
- Therefore, the original version seems to provide the better game experience from a players' perspective as they are engaging longer
- It is statistically significant that players in group B purchase more than players from group A. However, the result is not robust to extreme values
- Even though the CVR was higher for group B during the treatment time, it has not proven significant and players in group A and B convert equally likely
- The change in the game (version B) has not led players to convert significantly more
