samgjohn / ava-agile Goto Github PK

https://weareava.slack.com/files/U02DRA8TD/F0199NZBNQ2/faq_-_growth

The 4 most important elements you need to kick off a growth team are the following:

• Clean data set to track key metrics and goals
• Segmentation tools to be able to understand and segment the customer and activity at a granular level
• Rigorous experiment dashboard to analyze the experiment results and the statistical significance behind them
• Peer review process to discuss and analyze findings

It is critical for teams to have the right systems and tools in place to run experiments at scale. Especially key in the first year is the experiment dashboard. Experiment dashboards are essentially a single destination to track experiments/results, and allow for easy analysis by lots of people at the company. Dashboards contain:

• Experiment group metrics
• Control group metrics
• A set of metrics defined to track and measure statistical significance

The dashboard helps the team to run various experiments and test the results before proposing every single idea to be added to the product. As the growth team scales, the number of engineers increase and it becomes unwieldy without an experiment dashboard. A company at scale typically runs 1 experiment per growth engineer per week. With that future state in mind, it's vital to start early with a solid growth experiment dashboard. The dashboard also becomes an invaluable archive of past experiments that is also immensely helpful when adding new team members or iterating on past experiments.

100% of the experts we spoke with emphasized their decision to build their own internal tools at scale. Initially, you can use tools like Mixpanel, Optimizely, Superset and Chartio to track your experiments.

It can take several iterations to formalize the experiment dashboard. For example - one of the experts cited that the experiment dashboard was formalized after several iterations only after they had ~25 to 30 growth engineers on the team.

Peer review & Individual Experimentation

Teams often set up an internal experiment review process on a biweekly basis. Team members present their hypothesis and share the results of the experiment they ran to test the hypothesis. Peers ask a lot of questions to decide whether they agree or disagree with the findings. Growth teams that run 100+ experiments per year cite that only a third of their experiments turn out to be positive.

Though the success rate is only 20% to 30%, the point of this exercise is to encourage engineers to take more risks.

A common contention is whether engineers are allowed to run experiments independently. Companies in their early stages often encourage engineers to run growth experiments on their own. However some of them require PM oversight as they scale, especially as they get more rigid with quality standards.

Another important element is to make sure you set heuristics for the growth team. Growth teams are constantly testing hypotheses and running experiments. One of the most common heuristic experts use is: "Don't test things you wouldn't ship to everybody"

https://www.notion.so/ava/UTM-Attribution-Guide-c6aa233868bc4e84a0bb31f072ddfc68

• Need landing pages
• Need Tracking

Data alone cannot answer all the questions. It is equally important to have user researchers in place to really understand what is happening behind the numbers.

Your first 100M users will look a lot different from the second 100M users. Therefore it is important to do the following:

• Solicit real time feedback from users
• Use tools like Inspectlet to track UX
• Meet users outside of San Francisco, especially if it was your first core market. Other markets will look a lot different from SF
• Pay attention to how users use the product internationally. There may be cultural nuances in addition to language gaps (for example, people in Japan do not like to post photos of people without their permission and products may need to adapt to local taste).

Document every single use case. What is perfectly normal for one group can be very different for another group of users.
As you scale it is important to add dedicated user researchers to the growth team

• Identifying Loops
• Testing linear channels

https://guides.co/g/the-ultimate-growth-hacking-sourcebook/38840

Find script to export "who viewed your profile"

Steady increase in rank in this category with little increase in "Social networking"

To see if this 2 minute lag actually helps. https://www.notion.so/ava/Experimental-Backlog-9f60c0d0a3af42ce8ad9e94422b75911?showMoveTo=true&saveParent=true

https://guides.co/g/the-ultimate-growth-hacking-sourcebook/38830