Measure App Store success and debug funnel anomaly

Part A — Product case: measuring success for a new App Store

Shopify is introducing a Shopify App Store where merchants can browse and install apps created by third-party developers, some paid and some free. You are the Data Scientist supporting this launch.

1) Define success

Propose a framework for measuring success that includes:

• Primary (north-star) metric(s)
• Input/leading metrics (activation, engagement)
• Diagnostic metrics (funnel rates, segment cuts)
• Guardrails (latency, merchant churn, refunds/chargebacks, spam/fraud, support burden) Be clear about whose success you are optimizing for (merchants, developers, Shopify) and how you would balance the tradeoffs.

2) Data and instrumentation

Specify the data you would need and where it would come from.

• List the key event streams (for example, clickstream/browse/search, install/uninstall, subscription/billing, app usage, support tickets).
• Propose a minimal data model (example fact and dimension tables) that would support these metrics. Assume events arrive in near-real time; define any time windowing (such as daily in UTC) and identity rules (merchant_id, app_id, developer_id, session_id).

3) Experimentation plan

Design at least one experiment to improve App Store outcomes (for example, ranking algorithm, pricing surfaces, recommendation modules, onboarding prompts). Include:

• Unit of randomization (merchant vs session), eligibility, and duration
• Primary/secondary/guardrail metrics
• Key threats to validity (network effects, interference, novelty effects, selection bias)
• How you would analyze (e.g., CUPED, stratification) and make a ship/no-ship decision

---

Part B — Data interpretation and visualization: traffic spike with a worse funnel

You are given a dataset with 3 years of daily metrics for the App Store. You notice:

• A large traffic spike that is not explained by normal seasonality.
• Add-to-cart (ATC) rate drops sharply during the spike.
• Conversion rate drops slightly. Assume the table below (you may create derived fields such as YoY, WoW, and rolling averages):

Table: daily_app_store_metrics

• date (DATE)
• sessions (INT) — total visits to the App Store
• product_views (INT)
• add_to_cart (INT)
• purchases (INT)
• revenue (NUMERIC)
• channel (STRING) — e.g., organic, paid_search, email, affiliate, referral
• device_type (STRING) — desktop/mobile/tablet
• geo (STRING)
• merchant_tier (STRING) — e.g., trial/basic/plus
• landing_page (STRING)
• app_category (STRING)
• is_bot_suspected (BOOL) — if available

Tasks

1. Identify plausible hypotheses that could cause sessions ↑ while ATC rate ↓ and conversion ↓/flat (covering both product and data-quality causes).
2. Propose the most useful charts you would build (in Python or Google Sheets) to validate or refute your hypotheses.
3. Explain what follow-up data you would request if the dataset is insufficient. Define rates as:

• $ATC\_rate = add\_to\_cart / product\_views$ (or justify an alternative)
• $CVR = purchases / sessions$ Output expected: a structured investigation plan plus the key visualizations you would generate. Overview: This question assesses proficiency in product analytics, instrumentation design, experimentation planning, and funnel-level diagnostic analysis for an app marketplace, and sits squarely in the Analytics & Experimentation domain for data scientist roles.