Diagnose a non-significant experiment outcome

Your original design called for a two-sample t-test on the primary mean metric with significance level $\alpha = 0.05$ and 80% power. The baseline was 10.0, the standard deviation was $\sigma = 5.0$, and the planned minimum detectable effect was $+0.6$. After 14 days, the observed difference is $\Delta = +0.35$, with a 95% confidence interval of $[-0.05, +0.75]$.

a) Why might this result still be non-significant even with a large sample size?

b) Is computing post-hoc power appropriate here? If not, what should be reported instead, and why?

c) Under a loss function that penalizes false positives twice as heavily as false negatives, what decision would you make, and what next step would you take: extend the sample, reduce variance, or stop?

d) How would you revise the design—MDE, variance reduction, duration—for the next iteration?

Overview: This question assesses your grasp of A/B test interpretation, statistical power and confidence intervals, decision-making under asymmetric loss, and how to adjust experimental design in two-sample hypothesis testing.