Investigate SMS delivery-rate drop at Attentive
Problem: US SMS Delivery-Rate Drop Incident Response
On March 10, between 20:00 and 22:00 ET, Attentive experienced a significant incident: the US SMS delivery rate (Delivered/Attempted) dropped by 6.5 percentage points. This impact was more pronounced on two major US carriers, while Canada and the UK remained unaffected. Preceding this incident, on March 9, two changes were rolled out: a new link-shortener domain and updates to 10DLC registrations.
Your task is to develop a comprehensive triage and validation plan for this incident. The plan must address the following key areas:
- Metrics and Guardrails:
- Define delivery, acceptance, and failure metrics, leveraging carrier error codes.
- Specify relevant guardrails, including Click-Through Rate (CTR), unsubscribe rate, and complaint rate.
- Analysis Slices:
- List the precise dimensions for data analysis. These should include: by carrier, sender type (short code, toll-free, 10DLC), campaign/template, message length/character set, URL domain, vertical (specifically SHAFT-sensitive content), send-time, client, region, and new-vs-existing senders.
- Artifact Checks:
- Describe critical operational artifact checks. These include: duplicate sends, clock skew/timezone issues, logging gaps, changes in retry policies, throughput throttles, Messages Per Second (MPS) caps, and carrier queue backoffs.
- Hypotheses and Testing:
- Prioritize potential hypotheses for the delivery rate drop and outline how to test each one:
- (a) Carrier Filtering: Investigate using error-code mix shift and the delta between acceptance and delivery rates.
- (b) Link-Domain Reputation: Design an A/B test comparing the old versus the new domain, stratified by carrier and sender.
- (c) 10DLC Registration/Brand Score Issues: Propose an audit of TCR (The Campaign Registry) status and vetting outcomes.
- (d) Content/Template Changes: Suggest token-level diff analysis and application of spam-score models.
- Prioritize potential hypotheses for the delivery rate drop and outline how to test each one:
- Experiment Design:
- Specify the full experiment design, including: the randomization unit, methodology for sample sizing and Minimum Detectable Effect (MDE) for delivery-rate uplift, sequential monitoring strategies, and the use of holdouts.
- Propose a statistical model for attribution, such as hierarchical logistic regression controlling for carrier × sender × template × hour interactions.
- Action and Rollback Plan:
- Provide a clear action and rollback plan.
- Define escalation criteria for engaging with carriers.
- Outline the evidence required to confidently declare the incident resolved.
AI interview prep
Core dilemma
Rapidly diagnosing and mitigating a critical, multi-causal system failure in a high-throughput, external-dependency environment, while ensuring data integrity and preparing for future prevention.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Investigate SMS delivery-rate drop at Attentive
Problem: US SMS Delivery-Rate Drop Incident Response
On March 10, between 20:00 and 22:00 ET, Attentive experienced a significant incident: the US SMS delivery rate (Delivered/Attempted) dropped by 6.5 percentage points. This impact was more pronounced on two major US carriers, while Canada and the UK remained unaffected. Preceding this incident, on March 9, two changes were rolled out: a new link-shortener domain and updates to 10DLC registrations.
Your task is to develop a comprehensive triage and validation plan for this incident. The plan must address the following key areas:
- Metrics and Guardrails:
- Define delivery, acceptance, and failure metrics, leveraging carrier error codes.
- Specify relevant guardrails, including Click-Through Rate (CTR), unsubscribe rate, and complaint rate.
- Analysis Slices:
- List the precise dimensions for data analysis. These should include: by carrier, sender type (short code, toll-free, 10DLC), campaign/template, message length/character set, URL domain, vertical (specifically SHAFT-sensitive content), send-time, client, region, and new-vs-existing senders.
- Artifact Checks:
- Describe critical operational artifact checks. These include: duplicate sends, clock skew/timezone issues, logging gaps, changes in retry policies, throughput throttles, Messages Per Second (MPS) caps, and carrier queue backoffs.
- Hypotheses and Testing:
- Prioritize potential hypotheses for the delivery rate drop and outline how to test each one:
- (a) Carrier Filtering: Investigate using error-code mix shift and the delta between acceptance and delivery rates.
- (b) Link-Domain Reputation: Design an A/B test comparing the old versus the new domain, stratified by carrier and sender.
- (c) 10DLC Registration/Brand Score Issues: Propose an audit of TCR (The Campaign Registry) status and vetting outcomes.
- (d) Content/Template Changes: Suggest token-level diff analysis and application of spam-score models.
- Prioritize potential hypotheses for the delivery rate drop and outline how to test each one:
- Experiment Design:
- Specify the full experiment design, including: the randomization unit, methodology for sample sizing and Minimum Detectable Effect (MDE) for delivery-rate uplift, sequential monitoring strategies, and the use of holdouts.
- Propose a statistical model for attribution, such as hierarchical logistic regression controlling for carrier × sender × template × hour interactions.
- Action and Rollback Plan:
- Provide a clear action and rollback plan.
- Define escalation criteria for engaging with carriers.
- Outline the evidence required to confidently declare the incident resolved.