Predict Monthly Job-Change Risk

Context

You are tasked with developing a monthly predictive model to estimate the probability that a LinkedIn member will change jobs within a given month. The model's outputs will be integrated into various product features, including member notifications, job advertisement targeting, and tools designed for recruiters.

Tasks

1. Feature Specification

   • Identify and specify relevant features across several categories:
     • Profile and Job History: Examples include educational background (school, major, degree), normalized job title and seniority, tenure in current role and company, number of past career moves or promotions, industry, company size, and location/remote work status.
     • Member Behavior: Examples include recent profile edits, job search activities and applications, views of job or company pages, job alert subscriptions, connections or messages with recruiters/HR professionals, saved jobs, resume uploads, InMail activity, network growth, and days active on the platform.
     • Seasonality: Examples include month-of-year, peaks associated with graduation cycles, fiscal year-end hiring, and industry-specific hiring seasons.
     • External Demand: Examples include job postings volume by title and geographic location, applicant-per-opening ratios, trends in job postings, average time-to-fill for roles, unemployment rates, and signals from layoffs or news.

2. Label Definition, Data Structure, and Sampling

   • Define a hazard-style label using user-month rows: assign 1 if the user changes jobs in that specific month (and had not changed jobs in any prior month within the observation period), otherwise 0. Implement censoring for rows after a user's first job change or after a period of user inactivity.
   • Implement robust safeguards against data leakage, ensuring that features only incorporate data available before the start of the prediction month. This includes carefully handling backfilled profile updates and excluding any post-job-change behavioral data.
   • Address challenges related to class imbalance and the overall dataset size. Propose strategies such as downsampling negative examples with re-weighting, utilizing focal loss, or managing right-censoring effectively.

3. Evaluation and Fairness

   • Select appropriate evaluation metrics: designate PR-AUC (Precision-Recall Area Under the Curve) as the primary metric, supplemented by recall at a fixed precision threshold, calibration metrics (e.g., Brier score, reliability curves), and time-split backtests (e.g., rolling-origin evaluation).
   • Incorporate fairness checks across critical demographic and professional groups (e.g., region, industry, seniority level) to ensure performance and calibration parity.

4. Product Use and Experimentation

   • Describe in detail how the model's predictions will be leveraged in:
     • Member notifications (e.g., suggesting relevant jobs).
     • Job advertisement targeting (e.g., showing ads to members likely to change jobs).
     • Recruiter tools (e.g., identifying and prioritizing potential candidates).
   • Propose a comprehensive A/B testing plan to accurately measure the business impact of the model and to mitigate potential feedback loops. Consider techniques such as shadow scoring, persistent holdouts, incorporating treatment indicators into training data, cluster randomization, and advanced evaluation methods like Inverse Propensity Score (IPS) weighting or doubly-robust estimation.