Director, Experimentation

Job Description

The Job

The Director of Experimentation will oversee a growing team of testing managers to drive the quantitative experimentation practice across Product and Editorial teams at CNN Digital. Drive the exponential growth of experimentation velocity and the evolution of the practice across the organization in a cross-functional team environment. Build the necessary frameworks, workflows, methodologies, communications, and educating teams across the organization, leading to a hybrid model of democratized testing with centralized oversight.

The Daily

Grow and scale the function:
• Develop end-to-end experimentation process including prioritization, documentation, experiment design, hypothesis definition, KPI selection, requirements gathering, experiment coordination.
• Drive continuous improvement of the team and the product organization as the function matures, helping educate stakeholders, and sharing best practices with peers.
• Identify experiment-related product, technical, and analytics requirements to ensure proper execution and tracking of experiments.
• Ensure the maintenance and monitoring of the operational status of experiment queues, roadmaps, test schedules, and in-flight tests.
• Track and measure the program and its aggregate impact, including velocity, win rates, and overall impact to the business, in total and by a cross-functional team.

Partner with stakeholders across the business:
• Collaborate with the technology teams to identify, develop, implement, and adopt cutting-edge experimentation technology and tools to enhance CNN's capabilities.
• Partner with engineering and product teams to the instrument and enhance current testing and reporting capabilities.
• Develop and enforce standardized documentation, procedures, and governance of controlled experiments across the Product and Editorial organization.
• Provide ongoing organization-wide communications of current experiments, results, and insights derived from experimentation.
• Collaborate with Data Science to improve test analysis methodologies to derive higher quality insights and better recommendations for follow-up experiments.
• Lead, coach, and develop a group of Experimentation practitioners that support CNN Digital worldwide.

The Essentials
• MS in Statistics, Analytics, Mathematics, Engineering, Computer Science or other quantitative fields.
• 10+ years of progressively related experience.
• 5+ years of managerial experience, building and growing teams in product analytics environments.
• 8+ years of digital optimization and A/B and multivariate testing experience.
• Expert understanding of testing methodologies and statistics related to AB/MVT testing.
• Proficiency in statistical calculations concerning A/B testing includes confidence intervals, statistical significance, causal inference, sample ration mismatch.
• Experience with Adobe Analytics or similar analytics platforms.
• Experience with leading AB testing tools, such as Adobe Target, Optimizely, Google Optimize, SiteSpect, Conductrics, AB Tasty, etc.
• Tableau, Looker, Domo, or similar visualization and dashboarding tools is a plus.
• Advanced SQL, Python, or R skills.
• Experience with cloud storage platforms such as AWS, Databricks, Snowflake, Snowplow, DBT.

Jobcode: Reference SBJ-gwnnok-3-141-41-121-42 in your application.