23 Daily — According to saastr.com, the SaaStr team recently leveraged an AI agent named 10K to fully automate deliverability testing across three email service providers, encompassing over 500,000 emails. This large-scale AI-driven competition not only accelerated the traditionally labor-intensive testing process but also provided a multi-dimensional and accurate assessment of email delivery performance, offering businesses a more objective basis for selecting email services.
Large-scale email deliverability test surpasses human limits
Vendor bake-offs traditionally involve sending a few hundred test emails and judging performance based on dashboards and reports. However, SaaStr utilized the AI agent 10K to conduct deliverability testing on over 500,000 emails—an unprecedented scale unattainable by human teams. 10K detected every issue, conducted multivariate diagnostics, and provided actionable recommendations that were adopted on the same day.
The test focused on comparing deliverability across three email service providers, specifically using an AI Day campaign split into two matched groups of around 2,300 recipients each. With identical content and sending time, the challenger’s emails achieved 2.4 times more opens and 2.9 times more clicks than the incumbent provider, indicating that delivery quality was the key differentiator.
Detailed diagnostics reveal vendor system faults
During testing, AI agent 10K discovered that one provider’s email flow showed 'Completed' delivery status, but in fact over 80,000 delivery attempts failed; the vendor’s status page meanwhile indicated all systems green. Through four diagnostic sends altering single variables, 10K ruled out content, configuration, and recipient lists as cause and identified the issue as lying in the vendor’s transport layer.
In a traditional two-week trial scenario, team members might have accepted the 'Completed' dashboard status and moved on without further probing.
Editor's take: AI agents redefine enterprise email provider evaluations
This AI-driven large-scale email test demonstrates that enterprises can leverage automated, standardized processes to deeply understand email service provider performance. For organizations lacking dedicated deliverability teams, AI agents like 10K can rapidly perform complex evaluations that would otherwise require hiring consultants for weeks.
Verification and entrepreneurial opportunities
To verify this technology’s effectiveness, startups and companies can conduct controlled split tests under identical conditions using AI agents to continuously monitor multivariate data and step-level results, confirming actual deliverability and avoiding reliance on vendor self-reports.
This AI-powered deliverability testing service represents a significant opportunity for email marketing, SaaS providers, and related fields by enabling clients to precisely select providers, improve operational efficiency, and offer vendor differentiation in a competitive market.