Fighting email spam is a constant challenge in today’s business world. Our journey in improving spam detection has taken us from traditional methods to the forefront of AI-driven solutions. In this article, we share the evolution of our spam detection approach.
Our challenge: overcoming spam in Salesforce cases
First, let’s dive into the unique challenges we encountered with Microsoft 365 and Salesforce. We received emails via shared mailboxes such as info@ in Microsoft 365, which were automatically forwarded to Salesforce as cases. This process created an unexpected challenge: a lack of effective anti-spam filters. Without filters in Microsoft 365 and Salesforce, unfiltered spam emails flooded our Salesforce cases, leaving our teams inefficient and overloaded.
First steps: Limited protection with traditional AI
Our first attempts to tackle spam with traditional AI, such as SpamAssassin with Bayesian filters, yielded limited protection. We could only filter 25% of spam, which prompted us to continue looking for improvements.
ADA: a leap forward
Faced with these limitations, we took our first steps toward a more advanced AI solution in 2022. We developed a modified model based on ADA, a predecessor of ChatGPT 3.5. This model brought a significant improvement in our spam detection, with an accuracy rate of about 90%. A key advantage of this ADA-based model was that it could be gradually improved through fine-tuning, while the operational cost remained surprisingly low.
Transition from ADA to GPT-3.5 Turbo
With the end of ADA’s lifespan in sight, we looked for more advanced AI solutions and chose GPT-3.5 Turbo. Although this move was a step forward in technology, the standard version of GPT-3.5 Turbo did not meet our expectations, with an accuracy of only 60% in spam detection. This outcome highlighted the need for a more specialized approach.
Our own GPT-3.5 model: the breakthrough
We took matters into our own hands and developed a modified GPT-3.5 model. With 10,000 self-collected ham and spam emails, we achieved 99% accuracy. However, when attempting further fine-tuning of this model, we encountered challenges. By adding only spam, the model began to classify everything as spam, and vice versa with ham. Despite the theory that existing models can be further fine-tuned, this proved ineffective in practice.
A large-scale solution: 99.7% accuracy
Our solution? A large-scale approach. We trained a completely new model with 30,000 emails, combined with SpamAssassin. This yielded 99.7% accuracy, proving that comprehensive data sets are the key to successful AI.
Conclusion: the pillar of our email automation
Our journey in AI-driven spam detection highlights not only our ongoing innovation and adaptation, but also how this technology is the backbone of our overall email automation. By efficiently detecting and filtering spam as the first step, we avoid wasting valuable time and resources processing unwanted emails. This allows our advanced AI to focus on answering legitimate customer inquiries.
