Agentforce on Our Site – 100K Conversations Later, Here’s What We Learned

Four months ago, we reimagined the web experience for Salesforce.com by launching Agentforce, our platform for building AI agents, across the site. We’ve seen some exciting successes — our agent, which automatically pops up to chat throughout the site, has handled more than 100,000 conversations and influenced 30,000 sales leads — but we’ve also been on a valuable learning curve. Let’s take a look at our early wins and works-in-progress, and preview some new ways that we’re improving the agent.

One of the biggest wins: The artificial intelligence (AI) agent’s role has expanded in scope and value. While its main task initially was to answer questions about Salesforce and its products in a sophisticated and efficient way, it’s now also guiding users toward key conversion points like watching a product demo or signing up for a free product trial.  

“The job of the agent now is to not only answer questions, but surface relevant offers at the right time to help move someone through the sales funnel,” said Dana Greenberg, senior director of product management at Salesforce. 

Matt Matsuoka, vice president of product management at Salesforce, takes it a step further, saying the agent is not just a digital worker, but a digital marketer.

“The agent is a marketer on our website whose job is to convince people that there’s something there for them to explore and there’s something they may want to purchase,” he said. “It does this by delivering compelling information that provides value to users. 

“This was an epiphany, that the agent is not just a mechanism on the website — but an employee with a job to do.” 

Before we get to the rest of our findings, here’s a refresher on how Agentforce works on our site.

Agentforce on Salesforce.com – how it works

A key step is determining which authoritative content sources are best to feed an agent. 

Originally, the agent was grounded in a set of knowledge articles that were designed specifically for it —  they were separate from any existing content or content-generating process. Turns out this wasn’t ideal: The data was hard to scale, didn’t cover the breadth of knowledge the agent needed to impart, and lacked a formal updating process. So, we pivoted: The agent now uses three data streams of existing content — product catalog, website content, and customer stories catalog. 

These sources are ingested and mapped to Data Cloud, which then feeds Agentforce. In the future, additional knowledge sources from the website, including different content types like video, will be added to Agentforce, enhancing the richness of the experience. 

Agentforce uses the Atlas Reasoning Engine and retrieval augmented generation (RAG) to execute combined semantic search on structured data, like product catalogs, and unstructured data, like website content. If a user asks Agentforce about Sales Cloud, it pulls data from the unstructured website content to respond. If they ask, “How much does Sales Cloud cost?” Agentforce understands the context, selects the right product details from the structured product catalog, and delivers an accurate answer. It seamlessly links unstructured and structured data using existing text, identifiers, and tags.

Sharat Radhakrishnan, principal architect at Salesforce, said responses to certain queries — such as those concerning pricing, and terms and conditions — must be 100% deterministic 100% of the time. This means they can never vary from one query to another, and must be completely accurate. Responses to more open-ended queries, such as “How can I improve the efficiency of my service team?” can be probabilistic, meaning some variances in responses are acceptable.

What’s working

All that planning and groundwork has resulted in a completely different conversational experience than what our previous chatbot offered: The AI agent is smarter, more flexible, and more helpful. But how is Agentforce actually performing? We analyzed usage data and customer journeys to see where the agent is truly making a difference. Here’s what we’ve learned.

Site visitors are converting at higher rates

The number of visitors using the AI agent is roughly the same as those who used the legacy chatbot on the site, but visitors are now taking more actions. In other words, the agent is nurturing users to the next step.  

“There are two types of actions a user could take after chatting with Agentforce,” said Jesse Luke, senior manager of data enablement at Salesforce. “They could either transfer to a human sales development rep (SDR), or they could fill out a form for a demo, free trial, or to be contacted. Taken together, we’ve seen an 18% increase in these conversions from visitors using Agentforce.”

These next steps demonstrate a visitor’s interest, which gives our sales team an encouraging lead. Ultimately, these conversions are what fill the sales pipeline and help the business grow.

The number of people transferred to SDRs has expectedly declined – the AI agent is answering basic questions previously directed to them – but the people who are transferred are generally more qualified to buy. 

Cross-functional operating model = killer alignment

One of the biggest reasons AI pilots fail is a lack of alignment across all stakeholders. To avoid that, we created a cross-functional team of marketing technology, engineering, marketing, content operations, analytics, and digital experts. The team meets regularly to discuss milestones, challenges, new features, and more. The upside of this approach goes beyond better alignment. The collaboration brings a ton of diverse perspectives, helping us identify challenges or unique solutions that a siloed team might overlook.  

For example, the team devised new user experience strategies, such as the agent’s new ability to proactively start a conversation with a user. (More on that below.) Marketing’s perspective was important for the product experts, since we needed a mix of prompts: both commonly asked questions and questions that provide general awareness of Salesforce at early stages in the customer’s journey. 

Enablement is crucial to this integrated approach, particularly as the agent launched on all global sites. Enablement experts supported the internal rollout with FAQs and other guides to entice employees to use the agent, better understand how it works, and report any observed issues to an internal triage team. The enablement and SEO teams are also currently working on training materials to help content developers write for large language models ( LLMs). 

Content can be tailored to user’s needs

The interactions with your website’s AI agent are more than just helpful exchanges for users. They represent a rich stream of valuable business intelligence. Think about it: Every question a visitor asks is logged in detail. This gives you an unfiltered view into your audience’s needs, and it’s a fantastic way to pinpoint content gaps and plan future content. 

“We definitely want to use this data to inform content on the site and to understand what the agent is finding most relevant and surfacing most frequently,” said Greenberg. 

In the future, these insights could even help refine marketing messages and SEO strategies, and help understand customers’ needs with greater accuracy.

What needs refinement

Pulling back the curtain means sharing not just the successes, but the areas ripe for improvement. The journey to a truly seamless AI-powered web experience is an iterative one. So, while Agentforce is already showing value, we’ve also identified where it needs further refinement. First off: How can we encourage more of our millions of site visitors to take that initial step and begin a conversation?

You can lead a horse to water but …

In April, the website received over 9 million visitors; however, only a fraction initiated conversations with the AI agent, despite its prominent placement on nearly every page.

Clearly, there was an opportunity to make interaction easier. To encourage use, we added preset prompts (see below) to help users get the conversation started. While it’s too early to measure their full impact, our product teams are already exploring new ways to personalize and contextualize these prompts.

For example, a user tooling around on the Sales Cloud page might get prompts specific to that product. If a user is a known entity, the prompts would be personalized to their industry, browsing, and purchase history.

To make the agent more visually engaging (links and text are so last-generation), the team is also experimenting with adding rich content – like content cards with embedded links for certain queries. Below, you’ll see the agent’s answer to a question about pricing tiers for CRM.

The team is also thinking about how to  make the agent more noticeable – by surfacing entry points to engage with the agent in different ways like directly with page content or as a key utility in the website header.

“We’re looking at whether we should have agents persistent in the main navigation so it’s the primary way of exploring the site. Also, there may be opportunities to have other invocation points embedded in the content of a page,” said Greenberg. 

Another possibility: an agent that activates itself with a greeting, instead of relying on the user to initiate a conversation.

Output quality is always a work in progress

Regularly measuring our agent’s accuracy and helpfulness is fundamental to its success. Agentforce currently gets High or Very High quality ratings for nearly 70% of its “utterances,” or responses to user questions. It’s a promising start, but we’re working to improve performance so users have the highest degree of confidence in the agent. That’s why gaining deep, granular insights into its real-world performance is the cornerstone of our iterative development approach.

To get there, we use Agentforce Interaction Explorer to monitor real-time conversations, and pair those insights with web and campaign analytics to measure performance against KPIs. This gives us a clear picture of what’s working — and what needs work.

Agentforce web experience: Enhancing engagement and efficiency

The AI agent on Salesforce.com operates as a digital worker, enhancing the customer experience and streamlining the sales process. Its primary functions include helping users find the information they need, answering their questions, guiding them through the site, offering tailored recommendations based on their activity, and capturing relevant interaction data. When users demonstrate strong interest or have needs requiring human expertise, the agent routes them to an SDR, an approach that can potentially improve conversion rates.

Available around the clock, the AI agent offers consistent real-time support and generates actionable insights from user interactions. These capabilities for continuous engagement and data analysis help refine how we connect with customers, leading to potential gains in productivity for SDRs, and increased customer satisfaction. This AI-powered digital worker is all about making customer interactions smoother and your sales process smarter and more effective from start to finish.