Artificial Intelligence for Customer Attraction – Why and How to Develop It?
Author: Egor Kopylov, founder of Topspace project, more than 10 years of experience in Tech leadership, developer of five Yandex services and dozens of own IT projects.
According to a study by McKinsey, the level of AI implementation in business has grown to 72% in 2024 (in 2023, this figure was 55%). The use of generative AI, which creates texts, videos, and images on demand, has doubled to 65%.
Generative AI is particularly popular in marketing and sales, as well as in product and service development (57%).
Today, there are ready-made AI solutions for generating leads. But it is better for a business to develop its own AI, which will fully take into account its needs and product features. How to write your own AI for lead generation? This article will talk about it.
A New Approach to Lead Generation
AI tools for lead generation have been used in advertising for quite a long time. On the same advertising platforms of Google and Yandex. But they use traditional tools, that is, they create a profile of a potential client and show him ads based on his supposed interests.
A new approach to lead generation using AI involves first “getting to know” the user based on many parameters: gender, age, interests, publicly available messages (Internet, public chats, comments on social networks), etc. LLM models and machine learning models take in many parameters, analyze this text data and, based on it, create a customer profile. Then they generate messages for him and offer a relevant product/service.
A fairly wide range of machine learning models are used here – from classical ones that accept and analyze various types of parameters and categories, to natural language machine learning models – the so-called LLM models.
Thus, AI technologies help to more accurately profile users and determine the target audience of a business.
Why Build Your Own AI for Lead Generation
By developing its own AI solutions, the company fully controls the parameters and algorithms for training AI, ensures the confidentiality and security of clients' personal data. And then it will always be able to modify its AI tool to new market conditions and business scales.
Typically, large companies create their AI models from scratch or modify models from open sources. As a result, they receive a personalized tool – a chatbot or an intelligent system that becomes part of their business processes.
To do this, you need to have your own deep expertise in machine learning. For example, Perplexity itself prepared the data and refined the architecture based on the GPT-4o and Claude 3 language models. As a result, it created its own search engine and chatbot that can communicate with users on various topics. The principle of the chatbot is similar to Chat GPT. But Chat GPT solves more general work tasks, and Perplexity offers a highly personalized assistant.
Read more materials on this topic in Compass CIO
Stages of implementation of AI-system for lead generation
The first stage is the formation of a knowledge base
Developers work closely with businesses, receiving detailed product descriptions and their unique features from the company. Based on this knowledge base, developers can compare user requests with the business's offer.
The second stage is preparing the dataset
A training dataset is a structured array of information that will be used to test leads. The first dataset is formed from a set of available messages. It is loaded into a neural network. The neural network then selects the data that best matches the business product.
Then the developers manually separate the examples successfully selected by the neural network from the incorrect ones.
The third stage is further training of the model
The developers will further train the model based on the prepared dataset. Two key tools will help with this.
1. DSPy – an automatic prompt optimization tool from the Stanford NLP School. It can be used to automatically select the necessary words for the LLM model based on a selected data set. These words were entered by the business in text format at the first stage.
DSPy allows you to refine these queries by adding relevant examples to the description. The tool also eliminates unnecessary words that may prevent the LLM model from correctly classifying leads.
2. LoRA (low-rank adaptation) is a tool for refining open-source models. LoRA adds several parameters to the main model and allows you to automatically retrain it based on a prepared data set.
The fourth stage is launching the model
The developers evaluate the result and, if necessary, refine the model.
At this stage, they are more focused on tracking business metrics. For example, a typical chat may consist of 100,000 user messages. Developers analyze them and select 2,000 leads who wrote messages that match the business's product or service. They then pass this information on to the business, and sales staff contact potential customers and lead them to purchase.
In certain cases, developers create an AI product that can complete a deal on its own. In this case, the neural network generates messages to the client, trying to figure out his requests and sell the product.
Stage five – evaluation of results
The results are compared with the initial business request. The quantity and quality of leads are also assessed here.
Libraries for creating chatbots
There are many libraries that can be used to create chatbots. These libraries are divided into several categories.
1. Libraries for working with prompts. Prompts are text fragments that are entered into LLM models. The libraries are designed to transform natural language (user messages) into specialized types of data. The LLM model converts the received data into numbers, and the framework extracts these numbers from the text and provides them to the programmer.
2. Libraries for creating AI agents that can act autonomously or in collaboration. They also use text descriptions to enter into LLM models, but allow you to specify the behavior of mini-bots at a higher level. The developer must not only specify the desired data type, but also provide the bot with tools to perform the task from the main prompt.
3. Research libraries. Designed to create fine-tuning models using the DSPy and LoRA tools mentioned earlier. Developers usually do not engage in such deep research; this is the task of machine learning specialists.
How to choose a framework?
There are several important factors to consider when choosing an AI software tool.
First, you need to evaluate the complexity of learning the library in the context of the task to be solved. Some frameworks, such as LangChain in Python, may be too complex for simple tasks: predictive analytics, generating text and visual content, training chatbots, etc. In such cases, it is better to choose lighter libraries or frameworks from Microsoft.
Secondly, pay attention to how actively the model is maintained and how long it has been in development. This information can be found on GitHub. If the developers do not update the library, then its integration and support will take too much of your time.
Thirdly, it is necessary to evaluate the popularity and standardization of the tool. There are libraries that companies create for themselves and publish in the public domain. For example, PWA and LangChain are well-known frameworks for working with promts and LLM models that can be considered reliable.
If you have a small IT team, then choose more standardized and debugged tools for working with AI technologies.
Team of AI specialists
The size of the IT team depends on the complexity of AI projects. For applied solutions with a small fine-tuning model, five people will be enough to program the behavior of chatbots using well-known libraries. In this case, narrow ML specialists are not needed, since you use ready-made LM models through the API of large IT companies.
To deploy LM models on your own premises, you will need a team of 20 developers and your own technological base (data centers, servers). These are usually companies that provide other developers with access to their libraries and frameworks.
Deep technical expertise in LM development requires a team of 50-100 machine learning specialists who understand the model architecture and can further train it. These are usually vertically integrated companies that create models from scratch.
Developing a proprietary AI gives businesses a lead generation tool that fully takes into account the specifics of their product and customer needs and, consequently, increases sales effectiveness.