Improving answers’ accuracy with machine learning models for chatbot – a case study
How do we boost chatbot’s user retention with machine learning models? Read the case study of the cooperation with DRUID AI – a leading conversational AI platform provider – and find the answer.
Access to all sources of information within the enterprise is essential. User questions must be answered with information managed securely in structured and unstructured data sources.
DRUID AI virtual assistants are able to learn from any enterprise data sources and provide accurate responses, applying security and enterprise governance.
The disconnect between what you’re seeking and what you get feels like trying to catch a butterfly with a fishing net – you’re left with something, but it’s not what you had in mind.
That’s why constant updating of the knowledge of chatbot answers is crucial for maintaining users’ retention and value delivered.
We’re lucky enough that contemporary technology improves chatbot accuracy to almost perfection.
Read further to learn how a mixed project team, built with Synergy Codes’s and DRUID AI’s experts, achieved it.
About project
DRUID AI hired a machine learning team from Synergy Codes to construct a rich knowledge base consisting of information from various data sources.
The end product had to be able to search for answers within the knowledge base and use semantic logic. In this way, DRUID AI wanted to improve the chatbot’s response accuracy and keep a competitive edge.
Who is DRUID AI?
DRUID AI is a leader in conversational AI technology with over 25 years of experience in building enterprise-grade software solutions. Founded in 2018, the organization gathers more and more top software engineers and business consultants. That makes DRUID AI one of the fastest-growing AI companies in Central and Eastern Europe.
Their vital mission is to make chatbots more accessible for end-users and creators.
The main product – enterprise AI conversational platform – enables and simplifies designing, developing, and deploying chatbots. The NLU engine can accurately read users’ intents, sentiments, and system entities.
DRUID AI is on its way to becoming a significant player in the global conversational AI landscape, cooperating with top companies like Microsoft, UilPath, KPMG, and Deloitte.
Get to know the chatbot platform’s potential and functionalities in the medical industry.
DRUID AI’s goals and expectations
Becoming a leading conversational AI provider is a real challenge. Knowing that, DRUID AI’s product owners started to wonder how to achieve a competitive edge in this high-competitive environment.
The answer assumed not to create new shiny features but rather to focus on reinforcing the platform’s capabilities to retain current customers and win new deals.
The first attempt was creating a knowledge base – a rich database consisting of structured data (questions and answers imputed automatically from Excel and PDF files or manually by admin) and a vast amount of unstructured data, such as website content.
Additionally, the knowledge base had to be equipped with a mechanism enabling semantic search for answers. And it had to do it fast.
You can only be 100% sure if an idea is profitable once you test it.
So, at first, a product owner from DRUID AI decided to create a demo to check both:
- if the knowledge base is, in fact, a promising idea to win new deals,
- if the vendor’s tech experts are skilled enough to deliver an efficient knowledge base with implemented machine learning.
Scope and challenges
The project’s general requirements have been defined. However, they needed to be written down in a clear scope with a timeline.
Scope
- Building a knowledge base consisted of structured (questions and answers imputed by admin) and unstructured (website content) data leveraging machine learning for chatbot.
- Enabling semantic searching within the knowledge base.
- Building a demo of an application showing the capabilities of the knowledge base.
As you already know, DRUID AI wanted to supercharge its platform with a knowledge base consisting of structured and unstructured data.
However, the real impact of the platform’s usability was a semantic search.
So, first of all, the project team had to come up with an idea of how to implement this feature into the emerging knowledge base.
Afterward, they could switch to building a knowledge base and think about how to power it with structured and unstructured data.
Eventually, these solutions had to be covered in an application demo showing the knowledge base capabilities.
Challenges
The aim was bold, so the path to it had to be bumpy.
The biggest obstacle on the way was the dualistic nature of the knowledge base. Because it includes structured and unstructured data, the project team had to figure out two machine learning models leveraging for chatbot to parse this information.
Approach and solutions
Jakub Błaszczyk, Machine Learning Engineer at Synergy Codes
“At the beginning, the scope of the project was broad. The knowledge base had to include information structured into questions and answers and unstructured data from various websites. So, we had to divide the project into stages.”
First stage: imputing structured data leveraging machine learning models for chatbot
Jakub – the project team leader – launched the work by imputing structured data based on questions and answers delivered by the platform’s admin.
After parsing the data, he implemented a machine learning model. This time, Jakub decided to use a pre-made model because:
- there was no time and other resources to create an authorial training model from scratch,
- the pre-made model had a valid license and was adjusted to DRUID AI’s expectations.
This decision shortened the overall development time and enabled faster starting work on the second stage.
However, to bring DRUID AI an immediate value – Jakub at this stage went through all of the remaining phases:
- data handling,
- ML models preparation,
- semantic searching,
- business logic,
- building an application.
He invited Mateusz to the team – a web developer experienced in creating web applications. His role was supporting Jakub with creating a demo of the application that could present the knowledge base capabilities.
Enabling semantic searching within the knowledge base
To enable semantic searching for the answers to users’ questions within the knowledge base, its whole data and the text of questions had to be transformed into vector form.
To achieve this, the project team decided to use a language model. It can eliminate all grammatical or spelling mistakes in defining queries or in the knowledge base’s data.
Language models are taught on such big datasets that can reflect, almost ideally, the real usage of language, including grammatical or spelling mistakes.
Eventually, the knowledge base was able to search semantically for answers – it knew how to connect questions with answers and how to read synonyms.
Second stage: imputing unstructured data leveraging machine learning models for chatbot
The real challenge occurred while imputing unstructured data, specifically website content. Because websites can be created in various ways, coming up with one universal parset is difficult.
Jakub decided to consider only these parts of websites created with HTML5 semantics. Consequently, he divided the valuable content from useless parts of websites, such as navbars, footers, or pop-ups.
Another step was imputing gathered data into the knowledge base.
However, because the database included vast information from websites was so big, it needed to be parted into small chunks of information – paragraphs.
Jakub prepared a second model for parsing the data parted into paragraphs.
It differs from the one used in the first stage. The previous model had to compare usually one-sentence questions to the answers with similar lengths.
However, in this case, a one-sentence question had to be compared with paragraphs built with 10, 15, or even 20 sentences.
The knowledge base powered by this machine learning model could show, for instance, five paragraphs that include the most accurate information and the answer to the user’s question within them.
After dealing with indexing unstructured data, Jakub and Mateusz went through the same steps from stage first (data handling, semantic searching, and business logic), developing the application with another features.
The results
Let’s look closer at the results that ended the working process.
DRUID AI received:
- a Proof of Concept delivered within two days from the project’s launch that presented the skills of the project team,
- a demo of the application, delivered within six weeks from the project’s launch, presented the knowledge base’s capabilities for semantic searching for answers.
The finishing touches included preparing clear documentation of the project that explained how to integrate the application with DRUID AI’s platform through Open API.
The demo was successfully used as a real-value product and generated profits for the company.
We are proud of our two experts, Jakub and Mateusz, who brought impactful value to the DRUID AI business within the short term.
Alexandra Ciobotaru, Product Owner at DRUID AI
“I appreciate the teamwork. It was important to keep all developers aligned, as multiple competencies were involved: machine learning, Python development, solution architecture, and integrations. I appreciate the support we received from all our colleagues, their self-reliance, and the fact that they kept the project on track. Finding an innovative solution for dynamic requirements was valuable and allowed us to keep the deadlines with our product roadmap.”
However, it’s not the end.
The described project was our another cooperation with DRUID AI. In the past, we joined forces to redesign and improve the platform’s user experience.
If you’d like to know more, read a case study of that project.
We also have plans for the future because our cooperation continues.
