Chatbot Data Storage - Storing Conversations and User Inputs

There are so many ways to store data for chatbots. In this post, learn how you can store data for your chatbot conversations.

When building out your initial pilot for a chatbot, it’s important to just start getting the chatbot out there so you can learn what your users are expecting from your chatbot. Based on my personal experience with implementing chatbots for large enterprises, having a strong analytical foundation of how you collect data of chatbot conversations along with the extract, transform, and load (ETL) processes is really important.

Importance of Choosing Data Structure and Data Storage

The reason why you should spend time making a decision on how to store data for chatbots is because the new enhancements and additions of features to your chatbot should come from the analytics that are derived from your bot’s data.

When we were rolling out our initial chatbot, we didn’t have a sense of the breadth of the type of skills and features that our chatbot would need. This meant that our conversation data architecture had very specific things to the initial skills that our chatbot had that was also combined with general information that any conversation would have (like input and outputs). What this meant was that any new skill that didn’t need a specific field in the JSON would have a blank, we could choose to grow the JSON message, or we create an entirely new message completely.

This caused a lot of heart ache as we continued adding skills. At one point, we had considered moving from noSQL to a SQL database architecture.

The tradeoffs is whether you want to spend time upfront to get the data structure right (SQL) or if you want to quickly get going and have the ETL process figure out the data later (noSQL). While gathering data using JSON format makes it easier to collect data due to its inherent noSQL structure, it added more time in the ETL processing side before we could make sense of the data.

The Importance of Chatbot Standardized Analytics Capture and Reporting

While it may not seem like out of the box reporting will fit all needs, having standard reporting capabilities when you are implementing a new technology allows you to at least have a foundation to start from when measuring chatbot metrics. The problem that we had when implementing nascent chatbot technology was that we were building our own definition of what it meant for something to be a chatbot conversation, chatbot sessions, and other query related metrics.

Boutique Chatbot Platforms with Reporting Capabilities

If you are willing to go to a boutique chatbot platform (which generally translates into thousands of dollars), you may find that the platforms offer out of the box standard reports that may be more customized than what the large cloud providers have to offer. For example, here’s a link to Bot Framework Analytics reports. Kore.AI and SmartBot had a lot of reporting capabilities built in with the ability to create your own visualizations of analytics and data.

Overview of Fundamental Chatbot Database Setup and Analytics

You can extract and send chatbot data using APIs. The three main actions that your chatbot architecture will need is the ability to extract data or user input, identifying and setting up your database store, sending data to the database (POST), and visualizing the data.

User inputs and conversations with the chatbot will need to be extracted and stored in the database. The user inputs generally are the utterances provided from the user in the conversation with the chatbot. Entities and intents can then be tagged to the user input.
The database that you setup should be based on your decision on whether to focus more development effort on determining the structure of the data on the front-end or whether you will spend time using ETL processes to format the data to get ready for analysis.
Determine what APIs are available from your database to flow back and forth into your chatbot platform.
The data can then be retrieved using a data visualizer like Tableau, PowerBI, or QuickSight to better analyze the customer experience for improvements in data modeling, bot responses, and overall information that can help the bot.

Advanced Chatbot Database Setup Techniques to Consider

Storing Relationships Between Entities

Graph databases like LUCID and Neo4J can be used by Freebase to store relationships between entities. It has a graph of nodes and edges. Each node has attributes and values for those attributes. This also allows edges to have attributes similar to nodes, and an edge connecting two nodes to define relationships between the nodes.

More on ETL Processes for Chatbot Analysis

Chatbot conversations can be stored in a SQL database that is hosted on a cloud platform. For example, if you were planning on creating a chatbot within the Microsoft Teams platform, you could use CosmosDB, a noSQL database with open APIs, to store your conversations and use PowerBI to visualize the reports. The piece that is missing from this is the way you structure your JSON messages. If you do not have a clean and well thought out data structure, you will potentially need transformational processes for your data in order for your data to make sense for reports.

Summary About Chatbot Conversation Storage

Chatbot analytics are important to ensure that you understand what your customers and users are seeking from a chatbot channel. By being intentional on how and what you capture from a user experience, you’ll be able to mine insights from what your customers are trying to tell you about what they want from your chatbot. It might be more information about products and services, it may be that they prefer troubleshooting via the chatbot, or it might be telling you that your other channels aren’t solving their problems. Start by understanding what type of insights you will want to gain insights on, structure your chatbot data architecture in a way that will capture that information for each and every conversation, and then choose the database (SQL or noSQL) from there, that will enable you to have a solid foundation of being able to get the information you are looking for.

How to Store Data for Chatbots