how to store session data into custom storage in bot builder

bot framework save user data
implementing-a-sql-server-database-with-the-microsoft-bot-framework
bot builder doc
session.userdata bot framework
turnstate bot framework
await turncontext sendactivityasync
conversationstate
services addsingleton istorage

I am using node js to build chat bot.

I'm currently storing session data into Microsoft default storage, which have a limit of 64K per user session. I want to use my own storage to store session data. Here is what I got help from Microsoft developer.

I am able to store in document DB and Azure table.

Still, I am confused. How we will implement the IStorageClient interface to store in own DB?

Whenever I set session.UserData.name="" it should store in own DB.

I'm not sure if I got the issue; I understand what you are trying to do and it seems you have all the pieces in place.

First, you have to implement the IStorageClient interface. In your implementation, you write the logic to store things in your DB.

You can review the DocumentDB and Azure Tables implementations to get an idea of how that can be implemented.

Now, the question is.. if your custom storage is in Azure..., then you might be able to use the AzureBotStorage with your custom storage client. You instantiate your custom storage client, pass a reference to the AzureBotStorage and set that as the storage in the bot

// Azure DocumentDb State Store
var docDbClient = new azure.DocumentDbClient({
    host: process.env.DOCUMENT_DB_HOST,
    masterKey: process.env.DOCUMENT_DB_MASTER_KEY,
    database: process.env.DOCUMENT_DB_DATABASE,
    collection: process.env.DOCUMENT_DB_COLLECTION
});
var botStorage = new azure.AzureBotStorage({ gzipData: false }, docDbClient);

// Set Custom Store
bot.set('storage', botStorage);

If your custom storage is in any other place than Azure, then AzureBotStorage might not work for you. Please note that I'm not sure of this, you will have to check the code to confirm. From what I saw, it seems pretty generic, and so you might be able to reuse it and just implement IStorageClient. If that's not the case, you will have to also implement the IBotStorage interface.

At the end, AzureBotStorage is doing that. It implements IBotStorage and uses an IStorageClient to interact with the actual provider.

How to save MS bot state data and conversations ?, You would use a custom implementation like this // Create new storage with redis client var storage = new YourStorage() // this is just here for BotBuilder source for the default storage - https://github.com/Microsoft/BotBuilder/blob/ does except save to cosmos or whatever your storage engine is. session. For testing and prototyping purposes, you can use the Bot Builder Framework's in-memory data storage. For production bots, you can implement your own storage adapter or use one of Azure Extensions. The Azure Extensions allow you to store your bot's state data in either Table Storage, CosmosDB, or SQL.

I'm posting this answer on a very, very old thread just because it seems to be on the top results for implementing custom storage options for Bot Framework.

There's a considerably easier solution that does not involve using npms that have not been updated in a while that carry a risk of not being auditable for corporate environments or that will flood your console with huge amounts of log entries.

The easiest way is to create your own Data Storage code for your bot using the most up-to-date driver for the db of your choice by following the steps in this Undocumented Guide for Custom Data Storage in Microsoft Bot Framework

It took me about an hour to implement for MongoDB, including some learning curve for their newest driver.

In short, in case the link dies at some point, you can create a middleware that exports two key functions that Bot Framework is looking for: getData and saveData. Each of these have to have the following rough structure (from the link using Redis):

var storageKey = 'conversationId';
module.exports = {
getData : (data, callback) => {
    redisClient.get(data[storageKey], (err, reply) => {
        if (err || !reply) {
            callback(err, null);
        } else {
            callback(null, JSON.parse(reply));
        }
    })
},
saveData : (data, value, callback) => {
        redisClient.set(data[storageKey], JSON.stringify(value), (err, reply) => {
            if (err) {
                callback(err);
            } else if (reply === 'OK') {
                callback(null);
            }
        })
    }
}

Then, in your app.js, be sure to require() your middleware and set it accordingly:

const dbMiddleware = require('./path/to/middleware')

bot.set('storage', dbMiddleware)

how to store session data into custom storage in bot builder, I am using node js to build chat bot. I'm currently storing session data into Microsoft default storage, which have a limit of 64K per user session. I want to use my  For testing and prototyping purposes, you can use the Bot Builder Framework's in-memory data storage. For production bots, you can implement your own storage adapter or use one of Azure Extensions. The Azure Extensions allow you to store your bot's state data in either Table Storage, CosmosDB, or SQL.

I written code to store bot data into Mongo Db. Full code and details can find in BotBuilder-MongoDB

IstorageClient Interface implementation code:

"use strict";
var Consts = require('./Consts');
var mongodb_1 = require("mongodb");
var replaceDot_Atrate = require("./replaceDot");
var mongoDbConnection = require('./connection.js');
var conf = require('../config');
var conversational_collname = conf.db.conversationalCollection;

var IStorageClient = (function () {

    function IStorageClient(options) {
        this.options = options;
    }

    IStorageClient.prototype.retrieve = function (partitionKey, rowKey, callback) {
        var id = partitionKey + ',' + rowKey;
        if(rowKey!=="userData"){
            var query={"$and":[{"userid":id}]}
                mongoDbConnection(function(err,db) {
                var iterator= db.collection(conversational_collname).find(query);
                iterator.toArray(function (error, result, responseHeaders) {
                    if (error) {
                        console.log("Error",error)
                        callback(error, null, null);
                    }
                    else if (result.length == 0) {
                        callback(null, null, null);
                    }
                    else {
                        var document_1 = result[0];
                        var finaldoc=replaceDot_Atrate.substituteKeyDeep(document_1, /\@/g, '.');
                        finaldoc["id"]=id
                        callback(null, finaldoc, null);
                    }
                });
            }); 
        }
        else{
            var query={"$and":[{"userid":partitionKey}]}
            mongoDbConnection(function(err,db) { 

                var iterator= db.collection(conversational_collname).find(query);
                iterator.toArray(function (error, result, responseHeaders) {
                    if (error) {
                        callback(error, null, null);
                    }
                    else if (result.length == 0) {
                        //console.log("result length 0")
                        callback(null, null, null);
                    }
                    else {
                        var document_1 = result[0];
                        callback(null, document_1, null);
                    }
                });
            });
        }
    };

    IStorageClient.prototype.initialize = function (callback) {
        var _this = this;
        var client=mongodb_1.MongoClient;
        this.client = client;

        mongoDbConnection(function(err,database) {    
                _this.database = database;
                _this.collection = database.collection(conversational_collname);
                callback(null);
         });
    };

    IStorageClient.prototype.insertOrReplace = function (partitionKey, rowKey, entity, isCompressed, callback) {
        var id=partitionKey + ',' + rowKey
        var docDbEntity = { id: partitionKey + ',' + rowKey, data: entity, isCompressed: isCompressed };
        if(rowKey!=="userData"){
            var newEntitiy=replaceDot_Atrate.substituteKeyDeep(entity, /\./g, '@');
            var conditions1 = {
                'userid': id
            };
            var updateobj1 = {
                "$set": {"data":newEntitiy,"isCompressed":false}
            };   
            mongoDbConnection(function(error,db) {    
                db.collection(conversational_collname).update(conditions1,updateobj1,{upsert: true},function(err,res){
                callback(error, null,"");
            });
            });
        }
        else{
            var conditions = {
                'userid': partitionKey
            };
            var update = {
                "$set": {"data":entity}
            }
            mongoDbConnection(function(error,db) {    
                db.collection(conversational_collname).update(conditions,update,{upsert: true},function(err,res){
                callback(error, null,"");
           })
        });
        } 
    };


    IStorageClient.getError = function (error) {
        if (!error)
            return null;
        return new Error('Error Code: ' + error.code + ' Error Body: ' + error.body);
    };

    return IStorageClient;
}());
exports.IStorageClient = IStorageClient;

Manage state data (v3 JS) - Bot Service, Learn how to save and retrieve state data with the Bot Framework SDK for Node.​js. If you store a user's preferences, you can use that information to customize the you can use the Bot Builder Framework's in-memory data storage. In the Bot Framework SDK for Node.js, the session object exposes the  @willportnoy in #2073 (comment), we talked about activity logger, but here my requirement to store session data into custom storage. Means let's say session.userdata.name="aakash" it should store in my custom storage.

Here is another take on this library: mongo-bot-storage

import {MongoDbBotStorage, MongoDBStorageClient} from "mongo-bot-storage";

// use it like this 
bot.set("storage", new MongoDbBotStorage(new MongoDBStorageClient({
    url: "mongodb://localhost/mydb",
    mongoOptions: {}
})));

// or like this
MongoClient.connect("mongodb://localhost/mydb", (error, db) => {
    bot.set("storage", new MongoDbBotStorage(new MongoDBStorageClient({db})));
});

and then use the default session.userData, etc..

don't forget to

// Enable Conversation Data persistence
bot.set('persistConversationData', true);

Write directly to storage - Bot Service, Learn how to write directly to storage with the Bot Framework SDK for . data your bot uses to preserve a conversation, or data that comes from a source Bot.​Builder; using Microsoft.Bot.Schema; using System.Collections. Interact with your bot as you normally would. When you are done, go to Storage Explorer and view your saved state data. View data in Storage Explorer. To view the state data, open Storage Explorer and connect to Azure using your Azure Portal credential or connect directly to the Table using the storageName and storageKey and navigate to your tableName.

how we can store conversation state in our own database (sql or , bot.dialog('/profile', [ function (session) { builder.Prompts.text(session, 'Hi! @​aakashvit is your question on how to enable the custom storage providers Storing data in a database from within a Bot is no different from storing  Once your storage is set to point to your Blob Storage account, your bot code will now store and retrieve data from Blob Storage. Start your bot. Run your bot locally. Start the emulator and connect your bot. Next, start the emulator and then connect to your bot in the emulator: Click the Create new bot configuration link in the emulator "Welcome" tab.

6 Managing State Data, Corresponding to this bot framework lets us store data at a user level across all Listing 6.6 Manually calling the save method of the session object. 1 like their name so we could later access that name to give personalized replies using their name. Data from a userData storage container can be removed by setting the  This connection string is what will be used by your bot to call the storage service to save state data. Note: The key will automatically be provided in the connection string generated by Azure. Configuring the Bot for Azure Table. Click here for the full Azure Table bot sample on hosted Github.

Saving State data with BotBuilder-Azure in Node.js, Saving State data with BotBuilder-Azure in Node.js We've discussed strategies to store state data for bots in .NET, in In the dialog, we use the session object and pass in the custom AzureBotStorage object we created, which will override the MongoDB Implementation for Node.js custom state storage  Set up the Azure Table storage service. After you’ve logged into the Azure portal, create a new Azure Table storage service by clicking on New. Search for Storage account that implements the Azure Table. Fill in the fields, click the Create button at the bottom of the screen to deploy the new storage service.

Comments
  • Thanks For quick reply. I already tried to store into Document Db and azure table. both example are working fine for me. But i want to use mongo db which is located into my own server. I am new to node js. Got confused how I can implement IStorageClient interface. You can see what Microsoft devs are suggesting github.com/Microsoft/BotBuilder/issues/1943
  • I know what they are suggesting. But I don't understand your issue. You are looking for someone to write the code for you? You implement that interface and put the code to interact with Mongo DB. There isn't much more than that
  • yeah exactly . if some one provide how we will code it. I am very much familiar with Mongo. I can code node js with Mongo. But not able to get how i can use this interface.. I was looking for sample example of IStorageClient . Or at least some basic logic .Rest logic for Mongo code i will do .
  • But the examples are basically those I put you in the answer. I linked to the code to both the document DB and Azure table implementation. Is just an interface, you need to go implement 2 methods
  • While the linked repository may be an example of what you describe, you should summarize the relevant parts here.
  • very very bad idea, have you seen how many entries it creates? i was chatting with the bot it stored 100 records within 2 mins for a conversation. Imagine what happens when 1000s of your users are chatting simultaneously, what you wanna do is only store the user id and name the first time they chat
  • @PirateApp , I implemented interface which Microsoft provide, even you use Redis or Microsoft Document db, number of call will be same. And for your information, we are using this layer since last 1 year with 500 user approx, ans it is fast enough.
  • @AakashKag have you tried it with a bigger audience set by any chance? curious to see if any metrics are available
  • fixed it. though that clicking on a link and reading the information over there will be sufficient, as this one is an actual library and not some kind of hack together half a project thing in accepted answer
  • @PaulRysevets you copied my solution, What's new in your solution except npm?? In what way it is better then my solution ? github.com/Microsoft/BotBuilder/issues/3095
  • @AakashKag, did you, actually, read the answers on that thread that you linked? Do you understand what is "not that good" with your code?
  • I create npm module also, It is faster then my older solution npmjs.com/package/botbuilder-mongodb github.com/Manacola/msbotframework-mongo-middlelayer