In our previous article, we explored how to migrate data from MongoDB to Elasticsearch using Logstash. Given that Coco AI's backend also utilizes Elasticsearch for data storage, it's natural to consider whether we can directly transfer MongoDB data into Coco AI's Elasticsearch instance and leverage Coco AI's search capabilities. The answer is yes!

Connector Setup

Coco AI's Connector can be created through two methods: via API or through the management interface. Those who have completed the first part of the "Private Knowledge Base with Coco AI" series should already have access to the management platform. We will now proceed to create a Connector using the management interface. For API-based creation, please consult the official documentation.

Log into the management console, navigate to Settings -> Connector -> Add New

DataSource Configuration

In DataSource -> Add New -> MongoDB

Note the DataSource ID: d037kjj75bvg264k5pe0, which will be required for configuring Logstash later.

Elasticsearch Compatibility

Since we're connecting to Elasticsearch via Logstash, compatibility mode needs to be enabled. Modify the easyserach.yml file accordingly. Refer to the guide titled "How to Connect Elasticsearch 8 Using Logstash" for detailed instructions.

elasticsearch.api_compatibility: true
elasticsearch.api_compatibility_version: "8.9.0"

Logstash Configuration

We'll make minor adjustments to the previous Logstash configuration used for migrating MongoDB data to Elasticsearch. The updated configuration includes document source metadata and writes to the coco_document index.

input {
  jdbc{
    jdbc_driver_class => "Java::com.wisecoders.dbschema.mongodb.JdbcDriver"
    jdbc_driver_library => "/usr/share/logstash/driver/mongojdbc4.8.3.jar"
    jdbc_user => "user"
    jdbc_password => "pwd"
    jdbc_connection_string => "jdbc:mongodb://localhost:27017/test"
    statement => "db.collection_test.find({},{'_id': false})"
  }
}

filter {
    mutate {
        rename => {
            "[document][tags]" => "tags"
            "[document][summary]" => "summary"
            "[document][username]" => "owner.username"
            "[document][content]" => "content"
            "[document][category]" => "category"
            "[document][created]" => "created"
            "[document][url]" => "url"
            "[document][id]" => "id"
            "[document][title]" => "title"
        }
        remove_field => [ "document","@timestamp","@version" ]
        add_field => {
            "[source][type]" => "connector"
            "[source][name]" => "MongoDB Datasource"
            "[source][id]" => "d037kjj75bvg264k5pe0"
        }
    }
}

output {
    #stdout { }
    elasticsearch {
        hosts => ["https://127.0.0.1:9200"]
        index => "coco_document"
        manage_template => false
        ssl_verification_mode => none
        user => "admin"
        password => "coco-server"
    }
}

Testing Search Functionality

Once the data migration is complete, verify that documents can be successfully searched within Coco AI.