Automatically create data stream about logstash-output-opensearch HOT 3 OPEN

@moharana-subhashree I could be wrong, but I don't think the plugin is actually capable of knowing that you want a data stream here, and it implies that you should create that data stream index yourself. Does that work?

In that case we should 1) turn this into a feature request, 2) document this better in https://opensearch.org/docs/latest/tools/logstash/ship-to-opensearch/#data-streams

from logstash-output-opensearch.

hi @dblock,
as in the above post, it is concluded that the creation of data stream on ourselves is mandatory and doesn't support creating it dynamically by the plugin for now.
Recently i was trying below opensearch configuration as below to create a legacy index template as part of the opensearch configurations as below

output {
    opensearch {
      hosts => "http://localhost:9200"
      index => "logs"
      action => "create"
      document_id => "%{id0}"
      template_name => "cas-template"
      template => "/usr/share/logstash/pipeline/template/cas-template.json"
      manage_template => "true"
      template_overwrite => "true"
    }
}

I have a legacy index template format as below provided in the configurations:

    {
      "index_patterns": ["logs-*"],
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "properties": {
          "timestamp": {
            "type": "date"
          },
          "message": {
            "type": "text"
          }
        }
      },
      "aliases": {
        "logstash": {}
      }
    }

but with this configuration it ideally shall create a index template and that index template could be used for data stream creation, is my understanding correct? will this be a right way to proceed?

from logstash-output-opensearch.

but with this configuration it ideally shall create a index template and that index template could be used for data stream creation, is my understanding correct? will this be a right way to proceed?

Yes, I can't think of a reason why one would not want this, so this is a feature request. I'll rename the issue.

from logstash-output-opensearch.