fluent bit - Nested JSON parsing with elasticsearch - Stack Overflow

admin管理员组

文章数量:1244411

I am having issues with our AWS ecs -> fluentbit -> elasticsearch set up, specifically around nested json.

For example, if the log message is:

{
  "endpoint": "/process",
  "payload": {
     "body": {
       "success": "true",
       "items": [
          {"name": "item_one"},
          {"name": "item_two"}
       ]   
     }
  }
}

We would like the following fields to be parsed:

endpoint -> "process"

payload -> {"body": {"success": "true", "items": {"name": "item_one"}, {"name": "item_two"}]}

Only the top level key.

We set "index.mapping.depth.limit": 1 but this resulted in the logs being rejected by elasticsearch

    "status":400,
    "error":{
        "type": "illegal_argument_exception",
        "reason": "Limit of mapping depth [1] has been exceeded due to object field []"
    }

Is there a setting that will parse only the top level but accept the rest of the data as the body in elasticsearch?

Or is this something that should be solved in the fluentbit level?

本文标签： fluent bitNested JSON parsing with elasticsearchStack Overflow