Promtail config and setup for Docker Swarm containers - Stack Overflow

admin管理员组

文章数量:1122832

I have a Promtail and Docker Compose config and setup that works fine but when I try to follow same for Docker Swarm cluster, logs are not showing up.

I have searched online for a doc on working config and setup for Docker Swarm with Promtail and unfortunately I could not find anything to help.

Here is error from Promtail container in Docker Swarm

level=error ts=2024-11-21T00:52:59.569451224Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

level=error ts=2024-11-21T00:53:10.069885261Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

Here is Promtail config am using

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        # __path__: /var/log/*log
        __path__: /var/log/!(auth.log)*log

  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"] 
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'
    pipeline_stages:
      - cri: {}
      - multiline:
          firstline: ^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}
          max_wait_time: 3s
      # /
      - json:
          expressions:
            #message: message
            level: level
            #output: 'message'

and here is the Docker Swarm service I have the logging labels attached to

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"
    max-file: "1"
    tag: "{{.Name}}"

services:

  app-1:
    image: "app-1:v0.1"
    networks:
      - apps
    env_file:
      - ./env/.env.app-1
    deploy:
      mode: replicated
      replicas: 2
      placement:
        constraints:
          - node.labels.node != node-1
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    logging: *default-logging

networks:
  apps:
    external: true

But when I check Loki, I do not see the container and logstream labels in Loki dashboard like it does for Docker Compose one that works with the same above config

What am I doing wrong and what do I need to fix so I can see the container and logstream labels to filter to the container logs I want to view logs for?

Docker Swarm labels are not showing up in Loki

Docker Compose works fine and labels show up in Loki

I have a Promtail and Docker Compose config and setup that works fine but when I try to follow same for Docker Swarm cluster, logs are not showing up.

I have searched online for a doc on working config and setup for Docker Swarm with Promtail and unfortunately I could not find anything to help.

Here is error from Promtail container in Docker Swarm

level=error ts=2024-11-21T00:52:59.569451224Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

level=error ts=2024-11-21T00:53:10.069885261Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

Here is Promtail config am using

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        # __path__: /var/log/*log
        __path__: /var/log/!(auth.log)*log

  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"] 
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'
    pipeline_stages:
      - cri: {}
      - multiline:
          firstline: ^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}
          max_wait_time: 3s
      # https://grafana.com/docs/loki/latest/clients/promtail/stages/json/
      - json:
          expressions:
            #message: message
            level: level
            #output: 'message'

and here is the Docker Swarm service I have the logging labels attached to

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"
    max-file: "1"
    tag: "{{.Name}}"

services:

  app-1:
    image: "app-1:v0.1"
    networks:
      - apps
    env_file:
      - ./env/.env.app-1
    deploy:
      mode: replicated
      replicas: 2
      placement:
        constraints:
          - node.labels.node != node-1
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    logging: *default-logging

networks:
  apps:
    external: true

But when I check Loki, I do not see the container and logstream labels in Loki dashboard like it does for Docker Compose one that works with the same above config

What am I doing wrong and what do I need to fix so I can see the container and logstream labels to filter to the container logs I want to view logs for?

Docker Swarm labels are not showing up in Loki

Docker Compose works fine and labels show up in Loki

• docker-compose
• docker-swarm
• grafana-loki
• promtail

Share Improve this question Follow edited Jan 2 at 21:23 halfer 20.4k1919 gold badges108108 silver badges200200 bronze badges asked Nov 21, 2024 at 11:24 uberrebuuberrebu 4,2791111 gold badges5555 silver badges8787 bronze badges

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 0

The thing I find odd is you are reading logs from /var/log/* which IS a place logs are written on linux, but not the place container logs are principally written.

When writing logs in swarm, it's important to do two things:

First, include the com.docker.* labels in the json-log metadata. This entails adding the following to your daemon.json:

"log-driver": "json-file",
  "log-opts": {
    "labels-regex": "^com.docker.+",
  }

Then, I use the following promtail config.

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

This uses the labels added by the docker compose, docker service and/or docker stack commands to discern the compose or swarm service name etc.

issue fixed, i needed to deploy promtail on docker swarm cluster as global mode, not on 1 node only

so updated promtail to be global deploy on all nodes in the cluster so it can scrpae the logs from the nodes

issue was i only had it on just one node and thus will not work to scrape logs for the other nodes

  promtail:
    image: grafana/promtail:2.9.1
    command: "-config.file=/mnt/config/promtail-config.yaml"
    volumes:
      - ./promtail/promtail-config.yaml:/mnt/config/promtail-config.yaml
      - /var/log:/var/log
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    deploy:
      mode: global

本文标签： Promtail config and setup for Docker Swarm containersStack Overflow