Docker 容器日志监控方案 Grafana + Loki + Promtail

在前文中，我们安装了 Grafana，请参考容器监控方案 cAdvisor + Prometheus + Grafana。

安装 Loki 和 Promtail

1. 创建配置文件夹并进入

mkdir /etc/loki
cd /etc/loki

2. 下载配置文件

wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

3. 修改 Promatil Config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://172.17.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs: attrs
  - json:
      expressions:
        tag: tag
      source: attrs
  - regex:
      expression: '(?P<image_name>[^|]+).(?P<container_name>[^|]+).(?P<image_id>[^|]+).(?P<container_id>[^|]+)'
      source: tag
  - labels:
      image_name: image_name
      container_name: container_name
      image_id: image_id
      container_id: container_id
  - output:
      source: output

4. 修改 Docker 日志选项

修改 /etc/docker/daemon.json，如不存在请创建。

{
   "log-driver": "json-file",
   "log-opts": {
      "tag": "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
   }
}

然后执行以下命令重启 Docker：

sudo systemctl daemon-reload
sudo systemctl restart docker

注意：容器需要使用 Docker 默认的 log driver，否则将无法获取到日志。

5. 启动 Loki 和 Promtail

docker run --name loki -d -v /etc/loki:/mnt/config -p 3100:3100 grafana/loki:3.0.0 -config.file=/mnt/config/loki-config.yaml
docker run --name promtail -d -v /etc/loki:/mnt/config -v /var/log:/var/log -v /var/lib/docker/containers:/var/lib/docker/containers --link loki grafana/promtail:3.0.0 -config.file=/mnt/config/promtail-config.yaml