使用 ElastAlert 監控 Elasticsearch 發出通知

之前筆記 使用 Docker Compose 建立 ElastAlert 測試環境 紀錄到該如何使用 docker compose 快速建立 ElastAlert 的開發測試環境,今天要來筆記一下如何設定 ElastAlert 來發出警告

除了 Elasticsearch 外,最近也接著在進行 Prometheus 的相關設定測試,想不到過個幾天就覺得 ElastAlert 好陌生呀,想著如果不趕緊來紀錄一下 真的需要實際使用時還忘記怎麼設定就尷尬了

為了測試與驗證,我嘗試了 emailpost 兩種方法,今天會紀錄這兩種用法的設定方式

基本環境說明

  1. macOS Catalina 10.15.7
  2. docker desktop 3.1.0 (51484)
  3. Fiddler Everywhere 1.5.0
  4. docker images

    • elasticsearch:7.10.1
    • yowko/fluentd-elasticsearch:1.0.0
    • kibana:7.10.1
    • praecoapp/elastalert-server:20210104
  5. docker-compose

    詳細內容請參考之前筆記 使用 Docker Compose 建立 ElastAlert 測試環境

    version: '3.2'
    services:
      elasticsearch:
        image: elasticsearch:7.10.1
        volumes:
          - type: bind
            source: ./config/elasticsearch.yml
            target: /usr/share/elasticsearch/config/elasticsearch.yml
            read_only: true
        ports:
          - "9200:9200"
          - "9300:9300"
        environment:
          ES_JAVA_OPTS: "-Xmx256m -Xms256m"
          discovery.type: single-node
        networks:
          - elk
        healthcheck:
            test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
            interval: 30s
            timeout: 15s
            retries: 3
            start_period: 180s
      fluentd:
        image: yowko/fluentd-elasticsearch:1.0.0
        volumes:
          - type: bind
            source: ./config/fluent.conf
            target: /fluentd/etc/fluent.conf
        ports:
          - "24224:24224/tcp"
          - "24224:24224/udp"
          - "8888:8888"
        networks:
          - elk
        depends_on:
          - elasticsearch
      kibana:
        image: kibana:7.10.1
        volumes:
          - type: bind
            source: ./config/kibana.yml
            target: /usr/share/kibana/config/kibana.yml
            read_only: true
        ports:
          - "5601:5601"
        networks:
          - elk
        depends_on:
          - elasticsearch
        healthcheck:
            test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
            interval: 30s
            timeout: 15s
            retries: 3
            start_period: 200s
      elastalert:
        image: praecoapp/elastalert-server
        volumes:
          - type: bind
            source: ./elastalert/config/elastalert.yaml 
            target: /opt/elastalert/config.yaml
          - type: bind
            source: ./elastalert/config/config.json
            target: /opt/elastalert-server/config/config.json
          - type: bind
            source: ./elastalert/rules
            target: /opt/elastalert/rules
          - type: bind
            source: ./elastalert/rule_templates
            target: /opt/elastalert/rule_templates
        ports:
          - "3030:3030"
        networks:
          - elk
        depends_on:
          - elasticsearch
          - kibana
        #entrypoint: ["sh","/opt/elastalert-server/scripts/start.sh"]
        healthcheck:
            #test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
            test: ["CMD-SHELL", "sh /opt/elastalert-server/scripts/start.sh || exit     1"]
            interval: 30s
            timeout: 15s
            retries: 3
            start_period: 100s
    networks:
      elk:
        driver: bridge
    

設定方式

  • post 設定

    1. 設定 ElastAlert rule

      ```yaml
      name: test # rule 名稱
      type: frequency # rule 的類型 (細節可以參考 [Rule Types](https://elastalert.readthedocs.io/en/latest/ruletypes.html#ruletypes))
      index: logstash-* # 監控哪個 elasticsearch index
      num_events: 1 # 命中查詢條件次數
      buffer_time: # 查詢 elasticsearch 的時間區間
      minutes: 5
      run_every: # 查詢 elasticsearch 的頻率
      minutes: 1
      timeframe: # 目標時間範圍 (60 分鐘內事件發生次數;事件內容依 type 來決定)
      minutes: 60
      realert: # 相同警告多久才發送一次避免收到過多警告,預設一分鐘,`0` 為每次警告都發送
      minutes: 0
      filter: # 查詢條件 (詳細內容可以參考 [Common Filter Types](https://elastalert.readthedocs.io/en/latest/recipes/writing_filters.html#common-filter-types))
      - query:
          query_string:
          query: "message:fluentd worker"
      alert: # 警示通知類型 (詳細內容可以參考 [Alerts](https://elastalert.readthedocs.io/en/latest/ruletypes.html#alerts))
      - "post"
      http_post_url: "http://blog.yowko.com" # post 目標 url
      http_post_proxy: "192.168.80.3:8866" # proxy 非必要設定值,這邊是我用來攔 http post 內容用的
      ```
      

      這邊我使用 Fiddler Everywhere 來攔 http post 的內容

      1post

  • email 設定

    這邊收發 mail 透過 mailtrap service 來進行,詳細內容可以參考 [tool]沒有mail server怎麼測試寄送email?快放過你的gmail來看看有那些可以測試用的smtp mail server

    1. 設定 email 帳密

      ElastAlert 使用 email 通知時,如果需要帳號密碼,需要另外用 yaml 檔案格式紀錄帳號與密碼,再將檔案位置指定給 rule 使用,以下延續之前筆記的路徑規則設定

      • elastalert/config/emailauth.yaml

        user: "username"
        password: "password"
        
    2. 調整 docker compoe

      將 email 帳號密碼設定檔 mount 給 ElastAlert

      • 差異內容

        ```yaml
        - type: bind
        source: ./elastalert/config/emailauth.yaml
        target: /opt/elastalert/config/emailauth.yaml
        ```
        
      • elastalert 完整範例

        elastalert:
            image: praecoapp/elastalert-server 
            volumes:
            - type: bind
                source: ./elastalert/config/elastalert.yaml 
                target: /opt/elastalert/config.yaml
            - type: bind
                source: ./elastalert/config/config.json
                target: /opt/elastalert-server/config/config.json
            - type: bind
                source: ./elastalert/config/emailauth.yaml
                target: /opt/elastalert/config/emailauth.yaml
            - type: bind
                source: ./elastalert/rules
                target: /opt/elastalert/rules
            - type: bind
                source: ./elastalert/rule_templates
                target: /opt/elastalert/rule_templates
            ports:
            - "3030:3030"
            networks:
            - elk
            depends_on:
            - elasticsearch
            - kibana
            #entrypoint: ["sh","/opt/elastalert-server/scripts/start.sh"]
            healthcheck:
                #test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
                test: ["CMD-SHELL", "sh /opt/elastalert-server/scripts/start.sh || exit 1"]
                interval: 30s
                timeout: 15s
                retries: 3
                start_period: 100s
        
    3. 設定 ElastAlert rule

      ```yaml
      name: test # rule 名稱 
      type: frequency # rule 的類型 (細節可以參考 [Rule Types](https://elastalert.readthedocs.io/en/latest/ruletypes.        html#ruletypes))
      index: logstash-* # 監控哪個 elasticsearch index
      num_events: 1 # 命中查詢條件次數
      buffer_time: # 查詢 elasticsearch 的時間區間
      minutes: 5
      run_every: # 查詢 elasticsearch 的頻率
      minutes: 1
      timeframe: # 目標時間範圍 (60 分鐘內事件發生次數;事件內容依 type 來決定)
      minutes: 60
      realert: # 相同警告多久才發送一次避免收到過多警告,預設一分鐘,`0` 為每次警告都發送
      minutes: 0
      filter: # 查詢條件 (詳細內容可以參考 [Common Filter Types](https://elastalert.readthedocs.io/en/latest/recipes/        writing_filters.html#common-filter-types))
      - query:
          query_string:
          query: "message:fluentd worker"
      alert: # 警示通知類型 (詳細內容可以參考 [Alerts](https://elastalert.readthedocs.io/en/latest/ruletypes.        html#alerts))
      - "email"
      email: # email 發送目標
      - "elastalert@example.com"
      alert_subject: "test alert via email" # email subject
      smtp_host: smtp.mailtrap.io
      smtp_port: 2525
      smtp_auth_file: /opt/elastalert/config/emailauth.yaml # email 帳號密碼設定檔位置
      ```
      

      2email

心得

  1. rule 是 yaml 格式,且附檔名必需為 .yaml 結尾,可以透過 {elastalert ip or host}:{elastalert port}/rules 來確認 rule 是否成功加入

    • 成功

      3rulesuccess

    • 失敗

      4rulefail

  2. email 帳號密碼需要額外設定檔案有點麻煩

參考資訊

  1. 使用 Docker Compose 建立 ElastAlert 測試環境
  2. Rule Types
  3. Alerts
  4. Common Filter Types
  5. [tool]沒有mail server怎麼測試寄送email?快放過你的gmail來看看有那些可以測試用的smtp mail server