Prometheus-Alertmanager integration with MS-teams

As we know monitoring our infrastructure is one of the critical components of infrastructure management, which ensures the proper functioning of our applications and infrastructure. But it is of no use if we are not getting notifications for alarms and threats in our system. As a better practice, if we enable all of the notifications in a common work-space, it would be very helpful for our team to track the status and performance of our infrastructure.

Last week, all of a sudden my company chose to migrate from slack to MS-teams as a common chatroom. Which meant, now, notifications would also be configured to MS-teams. If you had search a bit, you will find that there isn’t any direct configuration for Ms-teams in alert manager as slack does. As a DevOps engineer I didn’t stop and looked beyond for more solutions and I found out that we need some proxy in between ALERTMANAGER and MS-teams for forwarding alerts and I proceeded to configure those.

There are a couple of tools, which we can use as a proxy, but I preferred to use prometheus-msteams, for a couple of reasons.

  • Well-structured documentation.
  • Easy to configure.
  • We have more control in hand, can customise alert notification and you can also configure to send notifications to multiple channels on MS-teams. Besides well-described documentation.
    I still faced some challenges and took half of the day of mine.

How it works?

Firstly, Prometheus sends an alert to ALERTMANAGER on basis of rules we configured in the Prometheus server. For instance, if memory usages of the server are more than 90%, it will generate an alert, and this alert will send to ALERTMANAGER by the Prometheus server. Afterward, ALERTMANGER will send this alert to prometheus-msteams which in turn send this alert in JSON format to MS-teams’s channel.

How to Run and Configure prometheus-msteams

We have multiple options to run prometheus-msteams

  1. Running on standalone Server (Using Binary)
  2. Running as a Docker Container

Running on Server

Firstly, you need to download the binary, click here to download the binary from the latest releases.

When you execute the binary with help on your system, you can see multiple options with description, which help us to run prometheus-msteams just like man-pages.

prometheus-msteams --help

you can run promethues-msteams service as follow.

./prometheus-msteams server \
    -l localhost \
    -p 2000 \
    -w "Webhook of MS-teams channel"

Above options explanation

  • -l: On which address prometheus-msteams going to listen, the default address is “”. In the above example, prometheus-msteams listening on the localhost.
  • -p: On which port prometheus-msteams going to listen, the default port is 2000
  • -w: The incoming webhook of MS-teams channel we are going to insert here.

Now you know how to run prometheus-msteams on the server, let’s configure it with ALERTMANAGER.

Step 1 (Creating Incoming Webhook)

Create a channel in Ms-teams where you want to send alerts. Click on connectors(found connectors in options of the channel), and then search for ‘incoming webhook’ connector, from where you can create a webhook of this channel. Incoming webhook is used to send notification from external services to track the activities.

Step 2 (Run prometheus-msteams)

Till now, you have an incoming webhook of a channel where you want to send the notification. After that, you need to setup prometheus-msteams, and run it.

To have more options in the future you can use config.yml to provide webhook. So that you can give multiple webhooks to send alerts to multiple channels in MS-teams in future if you need it.

$ sudo nano /opt/promethues-msteams/config.yml

Add webhooks as shown below. if you want to add another webhook, you can add right after first webhook.

  - alert_channel: "WEBHOOK URL"

The next step is to add a template for custom notification.

$ sudo nano /opt/prometheus-msteams/card.tmpl

Copy the following content in your file, or you can modify the following template as per your requirements. This template can be customized and uses the Go Templating Engine.

{{ define "teams.card" }}
  "@type": "MessageCard",
  "@context": "",
  "themeColor": "{{- if eq .Status "resolved" -}}2DC72D
                 {{- else if eq .Status "firing" -}}
                    {{- if eq .CommonLabels.severity "critical" -}}8C1A1A
                    {{- else if eq .CommonLabels.severity "warning" -}}FFA500
                    {{- else -}}808080{{- end -}}
                 {{- else -}}808080{{- end -}}",
  "summary": "Prometheus Alerts",
  "title": "Prometheus Alert ({{ .Status }})",
  "sections": [ {{$externalUrl := .ExternalURL}}
  {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
      "facts": [
        {{- range $key, $value := $alert.Annotations }}
          "name": "{{ reReplaceAll "_" "\\\\_" $key }}",
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        {{- end -}}
        {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
          "name": "{{ reReplaceAll "_" "\\\\_" $key }}",
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        {{- end }}
      "markdown": true
    {{- end }}
{{ end }}

Create prometheus-msteams user, and use --no-create-home and --shell /bin/false to restrict this user log into the server.

$ sudo useradd --no-create-home --shell /bin/false prometheus-msteams

Now, set the user and group ownership on the prometheus-msteams directorie, and prometheus-msteams binary to the prometheus-msteams user.

Create a service file to run prometheus-msteams as service with the following command.

$ sudo nano /etc/systemd/system/prometheus-msteams.service

The service file tells systemd to run prometheus-msteams as the prometheus-msteams user, with the configuration file located /opt/promethues-msteams/config.yml, and template file located in the same directory.

Copy the following content into prometheus-msteams.service file.


ExecStart=/usr/local/bin/prometheus-msteams -config-file /opt/prometheus-msteams/config.yml -template-file /opt/prometheus-msteams/card.tmpl


promethues-msteams listen on localhost on 2000 port, and you have to provide configuration file and template also.

To use the newly created service, reload systemd.

$ sudo systemctl daemon-reload

Now Start promethues-msteams.

$ sudo systemctl start prometheus-msteams.service

Check, whether the service is running or not.

$ sudo systemctl status prometheus-msteams

Lastly, enable the service to start on the boot.

$ sudo systemctl enable prometheus-msteams

Now, prometheus-msteams is up and running, we can configure ALERTMANAGER to send alerts to prometheus-msteams.

Step 3(Configure ALERTMANAGER)

Open alertmanager.yml file in your favorite editor.

$ sudo vim /etc/alertmanager/alertmanager.yml

you can configure ALERTMANAGER as shown below.

  resolve_timeout: 5m

  - '/etc/alertmanager/*.tmpl'

- name: alert_channel
  - url: 'http://localhost:2000/alert_channel'
    send_resolved: true

  group_by: ['critical','severity']
  group_interval: 5m
  group_wait: 30s
  receiver: alert_channel
  repeat_interval: 3h

In the above configuration, ALERTMANAGER is sending alerts to prometheus-msteams, which is listening on localhost, and we pass send_resolved, which will send resolved alerts.

The critical alert to MS-teams will look like below.

When alert resolved, it will look like below.

Note: The logs of prometheus-msteams created in /var/log/syslog file. In this file you will find every notification send by prometheus-msteams. Apart from this, if something went wrong, and you are not getting notification, you can debug in syslog file

As Docker Container

you can also run prometheus-msteams as container in your system. All configuration files of prometheus-msteams going to be the same, you just need to run the following command.

docker run -d -p 2000:2000 \
    --name="promteams"  \
    -v /opt/prometheus-msteams/config.yml:/tmp/config.yml \
    -e CONFIG_FILE="/tmp/config.yml" \
    -v /opt/prometheus-msteams/card.tmpl:/tmp/card.tmpl \
    -e TEMPLATE_FILE="/tmp/card.tmpl" \

Now that you are all set to get alerts in MS-teams channel, you can see that it isn’t as difficult as you originally thought. Ofcourse, this is not the only way to get alerts on MS-teams. You can always use different tool like prome2teams, etc. With this, I think we are ready to move ahead and explore other monitoring tools as well.

I hope this blog post explains everything clearly. I would really appreciate to get feedback in comments.

8 thoughts on “Prometheus-Alertmanager integration with MS-teams”

  1. Hey thanks for the post, however i’m struggling to get notifications in msteams. I set up everything in a Docker-Container (v1.1.4) just like you described, my alert is firing and the alertmanager shows it but nothing shows up in msteams. /var/log is empty in my container. Do you have any idea what might be the reason? 🙂


    1. Hi @ragemoody, You have to make sure that your alertmanager is sending alerts to prometheus-msteams firslty, for that you can check alertmanager.yml file, and add prometheus-msteams info in this file as mentioned. Afterwards prometheus-msteams will print logs on /var/log/syslog once it gets alerts from alertmanager


  2. Thanks for the clear blog. I had a lot of trouble figuring out why it would not work on my server though so I wanted to share that I had to update the prometheus-msteams commandline args as follows to get it to work with alertmanager:
    ExecStart=/usr/local/bin/prometheus-msteams \
    -template-file “/opt/prometheus-msteams/card.tmpl” \
    -config-file “/opt/prometheus-msteams/config.yml”

    Liked by 1 person

  3. Hi,
    when i specify the card.tmpl file in the alertmanager.yml file and restart the service, I get this error message: component = configuration msg = “one or more config change subscribers failed to apply new config”
    file = /opt/prometheus/alertmanager/alertmanager.yml err = “failed to parse template: card.tmpl: 36: function \” counter \ “not defined”
    Do you have any ideas why this could be?
    alertmanager version: 0.21.0
    prometheus-msteams version: v1.4.1


  4. Hi,
    when i specify the card.tmpl file in the alertmanager.yml file and restart the service, I get this error message: component = configuration msg = “one or more config change subscribers failed to apply new config”
    file = /opt/prometheus/alertmanager/alertmanager.yml err = “failed to parse template: card.tmpl: 36: function \” counter \ “not defined”
    Do you have any ideas why this could be?
    alertmanager version: 0.21.0
    prometheus-msteams version: v1.4.1


    1. Hi baloghszilveszter,
      With the following command, you can run prometheus-msteams service, and then you can add URL of this service in under webhook_configs in alertmanger.yml
      /usr/local/bin/prometheus-msteams -config-file /opt/prometheus-msteams/config.yml -template-file /opt/prometheus-msteams/card.tmpl
      As in the above command, card.tmpl pass with promtheus-msteams binary as of template for ms-teams notification. You can refer to prometheus-msteams.service, and alertmanger.yml file for setup.


      1. Hi,
        Thank you for your response. The problem was that the templates remained in alertmanager.yml. I commented on this, after that the service started correctly.
        The MS Temas channel is already receiving messages. 🙂

        Liked by 1 person

  5. Hi, thanks for great guide.

    I already have a prometheus-alertmanager working just fine with a Slack integration. Do you have any suggestion on how I can incorporate this guide, so I can integrate with both msteams and Slack with the same alerting rules, I already have defined?

    Thanks in advance 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s