Sending custom fields in prometheus alerts to pagerduty

alertmanager pagerduty example
prometheus alerting tutorial
alertmanager slack
prometheus alert rules
prometheus alert grouping
prometheus dynamic alert rules
prometheus alertmanager
alertmanager template variables

I have a scenario where I need to send the current time stamp from Prometheus to Pagerduty. I need to access the current time stamp from the Custom details in Pagerduty to write Event Rules based on the current time.

Currently I can see three fields inside Custom Details in Pagerduty ["firing", "num_firing", "num_resolved"]. All the alert related details are inside "firing" field but as a single string.

I thought of adding the time stamp to the Labels within the Prometheus event rules, but the problem is that I will not be able to retrieve that value from the "firing" field as it is not in a structured format.

Event details in Pagerduty received from Prometheus:

{
 "client": "AlertManager",
 "client_url": "http://********/",
 "description": "[FIRING:1] **************",
 "event_type": "trigger",
 "incident_key": "********",
 "service_key": "********",
 "details": {
  "firing": "Labels:\n - alertname = ******\n - datacenter = *****\n - instance = ******\n - instance_id = ******\n - instance_type = ******\n - job = ******\n - metrics = ******\n - node = ******\n - pod = ******\n - private_ip = ******\n - public_ip = ******\n - service = ******\n - severity = critical\nAnnotations:\n - description = ******\n - summary = ******\nSource: ******\n",
  "num_firing": "1",
  "num_resolved": "0",
  "resolved": ""
 }
}

I need to add the current time stamp in the same level as the "firing" field. Is there any way to do this via Prometheus alert manager configuration or via the Alert rules?

The standard prometheus way for adding additional metadata to alerts is through annotations.

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ query "time()" }}

This should add the timestamp annotation to the alert being fired.

Prometheus Integration Guide, Prometheus sends events to PagerDuty via a Prometheus Alertmanager Key (​generated in the In PagerDuty section, above) in the service_key field, then save You can also configure custom `routes` to send alerts to different `receivers`. Prometheus + PagerDuty Benefits. Send richly formatted event data from Prometheus to PagerDuty, allowing you to engage the right people, accelerate resolution and improve learning. Create high and low urgency incidents based on the severity of the event from the Prometheus event payload. How it Works.

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }} {{ end }}

I have solved same problem, as above. Cheers.!

Creating nested custom details from alertmanager, I'm trying to send alerts from alertmanager with nested custom details. Prometheus Alert Manager - Custom Details using rest API Alertmanger uses our v1 Event API (link), which does not support custom fields . You might  The fields are documented in the PagerDuty API documentation. href: <tmpl_string> source: <tmpl_string> alt: <tmpl_string> <link_config> The fields are documented in the PagerDuty API documentation. href: <tmpl_string> text: <tmpl_string> <pushover_config> Pushover notifications are sent via the Pushover API. # Whether or not to notify about resolved alerts.

You can directly add this as a custom detail to the alertmanager.yaml .

details:
  time: {{ query "time()" }}

Note that this adds the time from alertmanager's point of view so might contain a small delay.

Sending custom fields in prometheus alerts to pagerduty, Currently I can see three fields inside Custom Details in Pagerduty ["firing", "​num_firing", "num_resolved"]. All the alert related details are inside "firing" field but  This article is a follow up to Custom Alerts Using Prometheus Queries.In this post, we will also demo installing Prometheus and configuring Alertmanager to send emails when alerts are fired, but in a much simpler way – using Rancher all the way through.

Alertmanager, Which are the external systems Prometheus Alertmanager supports to notify? Hello, I have a little question. If I navigate to the alert page from PD, and then click on the alert link to see the more detailed page, I see the custom details. Is it possible to change the arrangement for the labels? Or format it for example with markdown or something like that? Just to adapt it to our needs a little bit and make it somehow more readable. We are using the prometheus

Step-by-step guide to setting up Prometheus Alertmanager with , server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts. The Alertmanager handles alerts sent by Prometheus servers and sends notifications about them to different receivers based on their labels. A receiver can be one of many different integrations such as PagerDuty, Slack, email, or a custom integration via the generic webhook interface (for example JIRA ).

Configuration, How long to initially wait to send a notification for a group # of alerts. Allows to wait for an The fields are documented in the PagerDuty API documentation. Strings. Prometheus creates and sends alerts to the Alertmanager which then sends notifications out to different receivers based on their labels. A receiver can be one of many integrations including: Slack, PagerDuty, email, or a custom integration via the generic webhook interface. The notifications sent to receivers are constructed via templates.

Comments
  • Hi, Thanks for your reply. I tried this (timestamp: {{ time() }}) and I am getting an error like " cannot unmarshal !!map into string". I enclosed it with double quotes like (timestamp: "{{ time() }}" ) and still getting an error like "function "time" not defined"
  • @SujaiSivasamy updated answer which was missing the query function that is needed to evaluate PromQL expressions in annotations.
  • still I am getting the same error as "cannot unmarshal !!map into string" while running the promtool check rules command. Is there anything to do with the versions that we use?
  • It passes the promtool validation now. But the value that I am getting for the query is "timestamp = time: [0xc46b3c7ec0]". Any idea regarding what went wrong?
  • Hey, I found that it's an hexadecimal value. When I converted it into decimal and to the human readable time via epoch converter, it gave me the time as "GMT: Sunday, September 15, 1996 5:51:24.928 PM", which is wrong. I am not sure what I missed here.