Troubleshoot webhook endpoint issues

Problem Symptoms

• In the Nylas dashboard webhook logs or via email you receive the following message:

  • Your Nylas webhook needs attention

  • Failed to connect to the webhook endpoint
• Your webhook endpoint failed to reply within 10 seconds

In low workload environment a single failure within 15 minutes can result in a Your Nylas webhook needs attention email . Please disregard the email.

Description

We wait 10 seconds for the host to respond, there are a few reasons why the webhook endpoint might be having issues, we have no visibility on these.

• IP blocked
• Network delays or packet loss in the route, typically transient
• Host CPU congestions - due to ingesting too many webhooks

Resolution

• Determine when it failed, see the webhook logs in the dashboard https://dashboard.nylas.com/applications/app/logs/ . 
  • We can't say why, simple no response within 10 seconds, you need investigate further.
• Troubleshoot - If recurring:
  • Check the host CPU
  • Check the route between your endpoint and our IP's
    You can test for packet loss / latency via traceroute or MTR to the IP of the host you entered as your webhook endpoint, ideally from a AWS host of the Nylas region - you can find the IP address range we use to send webhooks here

sudo mtr X.76.21.9 --report
Start: 2022-05-25T16:15:08+0200
HOST: ip-192-168-0-246.ec2.intern Loss%   Snt   Last   Avg  Best  Wrst StDev
1.|-- ip-192-168-0-1.ec2.intern  0.0%    10    1.4 190.8   1.2 1010. 385.9
2.|-- ip-192-168-2-254.ec2.inte  0.0%    10    2.2 275.9   1.5 973.5 437.0
3.|-- 195-190-228-188.fixed.kpn  0.0%    10   16.0 262.1  12.3 917.3 401.2
4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0.  <<<<<<< Note the packet loss

If the endpoint is replying within 10 seconds but with a non 200 response please test your endpoint by sending a sample payload.

{"deltas": [{"date": 1673894272, "object": "event", "type": "event.updated", "object_data": {"namespace_id": "9wgj92degdkdqghftxjc8papb", "account_id": "9wgj92degdkdqghftxjc8papb", "object": "event", "attributes": null, "id": "4usbfkr53z0mo3jl2vs4hosxj", "metadata": null}}]}

• Recovery - Best practice - Ensure you setup your application to handle webhook endpoint failures by re-enabling the webhook 

Resources

Webhook Best practices

Updated October 09, 2023 19:37