Webhook Delivery - Error Handling and Retry Logic
Delivery Sequence
An ideal system would have the order/sequence of the webhooks published in the same sequence as when actions/events happen. Punchh does not guarantee the delivery of events in the sequence in which they were created. It is possible that sometimes, due to architecture/network reasons (vizualization due to multiple nodes / worker threads / services), a latter event could get processed earlier than its parent/prior event, and the sequence of the messages in the outbound queue changes.
Prevent Message Loss
To prevent message loss, Punchh may publish some messages at least once. Thus, target webhook endpoints may receive the same event message more than once. The target system should make the event processing idempotent to manage it efficiently based on the content_id and timestamp validation.
Error Handling and Retry Mechanism
Punchh will deliver the webhook messages very soon after an event takes place in the system. In most cases, it will be within 50-60 seconds.
1. Punchh should receive either of the responses: HTTP 200 OK, or 202 Accepted.
2. Logs timeframe minimum is the last 30 minutes. Logs are available for a day under intervals of the last 30 minutes, 1 hour, 3 hours, 6 hours, 12 hours, and 24 hours. No logs are available after 7 days.
3. Retry when HTTP 429 (too many requests) is received. Webhooks retry 3 times at set time intervals after the first attempt (15, 45, 180 second intervals). Similarly, for timeouts (i.e., HTTP 408).
4. HTTP 3xx response codes (Redirects) will be treated as failures and will not be retried.
5. For server errors vizualization HTTP 500, webhooks will retry 7 times at set time intervals after the first attempt (60, 120, 300, 600, 1800, 3600, 7200 second intervals).
Circuit Breaker Process
The circuit breaker is used in case the connection is not established even after the retries.
If, after the retry period, the request still fails and the counter goes over 1000, the event (and any pending queued up events) will be suspended, and no new requests will be sent for that event until verified again.
An event that is marked suspended can be reactivated by sending a successful response to a ping webhook. The ping webhook can be manually sent to the webhooks URL via the webhook manager. Apart from that, Punchh will automatically try to ping the webhooks URL once in 5 minutes, 10 minutes, and then every 2 hours for 6 attempts. After that, the webhook will be deactivated.
If successfully verified, then revoke all suspended events (send revoke email 2), else continue to verify again.
Message Content Locale and Timestamp
1. The webhook message will be only in the ENU locale. It is the receiving system’s responsibility to convert to other locales for its other clients in different languages.
2. It will have a timestamp in UTC. It is the target system’s responsibility to convert the UTC timestamp to a different time zone.
Message Transformation to Target System Format Not in Scope
1. Punchh will publish the webhook in Punchh’s data structure in JSON.
2. Punchh will not do any transformation of the message to the target system’s message structure. It is the receiving system’s responsibility to transform the message to the applicable structure.