Retries and replays
How Cevoid reads your webhook responses, retries failed deliveries for about 20 hours, and lets you retry or replay an event from Outbox.
Cevoid keeps trying a delivery until your endpoint accepts it or the retries run out. After that, you can send it again from Outbox: retry the same delivery, or replay the event as a new delivery.
Respond and deduplicate
- Return any
2xxstatus once you have stored the event. Result: Cevoid stops retrying it. - Respond within 10 seconds and do slow work afterwards, for example in a job queue. A slower response counts as a timeout and is retried, even if you processed the event.
- Store
webhook-id(the same value asidin the body) and skip an ID you have already processed. Result: a retry after a lost response cannot create a duplicate on your side. - Order your own state by
created_ator by the timestamps of the resource indata, not by arrival. Events for the same order or profile can arrive in any sequence.
How Cevoid reads your response
| Your response | What Cevoid does | Can you retry it from Outbox? |
|---|---|---|
2xx | Marks the delivery sent | Not needed |
408, 425, 429 or any 5xx | Retries on the schedule below, later if Retry-After asks for it | Yes, once the retries have run out |
| No response within 10 seconds, or a network error | Retries on the schedule below | Yes, once the retries have run out |
401 or 403 | Stops at once | Yes |
Any other 3xx or 4xx | Stops at once; redirects are not followed | No: replay it instead |
| The URL fails the safety check before the attempt (a private address, for example) | Stops at once | No: fix the destination and replay |
Cevoid reads Retry-After as seconds or as an HTTP date, and only on the responses it retries.
Retries
A delivery gets up to seven attempts. After a failed attempt, Cevoid waits:
| Before attempt | Wait after the previous attempt |
|---|---|
| 2 | 1 minute |
| 3 | 5 minutes |
| 4 | 30 minutes |
| 5 | 2 hours |
| 6 | 6 hours |
| 7 | 12 hours |
- Each wait varies by up to 20% either way, so endpoints recovering from an outage do not get every retry at the same moment.
- When your response has a
Retry-Afterthat asks for a later time than the schedule, Cevoid waits until then. - Without
Retry-After, the last attempt comes about 20.6 hours after the first. If it fails too, the delivery stops and shows as incomplete in Outbox. - Every retry sends the same body with the same
webhook-idto the URL of the first attempt, signed again with a newwebhook-timestamp.
Retry and replay
Open a delivery in Outbox or in the destination's Event deliveries, and use the menu of the delivery:
- Retry sends the same event again: the same
webhook-idand the same body, to the URL of the first attempt. It is offered for a delivery that stopped for a reason it can recover from (see the table above), at most three times per delivery, and makes one attempt without automatic retries. The destination must be enabled. - Replay as new delivery sends the original event again as a new delivery: the same
id,webhook-id,created_atand body, to the destination's current URL, with automatic retries of its own. A receiver that already processed the event skips it when it deduplicates onwebhook-id. The destination must be enabled, and its URL must pass the safety check.
Use Retry after an outage or a fixed secret. Use a replay when the delivery cannot be retried or after you moved the endpoint.
Delivery log
Outbox lists every delivery and every attempt of it: when it ran, the HTTP status, how long it took and why it stopped. For each attempt it keeps:
- the first 2,048 bytes of your response body, with email addresses, bearer tokens and values of fields such as
password,secretandtokenmasked; - your response's
content-type,retry-afterandx-request-idheaders.
Return an x-request-id header to match an attempt with your own logs.
Troubleshooting
- Repeated events: deduplicate on
webhook-id. A slow response is the most common cause: answer within 10 seconds. - Retry is not offered: the delivery stopped on a
3xxor4xxother than401and403, or it was retried three times already. Replay it. - Delivery stopped without retries: check the status in Outbox.
401,403and other4xxstatuses stop a delivery at once. - Replay is refused: enable the destination, and check that its URL still resolves to a public address. A delivery that never made an attempt has no body to replay.