Retries and replays
How Cevoid reads your webhook responses, retries failed deliveries for about 20 hours, and lets you retry or replay an event from Outbox.
Cevoid keeps trying a delivery until your endpoint accepts it or the retries run out. After that, you can send it again from Outbox, as the same event or as a new one.
Respond and deduplicate
- Return any
2xxstatus once you have stored the event. Result: Cevoid stops retrying it. - Respond within 10 seconds and do slow work afterwards, for example in a job queue. A slower response counts as a timeout and is retried, even if you processed the event.
- Store
webhook-id(the same value asidin the body) and skip an ID you have already processed. Result: a retry after a lost response cannot create a duplicate on your side. - Order your own state by
created_ator by the timestamps of the resource indata, not by arrival. Events for the same order or profile can arrive in any sequence.
How Cevoid reads your response
| Your response | What Cevoid does | Can you retry it from Outbox? |
|---|---|---|
2xx | Marks the delivery sent | Not needed |
408, 425, 429 or any 5xx | Retries on the schedule below, later if Retry-After asks for it | Yes, once the retries have run out |
| No response within 10 seconds, or a network error | Retries on the schedule below | Yes, once the retries have run out |
401 or 403 | Stops at once | Yes |
Any other 3xx or 4xx | Stops at once; redirects are not followed | No: replay it instead |
| The URL fails the safety check before the attempt (a private address, for example) | Stops at once | No: fix the destination and replay |
Cevoid reads Retry-After as seconds or as an HTTP date, and only on the responses it retries.
Retries
A delivery gets up to seven attempts. After a failed attempt, Cevoid waits:
| Before attempt | Wait after the previous attempt |
|---|---|
| 2 | 1 minute |
| 3 | 5 minutes |
| 4 | 30 minutes |
| 5 | 2 hours |
| 6 | 6 hours |
| 7 | 12 hours |
- Each wait varies by up to 20% either way, so endpoints recovering from an outage do not get every retry at the same moment.
- When your response has a
Retry-Afterthat asks for a later time than the schedule, Cevoid waits until then. - Without
Retry-After, the last attempt comes about 20.6 hours after the first. If it fails too, the delivery stops and shows as incomplete in Outbox. - Every retry sends the same body with the same
webhook-idto the URL of the first attempt, signed again with a newwebhook-timestamp.
Retry and replay
Open a delivery in Outbox, or choose View deliveries in the destination's menu, and use the menu of the delivery:
- Retry sends the same event again: the same
webhook-idand the same body, to the URL of the first attempt. It is offered for a delivery that stopped for a reason it can recover from (see the table above), at most three times per delivery, and makes one attempt without automatic retries. The destination must be enabled. - Replay as new delivery sends the same event data as a new event: a new
idandwebhook-id, to the destination's current URL, with automatic retries of its own. Because the ID is new, your deduplication does not filter it.created_atkeeps the original event's time. The destination must be enabled, and its URL must pass the safety check.
Use Retry after an outage or a fixed secret. Use a replay when the delivery cannot be retried, after you moved the endpoint, or when you need an event again that was already delivered.
Delivery log
Outbox lists every delivery and every attempt of it: when it ran, the HTTP status, how long it took and why it stopped. For each attempt it keeps:
- the first 2,048 bytes of your response body, with email addresses, bearer tokens and values of fields such as
password,secretandtokenmasked; - your response's
content-type,retry-afterandx-request-idheaders.
Return an x-request-id header to match an attempt with your own logs.
Troubleshooting
- Repeated events: deduplicate on
webhook-id. A slow response is the most common cause: answer within 10 seconds. - Retry is not offered: the delivery stopped on a
3xxor4xxother than401and403, or it was retried three times already. Replay it. - Delivery stopped without retries: check the status in Outbox.
401,403and other4xxstatuses stop a delivery at once. - Replay is refused: enable the destination, and check that its URL still resolves to a public address. A delivery that never made an attempt has no body to replay.