Introduction #
We often use webhooks during our interactions with systems, sometimes unknowingly. But what exactly is a webhook, and how does it compare to other system design patterns like short or long polling?
A webhook is an HTTP/S call triggered between two systems, where the source system initiates a specific event that notifies the target application. A common example is when you swipe your credit card at a merchant’s Point of Sale (POS) terminal and immediately receive a notification on your phone about the transaction.
In contrast, using a short or long polling pattern for the same POS transaction would require the mobile application or client to constantly poll the status of your bank account for updates. This approach leads to unnecessary computing resource usage, increasing operational costs.
Benefits of Using Webhooks #
- Real-time communication
- Loosely coupled architecture
- No compute resource wastage, unlike short/long polling
- Encrypted communication in transit over HTTPS
- Tokenized access, allowing only authorized systems to invoke the webhook
In this blog, we’ll design a webhook system using AWS serverless services to achieve a highly scalable, cost-effective, resilient, and highly available system.
Architecture #
Note: You can also implement your own authentication system to secure this webhook endpoint using AWS Cognito or a SAML-based system (not shown in the architecture) at the API Gateway.
How Does This Serverless Webhook Work? #
-
DNS Configuration: The Domain Name System (DNS) for the webhook endpoint is configured using AWS Route 53, with an alias record pointing to the API Gateway endpoint.
-
API Gateway: This regional AWS service serves as the entry point for incoming requests, efficiently routing them to AWS SQS (Simple Queue Service). The API Gateway has a default rate limit of 10,000 requests per second (RPS) and a default burst limit of 5,000 requests.
-
SQS FIFO Queue: We use a First-In-First-Out (FIFO) queue because the ordering of every request is critical to the webhook system. Additionally, SQS FIFO provides content-based deduplication out of the box. If a request from the producer system is mistakenly triggered twice within a 5-minute span, it will be considered a duplicate and won’t reach the queue. SQS acts as a reliable buffer, decoupling components and ensuring no data loss during traffic spikes or service failures, allowing the system to retry as needed.
-
AWS Lambda: All messages received in the SQS FIFO queue are processed in order by AWS Lambda, which can scale up to 1,000 concurrent executions every 10 seconds. Lambda handles the actual processing of the webhook’s business logic. When dealing with potentially high volumes of requests, Lambda functions enhance scalability by allowing for event-driven, serverless code execution, automatically scaling based on demand. Logs generated are streamed to AWS CloudWatch Logs for monitoring and debugging.
-
Dead Letter Queue (DLQ): If a message isn’t processed timely or if the SQS queue receives a bad message, it moves to the Dead Letter Queue for reprocessing.
-
AWS Certificate Manager: This service manages SSL certificates, ensuring they are valid and rotated before expiration, guaranteeing that all HTTPS requests are encrypted in transit.
This combination of API Gateway, SQS, and Lambda enables robust, cost-effective, resilient, and scalable architectures capable of handling varying workloads and maintaining high availability under challenging conditions.
Conclusion #
Implementing a webhook using AWS serverless services provides a scalable, resilient, and cost-effective solution. By leveraging services like