Rapid growth of our customer base and the increased utility of our Bulk Messaging and Marketing Automation systems exposed the need to optimize these workflows. They were already event-driven, queued systems, but a flood of 10s of thousands of email and SMS messages were straining resources and not living up to our standards.
Background
Timley communication with interested leads is a critical part of converting website contacts into paying customers.
Market Muscles long provided an interface to configure a series of SMS and Email communications that follow up with a lead spread out over a period of time. This is a familiar feature from large scale systems like SalesForce, Hubspot, and Marketo.
As the company's marketshare grew, Market Muscles' systems struggled with keeping features competitive, and handling larger amounts of communication and critical timeliness.
Studying competitors and similar tools gave me an initial foundation, then being able to talk to customers about how they would want to use it helped me find the right starting goals for how to expand this suite of features.
Product
As we iterated, the product gave businesses several paths for communication:
- The ability to send bulk messages to all of their contacts, or select segments of recipients based on tags and other filters.
- Automated workflows that could be assigned to contacts, optionally filtered by tags. These workflows were a combination of Email, SMS, and delays that would execute in order until completion or removal.
- Direct two-way SMS to a specific contact.
Across the three interfaces, the tool included features like processing merge tags and delaying send if the receiver's timezone hinted it was outside of reasonable business hours. Message sending was routed through Twilio and Sendgrid, with each platform notifying the application of status updates with WebHooks.
Bulk Messaging
We had dozens of customers sending to 10k+ contacts multiple times per week, so the process needed to be broken up into lots of distinct parts, all of which needed to be batched and/or queued. I broke the steps up into a chain of independent jobs that followed the logical order of building a large dataset:
- Circuit breaker: make sure sending is enabled
- Query contacts in a Typesense collection that match a json object in the request, i.e. filtering by tag, program enrollment, etc.
- Exclude opt-outs and then dedupe list.
- Create a batch of bulk message records for each contact in the collection.
- Broadcast an event that the batch started.
Each message in the batch is processed as its own job:
- Circuit breaker: make sure sending is enabled
- Multi-thread check: skip if this record is already pending.
- Check internal rate limit for sending service, delay if near limit
- If SMS, check if recipient's timezone is outside of account-defined "nice hours", delay if outside appropriate time
- Route to guardian(s) if no email/phone.
- Process merge tags for subject/body
- Store record w/ pending status
- Send message via Sendgrid or Twilio, as appropriate.
- Broadcast an event that the message was sent
Webhook listeners for Twilio and Sendgrid:
- Parse delivery updates, i.e. sent, rejected, delivered to device.
- Find and update database record
- Broadcast an event with the message update
Realtime observability UI
Soketi was configured to listen to events and broadcast a safelist of them to WebSocket channels listened to with Laravel Echo on the frontend. The UI would react to the events and display the progress, including realtime reports of how many were sent, any failures or bounces, and for text messages, if it was actually delivered to the device.
Automated Workflows
Workflows were a collection of actions that were followed a basic directed acyclic graph. These steps including communication steps for sending text or emails, control steps like delays, and internal actions like adding/removing tags, assigning tasks, etc.
Leaning into Event Driven Development, we were able to let the owner define a range of events that would trigger adding a contact to a campaign. The owner could target events like a new lead form submission, staff or another process adding a tag, or not attending a class for a defined time, and use that to add or remove a contact from a workflow.
The actual sending process was very close to the Bulk Message process described above. The main difference was a cron job that queried for contact_workflow records that required action in the next 15 minutes and dispatched them as new jobs.
Eventually a preference for utilizing action classes meant that most logic could be shared between Bulk and Workflows.
My Role
I owned this feature end-to-end and iterated on it repeatedly to improve the speed, user experience, and reliability. This was built in Laravel, using Redis for the queue driver, Soketi to communicate over WebSockets for UI updates, Typesense for contact segmenting, RabbitMQ to broker events, and Horizon to manage and monitor the queue.
Results
This became a "crown jewel" value proposition for the company. Several of the aspects set us apart from competitors and helped drive customers to the platform.