With the sucess of our products and effective marketing, we achieved multiple waves of new customers. Each one proved the importance of monitoring and redundant systems.
Overview
The first step into a distributed system was to create a VPN for the production environment. This connected all production systems, exposing only an Nginx Load Balancer to the public internet. Behind the load balancer, multiple php-fpm application servers handled synchronous traffic, while a separate collection of worker servers handled queued/async workloads. Additional servers provided Redis caches, and a MariaDB cluster.
The goal was to be able to provision new servers and remove as needed, with minimal configuration changes required. Most successfully, we were able to scale app servers and workers multiple times without disruption.
Eventually this would expand to include many other resources, including LLM microservices, Typesense, Soketi, and RabbitMQ.
Orchestration
Orchestration was a mix of manual management in AWS / DigitalOcean / Forge, and eventually utilizing Terraform with madewithlove/terraform-provider-laravelforge to move towards an Infrastructure as Code paradigm.