When your job is to deliver billions of messages to mobile devices around the world, chaos is inevitable. The key to doing it well is to design for when things go wrong.
Vibes CTO Brian Garofola, whose company delivered more than 6 billion texts, coupons, push notifications and other mobile messages last year for brands like Chipotle, Verizon and The Home Depot, knows a thing or two about that.
“To work on that scale, we’ve had to build a really robust infrastructure for messaging,” said Garofola. “We cause failures in our own test environment, see what happens and figure out how to plug the holes to make the system more resilient.”
We spoke with Garofola to learn more about chaos engineering, how products shape team structures and what it’s like to lead one of Chicago’s fastest-growing tech teams.
Things will go wrong — usually in ways we didn’t expect.”
What is the most interesting tech challenge your team faces?
We connect to more than 50 different mobile carriers, all of which have different intricacies when it comes to connections, throttling and maximum throughput. Our clients rely on us to deliver extremely large volumes of messages, regardless of carrier outages. We can’t just turn around and say: “Sorry, we dropped all your messages.” So we need to be capable of building up a backlog of millions and millions of messages and spread them out over time when the channel opens up again.
How do you build for that kind of resiliency?
Our fundamental philosophy is that things will go wrong — usually in ways we didn’t expect. We have to architect for the chaos. To do that, we think a lot about resilience and fault tolerance, and we run simulations of different sorts of outages in our test environment to avoid being surprised in production. It’s also a matter of being robust in our discussions of new features. It’s important to really think about what happens to the rest of the system if a feature breaks down.
Which technologies do you use?
Our core messaging platform is based on Java, and we use a lot of queueing technologies like HornetQ and RabbitMQ. Our web applications are built on Ruby on Rails and our databases are PostgreSQL. We recently started a migration of our data to Amazon Web Services, which we expect to be done by the end of the year. That will enable us to do streaming analytics and to introduce new machine learning capabilities.
Vibes is in a stage of rapid growth. How does that shape the way your team works?
Every problem is a scaling problem, both technically and organizationally. It’s important to start with a vision and make sure that people have a clear sense of where the company is growing. To communicate that vision, you need to be authentic. People won’t believe it if you aren’t.
The technical architecture is another challenge. Many startups have monolithic systems with a lot of components in one code base. As teams start to split off to handle different systems, there is a tendency for technical architecture to start mimicking team configuration — which often isn’t very efficient. To avoid that, you need to structure your teams based on your intended technical architecture.
Watching mobile payments is going to be interesting. How soon will we leave our physical wallets at home?”
How much do you expect your team to grow in the coming year?
Today, the Vibes engineering team is about 40 people. It’s hard to predict the future, but we expect to continue to grow at a rate of 25 to 30 percent per year.
What developments in mobile technologies are you watching in the years to come?
With Android there’s a huge push toward progressive web applications, while Apple continues to push for apps. How that plays out will have a big impact on how Vibes reaches consumers.
Watching mobile payments is also going to be interesting, because more and more consumers feel comfortable paying for things with their phones. How soon will we leave our physical wallets at home? We work really closely with Apple and Google, who both have really big plans for that space.