And we avoided messing up the communications with our users | #NovaTech
Have you ever received an email that looked something like this?
Or even worse, have you ever sent such an email from your platform?
Let’s be honest, we’ve all been there (probably not as bad as the example), a typo in one tag, a missing closing bracket, some style that got deleted by mistake, you name it. There are many things that can go wrong when sending emails, and the problem is that they are not easy to test.
I used to work in a company where the main purpose and end goal of the whole organization, was to, basically, send emails. We would do unit tests to make sure the data models were right, component testing to check that we were indeed calling the right methods, and also e2e testing ensuring that the right logs were written in a database and the emails were actually being sent. But, if somebody messed with the templates tags, or the image repository, or something like that, it usually took days and a bug report from a customer to realize that the emails were looking bad.
Of course, we could do manual testing, but that takes a lot of time, and having a healthy CI/CD process, we’re deploying features faster than it takes to test all the flows that involve sending emails.
We’re talking about automation here, how can we be sure in a quick, simple, and deterministic way that our emails are being sent out correctly, and not only that, also that they look as expected?
What we have
As you know, we use a combination of Stripo and AWS to decentralize the design and creation of email templates (if you don’t know, you might want to read this first). This provides us with great flexibility, since the communications team can change emails at will, and they get updated in real-time, reducing the bureaucracy for making simple changes. But… with great power comes great responsibility, and if a mistake is made there or in the backend, we might not notice it for a while.
We have unit and component testing in all of our backend code. But, since the templates are provided by an external service, we can’t test that in there. We can make sure in our unit tests that we’re indeed retrieving the template from a stub, and then calling the function to send it. For practical effects, we’re 100% sure that the template is being retrieved, and that the email is being sent, but we have no way of checking that the email looks “good”.
We also have an automated testing suite using Selenium and Cucumber-js that handles the E2E tests, but again, we’re using fake email accounts for every fake user we create, so those emails might be sent, but nobody is receiving them. Hence, we still don’t know if they look good or not.
Mailtrap to the rescue!
We were already using Mailtrap to handle emails in our sandbox environment. If you don’t know what Mailtrap is, it is a fake SMTP server that receives all the messages you send from an environment and puts them in a single mailbox. Mailtrap has a great free tier service and overall it works really well.
So… we had Mailtrap working, then all of our fake emails were actually getting somewhere. We then just needed to “look” at them from our E2E testing suite. Mailtrap actually provides an API where you can retrieve the messages received for a particular email address, and also get the raw HTML body of it. We had all the pieces in our hands, we just needed to combine them.
Putting it all together
When launching the tests from our E2E testing suite, any scenario that should send an email follows this process. Now, we are doing 2 things: first, making sure that the email was actually sent and arrived in the inbox; and second, we now have a screenshot attached to the test report.
Our reports are generated automatically whenever the E2E tests are launched, so we end up with a report that has screenshots of ALL the emails that are sent through our platform, populated with real “fake” data, and ready to be reviewed.
This would get sent to the product owner, or attached to the task tickets in JIRA to be reviewed by the stakeholders before releasing a feature into production. Also, we can run these tests on-demand, so, if the communications team is not sure about a change, they can run them and check the output to see that nothing got broken.
Conclusion
For us, this was of course a game-changer. By integrating the Mailtrap API into our E2E testing suite, we didn’t need to test each scenario manually with different email accounts and check the emails received. Then, the review process went from a very time-consuming, error-prone hell, to just going over a report, reviewing screenshots, and making sure they looked good.
By leveraging Selenium, cucumber-js, and Mailtrap API, we can be extremely confident about the content and correctness of our emails in an automated, and deterministic way. If there’s anything wrong with any part of our system, we would catch it pretty early in the process, before any users are affected, and this of course this translates into more effective and professional communications with our users.
