E2E testing using Selenium and Mailtrap | Nova hacks to send organisational emails part 2

Williams Aguilera
Williams Aguilera
Posted under Tech-Pills
Posted on September 7, 2021

And how we avoided messing up the communications with our users | #NovaTech

Have you ever received an email that looked something like this?

E2E testing using selenium and mailtrap

Or even worse, have you ever sent such an email from your platform? 

Let’s be honest, we’ve all been there (probably not as bad as the example), a typo in one tag, a missing closing bracket, some style that got deleted by mistake, you name it. There are many things that can go wrong when sending emails, and the problem is that they are not easy to test.

I used to work in a company where the main purpose and end goal of the whole organization, was to, basically, send emails. We would do unit tests to make sure the data models were right, component testing to check that we were indeed calling the right methods, and also e2e testing ensuring that the right logs were written in a database and the emails were actually being sent. But, if somebody messed with the templates tags, or the image repository, or something like that, it usually took days and a bug report from a customer to realize that the emails were looking bad.

Of course, we could do manual testing, but that takes a lot of time, and having a healthy CI/CD process, we’re deploying features faster than it takes to test all the flows that involve sending emails.

We’re talking about automation here, how can we be sure in a quick, simple, and deterministic way that our emails are being sent out correctly, and not only that, also that they look as expected?

 

WHAT WE HAVE

As you know, we use a combination of Stripo and AWS to decentralize the design and creation of email templates (if you don’t know, you might want to read this first). This provides us with great flexibility, since the communications team can change emails at will, and they get updated in real-time, reducing the bureaucracy for making simple changes. But… with great power comes great responsibility, and if a mistake is made there or in the backend, we might not notice it for a while.

We have unit and component testing in all of our backend code. But, since the templates are provided by an external service, we can’t test that in there. We can make sure in our unit tests that we’re indeed retrieving the template from a stub, and then calling the function to send it. For practical effects, we’re 100% sure that the template is being retrieved, and that the email is being sent, but we have no way of checking that the email looks “good”.

We also have an automated testing suite using Selenium and Cucumber-js that handles the E2E tests, but again, we’re using fake email accounts for every fake user we create, so those emails might be sent, but nobody is receiving them. Hence, we still don’t know if they look good or not.

 

MAILTRAP TO THE RESCUE! 

We were already using Mailtrap to handle emails in our sandbox environment. If you don’t know what Mailtrap is, it is a fake SMTP server that receives all the messages you send from an environment and puts them in a single mailbox. Mailtrap has a great free tier service and overall it works really well.

So… we had Mailtrap working, then all of our fake emails were actually getting somewhere. We then just needed to “look” at them from our E2E testing suite. Mailtrap actually provides an API where you can retrieve the messages received for a particular email address, and also get the raw HTML body of it. We had all the pieces in our hands, we just needed to combine them.

 

PUTTING IT ALL TOGETHER

When launching the tests from our E2E testing suite, any scenario that should send an email follows this process. Now, we are doing 2 things: first, making sure that the email was actually sent and arrived in the inbox; and second, we now have a screenshot attached to the test report.

Our reports are generated automatically whenever the E2E tests are launched, so we end up with a report that has screenshots of ALL the emails that are sent through our platform, populated with real “fake” data, and ready to be reviewed.

This would get sent to the product owner, or attached to the task tickets in JIRA to be reviewed by the stakeholders before releasing a feature into production. Also, we can run these tests on-demand, so, if the communications team is not sure about a change, they can run them and check the output to see that nothing got broken.

 

CONCLUSION

For us, this was of course a game-changer. By integrating the Mailtrap API into our E2E testing suite, we didn’t need to test each scenario manually with different email accounts and check the emails received. Then, the review process went from a very time-consuming, error-prone hell, to just going over a report, reviewing screenshots, and making sure they looked good.

By leveraging Selenium, cucumber-js, and Mailtrap API, we can be extremely confident about the content and correctness of our emails in an automated, and deterministic way. If there’s anything wrong with any part of our system, we would catch it pretty early in the process, before any users are affected, and this of course this translates into more effective and professional communications with our users.

Leave a Reply

Your email address will not be published. Required fields are marked *

E2E testing using selenium and mailtrap
Posted on September 7, 2021

E2E testing using Selenium and Mailtrap | Nova hacks to send organisational emails part 2

And how we avoided messing up the communications with our users | #NovaTech Have you ever received an email that looked something like this? Or even worse, have you ever sent such an email from your platform?  Let’s be honest, we’ve all been there (probably not as bad as the Continue reading…

Williams Aguilera
Williams Aguilera
Posted under Tech-Pills
Community Events
Posted on July 28, 2021

#NovaMoment with Eva Salas de Diego

With a Bachelor’s Degree in Biotechnology from HAN University of Applied Sciences and an MBA from Talento Farmacéutico y Sanitario Business School, Eva Salas de Diego looks forward to growing professionally in the Pharmaceutical sector alongside Nova. During her MBA, she was class president and also joined Farma Leaders Talento, Continue reading…

Manuela Vieytez
Manuela Vieytez
Posted under Nova Community
Steps to make your job more meaningful
Posted on July 8, 2021

A Nova way of making a difference with your career

Article author: Pablo Antonio Moreno Casares   Giving value to doing good Should you go for a career in tech, management consulting, or a completely different thing? If you are like me, you might feel a bit lost in how to approach career planning. There are tons of resources online, Continue reading…

Nova Talent
Nova Talent
Posted under Nova Community
Featured and Thumbnail images for blog
Posted on June 15, 2021

Decentralizing communications design with stripo and SES – Nova hacks to send organisational emails part 1

How we decreased bureaucracy and improved efficiency on our communications pipeline #NovaTech How do you handle your email templates? Do you store them in the database, or maybe even in code? Who can change those templates, and how? Is the process efficient, and clean? At some point or another, you Continue reading…

Williams Aguilera
Williams Aguilera
Posted under Tech-Pills