Review: CleanMyMailbox

CMM consists of an anti-spam bot that periodically checks your mailbox for spam and a web-based email inbox called a spambox that stores your spam for review. The bot examines the contents of the messages on your server to determine if they are likely to be spam. Each message receives a score based on the bot’s analysis, and if the score is high enough, the message is marked as spam.

CleanMyMailbox has several white- and blacklist options that allow you to tell the system to always or never filter mail depending upon the sender’s email address, the recipient’s email address, or the subject of the message.

Account creation

To sign up, CMM first asks the user to test their mail server for compatibility with the antispam bot. The user enters their email user name, password, mail server, and port and the bot attempts to make a connection. If it is successful, the user can sign up for the service.

While we can understand the reasons for this test, we felt that this added complexity to an already cumbersome signup process. The CMM signup form requested quite a bit of information about our mail server; many users may not know the name and port of their mail servers. It is possible to connect to most mail servers knowing only the user’s email address and password. And once the signup was complete, CleanMyMailbox assigned us a random user name, which we were allowed to change after we logged in. It would be better to allow users to select their own, easy to remember user name when they sign up.

Spambot settings

The antispam bot checks your mailbox for spam every 15 minutes. CMM suggests that users set their mail clients to check mail every 20 minutes so as not to remove mail from the server before the bot can examine it.

The most likely user for this service is a high-volume mail user. Such users would prefer to check their mail at shorter intervals and may find that 20 minutes is too long to wait between mail downloads.

If the bot were to check the mailbox at a shorter interval, this problem would be minimized, although this would come at the price of a higher load on the CleanMyMailbox servers.

Using CleanMyMailbox

Once you have an account with CMM, you mail is filtered without any further intervention. But to get full use out of the system, you will need to log into the CMM site on a regular basis to see what spam is being stored there and to adjust the filtering settings so that non-spam emails are not blocked as spam.

Logging in is made more difficult be the fact that the CMM home page does not have any obvious place to log in. To log into CMM, you must click on a secondary page first, where you will see a login item on the navigation menu.

Spam box

The heart of the CMM interface is the Spambox. Messages that are filtered as spam by the bot are moved to the spambox for later review. The spambox works in a similar method the inbox of Hotmail and other Web based email services. You can see who the mail is from, when it was sent, and even read the mail in the spambox if you would like.

Most email programs and Web based email sites allow the user to sort their email box by sender, subject, and date by clicking on the column headings. The spambox displays messages sorted by date, and does not allow the user to sort the messages in any other manner. This is unfortunate, as sorting would allow a user to manage their spambox more efficiently.

We like the fact that CMM allows you to restore messages from your spambox to your mail server. Messages can also be deleted from the spambox directly.

Messages can also be selected to have their senders added to a whitelist so that all future mail from that address will be ignored by the CMM bot. This approach may cause some usability issues since all other actions that can be performed from the spambox deal with messages, while this option works with the sender. There is also no apparent reason why the sender whilelist is available from the spambox while the other white- and blacklists are not.

To perform any of these actions on a message, you must click the appropriate checkbox next to the message and then click a button to process your choices. There is an extra, unlabeled checkbox that appears in the delete column. Clicking this box selects all of the messages for deletion.

Navigation within user account and spambox is inconsistent. There are three primary functions available to a user: View Spambox, Edit settings, and view stats. The button to edit settings is by itself on the top of the page while the button to view stats is centered at the bottom of the page.

Once you are editing settings or viewing stats, there is no way to return to the spambox without using the browser’s back button.

Spam filtering

Spam can be filtered into a special spam box on the CMM servers and viewed through their Web site or can be tagged with “** SPAM **” at the beginning of the subject.

CleanMyMailbox recommends that you use the spambox option. Once messages are in your spambox, you can make changes to your whitelist by simply selecting the message for addition to your whitelist.

Once a message has been altered to include the spam tag, your email client’s built in filters can remove the offending message from your mailbox. We tested this filtering method successfully using Eudora 5.1, Outlook XP, and Outlook Express.

Stats page

A statistics page tracks the number of messages that were caught as spam each month, along with an estimate of how much time CMM has saved you by removing your spam.

We would have like to see some other statistics such as how many messages were added to your whitelists and the percentage of your mail that was spam each month. These statistics would help users evaluate the effectiveness of the spam filters.

Settings

Users can change the mailbox that CleanMyMailbox filters and change how CMM filters their mail. The settings page is overly complex and some of the options on this page, like server port, would be better placed on the advanced settings page.

The advanced settings page contains options for changing the whitelists and blacklists and for changing the filters sensitivity. These options would be more accurately labeled “Filter Settings.” Calling them advanced settings may frighten some users away.

Near the end of our week of tests, a new feature appeared that allows the user to set the sensitivity of the bot. We didn’t have the opportunity to test this feature.

How CMM Did

The measurement for a spam filtering solution is how successful it is in recognizing spam. Not only must the filter recognize when a message is spam, but it must also recognize when it is not. A spam filter that removes a number of legitimate email messages could be considered worse than no filter at all.

In our one week test period, CMM successfully filtered out 53% of the total spam that was sent to our mailbox, but it also incorrectly tagged quite a few legitimate email messages. In fact, 37% of the messages filtered as spam were actually legitimate email messages.

The falsely tagged messages were mainly email discussion lists, and once we configured CMM to ignore mail from these lists, they were no longer filtered. Other filtered messages included email that we sent to someone else and used BCC to send a copy to ourselves.

We feel that the high percentage of false positives is misleading, since these would likely significantly decrease as we fine-tuned the system over time.

Conclusion

CleanMyMailbox is a good option for filtering spam if you don’t check your email often, but power users that check email every few minutes will find that CMM doesn’t filter spam often enough.

---

Want to make sure your site or product works the right way? Kalsey Consulting Group can perform a compelete product analysis for your company, including reports on usability, user interface design, and feature enhancements. Please contact us for more information.