Today we released the Emailchemy Memory Boost app, and I’m hoping it helps people deal with some of the gigantic email archives they’ve been writing to me about.

I originally designed Emailchemy to run in as little memory as possible, simply because at that time most computers had 32MB total of RAM. And, being that I was writing Emailchemy in Java, I had to be extra careful about how much heap space (“working memory”) Emailchemy required. This presented a specific requirement to not load an entire email message into memory for processing, because even back then, an email could be 10MB or larger with big attachments and most computers at the time would have choked under that weight. So, Emailchemy was designed to process the contents of an email in chunks, and it was able to run and convert any email in under 6MB of memory.

Even today when computers routinely ship with 1GB of RAM, I insist on sticking to highly efficient use of memory because it forces developers to write better code and it produces executables that can even run on lower-end, older computers. After all, not everybody buys a new laptop every 2 years.

Now, because Emailchemy is written in Java and therefore runs in a virtual machine, when the application starts it has to tell the virtual machine how much memory to grab from the host computer. The problem is that Java applications can’t see how much real memory the host computer has. So, we’ve intentionally kept the amount of memory that Emailchemy requests as low as possible — just high enough to work for most people’s needs. If we were to tell Emailchemy to grab too much memory, it would probably crash.

Even though Emailchemy processes the message content in chunks, for some formats it has to scan and build an index of all the little parts that make up a message in the original file, and that index has to be kept in memory throughout the conversion of all the messages.

So, we’ve bumped Emailchemy’s base memory footprint to 128MB just to accommodate these file formats. But, if a user has very large files, say bigger than a gigabyte or two, then Emailchemy really needs more memory to convert the email at the fastest rate possible. Then, if a user has extremely large files, say a 17GB PST file, Emailchemy really will need some more memory to even begin extracting and converting messages, because the index for such a file may take up the entirety of Emailchemy’s default memory allocation.

When people were writing in and asking for help with these super huge files, we tried to walk them through the process of editing a config file or starting Emailchemy from the command line to give it more memory. In many cases, the users were able to make the changes, but I can’t imagine trying to walk a marginally computer literate person through that exercise. So, it became very clear very fast that we needed to provide an easy way to change Emailchemy’s memory allocation.

But why a separate application? Why not build this into Emailchemy itself? Well, remember, Emailchemy can’t see how much memory the host computer has, so there was a risk that the user would choose a setting that asked for too much memory and thus prevent Emailchemy from launching again. If Emailchemy can’t start, then the user won’t be able to change the setting back, and Emailchemy wouldn’t be able to change it back itself. Sure, we could have posted instructions on how to completely uninstall and reinstall or manually change the setting back, but just putting this all in a separate application seemed like an easier way to go (for the users) in the long run. This way, if the user selects too much memory (which they might, because do average computer users really knows how much memory their computer has free? Probably not.), then they can always just run the memory setting application again and change Emailchemy’s memory back to the default setting.

Anyway, this is becoming a rather long-winded explanation of why the Emailchemy Memory Boost was needed. Ideally, most people shouldn’t need it, because it’s hard enough to explain why it’s needed at all.

So, in a nutshell, if you have really big email files to convert, and Emailchemy appears to be running slowly (say, only a message or two per second), or if Emailchemy stops completely when opening your giant file, try using the Emailchemy Memory Boost.

Tagged on: