Sending Multiple Emails

Weather Message's email routines send email sequentially using a background worker thread.  When an alarm matches a received product, a task is created for each email group associated with the alarm.  These tasks are added to the email queue for processing.

The email processor will generally work through the queue in short order.  If the email group has a large number of email addresses, it does take some time to send each email address to the smtp server.  While all of that is taking place the other email tasks are waiting.

I have completed the code to create a pool of threads.  This pool could be used to spawn multiple threads that will take a task from the email queue and start sending it to the smtp server.  The question for now is whether this will make the overall sending faster.   If the smtp server is throttled per connection, having multiple connections could make sending emails faster. 

I ran into a situation with a SNPP server that was slow per connection.  In this case, using multiple threads could have made that faster.

Now that the code is complete, I will be running some tests to see how this will work.  If the smtp server allows for multiple connects from one user, we might be able to deliver emails faster.  For users that only have a few groups, this would not have any affect.  For users that have bunches of email groups that are activated often, this may make delivery faster.

Note:  If the limiting factor is the internet pipe speed, there is nothing that we can do to make it faster.

I will report further as testing starts later this week.

Siren Controller - Activation Notice

The Weather Message Siren Controller now has the ability to send email notification of siren activations.  Previously, you would email the actual activation product.  Now the program can generate activation notificates whether automatic or manual.  The Siren Controller sends activation notices to Weather Message Server for processing like normal NWS products.

Here is a example manual activation to cancel the sirens:

NOXX02 KWXM 052011

SIRWXM

WEATHER MESSAGE SIREN CONTROLLER
ACTIVATION MESSAGE
311 PM CDT MON JUL 5 2010

SIREN ACTIVATION TYPE: MANUAL
ACTIVATION CLASS : Cancel

GROUPS:
1
2
 
SIRENS:
801
803
$$

This gives the agency feedback on the sirens and groups that were selected for activation.

National Weather Service Survey

The National Weather Service is asking users to complete a survey. In this survey you will have the option to selecting EMWIN or NOAAPort as your data source. I recommend that you take the opportunity to give the NWS feedback on your ingest method and their services and products.

The NWS is undertaking research on how satisfied you are with our products and services and would appreciate your feedback. The purpose of this research is to help the NWS improve its services. The survey will take approximately 15 minutes to complete and need not be finished all at once. We also encourage you to take a moment and complete one or more of the optional sections focused on Marine, Aviation, and Hazardous Weather.

The Survey is at the following link: https://svy.cfigroup.com/cgi-bin/qwebcorporate.dll?idx=3XRBCG

Backlog of files to be processed

A couple of users have reported a large backlog of files to be processed in their received files directory.  Under normal circumstances, this should not occur.  The processing of files depends on the number of alarms, number of connected clients, speed of the files being received, load on the computer and the speed of the computer itself. 

I have not seen this problem myself, however, I knew that the ingesting routine in Weather Message Server was ready for a make over.  I spend some time over a couple of days to rewrite the queue processing and ingest routines.  These changes will help with the backlog of files.

I moved the file processing to an independent thread. I moved the ingest directory read operation to another independent thread.  The ingest processing routine will now read all files in the directory at one time and queue them for the processing routine. The processing routine will then process them back-to-back without any internal delay.  The previous routine had some delay between each file.

I downloaded and processed the 3 hour EMWIN ftp archive.  This archive contains over 2400 files.  The server processed them in about 2 seconds.  The performance was definately improved.

I have to admit that I have a new 64bit Windows 7 computer with an i7 processor.  That cpu contains 4 cores with hyper-threading.  The i7 appears in the performance monitor as 8 cpu's.

A word of caution is always in order when creating additional threads.  Some think that the more treads you create the faster the processing.  That may not be the case for a single core computer.  Each thread has to share time on that one cpu.  You start to see the benefits of multiple threads when you have mulitple cores.

The changes will definately improve the program's processing speed, regardless of the number of cpu's.