Backlog of files to be processed

A couple of users have reported a large backlog of files to be processed in their received files directory.  Under normal circumstances, this should not occur.  The processing of files depends on the number of alarms, number of connected clients, speed of the files being received, load on the computer and the speed of the computer itself. 

I have not seen this problem myself, however, I knew that the ingesting routine in Weather Message Server was ready for a make over.  I spend some time over a couple of days to rewrite the queue processing and ingest routines.  These changes will help with the backlog of files.

I moved the file processing to an independent thread. I moved the ingest directory read operation to another independent thread.  The ingest processing routine will now read all files in the directory at one time and queue them for the processing routine. The processing routine will then process them back-to-back without any internal delay.  The previous routine had some delay between each file.

I downloaded and processed the 3 hour EMWIN ftp archive.  This archive contains over 2400 files.  The server processed them in about 2 seconds.  The performance was definately improved.

I have to admit that I have a new 64bit Windows 7 computer with an i7 processor.  That cpu contains 4 cores with hyper-threading.  The i7 appears in the performance monitor as 8 cpu's.

A word of caution is always in order when creating additional threads.  Some think that the more treads you create the faster the processing.  That may not be the case for a single core computer.  Each thread has to share time on that one cpu.  You start to see the benefits of multiple threads when you have mulitple cores.

The changes will definately improve the program's processing speed, regardless of the number of cpu's.