[SFDXA] LoTW heading for Statistical Cliff?

Tue Dec 11 11:50:13 EST 2012

Steve's right, but how do we convince the folks at Hq to change their 
thinking here?
They're stuck in a box, it seems.

Gil, W1RG

----- Original Message ----- 
From: "The Romagni's" <eromagni at gmail.com>
To: "Norman Alexander" <npalex at bellsouth.net>
Cc: "sfdxa" <sfdxa at mailman.qth.net>; "Mark Horowitz" <k2au at hotmail.com>
Sent: Tuesday, December 11, 2012 9:30 AM
Subject: Re: [SFDXA] LoTW heading for Statistical Cliff?

Let me jump here, I been involved in designing and rolling out more complex
systems than LOTW, environments that host more sensitive data, with a more
complex database design, and that have to comply with governmental
compliance rules such as PCI, and SOX audits, yet they have been less of a
mess of what the LOTW environment design looks like.

I had multiple discussions in the past with key players at the ARRL who are
in the front line for the design and implementation of LOTW, and in more
than one occasion I have pointed out to them the flaws and issues regarding
its designs and to be able to effectively process the amount of data that
gets uploaded, as it is now, an eventual bottleneck would occur (case in
point what is happening now).

I am not going to discuss the technical flaws, but overall the environment
that LOTW runs and how it was architected unless drastic changes are done
to the workflow, and environment architecture, this issue will not be
resolved by just adding cpu power, and will eventually as we see come down
to a halt as the backlog of processing data will continue to grow
exponentially.

In my day job, if I had design or even review a system like LOTW, I will
never would have given the ok by  QA,  as it will never had passed the QA
testing, and it will not has gone into production; I understand that the
ARRL treats the data as gold! And it wants to ensure the transparency of
it, but you can’t run a data mining system on a shoe string environment,
and it is worse when there is a clear lack of technical architectural
knowledge behind the design and even support.

Steve
W4DTA

On Tue, Dec 11, 2012 at 6:12 AM, Norman Alexander 
<npalex at bellsouth.net>wrote:

> There are those who do not think that the League has this LoTW processing
> issue
> under control.  See comments from W4TV, of Microham below.
>
>
> Norm
> ======================================================
>
> > Delay keeps growing. Up to 8 days just a few minutes ago.
>
> Agreed - 8 days 14 hours!  At the current rate of growth the
> delay will be 9 days before tomorrow.  Here are the latest
> numbers I have:
>
> Upload Time           Time Processed      Elapsed Time
> ============ ========= ========= ========= ========= =======
> 2012/11/23  18:00    2012/11/27  00:43    3  06:43:34
> 2012/11/28  22:32    2012/12/05  02:08    6  03:35:10
> 2012/11/29  18:38    2012/12/06  05:10    6  10:32:20
> 2012/11/30  00:58    2012/12/06  18:24    6  17:26:26
> 2012/11/30  17:04    2012/12/07  17:28    7  00:24:16
> 2012/12/01  08:20    2012/12/08  19:09    7  10:49:15
> 2012/12/01  12:30    2012/12/09  01:10    7  12:40:49
> 2012/12/02  05:00    2012/12/10  19:31    8  14:31:29
>
> The delay is growing out of control - between 12/1 and 112/2
> there were 2.5 times as many QSOs uploaded as LotW processed
> and LotW has been processing an *average* of 11,000 QSOs per
> minute or about 264,000 QSOs per day based on the information
> on the home page.
>
> If one plots a trend line, the delay is growing exponentially.
> The trend indicates a file uploaded *today* is not likely to
> be processed *for more than five weeks* based on the current
> rate of growth in the backlog.  Based on the processing time
> for files uploaded at 0500z on 12/2, there were about 2.25
> million unprocessed QSOs in the queue at that time - and that
> was 48 hours *after* the deadline for submitting CQWW CW logs.
> With the trend in growth between 11/25 and 12/2, the queue could
> hold as many *10 million* unprocessed QSOs (although that is
> unlikely)!
>
> I know the folks in Newington are saying "trust us," but their
> performance - and complete failure to provide *current data* -
> particularly the failure to provide accurate current status on
> the size of the processing queue - makes trust very difficult.
>
> 73,
>
> ... Joe, W4TV
>
>
>
>
> ________________________________
> From: Mark Horowitz <k2au at hotmail.com>
> To: sfdxa <sfdxa at mailman.qth.net>
> Sent: Tue, December 11, 2012 6:05:19 AM
> Subject: [SFDXA] FW: lotw
>
>
> Dec 10, 2012: LoTW Log Processing is About 8 Days Behind -- For more
> information
> on what has been happening with LoTW lately, please see this news item.
>  NEW...
> The bug that was causing the problem has been fixed.  But there is still a
> large
> queue of logs steadily being processed.  Thanks for your patience.
>
> HAVE A GREAT DAY MARK PLANTATION, FLORIDA
> ______________________________________________________________
> ______________________________________________________________
> South Florida DX Assoc.
> SFDXA WebSite: http://qsl.net/k4fk
> SFDXA Repeater 147.33+ 103.5 Tone
> To Post: mailto:SFDXA at mailman.qth.net
> Subscribe or UNSubScribe:
> http://mailman.qth.net/mailman/listinfo/sfdxa
> DX Net Wed 7:33 PM Repeater
>
> This list hosted by: http://www.qsl.net
> Please help support this email list: http://www.qsl.net/donate.html
>
______________________________________________________________
South Florida DX Assoc.
SFDXA WebSite: http://qsl.net/k4fk
SFDXA Repeater 147.33+ 103.5 Tone
To Post: mailto:SFDXA at mailman.qth.net
Subscribe or UNSubScribe:
http://mailman.qth.net/mailman/listinfo/sfdxa
DX Net Wed 7:33 PM Repeater

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html 