[SFDXA] LoTW heading for Statistical Cliff?

Tue Dec 11 11:04:34 EST 2012

nothing to do with money, it is all about proper system architectural
design, this can be done on a low budget using open source tools.

On Tue, Dec 11, 2012 at 9:47 AM, Pete Rimmel N8PR <n8pr1 at bellsouth.net>wrote:

> Steve, you had a lot more money to work with than they do,  I bet!
>
> 73,  PeteR
>
> -----Original Message----- From: The Romagni's
> Sent: Tuesday, December 11, 2012 9:30 AM
> To: Norman Alexander
> Cc: sfdxa ; Mark Horowitz
> Subject: Re: [SFDXA] LoTW heading for Statistical Cliff?
>
> Let me jump here, I been involved in designing and rolling out more complex
> systems than LOTW, environments that host more sensitive data, with a more
> complex database design, and that have to comply with governmental
> compliance rules such as PCI, and SOX audits, yet they have been less of a
> mess of what the LOTW environment design looks like.
>
> I had multiple discussions in the past with key players at the ARRL who are
> in the front line for the design and implementation of LOTW, and in more
> than one occasion I have pointed out to them the flaws and issues regarding
> its designs and to be able to effectively process the amount of data that
> gets uploaded, as it is now, an eventual bottleneck would occur (case in
> point what is happening now).
>
> I am not going to discuss the technical flaws, but overall the environment
> that LOTW runs and how it was architected unless drastic changes are done
> to the workflow, and environment architecture, this issue will not be
> resolved by just adding cpu power, and will eventually as we see come down
> to a halt as the backlog of processing data will continue to grow
> exponentially.
>
> In my day job, if I had design or even review a system like LOTW, I will
> never would have given the ok by  QA,  as it will never had passed the QA
> testing, and it will not has gone into production; I understand that the
> ARRL treats the data as gold! And it wants to ensure the transparency of
> it, but you can’t run a data mining system on a shoe string environment,
> and it is worse when there is a clear lack of technical architectural
> knowledge behind the design and even support.
>
> Steve
> W4DTA
>
>
> On Tue, Dec 11, 2012 at 6:12 AM, Norman Alexander <npalex at bellsouth.net
> >wrote:
>
>  There are those who do not think that the League has this LoTW processing
>> issue
>> under control.  See comments from W4TV, of Microham below.
>>
>>
>> Norm
>> ==============================**========================
>>
>> > Delay keeps growing. Up to 8 days just a few minutes ago.
>>
>> Agreed - 8 days 14 hours!  At the current rate of growth the
>> delay will be 9 days before tomorrow.  Here are the latest
>> numbers I have:
>>
>> Upload Time           Time Processed      Elapsed Time
>> ============ ========= ========= ========= ========= =======
>> 2012/11/23  18:00    2012/11/27  00:43    3  06:43:34
>> 2012/11/28  22:32    2012/12/05  02:08    6  03:35:10
>> 2012/11/29  18:38    2012/12/06  05:10    6  10:32:20
>> 2012/11/30  00:58    2012/12/06  18:24    6  17:26:26
>> 2012/11/30  17:04    2012/12/07  17:28    7  00:24:16
>> 2012/12/01  08:20    2012/12/08  19:09    7  10:49:15
>> 2012/12/01  12:30    2012/12/09  01:10    7  12:40:49
>> 2012/12/02  05:00    2012/12/10  19:31    8  14:31:29
>>
>> The delay is growing out of control - between 12/1 and 112/2
>> there were 2.5 times as many QSOs uploaded as LotW processed
>> and LotW has been processing an *average* of 11,000 QSOs per
>> minute or about 264,000 QSOs per day based on the information
>> on the home page.
>>
>> If one plots a trend line, the delay is growing exponentially.
>> The trend indicates a file uploaded *today* is not likely to
>> be processed *for more than five weeks* based on the current
>> rate of growth in the backlog.  Based on the processing time
>> for files uploaded at 0500z on 12/2, there were about 2.25
>> million unprocessed QSOs in the queue at that time - and that
>> was 48 hours *after* the deadline for submitting CQWW CW logs.
>> With the trend in growth between 11/25 and 12/2, the queue could
>> hold as many *10 million* unprocessed QSOs (although that is
>> unlikely)!
>>
>> I know the folks in Newington are saying "trust us," but their
>> performance - and complete failure to provide *current data* -
>> particularly the failure to provide accurate current status on
>> the size of the processing queue - makes trust very difficult.
>>
>> 73,
>>
>> ... Joe, W4TV
>>
>>
>>
>>
>> ______________________________**__
>> From: Mark Horowitz <k2au at hotmail.com>
>> To: sfdxa <sfdxa at mailman.qth.net>
>> Sent: Tue, December 11, 2012 6:05:19 AM
>> Subject: [SFDXA] FW: lotw
>>
>>
>> Dec 10, 2012: LoTW Log Processing is About 8 Days Behind -- For more
>> information
>> on what has been happening with LoTW lately, please see this news item.
>>  NEW...
>> The bug that was causing the problem has been fixed.  But there is still a
>> large
>> queue of logs steadily being processed.  Thanks for your patience.
>>
>> HAVE A GREAT DAY MARK PLANTATION, FLORIDA
>> ______________________________**______________________________**__
>> ______________________________**______________________________**__
>> South Florida DX Assoc.
>> SFDXA WebSite: http://qsl.net/k4fk
>> SFDXA Repeater 147.33+ 103.5 Tone
>> To Post: mailto:SFDXA at mailman.qth.net
>> Subscribe or UNSubScribe:
>> http://mailman.qth.net/**mailman/listinfo/sfdxa<http://mailman.qth.net/mailman/listinfo/sfdxa>
>> DX Net Wed 7:33 PM Repeater
>>
>> This list hosted by: http://www.qsl.net
>> Please help support this email list: http://www.qsl.net/donate.html
>>
>>  ______________________________**______________________________**__
> South Florida DX Assoc.
> SFDXA WebSite: http://qsl.net/k4fk
> SFDXA Repeater 147.33+ 103.5 Tone
> To Post: mailto:SFDXA at mailman.qth.net
> Subscribe or UNSubScribe:
> http://mailman.qth.net/**mailman/listinfo/sfdxa<http://mailman.qth.net/mailman/listinfo/sfdxa>
> DX Net Wed 7:33 PM Repeater
>
> This list hosted by: http://www.qsl.net
> Please help support this email list: http://www.qsl.net/donate.html
>