[Elecraft] Sherwood on ARRL Testing Methodology (LONG!)

[Bill Tippett]

         Rob Sherwood gave his permission to post this on the
FT-2000 list.  Since that is public information I am posting
it here also.  BTW I agree with his comments on the "new"
methodology.

                                 73,  Bill  W4ZV


What has gone wrong with the ARRL's new Product Reviews in QST?

For several years I participated as part of a group of hams who were 
trying to improve the testing of radios reviewed in QST.  Several 
reviews in the past had included questionable data on receiver 
performance.  A sincere effort was initiated to correct testing 
problems within the lab, and hopefully to also improve the "hands on" 
portion of the report.

As time went on, however, it appeared to me that the group had become 
fixated on minutia, and at the same time the League was unwilling to 
look at real problems in new radios being offered to the amateur 
radio operator.  I have not contributed lately in the steering 
committee, as I felt I was banging my head against the wall.

Some of the nonsense coming out of the League has been around for a 
long time, like the following quote from the 2004 review of the Icom 
IC-7800.  "I was able to hear calling US stations on back scatter 
that I don't believe I would have heard on the '930."  Did the 
reviewer bother to turn on his TS-930?  No, he just assumed he was 
hearing something unusual on the receiver being reviewed (or hyped) in QST.

Has anything improved in 2007?   The latest October review of the 
FT-2000D (200 watt version of the FT-2000 that was earlier reviewed 
in February) states the following: "Why would I need a 200 W 
transceiver?  After using it for a while, I was quite impressed with 
the extra punch the '2000D offered during routine CW and SSB contacts 
compared to the 100 W version."  How could anyone tell a difference 
of 3 dB, especially compared to operation of the FT-2000 eight months 
ago?  This kind of reporting is drivel.

What has changed in the ARRL reports?

Some of the changes are of minor interest, like measuring the noise 
figure of an HF radio.  Noise figure is generally used by VHF and UHF 
enthusiasts, but adding these data points certain hurts nothing.  Is 
noise figure, or noise floor, or sensitivity a significant issue in 
today's receivers?  The 75A-4 has an excellent noise floor, as 
reported in the January 2006 QST Annual Vintage Issue.  Few of us 
have such a quiet location that atmospheric and galactic noise don't 
overshadow the noise floor of a modern receiver.

What we did get was additional confusion in the ethereal world of 
third-order intercept (IP3), in place of real dynamic-range 
data.  The League used to measure it one way, then a second way, and 
now three ways.  Is this supposed to be helpful?  The old way 
(measured at the noise floor) was acceptable. The second way 
referenced an imprecise S5, now defined as -97 dBm, and a third new 
way at 0 dBm.  Zero dBm is really strong, something we don't likely 
ever see, unless we are working Field Day or Multi-Multi contests 
from near-by transmitters.  (I am assuming we are not living in 
Europe with their 5 megawatt AM broadcast transmitters.)

0 dBm is S9 + 73 dB, assuming any S meter reads that level 
accurately.  (The Flex 5000A would actually do that.)  On my IC-781, 
0 dBm reads S9 + 50 with 30 dB of internal attenuation, or something 
like S9 + 80 dB with the attenuators off, if the S meter  would read 
that high, which is does not.  What happens when you put two 0 dBm 
signals into an IC-781 at 20 kHz spacing?  The IMD reads S9 + 18 
dB.  At 2 kHz spacing the IMD reads S9 + 60 dB!  The 781 is not a 
radio with performance problems, so what do these new and improved 
measurements really mean?

If you look at the FT-2000 chart for IP3 at 2 kHz with the preamp 
off, you see the IP3, measured at the noise floor, is -19 dBm.  This 
is not a good number, particularly since a Yaesu radio with "IPO" 
enabled (no preamp) is similar to most other radios with the 10 dB 
attenuator enabled.  Yet if you measure the FT-2000 at 0 dBm, the IP3 
calculates out to +15 dBm, which sounds good.  This new information 
is meaningless at best, or misleading at its worst.  Why is the IP3 
so high at 0 dBM?  Because the inter-modulation is so strong (S9 + 60 
dB) the AGC has basically turned the gain of the radio off.

Most operators will run an FT-2000 with preamp 1 enabled, since it 
gives a reasonable noise floor, sensitivity and AGC threshold. Yet no 
information is available with this typical setting for the newly 
touted IP3 reporting method, which at 2 kHz would be about -30 dBm 
for the League's sample.  (The FT-2000 I measured was considerably 
worse.) To get a meaningful dynamic-range number, the reader now has 
to subtract two numbers.  Why is this important data now missing, or 
at least obfuscated?  Could it be the big advertisers in QST didn't 
like seeing 2 kHz dynamic-range numbers that are typically around 70 
dB? Only the League could take a measured 2 kHz dynamic range of 69 
dB at 2 kHz and calculate it into a +15 dBm intercept at 0 dBm.  Talk 
about smoke and mirrors!

The League is also going to differentiate between blocking (gain 
compression) and phase noise limited (a typical problem with 
synthesized radio).  A narrow band audio spectrum analyzer is needed 
to measure blocking this new way.  (I used this method on my Flex 
5000A report because of the phase noise.)  The ear is not going to 
hear what the analyzer sees, but the League may have made an 
improvement here.  At least the two measurements now will be differentiated.

What is the League completely missing?

Most new DSP radios have serious problems in QRN, and with any kind 
of transient impulse noise.  Has QST reported on these 
problems?  They have not said a word.  I gave a talk on this subject 
at the 2007 Dayton Hamvention, to try to point out that all is not 
well in the current state of radio design.  The IC-7000 is a prime 
example of a radio that is nearly useless in QRN, as is the 
FT-2000.  Every DSP-chip based radio designed in the last few years 
has an AGC problem to some extent.  Fast rise-time noises are 
improperly handled by the AGC, drastically exaggerating the impulse noise.

I recently finalized an AGC test, using an HP fast-rise-time pulse 
generator.  It basically approximates an electric fence.  The 
generator was set for one pulse per second.  The rise time was < 10 
nanosecond, with a duration of around 1 microsecond.  The level was 
set to 1 volt peak, to propagate a pulse well into the HF 
spectrum.  The first radio tested with this new method was the 
FT-2000, with preamp 1 enabled.  This produces a rather typical CW 
noise floor of -124 dBm, an SSB sensitivity reading of 0.3 uV, and an 
excellent AGC threshold of 1.3 uV. With a reference non-DSP IC-781 
that has similar specifications, the S meter on the pulse test read 
less than S1, barely moving the S meter.  On the FT-2000 the impulse 
noise read S7, pulse after pulse after pulse.

While many hams seem oblivious to these AGC problems, at least some 
operators are voicing their concern.  I was pleased to hear from a 
new ham at a recent Colorado hamfest describe his observations on his 
IC-7000.  Even though he had no past frame of reference from an 
analog radio, he noted how strangely his Icom reacted to the 
slightest click or tick.  Merely turning on a light switch would kick 
his S meter up many S units.  I noticed the exact same problem two 
years ago on all the DSP radios coming though my lab and ham shack.

When I queried the League on their review of the IC-7000, saying they 
totally missed the AGC problems on transient noise and QRN, they 
simply said they listened to it in December when there was no 
QRN.  What is their excuse this time on the FT-2000D?  This radio had 
to have been evaluated during the summer of 2007 when there was 
plenty of thunderstorm static.

What did the FT-2000D review happen to say about the dynamic range 
numbers with the different roofing filters?  After giving a full 
paragraph to explaining why narrower roofing filters are usually 
helpful, the League simply said, "We noticed little difference in 
performance between the 3 and 6 kHz roofing filers in any of the 
FT-2000s tested, though, at any signal spacing."

On the FT-2000 data recently posted on my web site, the dynamic range 
actually dropped from 90 dB at 20 kHz with the 6 kHz roofing filter 
to 81 dB with the 3 kHz roofing filter.  At 2 kHz spacing there was 
minimal difference, 63 dB (3 kHz filter) and 61 dB (6 kHz 
filter).  Dynamic range numbers in the low 60s are not acceptable for 
serious operators.

Finally, one more bizarre comment from the "hands-on" QST 
reviewer.  The u-Tune unit adds modest selectivity in the front end, 
and significant insertion loss, as seen by the degraded noise 
floor.  Yet the reviewer found the u-Tune unit to be helpful "on 20 
meters before the band closed with the u-Tune unit switched on."  One 
wonders why a little added RF selectivity and 10 dB insertion loss 
would help when the band was fading out.  If this statement is 
accurate, which I question, there is something seriously wrong with 
this radio beyond AGC and roofing-filter problems, a subject totally 
ignored by the review.

When will the day come when the information in QST is more than a 
fluff review, and a free multi-page advertisement for the manufacturer?

73,
Rob Sherwood
NC0B
Rev C1