[DSP-10] EME2 Post Processing program works, but no joy on EME detection.

[Courtney Duncan]

My EME2 post processing program is working.  It does this:

- Reads an input file that has directions about what else to do.
- Reads multiple EME-2 capture files (alternating HA and HE records).
- Throws out points based on overall noise (HE record, field 6, 'pave')
- Allows for inclusion intervals based on time (from HA records).

Here is the input file I have now:

PD Summary
PN 4000
PF 5_21_set.dat
PF 5_26_set.dat
PF 5_26_ris.dat
PF 5_27_set.dat
PF 5_27_ris.dat
PF 5_28_ris.dat
# PF 5_29_ris.dat #
PF 5_29_set.dat
PF 5_30_ris.dat
PF 6_01_set.dat
PF 6_02_set.dat
PF 6_03_set.dat
# End with a comment to prevent end-of-file parse errors. #

PD sets the "debugging level" so I can see enough but not too much. 
Levels are 'None', 'Summary', and 'Detail'.

PN sets the maximum noise (pave) for a record at 4000.  If it's more 
than that the record is skipped, as shown in the summary.  The parser 
only understands HA and HE records (plus my own instruction types PD, 
PN, PF, etc.).  Also, if it doesn't find 'rfgain 0' (100), 'do_window 
0' and 'fft_bw 0', it stops.  I don't know what I would do if these 
were different from those values, so I've done my runs consistently 
that way.  '5_29_ris.dat' is commented out there because it had 
'fft_bw 3'.

This is the output.

[Session started at 2006-06-03 23:59:34 -0700.]

Parsing file 5_21_set.dat
Parsing file 5_26_set.dat
Parsing file 5_26_ris.dat
Parsing file 5_27_set.dat
Parsing file 5_27_ris.dat
Parsing file 5_28_ris.dat
Parsing file 5_29_set.dat
Parsing file 5_30_ris.dat
Parsing file 6_01_set.dat
Parsing file 6_02_set.dat
Parsing file 6_03_set.dat
Points processed:          10314
Points skipped:            0
Points skipped for noise:  1082
Biggest noise value:       3999.21
noise 2586.9 444.063
-10 12.4485 6.14168 0.0499762
-9 12.4172 6.29723 0.0186553
-8 12.4278 6.21838 0.0292254
-7 12.4012 6.35744 0.0027019
-6 12.3454 6.40951 -0.053155
-5 12.3866 6.09527 -0.0119686
-4 12.4728 6.21543 0.0742558
-3 12.4175 6.43381 0.0189626
-2 12.3461 6.4257 -0.0524041
-1 12.4567 6.35012 0.0581833
0 12.4066 6.30611 0.00805186
1 12.3769 6.11324 -0.0216084
2 12.3777 6.26434 -0.0208788
3 12.3357 6.20749 -0.0628289
4 12.2798 6.15163 -0.118736
5 12.3852 6.2058 -0.0133623
6 12.3275 6.34129 -0.0710411
7 12.4208 6.21193 0.022303
8 12.5381 6.23488 0.139566
9 12.3923 6.23986 -0.00619064
10 12.4088 6.22445 0.0102923


DSP-10_EME2_Post has exited with status 0.


The first several lines show the individual session capture files 
read in (each renamed from UHFA_1.DAT).  Points processed are those 
included in the average below.  Points skipped are those passed over 
due to user-set inclusion intervals (PI directive, not used here, the 
intent is to exclude times when the moon is behind a hill or the 
noise level is hopeless or something.)  Points skipped due to noise 
are those skipped because they exceeded the 'PN 4000' directive. 
'Biggest noise value' is the highest level used below that threshold.

Now the interesting part.

The next line is my average and standard deviation on all the noise 
values (HE field 6, I ignore field 5, 'paveave').

And the remaining 21 lines are the bin averages, standard deviations, 
and deviation of the means from the mean of all the bins.

So, I'm thinking I declare success if bin 0 shows a higher value in 
that last column than the others, by maybe one sigma (around 6).  It 
is, however, as near zero as all the others (nearer, actually).  This 
is a non-detection and does match what I see on the yellow line in 
the individual tests.

These are just the numbers straight out of the files, I haven't taken 
logs or anything.

Here are a couple of questions:

- How do I convert 'pave' to something resembling the number in the 
lower right corner of my screen?  I tried 10 * log10( pave / 482 ) 
and several similar things without getting close.  (482 = 512 - 30, 
accounting for the number of bins you ignore at the bottom to avoid 
60 Hz, etc.)  My SpecAve is '16 4.5' and SpecAnl is 'Non 1200'.  On a 
dummy load I'm used to seeing about 9 dB in that display and about 
10-13 dB on the antenna, depending on azimuth.  (I make contacts at 
reasonable receive levels in this configuration.)

- In UHFA.C line 1421 it says:

		pd16[i]=bar_pwr=upow10(0.0002441406*
		   ((double)(raw_data[2*i]+256*raw_data[2*i+1])+LOG_OFFSET_D));

Here's what I think this means:  0.0002441406 is 1.0 / 4096.0, a 12 
bit shift to the right.  LOG_OFFSET_D is 12288.0 such that if all the 
raw_data bits are 0s, this becomes 3 and pd16[i] becomes 1000.0.  If 
both raw_data bytes are unsigned, then the range is 10^3 to 10^19 
which doesn't match what's in my file, so I'm thinking that the high 
order byte [2*i+1] is signed, making the range 10^-5 to 10^11, which 
is more like it.  It seems, however, like for this to give the right 
answer, the low order byte [2*i] needs to be treated as unsigned and 
I don't see where or if that's done.

But, it could be that something is happening in UHFA.C or in the DSP 
code that I wasn't able to find that makes this work like it should. 
What am I missing?

My 1-2 hour data taking tests have been fairly clean noise-wise on 
the moonset side.  It's pretty noisy to my east though so most of the 
rise files are full of  some real interference (16-20 dB lower right 
corner).  Most of this is being lopped off by 'PB 4000'.

Here are my plans:

- For the data.  I'm taking another moonset run right now and will 
add it to the set, but don't expect to see any improvement [checking 
yellow line.... nope].  I could go through and enforce my antenna 
pattern and horizon mask against allowed times, but wonder if I'd see 
enough improvement to be worth the trouble.

- For the station.  I have an old satellite antenna that I could set 
up on a wood step ladder for an all day (night) test sometime.  I'd 
just have to sequence the polarization relay out of the Brickette 
then go out and move the antenna about every half hour all day 
(night).  (Someday I'll buy or borrow a 500 W. amplifier and figure 
out what's really going on here!)

- For the program itself.  It's my first Xcode project on the Mac and 
I'm running right now out of the development/debugging environment. 
I don't even know how to make a standalone executable, but it's 
probably not hard (would be a Darwin command line).  I'm also told 
that there are ways to make plots and to cross compile for other 
platforms like PC.  I might try the source on linux sometime.  If 
someone is interested in using this, I'll slap a version number and 
GPL stuff on it and spend a couple of hours figuring out how to share.

There aren't many additional features I can imagine (I already have 
the 'trivial' ones and dislike the i/o part of programming so really 
don't want to mess with much more prettiness.)  I could understand 
better about the data types to do more meaningful calculations and 
could enable others, like HB or properly deal with things like 
non-zero fft_bw.

- About Meeus.  I looked through a 1991 copy of Meeus and pretty much 
decided that there shouldn't be much polynomial rot for several 
decades.  Towards the end I was just doing the simpler (truncated) 
problems and not finding anything that should lead to errors larger 
than have already been measured (8 - 16 Hz at 10 GHz, well sub-Hz at 
2 meters).  I still don't understand why InstanTrack gets such a 
different answer, but I think EME2 is working right.

Sorry for such a long post.  That's where things stand.

Courtney, n5bf/6