[hpsdr] Advice please on my approach to modifying PowerSDR to increase waterfall sensitivity

Sun Apr 25 20:19:22 PDT 2010

Hello, All,

Please note I am not a computer guy and read/write code at a very low 
level.  But I am trying to learn ;)

The HPSDR Atlas/Mercury/Penelope/Ozy/Excaliber/PowerSDR combination is a 
great radio!  The receiver hears very well, but the waterfall is not 
optimized for microwave weak signal CW work.  I have wanted to improve 
this performance, and as the first two steps in this direction I wanted to:

1.  Modify the waterfall palette to be more like Linrad, which has an 
extremely sensitive waterfall, and
2.  Enable the use of larger FFT sizes; e.g. 65536 and 131072, to 
maximize the sensitivity for very narrowband [e.g. CW] signals by making 
FFT bin sizes more appropriately narrow for such signals.

I am hoping for some comment by those smarter and more experienced than 
I as to whether the particulars of what I am doing are correct, and 
possibly some help with an error message I get when I try to make the 
FFT size 131072.

I modified display.c to accomplish #1, and put the new code segment at:
http://www.nitehawk.com/w3sz/LinradColorsforPowerSDR.rtf
It uses the Linrad 23 segment palette modified for windows.
This seems to be working fine.

The best way to increase the FFT size is not clear to me.
As a first attempt, I made the following changes to the DttSP code, as 
downloaded from the KD5TFD PowerSDR 1.10.4+ branch for HPSDR svn for 
code compiled using Visual Studio 2003 [not the new PowerSDR 1.19.3 code 
recently posted by Doug].  The code I used was from:
[svn://64.245.179.219/svn/repos_sdr_windows/PowerSDR/branches/kd5tfd].

The changes I made were:

1.  original line in ovsv.c:
buflen = nblock2(ncoef - 1), fftlen = 2 * buflen;
changed to:
buflen = nblock2(ncoef - 1), fftlen = 16 * buflen;

2.  original line in filter.c:
fftlen = nblock2 (size) * 2;
changed to:
fftlen = nblock2 (size) * 16;  		

3. original line in update.c:
int i, fftlen = 2 * uni[thread].buflen,rtn=0;
changed to:
int i, fftlen = 16 * uni[thread].buflen, rtn=0;

I also made simple modifications to main.c in fftw_wisdom, so it would 
do calculations for up to FFT size 131072.

I believe doing the above will make the FFT size equal to 16 times the 
DSP buffer size as set in PowerSDR, which I have set at 4096 on the DSP 
Setup page, using the combobox values.  If I am correct, then this sets 
the FFT size to 65536.

I suspect that there is a much better way to do this, but I have not 
figured out another approach that seems to work.

Also, I cannot get the FFT size to increase to 131072, which I think 
would require changing the multiplier from 16 to 32,  because when try 
to make fftlen > 16 * buflen, I get an error that is generated by the 
following lines in bufvec.c:
---------------------------
newvec_COMPLEX_fftw(int size, char *tag)
{
	COMPLEX *p = fftwf_malloc(size*sizeof(COMPLEX));
	if (!p)
	{
		if (tag && *tag)
			fprintf(stderr, "fftwf_malloc: %s\n", tag);
		else
			perror("fftw malloc");
		exit(1);
	}

	return p;
}
---------------------------

The exact error message generated is:

"fftwf_malloc: signal out z vec in newFiltOvSv" [after correcting the 
typo 'fftwf_mlloc' in the original source code].

I believe the error comes from execution of the following lines in ovsv.c:

FiltOvSv
newFiltOvSv(COMPLEX * coefs, int ncoef, int pbits) {
   int buflen, fftlen;
   FiltOvSv p;
   fftwf_plan pfwd, pinv;
   COMPLEX *zrvec, *zfvec, *zivec, *zovec;
   p = (FiltOvSv) safealloc(1, sizeof(filt_ov_sv), "new overlap/save 
filter");
//  buflen = nblock2(ncoef - 1), fftlen = 2 * buflen; //deleted
   buflen = nblock2(ncoef - 1), fftlen = 32 * buflen;  //added by w3sz
   zrvec = newvec_COMPLEX_fftw(fftlen, "raw signal vec in newFiltOvSv");
   zfvec = newvec_COMPLEX_fftw(fftlen, "filter z vec in newFiltOvSv");
   zivec = newvec_COMPLEX_fftw(fftlen, "signal in z vec in newFiltOvSv");
   zovec = newvec_COMPLEX_fftw(fftlen, "signal out z vec in newFiltOvSv");

I assume that when I make the multiplier larger than 16, I am exceeding 
either the range of an indexed variable, or more likely exceeding the 
size allocated for a variable such as zrvec, but I haven't been able to 
figure this out in more detail so I could work around it.

The complete debugger output when PowerSDR is run with the multiplier 32 
is below.  I think everything is 'OK' except for the last 3 lines, which 
i have separated from the rest of the output:

'DefaultDomain': Loaded 
'c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll', No symbols 
loaded.
'PowerSDR': Loaded 'C:\HPSDRTEST\PennyMerge\bin\Release\PowerSDR.exe', 
Symbols loaded.
'PowerSDR.exe': Loaded

'c:\windows\assembly\gac\system.windows.forms\1.0.5000.0__b77a5c561934e089\system.windows.forms.dll', 
No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\system\1.0.5000.0__b77a5c561934e089\system.dll', No 
symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\system.drawing\1.0.5000.0__b03f5f7f11d50a3a\system.drawing.dll', 
No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll', 
No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\system.xml\1.0.5000.0__b77a5c561934e089\system.xml.dll', 
No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\adodb\7.0.3300.0__b03f5f7f11d50a3a\adodb.dll', 
No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\hpsdrtest\pennymerge\bin\release\interop.adox.dll', No symbols loaded.
'PowerSDR.exe': Loaded 
'c:\windows\assembly\gac\custommarshalers\1.0.5000.0__b03f5f7f11d50a3a\custommarshalers.dll', 
No symbols loaded.
The thread '.NET SystemEvents' (0x171c) has exited with code 0 (0x0).
--------------------------------
fftwf_malloc: signal out z vec in newFiltOvSv
An unhandled exception of type 'System.InvalidOperationException' 
occurred in mscorlib.dll

Additional information: Handle is not initialized.
--------------------------------

As a reward for those of you who read this far, I put some gif files of 
PowerSDR screens and Linrad screens that show what I am talking about 
regarding the waterfall sensitivities at the following URLs:

The screens are taken using recordings of a 192 kHz swath of the 3456 
MHz band that has a strong birdie at 3456.000, with a weak signal source 
at 3456.044 and an even weaker signal at 3456.088.

These are most easily seen on the main Linrad waterfall, where they 
appear at 0. 44000, and 88000 respectively, as white, red, and 
yellow-green lines, respectively:
http://www.nitehawk.com/w3sz/linrad3456-apr252010comparison.gif

On the screen from the stock, 'original' PowerSDR for HDR with standard 
waterfall colors and FFT size, the signal at 'zero' in the original 
recording appears as a purple line at 9.959, the 044 weak signal source 
appears faintly at 10.003 as just a slight perturbation in the 
background, and the signal at 88 is not visible:
http://www.nitehawk.com/w3sz/OriginalPSDR4096.gif

With PowerSDR with the modified color scheme and the FFT size increased 
x 16, the zero signal is seen as a white line at 10.049, the 044 weak 
signal source is seen faintly, but better than with the stock PowerSDR, 
at about 10.093.  The signal at 88 on the original recording is still 
not visible:
http://www.nitehawk.com/w3sz/fftlenbuflenX16-nozoom3.gif

You can see that I still have a way to go to equal the Linrad 
waterfall's sensitivity.  Please note that Mercury actually does just as 
good a job at HEARING these signals as does Linrad.  It is just that the 
PowerSDR waterfall can't see them, which is a major problem in microwave 
work.  Once I have the FFT size optimized and have solved the click-tune 
problems I have caused, I will tweak the windowing, averaging, etc. 
further.  The FFT bin size for a CW signal will be optimal at less than 
1 Hz, so I am hoping that I can get FFT sizes of greater than 131072 
[maybe 262144] to work.  262144 would give me a bin size of 732 Hz. 
With 65536 by bin size is still not optimal, at 2930 Hz.  For those of 
you who are thinking of suggesting that I reduce the sampling rate below 
192000 so that I can reduce the bin width that way, let me just say that 
won't work for my microwave application. ;)

Thanks to all for any advice, and

-- 
73,

W3SZ
Roger Rehr
http://www.nitehawk.com/w3sz