[hpsdr] Degradation problem solved

[Dan Quigley]

Since the release of the new code, I have reported two issues.  

 

The first is a "cold start" problem where Mercury does not function well at
data rates about 48k.  Within 30 minutes or so, the problem works itself
out. This issue is still a mystery.

 

The second issue was a gradual "over time" increase in CPU % and related
degradation of performance , eventually to the point the system was
unusable.  Another interesting observation was how this degradation occurred
more quickly when I set the PowerSDR process priority higher than "normal".
I now know the root cause and can start the troubleshoot process to isolate
and fix it.  

 

When you look at Windows Task Manager and sort the running processes by CPU
(Processor Utilization), the System Idle Process is almost always at the top
of the list.  What you may not know is that "process" is really a roll-up of
several things.  Among other things, included in that CPU number, is
hardware interrupts and deferred procedure calls (DPCs).  You can see these
two items by using the Microsoft "SysInternals" Process Explorer available
here:

 

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

 

When my HPSDR system performance degraded and PowerSDR reported 100% CPU - I
used that program to peek into the PowerSDR process to see which threads
were consuming CPU and sent in a stack trace in hopes that Bill Tracy could
figure out what was going wrong.  As it turns out the problem is not
PowerSDR and is due to an over-temp condition in one of the processors
(most likely the central processor) on my laptop.

 

I mentioned DPCs earlier.  Microsoft defines them as  "A Deferred Procedure
Call (DPC)  is a queued call to a kernel-mode function that will usually be
executed at a later time. DPCs are used by drivers to schedule I/O
operations that do not have to take place in an ISR at a high IRQL, and can
instead be safely postponed until the processor IRQL has been lowered."

 

Processing of streaming data in real-time can be a challenging task for
Windows based applications and device drivers. This is because by design
Windows is not a real-time operating system. There is no guarantee that
tasks can be executed in a deterministic (timely) manner.

 

Audio or video data streams transferred from or to an external device are
typically handled by a kernel-mode device driver. Data processing in such
device drivers is interrupt-driven. Typically, the external hardware
periodically issues interrupts to request the driver to transfer the next
block of data. In Windows NT based systems (Windows 2000 and later) there is
a specific interrupt handling mechanism.   When a device driver cannot
process data immediately in its interrupt routine, it schedules a DPC.

 

Microsoft has a good article on what DPCs are here:

 

http://technet.microsoft.com/en-us/sysinternals/bb963898.aspx

 

Now back to my specific problem.  

 

Yesterday, when PowerSDR showed 100% CPU.  Normally, I could just cycle
PowerSDR and the problem would go away.  This was not the case yesterday.
PowerSDR would restart, then immediately show 100% CPU.   When I used
Process Explorer to look at the threads in PowerSDR, this time however, I
drilled into the System Idle Process  and observed that DPCs were consuming
a whopping 40% CPU!  A little time later with the help of Google - heat is
the culprit.

 

We're in the middle of an early heat-wave here in the Northwest.  It reached
about 90F in my shack yesterday afternoon and the poor little fan on my
laptop could not remove enough heat from the system to allow it to recover.
After an hour or so cool down - PowerSDR worked just fine. 

 

During my  research I ran across this cool (no pun intended)  site and
utility that may help others with latency problems: 

 

Thesycon's DPC Latency Checker is a free Windows tool that analyses the
capabilities of a computer system to handle real-time data streams properly.
It may help to find the cause for interruptions in real-time audio and video
streams, also known as drop-outs. The program supports Windows 2000, Windows
XP, Windows XP x64, Windows Server 2003, Windows Server 2003 x64, Windows
Vista, Windows Vista x64.

 

http://www.thesycon.de/deu/latency_check.shtml

 

 

73's

Dan (N7HQ)

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openhpsdr.org/pipermail/hpsdr-openhpsdr.org/attachments/20090603/3e1642db/attachment-0002.htm>