[hpsdr] FFT latency

[G3XJP]

I need your collective wisdom!  Is it not an inescapable fact that any 
serious FFT in the real-time path between your antenna and your ears 
must introduce an inherent and inevitable delay that will preclude a 
conversational VOX/QSK QSO?  I can see that it would not preclude 
listening to a monologue - or to watching spectral pictures on a 
display.  But anything more than about 1/5 sec is doomed to feel like 
Skype on a bad day - where you often end up having to say "over".  As a 
generality, am I missing something fundamental here?  Peter G3XJP