The fast real-time processing of radiometer signals can be successfully
implemented because of the existence of modern high performance ADC and
data acquisition systems. One can successfully fight therewith the
powerful spike jams, using sampling frequency 100--1000 times more than
required by sampling theorem. In thas case the powerful spikes could be
easily discarded by threshold processing. However things become complicated
if the spikes power gets to be comparable to the noise signal variance.
Articles (Erukhimov et al. 1988; 1990) hawe shown the reasonability of
using robust methods for raw radiometer data compression. Those
algorithms are effective also to fight not very powerful spikes as mentioned
above.
Analyzing articles considered in those algorithms one can see that we
deal with some alternative aspects, which are typical for most
statistical estimating tasks. On the one hand the trivial meaning average
allows us
to have the minimum dispersion of mean estimation for the normally distributed noise --
like signal. But it is absolutely non robust and mean estimation gets
senseless if even the single spike appears. On the other, hand the
median-like rank estimation are very robust to spike type contamination (up
to 50%), but there is loss in efficiency in case of normally distributed,
spike-free signal. The worst case is a median estimation upon the infinite
sampling: the efficiency drops to 0.637. The compromise Hodges-Lehmann (Hodges, Lehmann 1957)
estimation, which is used for post data processing at RATAN-600, has
0.95-0.98 efficiency, but it is robust just in the case of no more than 25%
contamination. Moreover, the calculation of this estimation is quite time
consuming, because the sorting of
values, where n is sampling amount, is involved. It makes some
difficulties for using such an estimation in real-time procedures.
However, there is a very promising way of solving this problem if one can formalize some rules to choose different algorithm on-the-fly, during raw data processing. It can be done, based on some general features of signal and spikes. There are hardware dynamic "spike-depression" devices, which are still used in some radiometers at RATAN-600. But because of analog type of these devices, only some sorts of analog threshold criteria could be used. Those criteria are completely unrobust and always need the frequently applied fine tuning, which is very difficult and undesirable for long term observation cycle.
The digital integration method with adaptive selection of the optimal algorithm is given in this article. The method is quite robust and based on some general prior information about signal and spikes.