MeteoIODoc 20240328.18c31bd1
mio::FilterDespikingPS Class Reference

Detailed Description

Despiking filter using phase space.

This filter iteratively finds and replaces spikes (outliers) in a data sequence until no more spikes are found. The filter was developed to despike Acoustic Doppler Velocimeter (ADV) data, but might also be useful to remove outliers from other signals.

Example of filter performance for one hour of ADV-data. Outliers are successfully removed by the filter.

Two versions of this filter are implemented:

  • The "Goring"-version is implemented according to following paper: Goring, D.G. and Nikora, V.I. (2002). "Despiking Acoustic Doppler Velocimeter Data" J. Hydraul. Eng., 128(1): 117-126
  • And the "Mori"-version is implemented according to this paper: Mori, N., Suzuki, T. and Kakuno, S. (2007). "Noise of Acoustic Doppler Velocimeter Data in Bubbly Flows" J. Eng. Mech., 133(1):122-125

Method:

  • Detection of the spikes:
    • calculate first and second derivatives of the whole signal
    • the data points plotted in phase space are enclosed by an ellipsoid defined by the standard deviations and the universal threshold (=sqrt(2*ln(number of data points))) (see figure below)
    • points outside this ellipsoid are designated as spikes (the Goring-implementation uses three 2D-projections of the ellipsoid to identify the outliers, while the Mori-implementation finds the outliers in 3D)
  • Replacement of the spikes:
    • find a fit for data points around the spike
    • replace the spike with a fitted value

It takes the following arguments

  • METHOD: decide which implementation to use: "Mori" or "Goring". The differences are small between both implementations. According to Mori the "Mori"-version performs slightly better. However, we suggest to use the Goring-method, because it is better tested (mandatory parameter).
  • SENSITIVITY: control the sensitivity of the filter. The universal threshold is divided by this value. The larger the sensitivity the smaller the threshold (=the smaller the ellipsoid) and thus the more spikes will be detected (optional, default=1).
  • INTERPOL_DEG: degree of the fit for the replacement. 1 is linear, 2 is quadratic, ..., 0 means "off" (removed spikes will be nodata and can be resampled later via [Interpolations1D]). This is useful in rare cases where the cubic fit happens to produce an unwanted peak by itself (optional, default=3 meaning cubic fit as proposed in the paper by Goring).
  • INTERPOL_PTS: number of points used for the fitting (replacement) of the spikes (optional, default=24 as in Goring's paper).

There are also some hard-coded parameters:

  • the maximum number of iterations for the spike detection. This is set to 50.
  • When fitting, extrapolation is set to false.
2D projections of the phase space plots. Points outside the ellipses are spikes.
Author
Thiemo Theile
Date
2018-08-15
VW::filter1 = despiking
VW::arg1::sensitivity = 1
VW::arg1::method = GORING
HS::filter1 = despiking
HS::arg1::sensitivity = 1.4
HS::arg1::method = GORING
HS::arg1::interpol_deg = 0 ;remove spikes

#include <FilterDespikingPS.h>

Public Member Functions

 FilterDespikingPS (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
 
virtual void process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)
 This function is the main function to do the filtering: More...
 
- Public Member Functions inherited from mio::ProcessingBlock
virtual ~ProcessingBlock ()
 
virtual void process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)=0
 
virtual void process (Date &dateStart, Date &dateEnd)
 
std::string getName () const
 
const ProcessingPropertiesgetProperties () const
 
const std::string toString () const
 
bool skipStation (const std::string &station_id) const
 Should the provided station be skipped in the processing? More...
 
bool noStationsRestrictions () const
 
const std::vector< DateRangegetTimeRestrictions () const
 

Additional Inherited Members

- Static Public Member Functions inherited from mio::ProcessingBlock
static void readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y)
 Read a data file structured as X Y value on each lines. More...
 
static void readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y1, std::vector< double > &Y2)
 Read a data file structured as X Y1 Y2 value on each lines. More...
 
static std::vector< double > readCorrections (const std::string &filter, const std::string &filename, const size_t &col_idx, const char &c_type, const double &init)
 Read a correction file applicable to repeating time period. More...
 
static std::vector< offset_specreadCorrections (const std::string &filter, const std::string &filename, const double &TZ, const size_t &col_idx=2)
 Read a correction file, ie a file structured as timestamps followed by values on each lines. More...
 
static std::map< std::string, std::vector< DateRange > > readDates (const std::string &filter, const std::string &filename, const double &TZ)
 Read a list of date ranges by stationIDs from a file. More...
 
- Protected Member Functions inherited from mio::ProcessingBlock
 ProcessingBlock (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
 protected constructor only to be called by children More...
 
- Static Protected Member Functions inherited from mio::ProcessingBlock
static void extract_dbl_vector (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< double > &ovec)
 
static void extract_dbl_vector (const unsigned int &param, const std::vector< const MeteoData * > &ivec, std::vector< double > &ovec)
 
- Protected Attributes inherited from mio::ProcessingBlock
const std::set< std::string > excluded_stations
 
const std::set< std::string > kept_stations
 
const std::vector< DateRangetime_restrictions
 
ProcessingProperties properties
 
const std::string block_name
 
- Static Protected Attributes inherited from mio::ProcessingBlock
static const double soil_albedo = .23
 
static const double snow_albedo = .85
 
static const double snow_thresh = .1
 parametrize the albedo from HS More...
 

Constructor & Destructor Documentation

◆ FilterDespikingPS()

mio::FilterDespikingPS::FilterDespikingPS ( const std::vector< std::pair< std::string, std::string > > &  vecArgs,
const std::string &  name,
const Config cfg 
)

Member Function Documentation

◆ process()

void mio::FilterDespikingPS::process ( const unsigned int &  param,
const std::vector< MeteoData > &  ivec,
std::vector< MeteoData > &  ovec 
)
virtual

This function is the main function to do the filtering:

  1. subtract the mean from the signal
  2. iteratively find and replace spikes in the signal
  3. add the mean back to the signal
    Parameters
    paramwhich value from the MeteoData should be processed (for example temperature)
    ivecthe input data (MeteoData-vector)
    ovecthe filtered data (MeteoData-vector)

Implements mio::ProcessingBlock.


The documentation for this class was generated from the following files: