MeteoIODoc  MeteoIODoc-2.8.0
Processing overview

The pre-processing infrastructure is described in ProcessingBlock (for its API). The goal of this page is to give an overview of the available filters and processing elements and their usage. Moreover, there is a special mode of operation where MeteoIO writes on the screen a line for each data point that gets modified (either filtered, resampled or generated). This is enabled by setting the DATA_QA_LOGS key to true in the [General] section. The outputs then look like the following:

[DATA_QA] Filtering WFJ2::ILWR::MIN_MAX 2016-01-18T09:00:00+01:00 [2016-W03-01]
[DATA_QA] Resampling WFJ1::VW_MAX::LINEAR 2015-10-12T13:00:00+01:00 [2015-W42-01]

Modes of operation

It should be noted that filters often have two modes of operations: soft or hard. In soft mode, all value that is rejected is replaced by the filter parameter's value. This means that for a soft min filter set at 0.0, all values less than 0.0 will be replaced by 0.0. In hard mode, all rejected values are replaced by nodata.

It is possible to disable a given Processing Element for specific stations, using the exclude or only options followed by a list of station IDs (see example below). This is supported automatically by all Processing Elements. Several Processing Elements take arguments describing a processing window (for example, FilterStdDev). In such a case, they take the window parameters arguments as defined in WindowedFilter::setWindowFParams().

Filtering section

The filters are specified for each parameter in the [Filters] section. This section contains a list of the various meteo parameters (see MeteoData) with their associated choice of filtering algorithms and optional parameters.The filters are applied serialy, in the order they are given in. An example of such section is given below:

TA::filter1 = min_max
TA::arg1::min = 230
TA::arg1::max = 330
RH::filter1 = min_max
RH::arg1::min = -0.2
RH::arg1::max = 1.2
RH::filter2 = min_max
RH::arg2::soft = true
RH::arg2::min = 0.0
RH::arg2::max = 1.0
PSUM::filter1 = min
PSUM::arg1::min = -0.1
PSUM::filter2 = min
PSUM::arg2::soft = true
PSUM::arg2::min = 0.
PSUM::filter3 = undercatch_wmo
PSUM::arg3::type = Hellmannsh
PSUM::arg3::exclude = DAV3 WFJ2

Available processing elements

New filters can easily be developed. The filters that are currently available are the following:

  • NONE: this does nothing (this is useful in an IMPORT to overwritte previous filters);
  • MIN: minimum check filter, see FilterMin
  • MAX: maximum check filter, see FilterMax
  • MIN_MAX: range check filter, see FilterMinMax
  • RATE: rate of change filter, see FilterRate
  • STD_DEV: reject data outside mean +/- k*stddev, see FilterStdDev
  • MAD: median absolute deviation, see FilterMAD
  • TUKEY: Tukey53H spike detection, based on median, see FilterTukey
  • DESPIKING: despiking in phase space according to Goring and Nikora (2002), see FilterDespikingPS
  • UNHEATED_RAINGAUGE: detection of snow melting in a rain gauge, see FilterUnheatedPSUM
  • NO_CHANGE: reject data that changes too little (low variance), see FilterNoChange
  • TIME_CONSISTENCY: reject data that changes too much , see FilterTimeconsistency
  • DETECT_GRASS: detection of grass growing under the snow height sensor, see FilterDeGrass
  • POTENTIALSW: ensuring physically realistic incoming short wave radiation, see FilterPotentialSW

Some data transformations are also supported besides filtering, both very basic and generic data transformations:

As well as more specific data transformations:

  • EXP_SMOOTHING: exponential smoothing of data, see ProcExpSmoothing
  • WMA_SMOOTHING: weighted moving average smoothing of data, see ProcWMASmoothing
  • IIR: Low Pass or High Pass critically damped filter, see ProcIIR
  • AGGREGATE: various data aggregation algorithms, see ProcAggregate
  • DEACCUMULATE: recompute instantaneous values from accumulated values, see ProcDeAccumulate
  • UNDERCATCH_WMO: WMO rain gauge correction for undercatch, using various correction models, see ProcUndercatch_WMO
  • UNDERCATCH_FORLAND: Forland1996 rain gauge correction for solid and liquid undercatch, using various correction models, see ProcUndercatch_Forland
  • UNDERCATCH_HAMON: Hamon1973 rain gauge correction for undercatch, see ProcUndercatch_Hamon
  • UNVENTILATED_T: unventilated temperature sensor correction, see ProcUnventilatedT
  • PSUM_DISTRIBUTE: distribute accumulated precipitation over preceeding timesteps, see ProcPSUMDistribute
  • SHADE: apply a shading mask to the Incoming or Reflected Short Wave Radiation, see ProcShade

A few filters can be applied to the timestamps themselves:

  • SUPPR: delete whole timesteps, see TimeSuppr
  • UNDST: correct timestamps that contain Daylight Saving Time back to Winter time, see TimeUnDST