COSTEO DE LAS ACTIVIDADES

The IDU scientific performance is assured by specific test campaigns fol- lowing the approved DPAC standards described in leaders CU1 [2012] and Guerra and leaders CU leaders [2013]. These tests are carried out regularly by the DPCB team in close collaboration with all the IDU contributors. For these tests, detailed analysis over the obtained results are done – even including the execution of reduced iterations with other systems.

As already commented in previous sections, IDU processes a huge amount of data and produces similarly a huge amount of output results. The con- tinuous and progressive check on the quality of these results is more than a desirable feature. However, the analysis of every calibration and parameter produced by IDU (as it is done for the test campaigns) is not affordable – it would have almost the same computational cost than the processing itself. For this reason, we aim for the design and implementation of a modular system able to assure the quality of the results up to a reasonable limit. First of all, we have implemented in each IDU task several built–in consis- tency checks over the input and output data. These are really basic checks for:

• verifying the consistency of the configuration parameters including its tracking along the full processing pipeline.

• verifying the consistency of the input data, so corrupted data or inconsistent input data combinations do not enter the pipeline and are not propagated to subsequent tasks.

• accounting for the number of outputs with respect to the inputs, so data lost is detected and properly handled – in general forcing a task failure.

Additionally, all IDU tasks integrates the Intermediate Data Validation (IDV) framework [Valles et al., 2012]. This framework provides several tools for the generation of statistical plots of different kinds. IDV provides:

Bar Histograms

Histograms for the characterisation of the frequency of parameters with limited number of values.

For example IPD processing status, counts of observations per row, etc.

1D Histograms

Histograms for the computation of the frequency of non discrete pa- rameters. This implementation supports the computation of several percentiles and the Robust Scatter Estimator (RSE).

2D Histograms

Histograms showing the distribution of values in a data set across the range of two parameters. They support static dimensions or abscis- sae dynamic allocation – in general for analysis of the evolution of a given parameter in function of a non restricted increasing parameter as the observation time. They can be normalised globally or locally for each abscissae bin. Percentiles and RSE as well as contours are supported.

Mainly used for the analysis of 2D dependencies or 2D density dis- tributions of two given parameters – usually the abscissae parameter is the magnitude, or some kind of distance; to last CI, to a reference observation/source, etc.

Sky Maps

Plots generated from a histogram based in the HEALPix tessellation and implementing the Hammer-Aitoff Projection. It can represent the pixel count, pixel density or the pixel mean value for a given measured parameter.

Mainly used to obtain the sky distribution of some particular ob- ject (sources, observations, etc.) or to analyse the alpha and delta dependency of some parameter mean value, i.e. the Astrophysical Background, proper motion, etc.

Sky Region Maps

regions.

Mainly used for the analysis of the IDU-XM and IDU-DC results. Focal Plane Region Maps

For the representation of the observations according to their AL and AC plane coordinates.

Mainly used for the analysis of the IDU-SCN and IDU-DC results. Round-Robin Database (RRD)

RRD are used to handle and plot time-series data like network band- width, temperatures, CPU load, etc. but also usable to handle Quaternion evolution, observation density, match distance evolution, etc.

Range Validators

Implement very basic range validation against the expected nominal parameters defined in the MDB ICD

TableStats

Collector of statistics for miscellaneous MDB table fields. Basically provides counters for the discrete values of predefined fields or for boolean flags/fields.

All Histograms and the SkyMaps, share a common framework allowing the split of the collected statistical data according to the FoV, CCD row, CCD strip, window class, source type, etc. This functionality is quite useful for restricting the origin of any features visible in the general plots – in that sense the user can see if some plot peculiarity or feature are present only in one of the FoV, rows or for a given source type.

It is worth mentioning that the Sky Region Plot can be generated di- rectly from the code but also we have implemented a graphical tool for the interactive generation of this kind of plots. This tool is called Sky- Explorer. This tool allows the loading and visualisation of all kinds of Cross-Match related data. It also implements the functionalities for sky navigation, zoom, distance measurement, animation of Gaia scans and

much more. The SkyExplorer has become an essential tool for the vali- dation of the IDU-XM but also for the IDU-DC. Several examples of this tool outputs have been already included in Section 4.3 in this chapter and more can be found in Chapter 6.

All these tools have also been integrated in IDT and they are used for its monitoring on a daily basis. A handful of examples of the plots obtained using these statistics tools have been included in Chapter 6.

With all the listed IDV features, the monitoring of the IDU scientific re- sults should be easy. The only thing pending is the definition of the best diagnostics for each specific task. Some examples could be:

• For all task in general:

– Range validation against expected nominal parameters. • For IDU-XM and IDU-DC:

– Monitoring of the amount of new sources created compared with previous executions of the IDU-XM.

– Monitoring of the time evolution of the Cross-Match AL/AC distance to the primary matched source per FoV.

– Check the evolution of matches to a predefined set of reference sources, to check if the overall transits have been assigned dif- ferently now, as compared to the previous cycle.

– Monitoring of the evolution of the number of spurious detection density for very bright sources.

– etc. • For IDU-IPD:

– Monitoring on the goodness–of–fit obtained.

– Comparison of the derived Image Parameters against the AGIS solution over a pre–selection of well–behaved sources.

– Cross—check of the residual from the previous statistic against the chromatic calibration residuals from AGIS.

– etc.

Most of these checks are still done manually by the operator but their progressive integration in the processing framework is envisaged so they can be performed automatically and summarising reports are provided to the user for inspection. At the time of writing this thesis, many efforts have been devoted on identifying the best diagnostics – the ones assuring the best control on the quality of the produced data – and some examples are provided in Chapter 7.

Finally, it is worth pointing out that the computational performance and the correct progress of the processing is also monitored. The detailed list of functionalities designed and implemented so far for monitoring the job performance and handling the jobs outcome is presented in Chapter 5

4.9

Conclusions

After describing in Chapter 3 the basis of the data reduction system, we have described in more detail some of the most important tasks involved in the astrometric reduction loop. Particular attention has been paid to the IDU-XM, the IDU-SCN and IDU-IPD where most of the work done is attributable to this thesis.

The LSF/PSF calibration has also been covered exhaustively. During years, it has been discussed and studied the possibility of producing a clean LSF/PSF library – free from the CTI effects – in conjunction with a Charge Distortion Model (CDM). In this scenario, the IPD would have been in charge of predicting the distorted image from both: the clean LSF- /PSF library and the CDM and then performing the Maximum-Likelihood Estimation (MLE) against the observed image. This direct modelling of the charge distortion would be more transparent, versatile and will be able

to cope more rigorously the illumination history. Unfortunately, this ap- proach has been deferred since no suitable CDM has been achieved yet and because the estimated development cost of this full direct modelling have been considered not affordable at the current stage of the LSF/PSF cali- bration developments. Instead an empirical modelling approach has been followed, implementing an Empirical LSF/PSF (ELSF) library directly ac- counting for all possible image distortion factors.

We have also described the IDU-IPD task, reasoning how this task brings together all the partial solutions of the several processing and calibration systems coming from different CUs, consolidating the starting of the as- trometric reduction iteration loops. This consolidation is essential for the improvement of the astrometric solution produced by AGIS.

It is worth pointing out the close cooperation with IfA-ROE team during this thesis. The four months stay with this team during 2010 and 2011 procured a solid basis for the design and implementation of most of the IDU tasks. This close cooperation continues and is fundamental for the progressive improvement of most of the IDU tasks.

It must be noted, that the current design of IDU tasks, and their implemen- tation later described in Chapter 4, fulfil to big extent all the requirements for the Gaia data exploitation.

Finally, an overview of the several monitoring and validation tools imple- mented in the frame of this thesis has been included. The autonomous validation and monitoring of the outputs is still an ongoing task but this fact can not be considered a major or stopping issue for the execution of any of the developed IDU tasks.

5

IDU Operation and