Variables Estadísticas

Catalogs with information about objects on the sky are amongst the most important data sets in astronomy. These catalogs consist of sets of sources and parameters (or attributes) that quantify properties of these sources. We introduce the Source Collection to handle such catalogs and give a simplified example.

Implementation Detail: Besides Source Collections, Astro-WISE also has the SourceList class to hold catalog data (section 3.1.3). For the purpose of this chap- ter, readers familiar with Astro-WISE can see a SourceList as a Source Collection with an image as progenitor (sections 2.5.1, 2.5.2).

2.2.1

Source Collection Description and Representations

The Source Collection is a Process Target for source catalogs. Processing a Source Collection means deriving the exact set of sources and the values of the attributes from its progenitors. When we refer to catalog data, we mean this processing result.

A Source Collection can be created without being processed. That is, the catalog data itself does not need to be created. We often use the terminology that a Source Collection represents sources and attributes, since it does not always contain sources or attributes.

Therefore it is insightful to see a Source Collection also as a representation of the action required to derive the catalog. We use the term operator to refer to this action in a conceptual sense. Processing a Source Collection then becomes evaluating this operator on other Process Targets, its progenitors. It is instructive to present the different ways to see a Source Collection that arise as a result of this catalog–operator duality.

Representation: Operator

Every Source Collection instance corresponds to an operation to create the catalog, called its operator. The operations have to be predefined by the information system, because this maximizes the amount of information that can be inferred from the data lineage. The Source Collection operators relevant for our research are summarized in section 2.4.1 and described in more detail in section 2.5.

In this thesis we mention the corresponding operator when referring to a Source Collection. For example, a Source Collection that applies a selection criterion to limit the sources of another Source Collection is called a Filter Sources Source Collection. In our Astro-WISE implementation we defined a separate Process Target class for each operator (section 3.3). Therefore, we also talk about the class of a Source Collection. Representation: Set of Source Objects

A source itself can also be seen as an object in the computer science reading of the term. This makes a Source Collection a set of source objects (figure 2.1b). A

parametrized property of a source can then be seen as an attribute of such an object. In this thesis we will usually use the term attribute instead of parameter, which originates from this object oriented approach, even though we do not use such source objects in our Astro-WISE implementation.

SourceCollection contains Source

Attribute1 Attribute2 AttributeN

(a) Entity-Relation Diagram

(b) Set (c) Table

Figure 2.1: Top: Entity Relation diagram of a Source Collection. Bottom: representations of two Source Collections. Source Collection A contains sources 12, 18, 44 and 67 of which two attributes are shown. Source Collection B represents a subset of two sources of Source

Collection A with the same attributes.

Representation: Table

The result of processing a Source Collection can be seen as a table, where every row corresponds to a source, and every column to a parameter (figure 2.1c). An information system is expected to handle and store the result of processing a Source Collections in this manner. This allows the information system to maintain the object oriented approach of the sources in a flat relational database.

Representation: Dependency Graphs

It is instructive to visualize Source Collections and their dependencies as connected nodes in a graph (figure 2.2b and 2.3 in the example in section 2.2.2). In the figures in this thesis the progenitors of a process target are shown above it. Therefore the data processing runs from top to bottom and the data lineage from bottom to top.

Such a dependency graph can conceptually be extended in both directions. The top nodes will contain photometric attributes and can be connected to nodes repre- senting frames that were used to measure these attributes. The bottom nodes can be connected to nodes representing hypothetical process targets for graphs or other analysis products.

2.2.2

Source Collection Example

We demonstrate the Source Collection concept with a simplified example of data pulling. We assume the existence of a Source Collection (labeled A) that contains apparent magnitudes and redshifts for a large set of galaxies. A scientist pulls a dataset with both absolute and apparent magnitudes for nearby galaxies.

Pulling of a Catalog

The scientist formulates a data pulling request (figure 2.2). Three pieces of informa- tion are specified:

ˆ The data set from which the sources should be selected: Source Collection A. ˆ The selection criterion for the sources: a redshift below 0.1.

ˆ The required attributes: absolute and apparent magnitudes. Creation of Dependencies

The information system creates the required Source Collections (figure 2.3a): ˆ Source Collection B is created to select all sources that match the given selection

criterion.

ˆ The information system determines that no absolute magnitudes have been defined for these sources and it creates Source Collection C to calculate absolute magnitudes from relative magnitudes.

ˆ The information system determines that the calculation can be performed on all sources in Source Collection A. Therefore, it optimizes for generality and uses Source Collection A as progenitor of Source Collection C, instead of B. The Source Collection is not yet processed at this stage.

ˆ Source Collections D and E are created to combine the attributes represented by different Source Collections and select the required ones.

Processing of Dependencies

The created dependency graph is optimized and processed (figure 2.3b):

ˆ The information system creates a temporary copy of the dependency graph in order to optimize it for scalability to fulfill the request as quickly as possible. ˆ It reorganizes the dependency graph to minimize the required processing by

placing the selection of sources before the calculation of absolute magnitudes. ˆ The information system retrieves the data of Source Collection b and uses this

to let Source Collection c process itself completely. The calculated attributes will be stored for future requests, because they cannot be derived on the fly. ˆ The other Source Collections are processed on the fly while retrieving the catalog

data of Source Collection e. The catalog data is subsequently returned to the scientist.

ID Operator Process Parameters (Set of Sources) Number

List of Attributes (a) Primitive A E x t e r n a l s r c s : ( a ) 1 0 0 0 0 0 RA, DEC m a g _ u , z

(b) Example (c) A scientist pulling data from the infor- mation system.

Figure 2.2: (a) A Source Collection primitive and (b) a representation of a Source Collection used in the example: Top left: a unique identifier of this Source Collection. Top right: the operator of the Source Collection. The second row represents the process parameters, if any. The next row describes the sources of the Source Collection. The number on the right is the number of sources and the letter between parenthesis represents the exact set of sources. Source Collections with the same symbol represent the exact same set of sources; different symbols might represent different sets. At the bottom: the names of the attributes that are represented by this Source Collection; in this case celestial coordinates, an apparent magnitude in the u band and a redshift. (c) The scientist formulating a data pulling request.

2.2.3

Key Features in Example

The example in section 2.2.2 highlights the key aspects of the Source Collections (section 2.3):

ˆ Catalog data is pulled and new Source Collections are created to compute at- tributes that do not yet exist (sections 2.3.3, 2.3.5).

ˆ The final catalog has full data lineage: any attribute value can be recalculated and the selection criterion is stored (section 2.3.1).

ˆ Calculations are defined as generally as applicable. Source Collection C can be reused if at a later stage absolute magnitudes are requested for another subset of Source Collection A (section 2.3.3).

ˆ The information system reorganizes the order of the Source Collections to pre- vent the calculation of unnecessary data (section 2.3.4). For more complex dependency graphs an algorithm to determine logical relations between sets of sources is used (section 2.3.6).

ˆ Source Collections C is processed partially by processing its smaller copy c entirely and sharing the result (sections 2.3.4, 2.3.1).

ˆ The calculation of the absolute magnitudes can be performed in Python, while the selection of data can be performed on the database (section 2.3.2).

A E x t e r n a l s r c s : ( a ) 1 0 0 0 0 0 RA, DEC m a g _ u , z B Filter S o u r c e s q u e r y : z < 0 . 1 s r c s : ( b ) 1 0 0 0 RA, DEC m a g _ u , z C A t t r i b u t e Calculator calc: AbsMag s r c s : ( a ) 1 0 0 0 0 0 a b s m a g _ u D C o n c a t e n a t e A t t r i b u t e s s r c s : ( b ) 1 0 0 0 RA, DEC m a g _ u , z a b s m a g _ u E S e l e c t A t t r i b u t e s a t t r s : m a g _ u , a b s m a g _ u s r c s : ( b ) 1 0 0 0 m a g _ u a b s m a g _ u

(a) Persistently Stored Source Collections

a E x t e r n a l s r c s : ( a ) 1 0 0 0 0 0 RA, DEC m a g _ u , z b Filter S o u r c e s q u e r y : z < 0 . 1 s r c s : ( b ) 1 0 0 0 RA, DEC m a g _ u , z c A t t r i b u t e Calculator calc: AbsMag s r c s : ( b ) 1 0 0 0 a b s m a g _ u e S e l e c t A t t r i b u t e s s r c s : ( b ) 1 0 0 0 m a g _ u d C o n c a t e n a t e A t t r i b u t e s s r c s : ( b ) 1 0 0 0 m a g _ u a b s m a g _ u

(b) Transient Source Collections used for Processing

Figure 2.3: Two dependency graphs of Source Collections, generated by the information system. Every box represents a Source Collection. The Source Collections on the left are persistently stored, where the Attribute Calculator is defined as general as applicable, to facilitate reuse. The Source Collections on the right are temporary and transient, where the Attribute Calculator is defined as specific as possible, to minimize the required processing.