Análisis estadístico de los datos obtenidos

As the vast majority of analytic frameworks are built on the Java Virtual Machine,

Cruciblemust target the JVM in the first instance; support for other languages

and interfaces is secondary. By targeting the JVM,Crucibleadditionally gains

the use of the vast library of existing open-source Java code. It would be possible

this chapter, however in our experience this results in extremely verbose code:

it is the goal of Crucibleto move the expression of an analytic to be as close

as possible to the user’s intended analysis, with a minimum of “scaffolding”. It

is therefore important that the design of the DSL facilitates the integration of both JVM primitives and other Java libraries.

Instead of designing the language semantics for a novel language from scratch,

Crucible’s DSL is built on the XText [36] language framework. Through XText’s use of XBase, an embeddable version of the XTend Java Virtual Machine

(JVM) language, Crucible avoids the need to implement a new parser and

design a Turing-complete language implementation. Crucially, the XBase syntax is not dissimilar to Java (with some higher level primitives and syntactic sugar).

Crucible’s extension to XBase provides a syntactic framework for modelling Processing Elements (PEs), while the syntax and semantics of standard XBase code are reused for each PE’s processing logic.

At a high level, aCrucibleanalytic (such as in Listing 4.1, a topology of

three linearly connected PEs) is structured similarly to a Java code file; it consists

of a package declaration for code organisation (line 1), a set of Java/Crucible

imports (lines 3-4), and then one or moreprocess declarations, each describing

a PE (lines 6, 14, 29). In theCrucibleDSL, each PE is modelled by a Java

class, with a name and an optional superclass. The body of a PE is divided into a set of unordered blocks:

• config– Compile-time configuration constants. The initialisation of these

may involve an arbitrary expression. These are transpiled toconst fields

in the Java class. (Lines 7, 15, 33);

• state– Runtime mutable state; shared globally between instances of this

PE. These variables may be declaredlocal, in which case their values are

stored only locally on instances of the PE. Section 4.1.4 discusses the use

of global state inCrucible. These are transpiled as instance variables on

4. Unified Secure On- and Off-Line Analytics 1 package eg . c o u n t e r 2 3 import c r u c i b l e . l i b . pe . F i l e S o u r c e 4 import c r u c i b l e . l i b . pe . F i l e S i n k 5

6 process Source extends F i l e S o u r c e {

7 config : {

8 Filename = ' / u s r / s h a r e / d i c t / words '

9 ReadLines = f a l s e // Read chars , not l i n e s

10 }

11 outputs : [ F i l e L i n e , F i l e C h a r a c t e r ] 12 }

13

14 process F i l t e r {

15 config : i n t N = 1500000 // For TopN c a l c u l a t i o n 16 state : i n t s e en = 0

17 output : Keys

18 input : Source . F i l e C h a r a c t e r > { 19 i f ( ( s e e n ) >= N) {

20 Keys . emit (' done ' > true, ' key ' > Character : : MIN_VALUE) 21 } e l s e i f ( Character : : i s L e t t e r ( c h a r a c t e r ) ) {

22 s e en = s e e n + 1

23 Keys . emit (' key ' > Character : : toUpperCase ( c h a r a c t e r ) ,

24 ' done ' > false, ' t o t a l ' > s e e n )

25 }

26 }

27 } 28

29 process CountingWriter extends F i l e S i n k { 30 output : R e s u l t s

31 state : counts = ('A '. charAt ( 0 ) . . 'Z '. charAt ( 0 ) ) 32 . toInvertedMap [ new AtomicInteger ] 33 config : Filename = ' counts . t x t '

34 input : F i l t e r . Keys > { 35 i f ( done ) {

36 l o g . i n f o ( counts . t o S t r i n g )

37 R e s u l t s . emit (' t o t a l ' > t o t a l , ' counts ' > counts as Map,

38 ' tstamp ' > System : : c u r r e n t T i m e M i l l i s )

39 }

40 counts . g e t ( key . charValue as i n t ) ? . incrementAndGet

41 }

42 input : CountingWriter . R e s u l t s > super 43 }

Listing 4.1: An ExampleCrucibleTopology fragment, counting the frequency

of characters in the input.

• output(s)– Declaration of the named output ports from the process. Each output is represented in the transpiled code as an instance variable

of theCrucible libraryOutput. (Lines 11, 17);

• input– A block which maps the qualified name of an output (in the form ProcessName.OutputName) to a block of code to execute upon arrival of a tuple from that port. The keys inferred to be present on the input tuple are present as variables in this code block. In the transpiled Java class,

each input is generated as areceivemethod, based on the qualified name

Figure 4.1: Components of theCrucibleSystem. Entries initalics are external dependencies.

Crucibletranspiles a topology described in the DSL into idiomatic Java

based on the CruciblePE Model (the bottom layer of Figure 4.1). This is in

contrast to many other JVM languages, such as Scala [79], which directly compile into unreadable bytecode. Compiler support is used to provide syntactic sugar for accessing global shared state and the security labelling mechanism, which are discussed in more detail in Section 4.1.3. The Code Generation component noted in Figure 4.1 is responsible for this transpilation process. It is built on

the XText Java Model Inferrer, which uses the syntax description (as listed in

Appendix A) to generate a parser and abstract syntax tree (AST) generator.

This AST is supplied to code implemented inCrucible for gathering tuple

types through XBase’s type inference, and generating Java classes in accordance with the description above. The result of this process is a series of Java classes

which inherit fromPEDefinitionand interact with the internalCrucibleJava

API, shown in Figure 4.2. These classes turn the variousCruciblekeywords

described above into class fields and methods – effectively generating for the

user the verbose Java “scaffolding” which Crucibleavoids. For example, the

87-line sample Cruciblefile in Listing 4.1 is transpiled to 560 lines of Java

across four separate classes; to give a sense of the complexity of these classes, the

class representing theFilterPE consists of 10 fields and 14publicmethods.