Dimensiones Funcionales

how often data needs to be ingested? How soon does data need to be available to downstream processing?

• Incremental updates: how will new data be added? Does it need to be appended to existing data? Or overwrite existing data?

• Data access and processing: will the data be used in processing? If so, will it be used in batch processing jobs? Or is random access to the data required?

• Source system and data structure: where is the data coming from? A relational database? Logs? Is it structured, semi-structured, un-structured data?

• Partitioning and splitting of data: how should data be partitioned after ingest? Does the data need to be ingested into multiple target systems — e.g. HDFS and HBase? • Storage format: what format will the data be stored in?

• Data transformation: does the data need to be transformed in flight?

We’ll start by talking about the first consideration, the timeliness requirements for the data being ingested.

Timeliness of data ingestion

When we talk about timeliness of data ingestion, we’re referring to the time lag from when data is available for ingestion to when it’s accessible to tools in the Hadoop eco‐ system. The time classification of an ingestion architecture will have a large impact on the storage medium and on the method of ingestion. For purposes of this discussion we’re not concerned with streaming processing or analytics, which we’ll discuss sepa‐ rately in Chapter five, but only with when the data is stored and available for processing in Hadoop.

In general, it’s recommended to use one of the following classifications before designing the ingestion architecture for an application:

• Macro Batch – This is normally anything over 15 minutes to hours, or even a daily job.

• Micro Batch – This is normally something that is fired off every 2 minutes or so, but no more than 15 minutes in total.

• Near Real Time Decision Support – This is considered to be “immediately action‐ able” by the recipient of the information, with data delivered in less than 2 minutes but greater than 2 seconds.

• Near Real-Time Event Processing – This is considered under 2 seconds, and can be as fast as 100 millisecond range.

• Real-Time – This term tends to be very over-used, and has many definitions. For purposes of this discussion it will be anything under 100 milliseconds.

An important thing to note with these time classifications is that as the implementation moves toward real-time, the complexity and cost of the implementation increases sub‐ stantially. Starting off at batch, e.g. using simple file transfers, is always a good idea — start simple before moving to more complex ingestion methods.

With more lenient timeliness requirements HDFS will likely be the preferred source as the primary storage location, and a simple file transfer or Sqoop jobs will often be suitable tools to ingest data. We’ll talk more about these options later, but these ingestion methods are well-suited for batch ingestion because of their simple implementations and the built-in validation checks that they provide out of the box. For example the

hadoop fs –put command will copy a file over and do a full checksum to confirm that

the data is copied over correctly.

One consideration when using the hdfs fs -put command or Sqoop is that the data

will land on HDFS in a format that might not be optimal for long-term storage and processing, so using these tools might require an additional batch process to get the data into the desired format. An example of where such an additional batch process would be required is loading Gzip files into HDFS. Although Gzip files can easily be stored in HDFS, and even processed with MapReduce and other processing frameworks on Ha‐ doop, as we discussed in the previous chapter Gzip files are not splittable. This will greatly impact the efficiency of processing these files, particularly as the files get larger. In this case a good solution is to store the files in Hadoop using a container format like SequenceFile or Avro that supports splittable compression.

As the requirements move from batch processing to more frequent updates, tools like Flume or Kafka need to be considered. Sqoop and file transfers are not going to be a good selection as the delivery requirements get faster than 2 minutes. Further, as the requirements get to less than 2 minutes the storage layer may need to change to HBase or Solr for more granular insertions and read operations. As the requirements get to the real-time category we need to think about memory first and permanent storage second. All of the parallelism in the world isn’t going to help for response requirements under 500 milliseconds as long as hard drives are in the process. At this point, we start to enter the realm of stream processing with tools like Storm or Spark Streaming. It should be emphasized that these tools are actually focused on data processing, as op‐ posed to data ingest like Flume or Sqoop. We’ll talk more about near real-time streaming with tools like Storm and Spark Streaming in chapter five. We’ll also discuss how HBase and Solr fit into the near real-time and real-time category in chapter five. Flume and Kafka will be discussed further in this chapter.