Evaluación proactiva del mantenimiento planificado

Use this procedure to update core-site.xml with the properties needed to integrate ECS HDFS with a Hadoop cluster that uses simple authentication mode.

Before you begin

You must have a set of user credentials that enable you to log in to Hadoop nodes and modify core-site.xml.

Configure HDFS in a non-secure Hadoop cluster

The location of core-site.xml depends on the distribution you are using. Table 32 core-site.xml locations

Hadoop

Distribution core-site.xml location Nodes toupdate

Pivotal HD /etc/ghpd/hadoop/conf ComputeMaster

and clients

Cloudera /etc/hadoop/conf All client nodes

Hortonworks /etc/hadoop/conf All nodes

core-site.xml resides on each node in the Hadoop cluster. You must modify the same properties in each instance. You can make the change in one node, and then use secure copy command (scp) to copy the file to the other nodes in the cluster.

See core_site.xml property reference for more information about each property you need to set.

Procedure

1. Log in to one of the HDFS nodes where core-site.xml is located. 2. Make a backup copy of core-site.xml.

cp core-site.xml core-site.backup

3. Using the text editor of your choice, open core-site.xml for editing.

Note

With Cloudera distributions it is better to use Cloudera Safety Valve, and with Hortonworks it is better to use Hortonworks Ambari, to make these changes so that they are persistent across the cluster.

4. Add the following properties and values to define the Java classes that implement the ECS HDFS file system:

<property> <name>fs.viprfs.impl</name> <value>com.emc.hadoop.fs.vipr.ViPRFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.viprfs.impl</name> <value>com.emc.hadoop.fs.vipr.ViPRAbstractFileSystem</value> </property>

5. Add the fs.vipr.installations property. In the following example, the value is set to Site1.

<property>

<name>fs.vipr.installations</name> <value>Site1</value>

</property>

6. Add the fs.vipr.installation.[installation_name].hosts property as a comma-separated list of ECS data nodes or load balancer IP addresses. In the following example, the installation_name is set to Site1.

Note

The use of a load balancer adds no value to a HDFS scenario as the client has the logic to connect to the nodes directly. For this reason, it is recommended that you provide a list of data nodes in this property, not the address of a load balancer. In addition, you should set the fs.vipr.installation.[installation_name].resolution to "dynamic".

<property>

<name>fs.vipr.installation.Site1.hosts</name>

<value>203.0.113.10,203.0.113.11,203.0.113.12</value> </property>

7. Add the fs.vipr.installation.[installation_name].resolution property, and set it to one of the following values:

Option Description

dynamic Use when accessing ECS data nodes directly without a load balancer. fixed Use when accessing ECS data nodes through a load balancer. In the following example, installation_name is set to Site1.

<property>

<name>fs.vipr.installation.Site1.resolution</name> <value>dynamic</value>

</property>

a. If you set fs.vipr.installation.[installation_name].resolution to dynamic, add the fs.vipr.installation.[installation_name].resolution.dynamic.time_to_live_ms property to specify how often to query ECS for the list of active nodes. In the following example, installation_name is set to Site1.

<property>

<name>fs.vipr.installation.Site1.resolution.dynamic.time_to_live _ms</name>

<value>900000</value> </property>

8. Locate the fs.defaultFS property and modify the value to specify the ECS file system URI. .

This setting is optional and you can specify the full file system URL to connect to the ECS ViPRFS.

Use the following format: viprfs://

<bucket_name.namespace.installation_name, where

l bucket_name: The name of the bucket that contains the data you want to use when

you run Hadoop jobs. If running in simple authentication mode, the owner of the bucket must grant permission to Everybody. In the following example, the bucket_name is set to mybucket.

l namespace: The tenant namespace where bucket_name resides. In the following

example, the namespace is set to mynamespace.

l installation_name: The value specified by the fs.vipr.installations property. In the

following example, installation_name is set to Site1. <property>

<name>fs.defaultFS</name>

Configure HDFS in a non-secure Hadoop cluster

<value>viprfs://mybucket.mynamespace.Site1/</value> </property>

9. Locate fs.permissions.umask-mode, and set the value to 022.

In some configurations, this property might not already exist. If it does not, then add it.

<property>

<name>fs.permissions.umask-mode</name> <value>022</value>

</property>

10.Add the fs.viprfs.auth.anonymous_translation property; use it to specify whether to map anonymously owned objects to the current user so the current user has permission to modify it.

Option Description

NONE (default) Do not map anonymously owned objects to the current user. CURRENT_USER Map anonymously owned objects to the current Unix user. <property>

<name>fs.viprfs.auth.anonymous_translation</name> <value>CURRENT_USER</value>

</property>

11.Add the fs.viprfs.auth.identity_translation property. It provides a way to assign users to a realm when Kerberos is not present.

Option Description

FIXED_REALM When specified, ECS HDFS gets the realm name from the value of the fs.vipr.auth.realm property.

NONE (default) ECS HDFS does no realm translation. <property>

<name>fs.viprfs.auth.identity_translation</name> <value>NONE</value>

</property>

12.If you set the fs.viprfs.auth.identity_translation property to FIXED_REALM, add the fs.viprfs.auth.realm property.

13.If you want to use the Pivotal HAWQ service, add the hawq.vipr.endpoint property. Specify the value using the following format:

bucket_name.namespace.installation_name. Where:

l bucket_name:The name of the bucket that contains the data you want to use when

you run Hadoop jobs. If running in simple authentication mode, the owner of the bucket must grant permission to Everybody. In the following example,

bucket_name is set to mybucket.

l namespace: The tenant namespace where bucket_name resides. In the following

example, the namespace is set to mynamespace.

l installation_name: The value specified by the fs.vipr.installations property. In the

You must be running a version of ECS that supports Pivotal HAWQ. For more information, see the Support Matrix.

<property>

<name>hawq.vipr.endpoint</name>

<value>mybucket.mynamespace.Site1</value> </property>

14.Save core-site.xml.

15.Update the core-site.xml on the required nodes in your Hadoop cluster. 16.If you are using a Cloudera distribution, use Cloudera Manager to update the core-

site.xml safety valve with the same set of properties and values. 17.Restart the Hadoop services.

Hadoop

Distribution Commands Pivotal HD ComputeMaster:

# service hadoop-yarn-resourcemanager restart

Data Nodes:

# service hadoop-hdfs-datanode restart # service hadoop-yarn-nodemanager restart

NameNode:

# service hadoop-yarn-nodemanager restart

When you configure the Pivotal Hadoop cluster to use ECS HDFS as the default file system (specified by fs.DefaultFS in core-

site.xml), you cannot use the icm_client's cluster start/stop functionality, instead, you must start all cluster services (except HDFS) individually. For example:

icm_client start -s yarn icm_client start -s zookeeper

and so on.

Cloudera Use Cloudera Manager to restart the HDFS and MapReduce services Apache # stop-all.sh

# start-all.sh

18.Test the configuration by running the following command to get a directory listing:

# hdfs dfs -ls viprfs://mybucket.mynamespace.Site1/

13/12/13 22:20:37 INFO vipr.ViPRFileSystem: Initialized ViPRFS for viprfs://mybucket.mynamespace.Site1/

If you have set fs.defaultFS, you can use:

# hdfs dfs -ls /

Configure HDFS in a non-secure Hadoop cluster

Edit HBASE hbase-site.xml

When you use HBASE with ECS HDFS, you must set the hbase.rootdir in hbase- site.xml to the same value as the core-site.xml fs.defaultFS property.

hbase-site.xml is located in one of the following locations: Table 33 hbase-site.xml locations

Hadoop Distribution hbase-site.xml location

Pivotal HD /etc/ghpd/hbase/conf/

Cloudera /etc/hbase/conf/

Hortonworks /etc/hbase/conf/

Procedure

1. Open hbase-site.xml.

2. Set the hbase.rootdir property to the same value as fs.defaultFS adding /hbase as the suffix.

3. Save your changes.

a. On Cloudera, add the hbase.rootdir property to the HBase Service Configuration Safety Valve for hbase-site.xml.

4. Restart the services for your distribution. HadoopDistribution Description

Pivotal HD Run this command on the hbase master node:

# service hbase-master restart

Run this command on the hbase region server:

# service hadoop-regionserver restart

Cloudera Use Cloudera manager to restart the HBase service. Hortonworks

# bin/start-hbase.sh

Example 1 hbase.rootdir entry

<property>

<name>hbase.rootdir</name>

<value>viprfs://testbucket.s3.testsite/hbase</value> </property>

CHAPTER 23