Table Of ContentHPE Reference Architecture for
Cloudera Enterprise 5.7 with HPE
Apollo 2000 and HPE Apollo 4200 Gen9
servers
Reference Architecture
Reference Architecture
Contents
Executive summary ................................................................................................................................................................................................................................................................................................................................ 3
Introduction ................................................................................................................................................................................................................................................................................................................................................... 3
Solution overview ..................................................................................................................................................................................................................................................................................................................................... 4
Storage nodes ....................................................................................................................................................................................................................................................................................................................................... 5
Compute nodes ................................................................................................................................................................................................................................................................................................................................... 5
Hadoop YARN ...................................................................................................................................................................................................................................................................................................................................... 5
Cloudera Enterprise overview ...................................................................................................................................................................................................................................................................................................... 5
Benefits of the HPE BDRA solution ........................................................................................................................................................................................................................................................................................ 6
Solution components ............................................................................................................................................................................................................................................................................................................................ 7
Edge node ................................................................................................................................................................................................................................................................................................................................................ 8
Management and head nodes ............................................................................................................................................................................................................................................................................................... 8
Compute nodes ................................................................................................................................................................................................................................................................................................................................... 9
Storage nodes ................................................................................................................................................................................................................................................................................................................................... 10
Power and cooling ........................................................................................................................................................................................................................................................................................................................ 10
Networking .......................................................................................................................................................................................................................................................................................................................................... 11
Capacity and sizing ............................................................................................................................................................................................................................................................................................................................ 12
Expanding the base configuration ................................................................................................................................................................................................................................................................................. 12
Best practices and configuration guidance for the solution ......................................................................................................................................................................................................................... 14
Setting up Cloudera Enterprise ........................................................................................................................................................................................................................................................................................ 14
Configuring compression ....................................................................................................................................................................................................................................................................................................... 18
Cloudera Impala..................................................................................................................................................................................................................................................................................................................................... 18
Installation ............................................................................................................................................................................................................................................................................................................................................ 18
Configuration ..................................................................................................................................................................................................................................................................................................................................... 18
YARN and Impala .......................................................................................................................................................................................................................................................................................................................... 19
Performance and configuration guidelines ............................................................................................................................................................................................................................................................ 19
Implementing a proof-of-concept ......................................................................................................................................................................................................................................................................................... 21
Summary ...................................................................................................................................................................................................................................................................................................................................................... 22
Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 23
Appendix B: Alternate compute node components ............................................................................................................................................................................................................................................. 29
Appendix C: Alternate storage node components ................................................................................................................................................................................................................................................ 30
Appendix D: HPE value added services and support ......................................................................................................................................................................................................................................... 31
Resources and additional links ................................................................................................................................................................................................................................................................................................ 33
Reference Architecture Page 3
Executive summary
This white paper describes a big data solution deploying Cloudera Enterprise, including Cloudera Impala, and using Hewlett Packard Enterprise
Big Data Reference Architecture (BDRA), HPE Apollo 2000 compute servers, and HPE Apollo 4200 storage servers as the key components of
the solution. This solution provides recognizable benefits of using Cloudera Enterprise on the HPE BDRA. In addition to simplifying the
procurement process, this paper also provides guidelines for configuring Cloudera Enterprise once the system has been deployed.
There is an ever-growing need for a scalable, modern architecture for the consolidation, storage, access, and processing of big data analytics. Big
data solutions using Hadoop are evolving from a simple model where each application was deployed on a dedicated cluster of identical nodes
into a complex model where applications are deployed on a cluster of asymmetric nodes. Analytics and processing engines themselves have
grown from MapReduce to a broader set now including Spark and SQL-based interfaces like Impala. By integrating the significant advances that
have occurred in fabrics, storage, container-based resource management, and workload-optimized servers since the inception of the Hadoop
architecture in 2005, HPE BDRA provides a cost-effective and flexible solution to optimize computing infrastructure in response to these ever-
changing requirements in the Hadoop ecosystem. This implementation focuses on Cloudera Enterprise along with Cloudera Impala. These
technologies, and guidelines for their configuration and usage, are described in this document.
Target audience: This paper is intended for decision makers, system and solution architects, system administrators and experienced users that
are interested in reducing design time or simplifying the purchase of a big data architecture containing both HPE and Cloudera components. An
intermediate knowledge of Apache Hadoop and scale-out infrastructure is recommended.
Document purpose: The purpose of this document is to describe the optimal way to configure Cloudera Enterprise on HPE BDRA along with
Cloudera Impala. This document takes the precedence over the previous version titled “HPE Big Data Reference Architecture for Cloudera
Enterprise: Using HPE Apollo 2000 and Apollo 4200 Gen9 servers”.
This reference architecture describes solution testing performed with Cloudera Enterprise 5.7 in April 2016.
Disclaimer: Products sold prior to the separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. on
November 1, 2015 may have a product name and model number that differ from current models
Introduction
As companies grow their big data implementations, they often find themselves deploying multiple clusters to support their needs. This could be
to support different big data environments (MapReduce, Spark, NoSQL DBs, MPP DBMSs, etc.), to support rigid workload partitioning for
departmental requirements or simply as a byproduct of multi-generational hardware. These multiple clusters often lead to data duplication and
movement of large amounts of data between systems to accomplish an organization’s business requirements. Many enterprise customers are
searching for ways to recapture some of the traditional benefits of a converged infrastructure, such as the ability to more easily share data
between different applications running on different platforms, the ability to scale compute and storage separately and the ability to rapidly
provision new servers without repartitioning data to achieve optimal performance.
To address these needs, HPE engineers challenged the traditional Hadoop architecture which always co-locates compute elements in a server
with data. While this approach works, the real power of Hadoop is that tasks run against specific slices of data without the need for coordination.
Consequently, the distributed lock management and distributed cache management found in older parallel database designs is no longer
required.
In fact, in a modern server, there is often more network bandwidth available to ship data off the server than there is bandwidth to disk. The HPE
BDRA for Cloudera Enterprise is deployed as an asymmetric cluster with some nodes dedicated to compute and others dedicated to Hadoop
Distributed File System (HDFS). Hadoop still works the same way – tasks still have complete ownership of their portion of the data and functions
are still being shipped to the data – but the computation is executed on a node optimized for this task and the file system operations are
executed on a node that is optimized for that work. Of particular interest is that this approach can actually perform better than a traditional
Hadoop cluster for several reasons. For more information on this architecture1 and the benefits that it provides, see the document at
https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-8931ENW
1 HPE BDRA, also known as, HPE Elastic Platform for Big Data Analytics (EPA)
Reference Architecture Page 4
Solution overview
HPE Big Data solutions provide world class performance and availability, with integrated software, services, infrastructure, and management – all
delivered as one tested configuration, described in more detail at hpe.com/info/hadoop. As shown below in Figure 1, the HPE BDRA solution
consists of a highly optimized configuration built using the unique servers offered by HPE – the HPE Apollo 4200 Gen9 for the high density
storage layer and the HPE Apollo 2000 system with HPE ProLiant XL170r nodes for the high density computational layer. The servers and
components described below offer a superb breed of Enterprise asymmetric Hadoop architecture designed and optimized for a variety of
workloads.
• HPE Apollo 4200 Gen9 server offers revolutionary storage density in a 2U form factor. When compared to previous generations, it provides
more than double the storage capacity than previous generations along with an unprecedented selection of processors to match for data
intensive workloads. HPE Apollo 4200 Gen9 server allows companies to save valuable data center space through its unique density-optimized
2U form factor which holds up to 28 LFF or 54 SFF hot plug drives.
• The HPE Apollo 2000 system offers a dense solution with up to four independent, hot-pluggable HPE ProLiant XL170r Gen9 server nodes in
a standard 2U chassis. It delivers twice the compute density as 1U rack-mount servers with front hot-pluggable drives and rear serviceable
nodes yielding very cost effective configurations for various workloads.
• HPE ProLiant DL360 servers include two sockets with eight-core processors and use Intel® Xeon® E5-2600 v3 product family to provide the
high performance required for the management services for Hadoop cluster.
The HPE iLO management engine on the servers contains HPE Integrated Lights-Out 4 (iLO 4) and features a complete set of embedded
management features for HPE Power/Cooling, Agentless Management, Active Health System, and Intelligent Provisioning which reduces node
and cluster level administration costs for Hadoop.
The reference architecture configuration is the result of a great deal of testing and optimization done by HPE engineers resulting in the right set
of software, drivers, firmware and hardware to yield extremely high density and performance. As shown in Figure 1, this architecture is changing
the economics of work distribution in big data.
Figure 1. HPE BDRA – changing the economics of work distribution in big data
Reference Architecture Page 5
In order to simplify the build for customers, HPE provides the exact bill of materials in this document to allow a customer to purchase this
complete solution. HPE recommends that customers purchase the option in which HPE Technical Services Consulting will install the prebuilt
operating system images, verify all firmware and versions are correctly installed, and run a suite of tests that verify that the configuration is
performing optimally. Once this has been done, the customer can perform a standard Cloudera Enterprise installation using the recommended
guidelines in this document.
The HPE BDRA design is anchored by the following HPE technologies.
Storage nodes
HPE Apollo 4200 Gen9 servers make up the HDFS storage layer, providing a single repository for big data. The HPE Apollo 4200 Gen9 server
offers the right mix between storage and compute performance in a 2U form factor. The 28 LFF disks have been adequately powered by 20
physical cores for the exclusive compute needs of HDFS. The storage controllers in the HPE Apollo 4200 support HPE Secure Encryption, an
HPE Smart Array controller-based data encryption solution that provides encryption for data at rest.
Compute nodes
HPE Apollo 2000 system nodes deliver a scalable, high-density layer for compute tasks and provide a framework for workload-optimization with
four HPE ProLiant XL170r nodes on a single chassis.
High-speed networking separates compute nodes and storage nodes, creating an asymmetric architecture that allows each tier to be scaled
individually; there is no commitment to a particular CPU/storage ratio. Since big data is no longer collocated with compute resources, Hadoop
does not need to achieve node locality. However, rack locality works in exactly the same way as in a traditional converged infrastructure; that is,
as long as you scale within a rack, overall scalability is not affected.
With compute and storage de-coupled, users gain many of the advantages of a traditional converged system. For example, you can scale
compute and storage independently, simply by adding compute nodes or storage nodes. Testing carried out by HPE indicates that most
workloads respond almost linearly to additional compute resources.
Hadoop YARN
YARN is a key feature for resource management in Hadoop cluster on HPE BDRA. It decouples MapReduce’s resource management and
scheduling capabilities from the data processing components, allowing Hadoop to support more varied processing approaches and a broader
array of applications.
Cloudera Enterprise overview
Founded in 2008, Cloudera was the first company to commercialize Apache Hadoop and to develop enterprise-grade solutions built on this
powerful open source technology. Today, Cloudera is the leading innovator in and largest contributor to the Hadoop open source software
community. Cloudera employs a “hybrid open” subscription software business model, affording customers all the benefits of open source
software, plus the features and support expected from traditional enterprise software such as security, data governance and system
management.
Reference Architecture Page 6
As shown in Figure 2, Cloudera Enterprise is built on top of Cloudera’s Enterprise Data Hub (EDH) software platform. In this way, it empowers
organizations to store, process and analyze all enterprise data, of whatever type, in any volume – creating remarkable cost-efficiencies as well as
enabling business transformation. It is one place to store all your data for as long as desired in its original fidelity. With Apache Hadoop at its core,
it is a new, more powerful and scalable data platform with the flexibility to run a variety of workloads – batch processing, interactive SQL,
enterprise search, advanced analytics – together with the robust security, governance, data protection, and management that enterprises require.
For detailed information on Cloudera Enterprise, see cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise.html.
Figure 2. Cloudera Enterprise
Benefits of the HPE BDRA solution
While the most obvious benefits of the HPE BDRA solution center on density and price/performance, other benefits include:
• Elasticity – HPE BDRA is designed for flexibility. Compute nodes can be allocated very flexibly without redistributing data; for example, nodes
can be allocated by time-of-day or even for a single job. The organization is no longer committed to yesterday’s CPU/storage ratios, leading to
much more flexibility in design and cost. Moreover, with HPE BDRA, the system is only grown where needed.
• Consolidation – HPE BDRA is based on HDFS, which has enough performance and can scale to large enough capacities to be the single
source for big data within any organization. Various pools of data currently being used in big data projects can be consolidated into a single,
central repository. YARN-compliant workloads access the big data directly via HDFS; other workloads can access the same data via
appropriate connectors.
• Workload-optimization – There is no one go-to software for big data; instead, there is a federation of data management tools. After selecting
the appropriate tool to meet organizational requirements, run the job using the compute nodes that are best suited for the workload, such as a
processor with high core count, more memory, or both.
• Enhanced capacity management – Compute nodes can be provisioned on the fly, while storage nodes now constitute a smaller subset of the
cluster and, as such, are less costly to overprovision. In addition, managing a single data repository rather than multiple different clusters
reduces overall management costs.
Reference Architecture Page 7
• Faster time-to-solution – Processing big data typically requires the use of multiple data management tools. When these tools are deployed
on conventional Hadoop clusters with dedicated – often fragmented – copies of the data, time-to-solution can be lengthy. With HPE BDRA,
data is not fragmented but consolidated in a single data lake, allowing different tools to access the same data. Thus, more time can be spent
on analysis, less on shipping data; therefore, time-to-solution is typically faster.
Solution components
Figure 3 provides a basic conceptual diagram of HPE BDRA.
Figure 3. HPE BDRA concept
Reference Architecture Page 8
For full BOM listings of products selected for the proof-of-concept, refer to Appendix A: Bill of materials of this white paper.
Minimum configuration
Figure 4 shows a minimum HPE BDRA configuration, with 12 worker nodes and 3 storage nodes housed in a single 42U rack.
Figure 4. Base HPE BDRA configuration, with detailed lists of components
Best practice
HPE recommends starting with three HPE Apollo 2000 chassis consisting of 12 HPE ProLiant XL170r nodes with 128 GB of memory for each
node, combined with three HPE Apollo 4200 Gen9 servers as storage nodes. For performance sensitive Impala installations, consider increasing
memory to 256 GB in the compute nodes.
The following nodes are used in the base HPE BDRA configuration.
Edge node
The HPE BDRA solution includes a multi-homed network server called an edge node. It acts as a gateway node between the cluster’s private
VLAN and the external routable network. Any application that requires both external network access and cluster-private-network can run on this
server. When significant storage and network bandwidth are required, use more additional HPE ProLiant DL360 Gen9 servers as edge nodes.
Management and head nodes
Three HPE ProLiant DL360 Gen9 servers are configured as management/head nodes. Refer to Table 2 in the Node components section for the
suggested software components for these nodes.
Management and head nodes store important metadata information for the cluster and hence for greater data protection and performance a
RAID configuration is suggested for data storage. RAID 1 is recommended for the OS and NameNode metadata. RAID 10 is recommended for
Reference Architecture Page 9
Database data. As ZooKeeper and Quorum Journal Nodes are extremely latency sensitive, they should be configured with a single disk each.
They are best implemented with a single spindle, outside any RAID set (if RAID is a must for some reason, use RAID 0).
Compute nodes
The base HPE BDRA configuration features three HPE Apollo 2000 chassis containing a total of 12 HPE ProLiant XL170r Gen9 hot-pluggable
server nodes. The HPE ProLiant XL170r Gen9 is shown in Figure 5. The rear and front of the HPE Apollo 2000 chassis is shown in Figure 6.
Figure 5. HPE ProLiant XL170r Gen9
Figure 6. HPE Apollo 2000 chassis (rear and front)
The HPE Apollo 2000 system is a dense solution with four independent HPE ProLiant XL170r Gen9 hot-pluggable server nodes in a standard
2U chassis. Each HPE ProLiant XL170r Gen9 server node is serviced individually without impacting the operation of other nodes sharing the
same chassis to provide increased server uptime. Each server node harnesses the performance of 2133 MHz memory (16 DIMM slots per node)
and dual Intel Xeon E5-2600 v3 processors in a very efficient solution that shares both power and cooling infrastructure. Other features of the
HPE ProLiant XL170r Gen9 server include:
• Support for high-performance memory (DDR4) and Intel Xeon E5-2600 v3 processor up to 18C, 145W
• Additional PCIe riser options for flexible and balanced I/O configurations
• FlexibleLOM feature for additional network expansion options
• Support for dual M.2 drives
For more information on HPE Apollo 2000 chassis, visit
hpe.com/us/en/product-catalog/servers/proliant-servers/pip.hpe-apollo-r2000-chassis.7832023.html
For more information on the HPE ProLiant XL170r Gen9 server, visit
hpe.com/us/en/product-catalog/servers/proliant-servers/pip.hpe-proliant-xl170r-gen9-server.7799270.html
Each of these compute nodes typically runs YARN NodeManagers or Impala daemons.
Reference Architecture Page 10
Storage nodes
There are three HPE Apollo 4200 Gen9 servers. Each server is configured with 28 LFF disks; each typically runs HDFS DataNode. The HPE
Apollo 4200 Gen9 server is shown in Figure 7.
Figure 7. HPE Apollo 4200 Gen9 server
The HPE Apollo 4200 Gen9 server offers revolutionary storage density in a 2U form factor for data intensive workloads such as Apache Spark
on Hadoop HDFS. The HPE Apollo 4200 Gen9 server allows you to save valuable data center space through its unique density-optimized 2U
form factor which holds up to 28 LFF disks and with capacity for up to 224 TB per server. It has the ability to grow your big data solutions with
its density-optimized infrastructure that is ready to scale. Another benefit is that the HPE Apollo 4200 fits easily into standard racks with a depth
of 32-inches per server – no special racks are required.
For more detailed information, visit hpe.com/us/en/product-catalog/servers/proliant-servers/pip.hpe-apollo-4200-gen9-server.8261831.html
Key point
The storage controllers in the HPE Apollo 4200 and HPE ProLiant XL170r support HPE Secure Encryption. Secure Encryption is an HPE Smart
Array controller-based data encryption solution. It provides encryption for data at rest, an important component for complying with government
regulations having data privacy requirements, such as HIPAA and Sarbanes-Oxley. This optional feature enhances the functionality of the
storage controller in cases where encryption is a required feature of the solution.
Best practice
HPE recommends starting with a minimum three HPE Apollo 2000 chassis, with 12 compute nodes and three to five HPE Apollo 4200 Gen9
servers as storage nodes.
Note
Cloudera recommends a minimum of five storage nodes in production deployments for optimal utilization of nodes in the context of HDFS
replication and fault tolerance.
Power and cooling
When planning large clusters, it is important to properly manage power redundancy and distribution. To ensure servers and racks have adequate
power redundancy, HPE recommends that each chassis (HPE Apollo 2000) and each HPE Apollo 4200 Gen9 server should have a backup
power supply and each rack should have at least two Power Distribution Units (PDUs).
There is additional cost associated with procuring redundant power supplies; however, the need for redundancy is less critical in larger clusters
where the inherent redundancy within CDH ensures there would be less impact in the event of a failure.
Description:5. Cloudera Enterprise overview . 5. Benefits of the HPE BDRA solution . Best practices and configuration guidance for the solution . The HPE iLO management engine on the servers contains HPE Integrated the customer can perform a standard Cloudera Enterprise installation using the
