Hadoop

Hadoop

Enhanced insights for Hadoop HDFS and Yarn services

Extension

Product information
Release notes

Overview

Hadoop monitoring in Dynatrace provides a high-level overview of the main Hadoop components within your cluster. These enhanced insights provides additional metrics directly from the HDFS and YARN services.

Use cases

The extension enables insights into the overall health of Hadoop HDFS and YARN services

Get started

Activate this extension in your Dynatrace environment from the in-product Hub and simply select which OneAgents to enable this on.

Details

This extension provides extra Hadoop metrics through the use of JMX queries.

HDFS - Hadoop Distributed File System
- Improved visibility into the health of your HDFS NameNodes and DataNodes
YARN - Yet Another Resource Negotiator
- Improved visibility into the health of your YARN NodeManagers, ResourceManagers and MRAppMaster services

Compatibility information

Hadoop version 2.4.1+
Linux OS
For full Hadoop visibility, OneAgent must be installed on all machines running the following Hadoop processes: NameNode, ResourceManager, NodeManager, DataNode, and MRAppMaster

Subscribe to new releases

Copy to clipboard

Extension content

Content typeNumber of items included

screen injections

1

generic relationship

1

dashboards

1

metric metadata

91

generic type

1

screen entities lists

1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included

Metric name	Metric key	Description	Unit
FilesAppended	hadoop.hdfs.namenode.FilesAppended.count	Total number of files appended	Count
FilesCreated	hadoop.hdfs.namenode.FilesCreated.count	Total number of files and directories created by create or mkdir operations	Count
FilesDeleted	hadoop.hdfs.namenode.FilesDeleted.count	Total number of files and directories deleted by delete or rename operations	Count
FilesRenamed	hadoop.hdfs.namenode.FilesRenamed.count	Total number of rename operations (NOT number of files/dirs renamed)	Count

Metric name	Metric key	Description	Unit
NodeManager AllocatedContainers	hadoop.yarn.nodemanager.AllocatedContainers	NodeManager Current number of allocated containers	Count
AllocatedGB	hadoop.yarn.nodemanager.AllocatedGB	Current allocated memory in GB	GigaByte
AvailableGB	hadoop.yarn.nodemanager.AvailableGB	Current available memory in GB	GigaByte
ContainersCompleted	hadoop.yarn.nodemanager.ContainersCompleted.count	Total number of successfully completed containers	Count
ContainersFailed	hadoop.yarn.nodemanager.ContainersFailed.count	Total number of failed containers	Count
ContainersIniting	hadoop.yarn.nodemanager.ContainersIniting.count	Current number of initializing containers	Count
ContainersKilled	hadoop.yarn.nodemanager.ContainersKilled.count	Total number of killed containers	Count
ContainersLaunched	hadoop.yarn.nodemanager.ContainersLaunched.count	Total number of launched containers	Count
ContainersRunning	hadoop.yarn.nodemanager.ContainersRunning.count	Current number of running containers	Count

Metric name	Metric key	Description	Unit
DataNode CacheCapacity	hadoop.hdfs.datanode.CacheCapacity	The cache capacity of the DataNode	Byte
DataNode CacheUsed	hadoop.hdfs.datanode.CacheUsed	The cache used the DataNode	Byte
DataNode Capacity	hadoop.hdfs.datanode.Capacity	Current raw capacity of the DataNodes in bytes	Byte
DataNode DfsUsed	hadoop.hdfs.datanode.DfsUsed	The storage space that has been used up by HDFS.	Byte
DataNode NumBlocksCached	hadoop.hdfs.datanode.NumBlocksCached	The number of blocks cached on the DataNode	Count
DataNode NumBlocksFailedToCache	hadoop.hdfs.datanode.NumBlocksFailedToCache	The number of blocks that failed to cache on the DataNode	Count
DataNode NumBlocksFailedToUncache	hadoop.hdfs.datanode.NumBlocksFailedToUncache	The number of failed blocks to remove from cache.	Count
DataNode NumFailedVolumes	hadoop.hdfs.datanode.NumFailedVolumes	Number of failed volumes.	Count
DataNode Remaining	hadoop.hdfs.datanode.Remaining	The remaining DataNode disk space left in Percent	Percent

Metric name	Metric key	Description	Unit
DataNode BlocksRead	hadoop.hdfs.datanode.BlocksRead.count	Total number of blocks read from DataNode	Count
DataNode BlocksRemoved	hadoop.hdfs.datanode.BlocksRemoved.count	Total number of blocks removed from DataNode	Count
DataNode BlocksReplicated	hadoop.hdfs.datanode.BlocksReplicated.count	Total number of blocks replicated	Count
DataNode BlocksVerified	hadoop.hdfs.datanode.BlocksVerified.count	Total number of blocks verified	Count
DataNode BlocksWritten	hadoop.hdfs.datanode.BlocksWritten.count	Total number of blocks written to DataNode	Count
DataNode BytesRead	hadoop.hdfs.datanode.BytesRead.count	Total number of bytes read from DataNode	Byte
DataNode BytesWritten	hadoop.hdfs.datanode.BytesWritten.count	Total number of bytes written to DataNode	Byte

Metric name	Metric key	Description	Unit
ShuffleConnections	hadoop.yarn.nodemanager.ShuffleConnections.count	Number of Shuffle connections	Count
ShuffleOutputBytes	hadoop.yarn.nodemanager.ShuffleOutputBytes.count	Total shuffle output bytes	Byte
ShuffleOutputsFailed	hadoop.yarn.nodemanager.ShuffleOutputsFailed.count	Number of failed shuffle outputs	Count
ShuffleOutputsOK	hadoop.yarn.nodemanager.ShuffleOutputsOK.count	Number of OK shuffle outputs	Count

Metric name	Metric key	Description	Unit
CapacityTotal	hadoop.hdfs.namenode.CapacityTotal	Current raw capacity of DataNodes in bytes	Byte
CapacityUsed	hadoop.hdfs.namenode.CapacityUsed	Current used capacity across all DataNodes in bytes	Byte
CapacityRemaining	hadoop.hdfs.namenode.CapacityRemaining	Current remaining capacity in bytes	Byte
TotalLoad	hadoop.hdfs.namenode.TotalLoad	Current number of connections	Count
BlocksTotal	hadoop.hdfs.namenode.BlocksTotal	Current number of allocated blocks in the system	Count
PendingDeletionBlocks	hadoop.hdfs.namenode.PendingDeletionBlocks	Current number of blocks pending deletion	Count
CorruptBlocks	hadoop.hdfs.namenode.CorruptBlocks	Current number of blocks with corrupt replicas.	Count
CapacityUsedNonDFS	hadoop.hdfs.namenode.CapacityUsedNonDFS	Current space used by DataNodes for non DFS purposes in bytes	Count

Metric name	Metric key	Description	Unit
JobsCompleted	hadoop.yarn.mrappmaster.JobsCompleted	Number of completed jobs	Count
JobsFailed	hadoop.yarn.mrappmaster.JobsFailed	Number of failed jobs	Count
JobsKilled	hadoop.yarn.mrappmaster.JobsKilled	Number of killed jobs	Count
JobsPreparing	hadoop.yarn.mrappmaster.JobsPreparing	Number of preparing jobs	Count
JobsRunning	hadoop.yarn.mrappmaster.JobsRunning	Number of running jobs	Count
MapsCompleted	hadoop.yarn.mrappmaster.MapsCompleted	Number of maps completed	Count
MapsFailed	hadoop.yarn.mrappmaster.MapsFailed	Number of maps failed	Count
MapsKilled	hadoop.yarn.mrappmaster.MapsKilled	Number of maps killed	Count
MapsRunning	hadoop.yarn.mrappmaster.MapsRunning	Number of maps running	Count
MapsWaiting	hadoop.yarn.mrappmaster.MapsWaiting	Number of maps waiting	Count
ReducesCompleted	hadoop.yarn.mrappmaster.ReducesCompleted	Number of completed reduces	Count
ReducesFailed	hadoop.yarn.mrappmaster.ReducesFailed	Number of failed reduces	Count
ReducesKilled	hadoop.yarn.mrappmaster.ReducesKilled	Number of killed reduces	Count
ReducesRunning	hadoop.yarn.mrappmaster.ReducesRunning	Number of running reduces	Count
ReducesWaiting	hadoop.yarn.mrappmaster.ReducesWaiting	Number of waiting reduces	Count

Metric name	Metric key	Description	Unit
FilesTotal	hadoop.hdfs.namenode.FilesTotal	Current number of files and directories	Count
PendingReplicationBlocks	hadoop.hdfs.namenode.PendingReplicationBlocks	Current number of blocks pending to be replicated	Count
UnderReplicatedBlocks	hadoop.hdfs.namenode.UnderReplicatedBlocks	Current number of blocks under replicated	Count
ScheduledReplicationBlocks	hadoop.hdfs.namenode.ScheduledReplicationBlocks	Current number of blocks scheduled for replications	Count
NumLiveDataNodes	hadoop.hdfs.namenode.NumLiveDataNodes	Number of datanodes which are currently live	Count
NumDeadDataNodes	hadoop.hdfs.namenode.NumDeadDataNodes	Number of datanodes which are currently dead	Count
NumDecomLiveDataNodes	hadoop.hdfs.namenode.NumDecomLiveDataNodes	Number of datanodes which have been decommissioned and are now live	Count
NumDecomDeadDataNodes	hadoop.hdfs.namenode.NumDecomDeadDataNodes	Number of datanodes which have been decommissioned and are now dead	Count
VolumeFailuresTotal	hadoop.hdfs.namenode.VolumeFailuresTotal	Total number of volume failures across all Datanodes	Count
EstimatedCapacityLostTotal	hadoop.hdfs.namenode.EstimatedCapacityLostTotal	An estimate of the total capacity lost due to volume failures	Count
NumDecommissioningDataNodes	hadoop.hdfs.namenode.NumDecommissioningDataNodes	Number of datanodes in decommissioning state	Count
NumStaleDataNodes	hadoop.hdfs.namenode.NumStaleDataNodes	Number of datanodes marked stale due to delayed hearbeat.	Count

Metric name	Metric key	Description	Unit
NumActiveNMs	hadoop.yarn.resourcemanager.NumActiveNMs	Current number of active NodeManagers	Count
NumDecommissioningNMs	hadoop.yarn.resourcemanager.NumDecommissionedNMs	Current number of NodeManagers being decommissioned	Count
NumLostNMs	hadoop.yarn.resourcemanager.NumLostNMs	Current number of lost NodeManagers for not sending heartbeats	Count
NumRebootedNMs	hadoop.yarn.resourcemanager.NumRebootedNMs	Current number of rebooted NodeManagers	Count
NumUnhealthyNMs	hadoop.yarn.resourcemanager.NumUnhealthyNMs	Current number of unhealthy NodeManagers	Count

Metric name	Metric key	Description	Unit
NumberOfMissingBlocks	hadoop.hdfs.namenode.NumberOfMissingBlocks	-	Count
CacheCapacity	hadoop.hdfs.namenode.CacheCapacity	The total cache capacity of all DataNodes	Byte
CacheUsed	hadoop.hdfs.namenode.CacheUsed	The total cache used by all DataNodes	Byte

Metric name	Metric key	Description	Unit
AllocatedContainers	hadoop.yarn.resourcemanager.AllocatedContainers	Queue Current number of allocated containers	Count
AllocatedMB	hadoop.yarn.resourcemanager.AllocatedMB	Current allocated memory in MB	MegaByte
AllocatedVCores	hadoop.yarn.resourcemanager.AllocatedVCores	Current allocated CPU in virtual cores	Count
AppsCompleted	hadoop.yarn.resourcemanager.AppsCompleted.count	Total number of completed applications	Count
AppsFailed	hadoop.yarn.resourcemanager.AppsFailed.count	Total number of failed applications	Count
AppsKilled	hadoop.yarn.resourcemanager.AppsKilled.count	Total number of killed applications	Count
AppsPending	hadoop.yarn.resourcemanager.AppsPending.count	Current number of applications that have not yet been assigned by any containers	Count
AppsRunning	hadoop.yarn.resourcemanager.AppsRunning.count	Current number of running applications	Count
AppsSubmitted	hadoop.yarn.resourcemanager.AppsSubmitted.count	Total number of submitted applications	Count
AvailableMB	hadoop.yarn.resourcemanager.AvailableMB	Current available memory in MB	Count
AvailableVCores	hadoop.yarn.resourcemanager.AvailableVCores	Current available CPU in virtual cores	Count
PendingMB	hadoop.yarn.resourcemanager.PendingMB	Current memory requests in MB that are pending to be fulfilled by the scheduler	MegaByte
PendingVCores	hadoop.yarn.resourcemanager.PendingVCores	Current CPU requests in virtual cores that are pending to be fulfilled by the scheduler	Count
ReservedMB	hadoop.yarn.resourcemanager.ReservedMB	Current reserved memory in MB	MegaByte
ReservedVCores	hadoop.yarn.resourcemanager.ReservedVCores	Current reserved CPU in virtual cores	Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.

ReleaseDate

Full version history

No release notes

Discover recent additions to Dynatrace

Problems

Analyze abnormal system behavior and performance problems detected by Davis AI.

Logs

Explore all your logs without writing a single query.

Security Investigator

Fast and precise forensics for security and logs on Grail data with DQL queries.

Business Flow

Track, analyze, and optimize your critical business processes.

Cost & Carbon Optimization

Track, analyze, and optimize your IT carbon footprint and public cloud costs.

Davis Anomaly Detection

Detect anomalies in timeseries using the Davis AI

Analyze your data

Understand your data better with deep insights and clear visualizations.

Notebooks

Create powerful, data-driven documents for custom analytics and collaboration.

Dashboards

Transform complex data into clear visualizations with custom dashboards.

Automate your processes

Turn data and answers into actions, securely, and at scale.

Workflows

Automate tasks in your IT landscape, remediate problems, and visualize processes

Jira

Create, query, comment, transition, and resolve Jira tickets within workflows.

Slack

Automate Slack messaging for security incidents, attacks, remediation, and more.

Secure your cloud application

See vulnerabilities and attacks in your environment.

Security Overview

Get a comprehensive overview of the security of your applications.

Code-Level Vulnerabilities

Detect vulnerabilities in your code in real time.

Threats & Exploits

Understand, triage, and investigate application security findings and alerts.

Are you looking for something different?

We have hundreds of apps, extensions, and other technologies to customize your environment

Leverage our newest innovations of Dynatrace Saas

Kick-start your app creation

Kick-start your app creation

Whether you’re a beginner or a pro, Dynatrace Developer has the tools and support you need to create incredible apps with minimal effort.

Go to Dynatrace Developer

Upgrading from Dynatrace Managed to SaaS

Upgrading from Dynatrace Managed to SaaS

Drive innovation, speed, and agility in your organization by seamlessly and securely upgrading.

Log Management and Analytics

Log Management and Analytics

Innovate faster and more efficiently with unified log management and log analytics for actionable insights and automation.