One of the reasons that this failure occurs is because of the incorrect client setting of yarn.resourcemanager.cluster-id
that ends up not matching the one in the cluster. This often results in a cryptic error that looks like this:
Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40242 for hari)
Unfortunately, the original IOException
seems to get completely lost, neither logged on the server nor gets seen on the client. When I put an extra trace log line to print the exception, I got this:
2016-06-06 12:57:57,394 ERROR [tionTokenRenewer #50] security.DelegationTokenRenewer - Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40663 for sfdc)
java.io.IOException: Unable to map logical nameservice URI 'hdfs://test-local-EMPTY' to a NameNode. Local configuration does not have a failover proxy provider configured.
at org.apache.hadoop.hdfs.DFSCl