Hadoop + Hive Build/Runtime issues for Metering Operator -- trying to get the tip of Hadoop 3.3 branch and latest Hive master (Oct 15, 2020) to work to see if will fix a bug related to S3 bucket contents deletion

gistfile1.txt

│ Exception in thread "main" 2020-10-13T17:23:10.464510059Z java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V2020-10-13T17:23:10.464518738Z        │
│     at org.apache.hadoop.conf.Configuration.set(Configuration.java:1382)2020-10-13T17:23:10.464616395Z                                                                                                                 │
│     at org.apache.hadoop.conf.Configuration.set(Configuration.java:1363)2020-10-13T17:23:10.464630106Z                                                                                                                 │
│     at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)2020-10-13T17:23:10.464669465Z                                                                                                                         │
│     at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)                                                                                                                                                │
│     at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)2020-10-13T17:23:10.464698814Z                                                                                                                         │
│     at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4047)2020-10-13T17:23:10.464708189Z                                                                                                               │
│     at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:4010)                                                                                                                                                 │
│     at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7002)2020-10-13T17:23:10.464737483Z                                                                                                      │
│     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                                     │
│     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)2020-10-13T17:23:10.464765419Z                                                                                                     │
│     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                           │
│     at java.lang.reflect.Method.invoke(Method.java:498)2020-10-13T17:23:10.464793618Z                                                                                                                                  │
│     at org.apache.hadoop.util.RunJar.run(RunJar.java:323)2020-10-13T17:23:10.464803494Z                                                                                                                                │
│     at org.apache.hadoop.util.RunJar.main(RunJar.java:236)                                                                                                                                                             │
│ stream closed

Having a problem where Hadoop version created by checkout tag rel/release-3.3.0 and cherry pick all "our" commits creates a fully working Hadoop and tests okay with Hive 3.1.2 (through Metering deploy and reportdatasources data being added).

But... PR has needs-rebase, likely due to these not being the same:

hadoop from_rel-3.3.0 $ git show-ref --heads -s origin master
6ace76f403981964e7b6714530f5e01948c10b09
hadoop from_rel-3.3.0 $ git merge-base origin/master from_rel-3.3.0
49c747ab187d0650143205ba57ca19607ec4c6bd

Trying to rebase instead onto 3.2.1, but we run into:

Caused by: org.apache.maven.plugin.MojoExecutionException: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 3.7.1', expected version is '2.5.0'

and the whole point of this move is to move to newer protobuf

But this rebased onto 3.2.1 still fails with error I was also seeing on the bigger jump to 3.3.0:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-common: Compilation failure: Compilation failure:
[ERROR] /build/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java:[162,38] cannot find symbol
[ERROR] symbol:   class AbstractDelegationTokenSelector
[ERROR] location: class org.apache.hadoop.crypto.key.kms.KMSClientProvider
[ERROR] /build/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java:[1010,35] cannot find symbol
[ERROR] symbol:   method selectToken(org.apache.hadoop.io.Text,java.util.Collection<org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier>
>)
[ERROR] location: variable INSTANCE of type org.apache.hadoop.crypto.key.kms.KMSClientProvider.TokenSelector

Above is possible related to this commit:

752092a8608 Revert "HADOOP-14445. Delegation tokens are not shared between KMS instances. Contributed by Xiao Chen and Rushabh S Shah."

A New Beginning....
Successfully used roughly the kube/master Dockerfile and rebased from 3.1.1 to 3.1.4 just to see how that would go and maybe more nearly align with what has to be the next stop for this change, 3.2.0

https://github.com/bentito/hadoop/blob/yarb_rel_3.1.4/Dockerfile

rebase to 3.1.4 and 3.2.0 common ancestor compiled successfully:

git pull --rebase upstream  a39296260f8c77f3808e27b43c623a0edebe4a17

This branch is: bentito/yarb_rel_3.1.4-3.2.0

After the, now, shorter rebase from ☝️ to upstream/branch_3.2.0 seeing this error on build:

[ERROR] /build/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:[529,54] cannot find symbol

Built 3.3.0 with a "our" Dockerfile that moves:
cmake -> 3.19.1;
mvn -> 3.3.9;
protobuf -> 3.7.1;
this branch is: hadoop-3.3.0-from-3.2.2

So we still face a needs-rebase tag. Going with

git cherry-pick 2b9a8c1d3a2^..6ace76f4039

onto hadoop-3.3.0-from-3.2.2 to pickup everything of "ours" from kube-reporting/hadoop/master

Seeing this as before on this path:

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: User hadoop is not allowed to perform this API call                                                                                                 ```