Created
September 8, 2018 13:10
-
-
Save rfum/6891bb390ce031da9e3086a27d1499a5 to your computer and use it in GitHub Desktop.
Flink + Minio Error creating file system
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Starting standalonesession as a console application on host flink-job-manager-1. | |
| 2018-09-08 11:56:26,025 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:56:26,027 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC) | |
| 2018-09-08 11:56:26,027 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink | |
| 2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found> | |
| 2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13 | |
| 2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes | |
| 2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre | |
| 2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: | |
| 2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:56:26,034 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2018-09-08 11:56:26,052 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:56:26,053 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m | |
| 2018-09-08 11:56:26,054 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m | |
| 2018-09-08 11:56:26,054 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40 | |
| 2018-09-08 11:56:26,055 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2018-09-08 11:56:26,055 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper | |
| 2018-09-08 11:56:26,056 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom | |
| 2018-09-08 11:56:26,056 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000 | |
| 2018-09-08 11:56:26,057 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| 2018-09-08 11:56:26,058 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 | |
| 2018-09-08 11:56:26,060 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true | |
| 2018-09-08 11:56:26,060 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124 | |
| 2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log | |
| 2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE | |
| 2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** | |
| 2018-09-08 11:56:26,062 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:56:26,149 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint. | |
| 2018-09-08 11:56:26,150 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. | |
| 2018-09-08 11:56:26,182 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. | |
| 2018-09-08 11:56:26,189 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context. | |
| 2018-09-08 11:56:26,197 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:56:26,209 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:56:26,210 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. | |
| 2018-09-08 11:56:26,223 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013 | |
| 2018-09-08 11:56:27,020 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2018-09-08 11:56:27,093 INFO akka.remote.Remoting - Starting remoting | |
| 2018-09-08 11:56:27,224 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:44013] | |
| 2018-09-08 11:56:27,231 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://[email protected]:44013 | |
| 2018-09-08 11:56:28,288 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob | |
| 2018-09-08 11:56:28,296 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2018-09-08 11:56:28,296 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace. | |
| 2018-09-08 11:56:28,389 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-1.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre | |
| 2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib | |
| 2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink | |
| 2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5851bd4f | |
| 2018-09-08 11:56:28,425 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181 | |
| 2018-09-08 11:56:28,430 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session | |
| 2018-09-08 11:56:28,442 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0001, negotiated timeout = 40000 | |
| 2018-09-08 11:56:28,444 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-6408d98a-52ff-4a26-8d82-de0384825640 | |
| 2018-09-08 11:56:28,449 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2018-09-08 11:56:28,453 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 | |
| 2018-09-08 11:56:28,472 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2018-09-08 11:56:28,478 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-0d513fbd-0341-4cd0-b62b-72d21e7a942a, expiration time 3600000, maximum cache size 52428800 bytes. | |
| 2018-09-08 11:56:28,511 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-87a55d4d-8256-47eb-8d4a-612093d15f44 | |
| 2018-09-08 11:56:28,528 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address' | |
| 2018-09-08 11:56:28,529 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-bd38d844-bc95-490e-b1b2-83325257d8d2/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. | |
| 2018-09-08 11:56:28,531 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-bd38d844-bc95-490e-b1b2-83325257d8d2/flink-web-upload for file uploads. | |
| 2018-09-08 11:56:28,535 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. | |
| 2018-09-08 11:56:29,047 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. | |
| 2018-09-08 11:56:29,047 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'. | |
| 2018-09-08 11:56:29,285 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 | |
| 2018-09-08 11:56:29,286 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. | |
| 2018-09-08 11:56:29,325 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081. | |
| 2018-09-08 11:56:29,343 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager . | |
| 2018-09-08 11:56:29,399 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . | |
| 2018-09-08 11:56:29,419 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. | |
| 2018-09-08 11:56:29,421 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 11:56:29,430 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. | |
| 2018-09-08 11:56:29,432 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. | |
| 2018-09-08 12:58:52,708 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated] | |
| 2018-09-08 12:59:34,016 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://[email protected]:44013/user/dispatcher was granted leadership with fencing token 62abf6d0-8509-4ea9-9cb5-1a70b37e7eb5 | |
| 2018-09-08 12:59:34,016 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs. | |
| 2018-09-08 12:59:34,021 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 was granted leadership with leaderSessionID=25d1bd81-5ab0-4f54-9908-241a31905109 | |
| 2018-09-08 12:59:34,075 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://[email protected]:44013/user/resourcemanager was granted leadership with fencing token 88ea0b9394bde28a8629de01c7b94903 | |
| 2018-09-08 12:59:34,077 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2018-09-08 11:58:21,842 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:58:21,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC) | |
| 2018-09-08 11:58:21,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink | |
| 2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found> | |
| 2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13 | |
| 2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes | |
| 2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre | |
| 2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available | |
| 2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: | |
| 2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m | |
| 2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m | |
| 2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode | |
| 2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host | |
| 2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:58:21,850 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2018-09-08 11:58:21,872 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m | |
| 2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m | |
| 2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40 | |
| 2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2018-09-08 11:58:21,877 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper | |
| 2018-09-08 11:58:21,877 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom | |
| 2018-09-08 11:58:21,878 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000 | |
| 2018-09-08 11:58:21,878 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| 2018-09-08 11:58:21,879 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 | |
| 2018-09-08 11:58:21,880 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true | |
| 2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124 | |
| 2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log | |
| 2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE | |
| 2018-09-08 11:58:21,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** | |
| 2018-09-08 11:58:21,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:58:22,003 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint. | |
| 2018-09-08 11:58:22,006 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. | |
| 2018-09-08 11:58:22,034 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. | |
| 2018-09-08 11:58:22,043 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context. | |
| 2018-09-08 11:58:22,056 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:58:22,070 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:58:22,071 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. | |
| 2018-09-08 11:58:22,086 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:34357 | |
| 2018-09-08 11:58:22,839 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2018-09-08 11:58:22,932 INFO akka.remote.Remoting - Starting remoting | |
| 2018-09-08 11:58:23,091 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:34357] | |
| 2018-09-08 11:58:23,098 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://[email protected]:34357 | |
| 2018-09-08 11:58:24,303 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob | |
| 2018-09-08 11:58:24,309 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2018-09-08 11:58:24,310 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace. | |
| 2018-09-08 11:58:24,400 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2018-09-08 11:58:24,416 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT | |
| 2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-3.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 | |
| 2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure | |
| 2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink | |
| 2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink | |
| 2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink | |
| 2018-09-08 11:58:24,420 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5851bd4f | |
| 2018-09-08 11:58:24,442 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181 | |
| 2018-09-08 11:58:24,461 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181, initiating session | |
| 2018-09-08 11:58:24,484 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-1c94e424-8f4d-470d-bcb8-c122e8b3e56c | |
| 2018-09-08 11:58:24,486 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181, sessionid = 0x365b903c2fe0001, negotiated timeout = 40000 | |
| 2018-09-08 11:58:24,492 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 | |
| 2018-09-08 11:58:24,500 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2018-09-08 11:58:24,520 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2018-09-08 11:58:24,527 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-cae866a9-4109-49c0-9e81-b5be77c48ab9, expiration time 3600000, maximum cache size 52428800 bytes. | |
| 2018-09-08 11:58:24,589 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-c62109a6-032d-4541-acde-937375922162 | |
| 2018-09-08 11:58:24,638 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address' | |
| 2018-09-08 11:58:24,641 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-c7a37d10-072f-46c8-923b-6401836c8d5d/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. | |
| 2018-09-08 11:58:24,652 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-c7a37d10-072f-46c8-923b-6401836c8d5d/flink-web-upload for file uploads. | |
| 2018-09-08 11:58:24,658 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. | |
| 2018-09-08 11:58:25,194 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. | |
| 2018-09-08 11:58:25,195 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'. | |
| 2018-09-08 11:58:25,388 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 | |
| 2018-09-08 11:58:25,388 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. | |
| 2018-09-08 11:58:25,431 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081. | |
| 2018-09-08 11:58:25,453 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager . | |
| 2018-09-08 11:58:25,542 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . | |
| 2018-09-08 11:58:25,571 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. | |
| 2018-09-08 11:58:25,573 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 11:58:25,575 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. | |
| 2018-09-08 11:58:25,577 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. | |
| 2018-09-08 12:58:52,702 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated] | |
| 2018-09-08 12:59:34,175 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 12:59:34,248 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,314 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,386 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,454 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,525 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,595 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,665 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,735 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,812 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,893 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Starting taskexecutor as a console application on host flink-task-manager-0. | |
| 2018-09-08 12:53:19,496 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- | |
| 2018-09-08 12:53:19,497 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC) | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: flink | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found> | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13 | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /docker-java-home/jre | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options: | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments: | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /opt/flink/conf | |
| 2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- | |
| 2018-09-08 12:53:19,503 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2018-09-08 12:53:19,510 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576. | |
| 2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40 | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom | |
| 2018-09-08 12:53:19,527 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000 | |
| 2018-09-08 12:53:19,528 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| 2018-09-08 12:53:19,529 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 | |
| 2018-09-08 12:53:19,530 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true | |
| 2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/task.log | |
| 2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124 | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** | |
| 2018-09-08 12:53:19,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 12:53:19,556 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. | |
| 2018-09-08 12:53:19,575 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 12:53:19,603 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 12:53:21,263 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob | |
| 2018-09-08 12:53:21,269 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2018-09-08 12:53:21,270 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace. | |
| 2018-09-08 12:53:21,381 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT | |
| 2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-task-manager-0 | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink | |
| 2018-09-08 12:53:21,408 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2631f68c | |
| 2018-09-08 12:53:21,465 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181 | |
| 2018-09-08 12:53:21,489 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session | |
| 2018-09-08 12:53:21,493 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:21,506 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0003, negotiated timeout = 40000 | |
| 2018-09-08 12:53:21,519 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. | |
| 2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics | |
| 2018-09-08 12:53:21,633 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.164:36411. | |
| 2018-09-08 12:53:21,638 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:21,639 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address 'flink-task-manager-0' (10.244.3.165) for communication. | |
| 2018-09-08 12:53:21,643 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Starting AkkaRpcService at flink-task-manager-0:0. | |
| 2018-09-08 12:53:22,281 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2018-09-08 12:53:22,371 INFO akka.remote.Remoting - Starting remoting | |
| 2018-09-08 12:53:22,616 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-task-manager-0:44689] | |
| 2018-09-08 12:53:22,692 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2018-09-08 12:53:22,699 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-741f05fd-7a5f-4b20-ab11-66098ddd7bb5 | |
| 2018-09-08 12:53:22,705 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-51498ac0-f068-4522-bfde-dc6047d851dd | |
| 2018-09-08 12:53:22,713 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: flink-task-manager-0/10.244.3.165, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 40 (manual), number of client threads: 40 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)] | |
| 2018-09-08 12:53:22,811 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 29 GB, usable 14 GB (48.28% usable) | |
| 2018-09-08 12:53:23,045 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768). | |
| 2018-09-08 12:53:23,173 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. | |
| 2018-09-08 12:53:23,174 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. | |
| 2018-09-08 12:53:23,175 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components. | |
| 2018-09-08 12:53:23,249 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 73 ms). | |
| 2018-09-08 12:53:23,368 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 118 ms). Listening on SocketAddress /10.244.3.165:40913. | |
| 2018-09-08 12:53:23,369 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (634 MB), memory will be allocated lazily. | |
| 2018-09-08 12:53:23,375 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-908bca06-9e34-4ebc-882f-055d76bf3454 for spill files. | |
| 2018-09-08 12:53:23,443 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms | |
| 2018-09-08 12:53:23,454 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 . | |
| 2018-09-08 12:53:23,472 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:23,473 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service. | |
| 2018-09-08 12:53:23,475 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-5e7e5b20-b71a-4e31-9766-da70e668619f | |
| 2018-09-08 12:53:23,489 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://[email protected]:36411/user/resourcemanager(a5a439df914aa590bdd27cbd63084651). | |
| 2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration | |
| 2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms) | |
| 2018-09-08 12:53:23,895 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 2 (timeout=200ms) | |
| 2018-09-08 12:53:23,937 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://[email protected]:36411/user/resourcemanager under registration id d223747355b4842abc73660ffa585ccb. | |
| 2018-09-08 12:58:52,701 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated] | |
| 2018-09-08 12:59:34,076 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager connection a4355e1fb750ab2dc11bedbaddd35c38. | |
| 2018-09-08 12:59:34,102 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:36411] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:36411]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 12:59:34,122 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://[email protected]:44013/user/resourcemanager(88ea0b9394bde28a8629de01c7b94903). | |
| 2018-09-08 12:59:34,128 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local] | |
| 2018-09-08 12:59:34,129 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 12:59:44,155 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 12:59:44,155 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 12:59:54,186 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 12:59:54,186 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:04,234 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:04,234 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 13:00:14,262 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:14,262 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known] | |
| 2018-09-08 13:00:24,280 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:00:24,281 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:24,282 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:00:34,299 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:00:34,301 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:00:34,301 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:44,319 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:00:44,319 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:00:44,320 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:54,338 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:00:54,339 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:00:54,340 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:04,358 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:04,359 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:04,359 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:01:14,378 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:01:14,378 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:14,380 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:24,399 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:24,400 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:24,402 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:01:34,419 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:34,420 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:34,421 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:01:44,442 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:44,444 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:01:44,445 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:54,460 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:01:54,461 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:01:54,462 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:04,477 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:04,478 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:02:04,478 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:14,499 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:14,501 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:02:14,501 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:24,519 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:02:24,520 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:24,520 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:34,538 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:34,539 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:02:34,540 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:44,559 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:44,560 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:02:44,561 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:54,578 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:02:54,579 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:02:54,579 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:04,598 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:04,599 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:04,601 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:03:14,618 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:14,620 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:14,620 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:03:24,639 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:24,639 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:03:24,640 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:34,660 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:34,661 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:34,661 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:03:44,678 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:44,679 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:44,679 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. | |
| 2018-09-08 13:03:54,700 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013 | |
| 2018-09-08 13:03:54,701 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013] | |
| 2018-09-08 13:03:54,707 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://[email protected]:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:44013/user/resourcemanager.. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Starting standalonesession as a console application on host flink-job-manager-0. | |
| 2018-09-08 11:55:37,829 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:55:37,830 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC) | |
| 2018-09-08 11:55:37,830 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink | |
| 2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found> | |
| 2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13 | |
| 2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes | |
| 2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre | |
| 2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available | |
| 2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: | |
| 2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m | |
| 2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m | |
| 2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host | |
| 2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster | |
| 2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- | |
| 2018-09-08 11:55:37,836 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2018-09-08 11:55:37,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m | |
| 2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m | |
| 2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40 | |
| 2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper | |
| 2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom | |
| 2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000 | |
| 2018-09-08 11:55:37,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| 2018-09-08 11:55:37,858 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 | |
| 2018-09-08 11:55:37,859 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true | |
| 2018-09-08 11:55:37,859 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124 | |
| 2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log | |
| 2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE | |
| 2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** | |
| 2018-09-08 11:55:37,861 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 11:55:37,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint. | |
| 2018-09-08 11:55:37,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. | |
| 2018-09-08 11:55:38,002 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. | |
| 2018-09-08 11:55:38,013 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context. | |
| 2018-09-08 11:55:38,022 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:55:38,038 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 11:55:38,038 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. | |
| 2018-09-08 11:55:38,061 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411 | |
| 2018-09-08 11:55:38,873 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2018-09-08 11:55:38,969 INFO akka.remote.Remoting - Starting remoting | |
| 2018-09-08 11:55:39,117 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:36411] | |
| 2018-09-08 11:55:39,130 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://[email protected]:36411 | |
| 2018-09-08 11:55:40,597 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob | |
| 2018-09-08 11:55:40,603 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2018-09-08 11:55:40,604 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace. | |
| 2018-09-08 11:55:40,771 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT | |
| 2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 | |
| 2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre | |
| 2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib | |
| 2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2018-09-08 11:55:40,803 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2018-09-08 11:55:40,803 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure | |
| 2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink | |
| 2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink | |
| 2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink | |
| 2018-09-08 11:55:40,805 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@29caf222 | |
| 2018-09-08 11:55:40,875 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181 | |
| 2018-09-08 11:55:40,889 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session | |
| 2018-09-08 11:55:40,890 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-2a386297-ce71-42d9-a968-8c66117cea0a | |
| 2018-09-08 11:55:40,903 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0000, negotiated timeout = 40000 | |
| 2018-09-08 11:55:40,913 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2018-09-08 11:55:40,917 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 | |
| 2018-09-08 11:55:40,949 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2018-09-08 11:55:40,955 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-4ef201cf-7028-4a71-9041-83de535b3147, expiration time 3600000, maximum cache size 52428800 bytes. | |
| 2018-09-08 11:55:41,001 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-b9468b26-b300-4c2e-ac54-dc3acee1e610 | |
| 2018-09-08 11:55:41,020 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address' | |
| 2018-09-08 11:55:41,032 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-92eb8b33-ac65-4c24-a42b-5c64fb4da5a2/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. | |
| 2018-09-08 11:55:41,034 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-92eb8b33-ac65-4c24-a42b-5c64fb4da5a2/flink-web-upload for file uploads. | |
| 2018-09-08 11:55:41,040 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. | |
| 2018-09-08 11:55:41,461 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. | |
| 2018-09-08 11:55:41,462 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'. | |
| 2018-09-08 11:55:41,689 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 | |
| 2018-09-08 11:55:41,689 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. | |
| 2018-09-08 11:55:41,741 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081. | |
| 2018-09-08 11:55:41,770 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager . | |
| 2018-09-08 11:55:41,783 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 was granted leadership with leaderSessionID=0d817ea6-85a8-4af3-ab02-0cae2b2490dd | |
| 2018-09-08 11:55:41,836 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . | |
| 2018-09-08 11:55:41,873 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. | |
| 2018-09-08 11:55:41,880 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 11:55:41,888 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. | |
| 2018-09-08 11:55:41,890 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. | |
| 2018-09-08 11:55:41,902 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://[email protected]:36411/user/resourcemanager was granted leadership with fencing token a5a439df914aa590bdd27cbd63084651 | |
| 2018-09-08 11:55:41,903 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. | |
| 2018-09-08 11:55:41,908 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://[email protected]:36411/user/dispatcher was granted leadership with fencing token bc6db51a-f6d7-4b5e-a201-8b5fccc37b4a | |
| 2018-09-08 11:55:41,908 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs. | |
| 2018-09-08 12:53:23,924 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Replacing old registration of TaskExecutor 5737ffe2496c7432429419864fbb76a0. | |
| 2018-09-08 12:53:23,924 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Unregister TaskManager a0ddec1a885cee59adf6504db55f73fe from the SlotManager. | |
| 2018-09-08 12:53:23,925 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - The target with resource ID 5737ffe2496c7432429419864fbb76a0 is already been monitored. | |
| 2018-09-08 12:53:23,978 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 5737ffe2496c7432429419864fbb76a0 under d223747355b4842abc73660ffa585ccb at the SlotManager. | |
| 2018-09-08 12:54:10,259 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager b0f6f22da68a2e2e997b219f10fe5828 under f1f06580d44e554ad6375d5aff799c26 at the SlotManager. | |
| 2018-09-08 12:55:00,634 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 286dbce39f7cfd3178979012f5015873 under 9814c8f1bc76eb99c57c796884621491 at the SlotManager. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Starting taskexecutor as a console application on host flink-task-manager-0. | |
| 2018-09-08 12:53:19,496 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- | |
| 2018-09-08 12:53:19,497 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC) | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: flink | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found> | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13 | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes | |
| 2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /docker-java-home/jre | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options: | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M | |
| 2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments: | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir | |
| 2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /opt/flink/conf | |
| 2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- | |
| 2018-09-08 12:53:19,503 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2018-09-08 12:53:19,510 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576. | |
| 2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| 2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m | |
| 2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40 | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper | |
| 2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom | |
| 2018-09-08 12:53:19,527 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000 | |
| 2018-09-08 12:53:19,528 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| 2018-09-08 12:53:19,529 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 | |
| 2018-09-08 12:53:19,530 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true | |
| 2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/task.log | |
| 2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124 | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE | |
| 2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** | |
| 2018-09-08 12:53:19,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125 | |
| 2018-09-08 12:53:19,556 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. | |
| 2018-09-08 12:53:19,575 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 12:53:19,603 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. | |
| 2018-09-08 12:53:21,263 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob | |
| 2018-09-08 12:53:21,269 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2018-09-08 12:53:21,270 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace. | |
| 2018-09-08 12:53:21,381 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT | |
| 2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-task-manager-0 | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar::: | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink | |
| 2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink | |
| 2018-09-08 12:53:21,408 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2631f68c | |
| 2018-09-08 12:53:21,465 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181 | |
| 2018-09-08 12:53:21,489 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session | |
| 2018-09-08 12:53:21,493 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:21,506 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0003, negotiated timeout = 40000 | |
| 2018-09-08 12:53:21,519 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. | |
| 2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics | |
| 2018-09-08 12:53:21,633 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.164:36411. | |
| 2018-09-08 12:53:21,638 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:21,639 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address 'flink-task-manager-0' (10.244.3.165) for communication. | |
| 2018-09-08 12:53:21,643 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Starting AkkaRpcService at flink-task-manager-0:0. | |
| 2018-09-08 12:53:22,281 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2018-09-08 12:53:22,371 INFO akka.remote.Remoting - Starting remoting | |
| 2018-09-08 12:53:22,616 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-task-manager-0:44689] | |
| 2018-09-08 12:53:22,692 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2018-09-08 12:53:22,699 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-741f05fd-7a5f-4b20-ab11-66098ddd7bb5 | |
| 2018-09-08 12:53:22,705 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-51498ac0-f068-4522-bfde-dc6047d851dd | |
| 2018-09-08 12:53:22,713 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: flink-task-manager-0/10.244.3.165, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 40 (manual), number of client threads: 40 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)] | |
| 2018-09-08 12:53:22,811 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 29 GB, usable 14 GB (48.28% usable) | |
| 2018-09-08 12:53:23,045 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768). | |
| 2018-09-08 12:53:23,173 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. | |
| 2018-09-08 12:53:23,174 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. | |
| 2018-09-08 12:53:23,175 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components. | |
| 2018-09-08 12:53:23,249 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 73 ms). | |
| 2018-09-08 12:53:23,368 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 118 ms). Listening on SocketAddress /10.244.3.165:40913. | |
| 2018-09-08 12:53:23,369 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (634 MB), memory will be allocated lazily. | |
| 2018-09-08 12:53:23,375 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-908bca06-9e34-4ebc-882f-055d76bf3454 for spill files. | |
| 2018-09-08 12:53:23,443 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms | |
| 2018-09-08 12:53:23,454 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 . | |
| 2018-09-08 12:53:23,472 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. | |
| 2018-09-08 12:53:23,473 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service. | |
| 2018-09-08 12:53:23,475 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-5e7e5b20-b71a-4e31-9766-da70e668619f | |
| 2018-09-08 12:53:23,489 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://[email protected]:36411/user/resourcemanager(a5a439df914aa590bdd27cbd63084651). | |
| 2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration | |
| 2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms) | |
| 2018-09-08 12:53:23,895 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 2 (timeout=200ms) | |
| 2018-09-08 12:53:23,937 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://[email protected]:36411/user/resourcemanager under registration id d223747355b4842abc73660ffa585ccb. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM flink:latest | |
| WORKDIR /opt/flink | |
| COPY flink-conf.yaml conf | |
| COPY masters conf | |
| RUN cp opt/flink-s3-fs-presto-1.6.0.jar lib && cp opt/flink-s3-fs-hadoop-1.6.0.jar lib | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ################################################################################ | |
| # Licensed to the Apache Software Foundation (ASF) under one | |
| # or more contributor license agreements. See the NOTICE file | |
| # distributed with this work for additional information | |
| # regarding copyright ownership. The ASF licenses this file | |
| # to you under the Apache License, Version 2.0 (the | |
| # "License"); you may not use this file except in compliance | |
| # with the License. You may obtain a copy of the License at | |
| # | |
| # http://www.apache.org/licenses/LICENSE-2.0 | |
| # | |
| # Unless required by applicable law or agreed to in writing, software | |
| # distributed under the License is distributed on an "AS IS" BASIS, | |
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| # See the License for the specific language governing permissions and | |
| # limitations under the License. | |
| ################################################################################ | |
| # | |
| # #rfum : | |
| # | |
| #============================================================================== | |
| # Common | |
| #============================================================================== | |
| # The external address of the host on which the JobManager runs and can be | |
| # reached by the TaskManagers and any clients which want to connect. This setting | |
| # is only used in Standalone mode and may be overwritten on the JobManager side | |
| # by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable. | |
| # In high availability mode, if you use the bin/start-cluster.sh script and setup | |
| # the conf/masters file, this will be taken care of automatically. Yarn/Mesos | |
| # automatically configure the host name based on the hostname of the node where the | |
| # JobManager runs. | |
| jobmanager.rpc.address: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| # The RPC port where the JobManager is reachable. | |
| #jobmanager.rpc.port: 6123 | |
| # The heap size for the JobManager JVM | |
| jobmanager.heap.size: 1024m | |
| # The heap size for the TaskManager JVM | |
| taskmanager.heap.size: 1024m | |
| # The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline. | |
| taskmanager.numberOfTaskSlots: 40 | |
| # The parallelism used for programs that did not specify and other parallelism. | |
| parallelism.default: 1 | |
| # The default file system scheme and authority. | |
| # | |
| # By default file paths without scheme are interpreted relative to the local | |
| # root file system 'file:///'. Use this to override the default and interpret | |
| # relative paths relative to a different file system, | |
| # for example 'hdfs://mynamenode:12345' | |
| # | |
| # fs.default-scheme | |
| #============================================================================== | |
| # High Availability | |
| #============================================================================== | |
| # The high-availability mode. Possible options are 'NONE' or 'zookeeper'. | |
| # | |
| high-availability: zookeeper | |
| high-availability.cluster-id: cusotm | |
| # The path where metadata for master recovery is persisted. While ZooKeeper stores | |
| # the small ground truth for checkpoint and leader election, this location stores | |
| # the larger objects, like persisted dataflow graphs. | |
| # | |
| # Must be a durable file system that is accessible from all nodes | |
| # (like HDFS, S3, Ceph, nfs, ...) | |
| # | |
| high-availability.storageDir: s3://minio:9000 # minio is the name of ClusterIP service which has only 1 pod!!! | |
| # The list of ZooKeeper quorum peers that coordinate the high-availability | |
| # setup. This must be a list of the form: | |
| # "host1:clientPort,host2:clientPort,..." (default clientPort: 2181) | |
| # | |
| high-availability.zookeeper.quorum: flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| # ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes | |
| # It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE) | |
| # The default value is "open" and it can be changed to "creator" if ZK security is enabled | |
| # | |
| #high-availability.zookeeper.client.acl: open | |
| #============================================================================== | |
| # Fault tolerance and checkpointing | |
| #============================================================================== | |
| # The backend that will be used to store operator state checkpoints if | |
| # checkpointing is enabled. | |
| # | |
| # Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the | |
| # <class-name-of-factory>. | |
| # | |
| # state.backend: filesystem | |
| # Directory for checkpoints filesystem, when using any of the default bundled | |
| # state backends. | |
| # | |
| # state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints | |
| # Default target directory for savepoints, optional. | |
| # | |
| # state.savepoints.dir: hdfs://namenode-host:port/flink-checkpoints | |
| # Flag to enable/disable incremental checkpoints for backends that | |
| # support incremental checkpoints (like the RocksDB state backend). | |
| # | |
| # state.backend.incremental: false | |
| #============================================================================== | |
| # Web Frontend | |
| #============================================================================== | |
| # The address under which the web-based runtime monitor listens. | |
| # | |
| #jobmanager.web.address: 0.0.0.0 | |
| # The port under which the web-based runtime monitor listens. | |
| # A value of -1 deactivates the web server. | |
| rest.port: 8081 | |
| # Flag to specify whether job submission is enabled from the web-based | |
| # runtime monitor. Uncomment to disable. | |
| #jobmanager.web.submit.enable: false | |
| #============================================================================== | |
| # Advanced | |
| #============================================================================== | |
| # Override the directories for temporary files. If not specified, the | |
| # system-specific Java temporary directory (java.io.tmpdir property) is taken. | |
| # | |
| # For framework setups on Yarn or Mesos, Flink will automatically pick up the | |
| # containers' temp directories without any need for configuration. | |
| # | |
| # Add a delimited list for multiple directories, using the system directory | |
| # delimiter (colon ':' on unix) or a comma, e.g.: | |
| # /data1/tmp:/data2/tmp:/data3/tmp | |
| # | |
| # Note: Each directory entry is read from and written to by a different I/O | |
| # thread. You can include the same directory multiple times in order to create | |
| # multiple I/O threads against that directory. This is for example relevant for | |
| # high-throughput RAIDs. | |
| # | |
| # io.tmp.dirs: /tmp | |
| # Specify whether TaskManager's managed memory should be allocated when starting | |
| # up (true) or when memory is requested. | |
| # | |
| # We recommend to set this value to 'true' only in setups for pure batch | |
| # processing (DataSet API). Streaming setups currently do not use the TaskManager's | |
| # managed memory: The 'rocksdb' state backend uses RocksDB's own memory management, | |
| # while the 'memory' and 'filesystem' backends explicitly keep data as objects | |
| # to save on serialization cost. | |
| # | |
| # taskmanager.memory.preallocate: false | |
| # The classloading resolve order. Possible values are 'child-first' (Flink's default) | |
| # and 'parent-first' (Java's default). | |
| # | |
| # Child first classloading allows users to use different dependency/library | |
| # versions in their application than those in the classpath. Switching back | |
| # to 'parent-first' may help with debugging dependency issues. | |
| # | |
| # classloader.resolve-order: child-first | |
| # The amount of memory going to the network stack. These numbers usually need | |
| # no tuning. Adjusting them may be necessary in case of an "Insufficient number | |
| # of network buffers" error. The default min is 64MB, teh default max is 1GB. | |
| # | |
| # taskmanager.network.memory.fraction: 0.1 | |
| # taskmanager.network.memory.min: 64mb | |
| # taskmanager.network.memory.max: 1gb | |
| #============================================================================== | |
| # Flink Cluster Security Configuration | |
| #============================================================================== | |
| # Kerberos authentication for various components - Hadoop, ZooKeeper, and connectors - | |
| # may be enabled in four steps: | |
| # 1. configure the local krb5.conf file | |
| # 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit) | |
| # 3. make the credentials available to various JAAS login contexts | |
| # 4. configure the connector to use JAAS/SASL | |
| # The below configure how Kerberos credentials are provided. A keytab will be used instead of | |
| # a ticket cache if the keytab path and principal are set. | |
| # security.kerberos.login.use-ticket-cache: true | |
| # security.kerberos.login.keytab: /path/to/kerberos/keytab | |
| # security.kerberos.login.principal: flink-user | |
| # The configuration below defines which JAAS login contexts | |
| # security.kerberos.login.contexts: Client,KafkaClient | |
| #============================================================================== | |
| # ZK Security Configuration | |
| #============================================================================== | |
| # Below configurations are applicable if ZK ensemble is configured for security | |
| zookeeper.sasl.disable: true | |
| # Override below configuration to provide custom ZK service name if configured | |
| #zookeeper.sasl.service-name: zookeeper | |
| # The configuration below must match one of the values set in "security.kerberos.login.contexts" | |
| #zookeeper.sasl.login-context-name: Client | |
| #============================================================================== | |
| # HistoryServer | |
| #============================================================================== | |
| # The HistoryServer is started and stopped via bin/historyserver.sh (start|stop) | |
| # Directory to upload completed jobs to. Add this directory to the list of | |
| # monitored directories of the HistoryServer as well (see below). | |
| #jobmanager.archive.fs.dir: hdfs:///completed-jobs/ | |
| # The address under which the web-based HistoryServer listens. | |
| #historyserver.web.address: 0.0.0.0 | |
| # The port under which the web-based HistoryServer listens. | |
| #historyserver.web.port: 8082 | |
| # Comma separated list of directories to monitor for completed jobs. | |
| #historyserver.archive.fs.dir: hdfs:///completed-jobs/ | |
| # Interval in milliseconds for refreshing the monitored directories. | |
| #historyserver.archive.fs.refresh-interval: 10000 | |
| blob.server.port: 6124 | |
| query.server.port: 6125 | |
| log.file: /data/flink/job.log | |
| s3.access-key: AKIAIOSFODNN7EXAMPLE | |
| s3.secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 | |
| flink-job-manager-1.flink-job-svc.flink.svc.cluster.local:8081 | |
| flink-job-manager-2.flink-job-svc.flink.svc.cluster.local:8081 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: apps/v1 | |
| kind: StatefulSet | |
| metadata: | |
| name: flink-job-manager | |
| namespace: flink | |
| spec: | |
| selector: | |
| matchLabels: | |
| app: flink | |
| role: job # has to match .spec.template.metadata.labels | |
| serviceName: "flink-job-svc" # "flink-job-svc-headless" | |
| replicas: 4 # by default is 1 | |
| template: | |
| metadata: | |
| labels: | |
| app: flink # has to match .spec.selector.matchLabels | |
| role: job | |
| spec: | |
| terminationGracePeriodSeconds: 10 | |
| restartPolicy: Always | |
| imagePullSecrets: | |
| - name: customn-cr-secret | |
| containers: | |
| - name: flink | |
| image: customcr.azurecr.io/flink-job:v1 | |
| args: | |
| - jobmanager | |
| ports: | |
| - containerPort: 6123 | |
| name: rpc | |
| - containerPort: 6124 | |
| name: blob | |
| - containerPort: 6125 | |
| name: query | |
| - containerPort: 8081 | |
| name: ui | |
| volumeMounts: | |
| - name: checkpoint | |
| mountPath: /data/flink/checkpoints | |
| readinessProbe: | |
| exec: | |
| command: | |
| - ls | |
| - /data/flink/checkpoints | |
| initialDelaySeconds: 5 | |
| periodSeconds: 5 | |
| imagePullPolicy: Always | |
| volumeClaimTemplates: | |
| - metadata: | |
| name: checkpoint | |
| spec: | |
| accessModes: [ "ReadWriteOnce" ] | |
| storageClassName: "custom-ssd-gen" | |
| resources: | |
| requests: | |
| storage: 20Gi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| NAME READY STATUS RESTARTS AGE | |
| flink-job-manager-0 1/1 Running 0 1h | |
| flink-job-manager-1 1/1 Running 0 1h | |
| flink-job-manager-2 1/1 Running 0 1h | |
| flink-job-manager-3 1/1 Running 0 1h | |
| flink-task-manager-0 1/1 Running 0 5m | |
| flink-task-manager-1 1/1 Running 0 5m | |
| flink-task-manager-2 1/1 Running 0 4m | |
| flink-zk-zookeeper-0 1/1 Running 1 2d | |
| flink-zk-zookeeper-1 1/1 Running 1 2d | |
| flink-zk-zookeeper-2 1/1 Running 1 2d | |
| minio-55c776944c-jl9x5 1/1 Running 1 2d |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: Service | |
| metadata: | |
| name: flink-job-svc | |
| namespace: flink | |
| spec: | |
| clusterIP: "None" | |
| ports: | |
| - name: rpc | |
| port: 6123 | |
| - name: blob | |
| port: 6124 | |
| - name: query | |
| port: 6125 | |
| - name: ui | |
| port: 8081 | |
| selector: | |
| app: flink | |
| role: job |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM flink:latest | |
| WORKDIR /opt/flink | |
| COPY flink-conf.yaml conf | |
| COPY masters conf | |
| RUN cp opt/flink-s3-fs-presto-1.6.0.jar lib && cp opt/flink-s3-fs-hadoop-1.6.0.jar lib |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ################################################################################ | |
| # Licensed to the Apache Software Foundation (ASF) under one | |
| # or more contributor license agreements. See the NOTICE file | |
| # distributed with this work for additional information | |
| # regarding copyright ownership. The ASF licenses this file | |
| # to you under the Apache License, Version 2.0 (the | |
| # "License"); you may not use this file except in compliance | |
| # with the License. You may obtain a copy of the License at | |
| # | |
| # http://www.apache.org/licenses/LICENSE-2.0 | |
| # | |
| # Unless required by applicable law or agreed to in writing, software | |
| # distributed under the License is distributed on an "AS IS" BASIS, | |
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| # See the License for the specific language governing permissions and | |
| # limitations under the License. | |
| ################################################################################ | |
| # | |
| # #rfum : | |
| # | |
| #============================================================================== | |
| # Common | |
| #============================================================================== | |
| # The external address of the host on which the JobManager runs and can be | |
| # reached by the TaskManagers and any clients which want to connect. This setting | |
| # is only used in Standalone mode and may be overwritten on the JobManager side | |
| # by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable. | |
| # In high availability mode, if you use the bin/start-cluster.sh script and setup | |
| # the conf/masters file, this will be taken care of automatically. Yarn/Mesos | |
| # automatically configure the host name based on the hostname of the node where the | |
| # JobManager runs. | |
| jobmanager.rpc.address: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local | |
| # The RPC port where the JobManager is reachable. | |
| jobmanager.rpc.port: 6123 | |
| # The heap size for the JobManager JVM | |
| jobmanager.heap.size: 1024m | |
| # The heap size for the TaskManager JVM | |
| taskmanager.heap.size: 1024m | |
| # The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline. | |
| taskmanager.numberOfTaskSlots: 40 | |
| # The parallelism used for programs that did not specify and other parallelism. | |
| parallelism.default: 1 | |
| # The default file system scheme and authority. | |
| # | |
| # By default file paths without scheme are interpreted relative to the local | |
| # root file system 'file:///'. Use this to override the default and interpret | |
| # relative paths relative to a different file system, | |
| # for example 'hdfs://mynamenode:12345' | |
| # | |
| # fs.default-scheme | |
| #============================================================================== | |
| # High Availability | |
| #============================================================================== | |
| # The high-availability mode. Possible options are 'NONE' or 'zookeeper'. | |
| # | |
| high-availability: zookeeper | |
| # The path where metadata for master recovery is persisted. While ZooKeeper stores | |
| # the small ground truth for checkpoint and leader election, this location stores | |
| # the larger objects, like persisted dataflow graphs. | |
| # | |
| # Must be a durable file system that is accessible from all nodes | |
| # (like HDFS, S3, Ceph, nfs, ...) | |
| # | |
| high-availability.cluster-id: custom | |
| high-availability.storageDir: s3://minio:9000 # minio is the name of ClusterIP service which has only 1 pod!!! | |
| # The list of ZooKeeper quorum peers that coordinate the high-availability | |
| # setup. This must be a list of the form: | |
| # "host1:clientPort,host2:clientPort,..." (default clientPort: 2181) | |
| # | |
| high-availability.zookeeper.quorum: flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 | |
| # ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes | |
| # It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE) | |
| # The default value is "open" and it can be changed to "creator" if ZK security is enabled | |
| # | |
| #high-availability.zookeeper.client.acl: open | |
| #============================================================================== | |
| # Fault tolerance and checkpointing | |
| #============================================================================== | |
| # The backend that will be used to store operator state checkpoints if | |
| # checkpointing is enabled. | |
| # | |
| # Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the | |
| # <class-name-of-factory>. | |
| # | |
| # state.backend: filesystem | |
| # Directory for checkpoints filesystem, when using any of the default bundled | |
| # state backends. | |
| # | |
| # state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints | |
| # Default target directory for savepoints, optional. | |
| # | |
| # state.savepoints.dir: hdfs://namenode-host:port/flink-checkpoints | |
| # Flag to enable/disable incremental checkpoints for backends that | |
| # support incremental checkpoints (like the RocksDB state backend). | |
| # | |
| # state.backend.incremental: false | |
| #============================================================================== | |
| # Web Frontend | |
| #============================================================================== | |
| # The address under which the web-based runtime monitor listens. | |
| # | |
| #jobmanager.web.address: 0.0.0.0 | |
| # The port under which the web-based runtime monitor listens. | |
| # A value of -1 deactivates the web server. | |
| rest.port: 8081 | |
| # Flag to specify whether job submission is enabled from the web-based | |
| # runtime monitor. Uncomment to disable. | |
| #jobmanager.web.submit.enable: false | |
| #============================================================================== | |
| # Advanced | |
| #============================================================================== | |
| # Override the directories for temporary files. If not specified, the | |
| # system-specific Java temporary directory (java.io.tmpdir property) is taken. | |
| # | |
| # For framework setups on Yarn or Mesos, Flink will automatically pick up the | |
| # containers' temp directories without any need for configuration. | |
| # | |
| # Add a delimited list for multiple directories, using the system directory | |
| # delimiter (colon ':' on unix) or a comma, e.g.: | |
| # /data1/tmp:/data2/tmp:/data3/tmp | |
| # | |
| # Note: Each directory entry is read from and written to by a different I/O | |
| # thread. You can include the same directory multiple times in order to create | |
| # multiple I/O threads against that directory. This is for example relevant for | |
| # high-throughput RAIDs. | |
| # | |
| # io.tmp.dirs: /tmp | |
| # Specify whether TaskManager's managed memory should be allocated when starting | |
| # up (true) or when memory is requested. | |
| # | |
| # We recommend to set this value to 'true' only in setups for pure batch | |
| # processing (DataSet API). Streaming setups currently do not use the TaskManager's | |
| # managed memory: The 'rocksdb' state backend uses RocksDB's own memory management, | |
| # while the 'memory' and 'filesystem' backends explicitly keep data as objects | |
| # to save on serialization cost. | |
| # | |
| # taskmanager.memory.preallocate: false | |
| # The classloading resolve order. Possible values are 'child-first' (Flink's default) | |
| # and 'parent-first' (Java's default). | |
| # | |
| # Child first classloading allows users to use different dependency/library | |
| # versions in their application than those in the classpath. Switching back | |
| # to 'parent-first' may help with debugging dependency issues. | |
| # | |
| # classloader.resolve-order: child-first | |
| # The amount of memory going to the network stack. These numbers usually need | |
| # no tuning. Adjusting them may be necessary in case of an "Insufficient number | |
| # of network buffers" error. The default min is 64MB, teh default max is 1GB. | |
| # | |
| # taskmanager.network.memory.fraction: 0.1 | |
| # taskmanager.network.memory.min: 64mb | |
| # taskmanager.network.memory.max: 1gb | |
| #============================================================================== | |
| # Flink Cluster Security Configuration | |
| #============================================================================== | |
| # Kerberos authentication for various components - Hadoop, ZooKeeper, and connectors - | |
| # may be enabled in four steps: | |
| # 1. configure the local krb5.conf file | |
| # 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit) | |
| # 3. make the credentials available to various JAAS login contexts | |
| # 4. configure the connector to use JAAS/SASL | |
| # The below configure how Kerberos credentials are provided. A keytab will be used instead of | |
| # a ticket cache if the keytab path and principal are set. | |
| # security.kerberos.login.use-ticket-cache: true | |
| # security.kerberos.login.keytab: /path/to/kerberos/keytab | |
| # security.kerberos.login.principal: flink-user | |
| # The configuration below defines which JAAS login contexts | |
| # security.kerberos.login.contexts: Client,KafkaClient | |
| #============================================================================== | |
| # ZK Security Configuration | |
| #============================================================================== | |
| zookeeper.sasl.disable: true | |
| # Below configurations are applicable if ZK ensemble is configured for security | |
| # Override below configuration to provide custom ZK service name if configured | |
| # zookeeper.sasl.service-name: zookeeper | |
| # The configuration below must match one of the values set in "security.kerberos.login.contexts" | |
| # zookeeper.sasl.login-context-name: Client | |
| #============================================================================== | |
| # HistoryServer | |
| #============================================================================== | |
| # The HistoryServer is started and stopped via bin/historyserver.sh (start|stop) | |
| # Directory to upload completed jobs to. Add this directory to the list of | |
| # monitored directories of the HistoryServer as well (see below). | |
| #jobmanager.archive.fs.dir: hdfs:///completed-jobs/ | |
| # The address under which the web-based HistoryServer listens. | |
| #historyserver.web.address: 0.0.0.0 | |
| # The port under which the web-based HistoryServer listens. | |
| #historyserver.web.port: 8082 | |
| # Comma separated list of directories to monitor for completed jobs. | |
| #historyserver.archive.fs.dir: hdfs:///completed-jobs/ | |
| # Interval in milliseconds for refreshing the monitored directories. | |
| #historyserver.archive.fs.refresh-interval: 10000 | |
| log.file: /data/flink/task.log | |
| blob.server.port: 6124 | |
| query.server.port: 6125 | |
| s3.access-key: AKIAIOSFODNN7EXAMPLE | |
| s3.secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 | |
| flink-job-manager-1.flink-job-svc.flink.svc.cluster.local:8081 | |
| flink-job-manager-2.flink-job-svc.flink.svc.cluster.local:8081 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: apps/v1 | |
| kind: StatefulSet | |
| metadata: | |
| name: flink-task-manager | |
| namespace: flink | |
| spec: | |
| serviceName: "" | |
| selector: | |
| matchLabels: | |
| app: flink | |
| role: task | |
| updateStrategy: | |
| type: RollingUpdate | |
| replicas: 3 | |
| template: | |
| metadata: | |
| labels: | |
| app: flink | |
| role: task | |
| spec: | |
| terminationGracePeriodSeconds: 10 | |
| imagePullSecrets: | |
| - name: custom-cr-secret | |
| containers: | |
| - name: flink | |
| image: customcr.azurecr.io/flink-task:v1 | |
| args: | |
| - taskmanager | |
| ports: | |
| - containerPort: 6121 | |
| name: data | |
| - containerPort: 6122 | |
| name: rpc | |
| - containerPort: 6125 | |
| name: query | |
| volumeMounts: | |
| - name: checkpoint | |
| mountPath: /data/flink/checkpoints | |
| readinessProbe: | |
| exec: | |
| command: | |
| - ls | |
| - /data/flink/checkpoints | |
| initialDelaySeconds: 5 | |
| periodSeconds: 5 | |
| imagePullPolicy: Always | |
| volumeClaimTemplates: | |
| - metadata: | |
| name: checkpoint | |
| spec: | |
| accessModes: [ "ReadWriteOnce" ] | |
| storageClassName: "custom-ssd-gen" | |
| resources: | |
| requests: | |
| storage: 20Gi |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment