Skip to content

Instantly share code, notes, and snippets.

@hmcgowan

hmcgowan/gist:4f9b1efd4c0c98b4cd446f27d1474d6c

Created November 14, 2023 22:50
Show Gist options
  • Save hmcgowan/4f9b1efd4c0c98b4cd446f27d1474d6c to your computer and use it in GitHub Desktop.
Save hmcgowan/4f9b1efd4c0c98b4cd446f27d1474d6c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Attaching to opensearch-dashboards, opensearch-node1, opensearch-node2
opensearch-node1 | Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
opensearch-node1 | **************************************************************************
opensearch-node1 | ** This tool will be deprecated in the next major release of OpenSearch **
opensearch-node1 | ** https://github.com/opensearch-project/security/issues/1755 **
opensearch-node1 | **************************************************************************
opensearch-node1 | OpenSearch Security Demo Installer
opensearch-node1 | ** Warning: Do not use on production or public reachable systems **
opensearch-node1 | Basedir: /usr/share/opensearch
opensearch-node2 | Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
opensearch-node2 | **************************************************************************
opensearch-node2 | ** This tool will be deprecated in the next major release of OpenSearch **
opensearch-node2 | ** https://github.com/opensearch-project/security/issues/1755 **
opensearch-node2 | **************************************************************************
opensearch-node1 | OpenSearch install type: rpm/deb on Amazon Linux release 2023 (Amazon Linux)
opensearch-node1 | OpenSearch config dir: /usr/share/opensearch/config
opensearch-node1 | OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
opensearch-node1 | OpenSearch bin dir: /usr/share/opensearch/bin
opensearch-node1 | OpenSearch plugins dir: /usr/share/opensearch/plugins
opensearch-node1 | OpenSearch lib dir: /usr/share/opensearch/lib
opensearch-node1 | Detected OpenSearch Version: x-content-2.11.0
opensearch-node1 | Detected OpenSearch Security Version: 2.11.0.0
opensearch-node2 | OpenSearch Security Demo Installer
opensearch-node1 | /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
opensearch-node1 | Enabling OpenSearch Security Plugin
opensearch-node1 | Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
opensearch-node2 | ** Warning: Do not use on production or public reachable systems **
opensearch-node2 | Basedir: /usr/share/opensearch
opensearch-node2 | OpenSearch install type: rpm/deb on Amazon Linux release 2023 (Amazon Linux)
opensearch-node2 | OpenSearch config dir: /usr/share/opensearch/config
opensearch-node2 | OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
opensearch-node2 | OpenSearch bin dir: /usr/share/opensearch/bin
opensearch-node2 | OpenSearch plugins dir: /usr/share/opensearch/plugins
opensearch-node2 | OpenSearch lib dir: /usr/share/opensearch/lib
opensearch-node2 | Detected OpenSearch Version: x-content-2.11.0
opensearch-node2 | Detected OpenSearch Security Version: 2.11.0.0
opensearch-node2 | /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
opensearch-node2 | Enabling OpenSearch Security Plugin
opensearch-node2 | Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
opensearch-dashboards | Disabling OpenSearch Security Dashboards Plugin
opensearch-dashboards | Removing securityDashboards...
opensearch-dashboards | Plugin removal complete
opensearch-node2 | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-node2 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
opensearch-node2 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
opensearch-node2 | WARNING: System::setSecurityManager will be removed in a future release
opensearch-node1 | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-node1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
opensearch-node1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
opensearch-node1 | WARNING: System::setSecurityManager will be removed in a future release
opensearch-node2 | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-node2 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
opensearch-node2 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
opensearch-node2 | WARNING: System::setSecurityManager will be removed in a future release
opensearch-node1 | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-node1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
opensearch-node1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
opensearch-node1 | WARNING: System::setSecurityManager will be removed in a future release
opensearch-node2 | [2023-11-14T22:47:56,461][INFO ][o.o.n.Node ] [opensearch-node2] version[2.11.0], pid[31], build[tar/4dcad6dd1fd45b6bd91f041a041829c8687278fa/2023-10-13T02:56:10.311811371Z], OS[Linux/6.4.16-linuxkit/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
opensearch-node2 | [2023-11-14T22:47:56,462][INFO ][o.o.n.Node ] [opensearch-node2] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
opensearch-node2 | [2023-11-14T22:47:56,463][INFO ][o.o.n.Node ] [opensearch-node2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12965831683497626421, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
opensearch-node1 | [2023-11-14T22:47:56,463][INFO ][o.o.n.Node ] [opensearch-node1] version[2.11.0], pid[31], build[tar/4dcad6dd1fd45b6bd91f041a041829c8687278fa/2023-10-13T02:56:10.311811371Z], OS[Linux/6.4.16-linuxkit/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
opensearch-node1 | [2023-11-14T22:47:56,466][INFO ][o.o.n.Node ] [opensearch-node1] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
opensearch-node1 | [2023-11-14T22:47:56,467][INFO ][o.o.n.Node ] [opensearch-node1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12457543753141691015, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
opensearch-node1 | [2023-11-14T22:47:57,475][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-node1] SSL dual mode is disabled
opensearch-node1 | [2023-11-14T22:47:57,476][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
opensearch-node2 | [2023-11-14T22:47:57,484][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-node2] SSL dual mode is disabled
opensearch-node2 | [2023-11-14T22:47:57,485][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
opensearch-node1 | [2023-11-14T22:47:58,059][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-node1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
opensearch-node2 | [2023-11-14T22:47:58,059][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-node2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
opensearch-node2 | [2023-11-14T22:47:58,492][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-node2] ReindexPlugin reloadSPI called
opensearch-node2 | [2023-11-14T22:47:58,493][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-node2] Unable to find any implementation for RemoteReindexExtension
opensearch-node1 | [2023-11-14T22:47:58,496][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-node1] ReindexPlugin reloadSPI called
opensearch-node1 | [2023-11-14T22:47:58,497][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-node1] Unable to find any implementation for RemoteReindexExtension
opensearch-node1 | [2023-11-14T22:47:58,523][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
opensearch-node2 | [2023-11-14T22:47:58,523][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
opensearch-node2 | [2023-11-14T22:47:58,564][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
opensearch-node1 | [2023-11-14T22:47:58,564][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
opensearch-node2 | [2023-11-14T22:47:58,565][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
opensearch-node1 | [2023-11-14T22:47:58,565][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
opensearch-node2 | [2023-11-14T22:47:58,566][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
opensearch-node1 | [2023-11-14T22:47:58,566][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node1] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
opensearch-node2 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [aggs-matrix-stats]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [analysis-common]
opensearch-node1 | [2023-11-14T22:47:58,570][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [aggs-matrix-stats]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [geo]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [ingest-common]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [ingest-geoip]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [ingest-user-agent]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [analysis-common]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [geo]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [ingest-common]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [ingest-geoip]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [ingest-user-agent]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [lang-expression]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [lang-expression]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [lang-mustache]
opensearch-node2 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [lang-painless]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [mapper-extras]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [lang-mustache]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [lang-painless]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [mapper-extras]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [opensearch-dashboards]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [parent-join]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [percolator]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [rank-eval]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [reindex]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [opensearch-dashboards]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [parent-join]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [percolator]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [rank-eval]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [reindex]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [repository-url]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [search-pipeline-common]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [systemd]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded module [transport-netty4]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [repository-url]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [search-pipeline-common]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [systemd]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded module [transport-netty4]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-alerting]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-alerting]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-anomaly-detection]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-asynchronous-search]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-anomaly-detection]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-cross-cluster-replication]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-custom-codecs]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-geospatial]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-index-management]
opensearch-node2 | [2023-11-14T22:47:58,573][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-job-scheduler]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-knn]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-asynchronous-search]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-ml]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-neural-search]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-notifications]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-notifications-core]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-observability]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-cross-cluster-replication]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-performance-analyzer]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-custom-codecs]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-geospatial]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-index-management]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-reports-scheduler]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-job-scheduler]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-security]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-security-analytics]
opensearch-node2 | [2023-11-14T22:47:58,574][INFO ][o.o.p.PluginsService ] [opensearch-node2] loaded plugin [opensearch-sql]
opensearch-node1 | [2023-11-14T22:47:58,571][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-knn]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-ml]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-neural-search]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-notifications]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-notifications-core]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-observability]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-performance-analyzer]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-reports-scheduler]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-security]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-security-analytics]
opensearch-node1 | [2023-11-14T22:47:58,572][INFO ][o.o.p.PluginsService ] [opensearch-node1] loaded plugin [opensearch-sql]
opensearch-node1 | [2023-11-14T22:47:58,628][INFO ][o.o.e.ExtensionsManager ] [opensearch-node1] ExtensionsManager initialized
opensearch-node2 | [2023-11-14T22:47:58,628][INFO ][o.o.e.ExtensionsManager ] [opensearch-node2] ExtensionsManager initialized
opensearch-node1 | [2023-11-14T22:47:58,642][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-node1] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node2 | [2023-11-14T22:47:58,642][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-node2] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node2 | [2023-11-14T22:47:58,655][INFO ][o.o.e.NodeEnvironment ] [opensearch-node2] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vda1)]], net usable_space [4.9gb], net total_space [58.3gb], types [ext4]
opensearch-node2 | [2023-11-14T22:47:58,655][INFO ][o.o.e.NodeEnvironment ] [opensearch-node2] heap size [512mb], compressed ordinary object pointers [true]
opensearch-node1 | [2023-11-14T22:47:58,656][INFO ][o.o.e.NodeEnvironment ] [opensearch-node1] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vda1)]], net usable_space [4.9gb], net total_space [58.3gb], types [ext4]
opensearch-node1 | [2023-11-14T22:47:58,656][INFO ][o.o.e.NodeEnvironment ] [opensearch-node1] heap size [512mb], compressed ordinary object pointers [true]
opensearch-node1 | [2023-11-14T22:47:58,743][INFO ][o.o.n.Node ] [opensearch-node1] node name [opensearch-node1], node ID [0vFCdHRxTiOGrFalS5m2Lw], cluster name [opensearch-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
opensearch-node2 | [2023-11-14T22:47:58,744][INFO ][o.o.n.Node ] [opensearch-node2] node name [opensearch-node2], node ID [PiGFR-S1R564VXiElofAnQ], cluster name [opensearch-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSource\" is disabled."}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""}
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [50] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,securityAnalyticsDashboards,home,apmOss,savedObjects,reportsDashboards,searchRelevanceDashboards,dashboard,mlCommonsDashboards,visualizations,visTypeVega,visTypeTable,visTypeTimeline,visTypeMarkdown,visAugmenter,alertingDashboards,anomalyDetectionDashboards,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,indexManagementDashboards,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,dataExplorer,charts,visTypeVislib,visTypeTagcloud,visTypeMetric,visTypeTimeseries,observabilityDashboards,discover,savedObjectsManagement,bfetch]"}
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.21.0.3:9200"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:47:59Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
opensearch-node2 | [2023-11-14T22:48:00,126][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-node2] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node1 | [2023-11-14T22:48:00,135][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-node1] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node1 | [2023-11-14T22:48:00,586][INFO ][o.o.n.p.NeuralSearch ] [opensearch-node1] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
opensearch-node2 | [2023-11-14T22:48:00,626][INFO ][o.o.n.p.NeuralSearch ] [opensearch-node2] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
opensearch-node1 | [2023-11-14T22:48:01,077][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-node1] Registered memory breaker.
opensearch-node2 | [2023-11-14T22:48:01,084][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-node2] Registered memory breaker.
opensearch-node2 | [2023-11-14T22:48:01,338][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node2] Registered ML memory breaker.
opensearch-node1 | [2023-11-14T22:48:01,337][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node1] Registered ML memory breaker.
opensearch-node1 | [2023-11-14T22:48:01,338][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node1] Registered ML disk breaker.
opensearch-node1 | [2023-11-14T22:48:01,338][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node1] Registered ML native memory breaker.
opensearch-node2 | [2023-11-14T22:48:01,338][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node2] Registered ML disk breaker.
opensearch-node2 | [2023-11-14T22:48:01,338][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-node2] Registered ML native memory breaker.
opensearch-node2 | [2023-11-14T22:48:01,397][INFO ][o.r.Reflections ] [opensearch-node2] Reflections took 29 ms to scan 1 urls, producing 20 keys and 51 values
opensearch-node1 | [2023-11-14T22:48:01,397][INFO ][o.r.Reflections ] [opensearch-node1] Reflections took 29 ms to scan 1 urls, producing 20 keys and 51 values
opensearch-node2 | [2023-11-14T22:48:01,452][WARN ][o.o.s.p.SQLPlugin ] [opensearch-node2] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
opensearch-node1 | [2023-11-14T22:48:01,452][WARN ][o.o.s.p.SQLPlugin ] [opensearch-node1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
opensearch-node1 | [2023-11-14T22:48:01,483][WARN ][o.o.s.p.SQLPlugin ] [opensearch-node1] Async Query APIs are disabled as plugins.query.executionengine.spark.config is not configured properly in cluster settings. Please configure and restart the domain to enable Async Query APIs
opensearch-node2 | [2023-11-14T22:48:01,483][WARN ][o.o.s.p.SQLPlugin ] [opensearch-node2] Async Query APIs are disabled as plugins.query.executionengine.spark.config is not configured properly in cluster settings. Please configure and restart the domain to enable Async Query APIs
opensearch-node2 | [2023-11-14T22:48:01,912][INFO ][o.o.t.NettyAllocator ] [opensearch-node2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
opensearch-node1 | [2023-11-14T22:48:01,912][INFO ][o.o.t.NettyAllocator ] [opensearch-node1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
opensearch-node2 | [2023-11-14T22:48:01,980][INFO ][o.o.d.DiscoveryModule ] [opensearch-node2] using discovery type [zen] and seed hosts providers [settings]
opensearch-node1 | [2023-11-14T22:48:01,981][INFO ][o.o.d.DiscoveryModule ] [opensearch-node1] using discovery type [zen] and seed hosts providers [settings]
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:02Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.21.0.3:9200"}
opensearch-node2 | [2023-11-14T22:48:02,269][WARN ][o.o.g.DanglingIndicesState] [opensearch-node2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
opensearch-node1 | [2023-11-14T22:48:02,281][WARN ][o.o.g.DanglingIndicesState] [opensearch-node1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
opensearch-node1 | [2023-11-14T22:48:02,521][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node1] PerformanceAnalyzer Enabled: false
opensearch-node2 | [2023-11-14T22:48:02,521][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false
opensearch-node1 | [2023-11-14T22:48:02,545][INFO ][o.o.n.Node ] [opensearch-node1] initialized
opensearch-node2 | [2023-11-14T22:48:02,545][INFO ][o.o.n.Node ] [opensearch-node2] initialized
opensearch-node2 | [2023-11-14T22:48:02,546][INFO ][o.o.n.Node ] [opensearch-node2] starting ...
opensearch-node1 | [2023-11-14T22:48:02,546][INFO ][o.o.n.Node ] [opensearch-node1] starting ...
opensearch-node2 | [2023-11-14T22:48:02,568][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [windows_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [waf_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,569][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [windows_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [waf_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [vpcflow_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [test_windows_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [s3_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [vpcflow_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [test_windows_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [s3_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_web_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_web_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_proxy_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_macos_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_compliance_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_cloud_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_proxy_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_macos_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_compliance_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_apt_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_cloud_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_application_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [okta_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [network_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [netflow_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [m365_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_apt_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [linux_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [gworkspace_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [github_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [dns_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [others_application_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [okta_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [cloudtrail_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [azure_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [apache_access_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [network_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [ad_ldap_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,581][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [netflow_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,582][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [m365_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,582][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [linux_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,583][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [gworkspace_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,583][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [github_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,583][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [dns_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [cloudtrail_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [azure_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,585][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [apache_access_logtype.json] log type
opensearch-node2 | [2023-11-14T22:48:02,585][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node2] Loaded [ad_ldap_logtype.json] log type
opensearch-node1 | [2023-11-14T22:48:02,676][INFO ][o.o.t.TransportService ] [opensearch-node1] publish_address {172.21.0.3:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node1 | [2023-11-14T22:48:02,677][INFO ][o.o.t.TransportService ] [opensearch-node1] Remote clusters initialized successfully.
opensearch-node2 | [2023-11-14T22:48:02,708][INFO ][o.o.t.TransportService ] [opensearch-node2] publish_address {172.21.0.4:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node2 | [2023-11-14T22:48:02,709][INFO ][o.o.t.TransportService ] [opensearch-node2] Remote clusters initialized successfully.
opensearch-node2 | [2023-11-14T22:48:02,892][INFO ][o.o.b.BootstrapChecks ] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node2 | [2023-11-14T22:48:02,894][INFO ][o.o.c.c.Coordinator ] [opensearch-node2] cluster UUID [HO03mPPbSVKWBq6Vn7Y2iA]
opensearch-node1 | [2023-11-14T22:48:02,897][INFO ][o.o.b.BootstrapChecks ] [opensearch-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node1 | [2023-11-14T22:48:02,901][INFO ][o.o.c.c.Coordinator ] [opensearch-node1] cluster UUID [HO03mPPbSVKWBq6Vn7Y2iA]
opensearch-node2 | [2023-11-14T22:48:03,033][INFO ][o.o.c.r.a.AllocationService] [opensearch-node2] updating number_of_replicas to [0] for indices [.opensearch-observability, .plugins-ml-config, .opensearch-sap-log-types-config]
opensearch-node1 | [2023-11-14T22:48:03,039][INFO ][o.o.c.s.MasterService ] [opensearch-node1] elected-as-cluster-manager ([2] nodes joined)[{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, {opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 3, version: 21, delta: cluster-manager node changed {previous [], current [{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}
opensearch-node2 | [2023-11-14T22:48:03,043][INFO ][o.o.c.s.MasterService ] [opensearch-node2] elected-as-cluster-manager ([1] nodes joined)[{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 21, delta: cluster-manager node changed {previous [], current [{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
opensearch-node1 | [2023-11-14T22:48:03,041][WARN ][o.o.c.s.MasterService ] [opensearch-node1] failing [elected-as-cluster-manager ([2] nodes joined)[{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, {opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [21]
opensearch-node1 | org.opensearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer cluster-manager for term 3 while handling publication
opensearch-node1 | at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1272) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService.publish(MasterService.java:349) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:331) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
opensearch-node1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
opensearch-node1 | at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node1 | [2023-11-14T22:48:03,049][INFO ][o.o.c.c.JoinHelper ] [opensearch-node1] failed to join {opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=2, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=20, sourceNode={opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
opensearch-node1 | org.opensearch.transport.RemoteTransportException: [opensearch-node1][172.21.0.3:9300][internal:cluster/coordination/join]
opensearch-node1 | Caused by: org.opensearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer cluster-manager for term 3 while handling publication
opensearch-node1 | at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1272) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService.publish(MasterService.java:349) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:331) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.11.0.jar:2.11.0]
opensearch-node1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
opensearch-node1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
opensearch-node1 | at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node2 | [2023-11-14T22:48:03,057][INFO ][o.o.c.c.JoinHelper ] [opensearch-node2] failed to join {opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=2, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=20, sourceNode={opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
opensearch-node2 | org.opensearch.transport.RemoteTransportException: [opensearch-node1][172.21.0.3:9300][internal:cluster/coordination/join]
opensearch-node2 | Caused by: org.opensearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer cluster-manager for term 3 while handling publication
opensearch-node2 | at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1272) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.cluster.service.MasterService.publish(MasterService.java:349) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:331) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node2 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
opensearch-node2 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
opensearch-node2 | at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node2 | [2023-11-14T22:48:03,081][INFO ][o.o.c.s.ClusterApplierService] [opensearch-node2] cluster-manager node changed {previous [], current [{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 21, reason: Publication{term=4, version=21}
opensearch-node2 | [2023-11-14T22:48:03,087][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node2] Cluster is not recovered yet.
opensearch-node2 | [2023-11-14T22:48:03,092][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,110][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch-node2] Cache cluster manager node onClusterManager time: 1700002083110
opensearch-node2 | [2023-11-14T22:48:03,118][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node2] Config override setting update called with empty string. Ignoring.
opensearch-node2 | [2023-11-14T22:48:03,123][INFO ][o.o.d.PeerFinder ] [opensearch-node2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]
opensearch-node2 | [2023-11-14T22:48:03,124][INFO ][o.o.c.r.a.AllocationService] [opensearch-node2] updating number_of_replicas to [1] for indices [.opensearch-observability, .plugins-ml-config, .opensearch-sap-log-types-config]
opensearch-node2 | [2023-11-14T22:48:03,125][INFO ][o.o.c.s.MasterService ] [opensearch-node2] node-join[{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} join existing leader, {opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} join existing leader], term: 4, version: 22, delta: added {{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}}
opensearch-node2 | [2023-11-14T22:48:03,131][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {172.21.0.4:9200}, bound_addresses {0.0.0.0:9200}
opensearch-node2 | [2023-11-14T22:48:03,131][INFO ][o.o.n.Node ] [opensearch-node2] started
opensearch-node2 | [2023-11-14T22:48:03,132][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Node started
opensearch-node2 | [2023-11-14T22:48:03,133][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] 0 OpenSearch Security modules loaded so far: []
opensearch-node1 | [2023-11-14T22:48:03,183][INFO ][o.o.c.s.ClusterApplierService] [opensearch-node1] cluster-manager node changed {previous [], current [{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}, term: 4, version: 22, reason: ApplyCommitRequest{term=4, version=22, sourceNode={opensearch-node2}{PiGFR-S1R564VXiElofAnQ}{uUfsXoecR7ircER-Rl8uwA}{172.21.0.4}{172.21.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}
opensearch-node1 | [2023-11-14T22:48:03,185][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node1] Cluster is not recovered yet.
opensearch-node1 | [2023-11-14T22:48:03,185][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node1 | [2023-11-14T22:48:03,200][INFO ][o.o.m.a.MLModelAutoReDeployer] [opensearch-node1] Model auto reload configuration is false, not performing auto reloading!
opensearch-node1 | [2023-11-14T22:48:03,206][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node1] Config override setting update called with empty string. Ignoring.
opensearch-node2 | [2023-11-14T22:48:03,207][INFO ][o.o.c.s.ClusterApplierService] [opensearch-node2] added {{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}}, term: 4, version: 22, reason: Publication{term=4, version=22}
opensearch-node2 | [2023-11-14T22:48:03,207][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node2] Cluster is not recovered yet.
opensearch-node2 | [2023-11-14T22:48:03,207][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node1 | [2023-11-14T22:48:03,208][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node1] publish_address {172.21.0.3:9200}, bound_addresses {0.0.0.0:9200}
opensearch-node1 | [2023-11-14T22:48:03,208][INFO ][o.o.n.Node ] [opensearch-node1] started
opensearch-node1 | [2023-11-14T22:48:03,208][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] Node started
opensearch-node1 | [2023-11-14T22:48:03,208][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] 0 OpenSearch Security modules loaded so far: []
opensearch-node2 | [2023-11-14T22:48:03,218][WARN ][o.o.c.r.a.DiskThresholdMonitor] [opensearch-node2] Putting index create block on cluster as all nodes are breaching high disk watermark. Number of nodes above high watermark: 2.
opensearch-node2 | [2023-11-14T22:48:03,228][INFO ][o.o.m.a.MLModelAutoReDeployer] [opensearch-node2] Model auto reload configuration is false, not performing auto reloading!
opensearch-node1 | [2023-11-14T22:48:03,229][INFO ][o.o.d.PeerFinder ] [opensearch-node1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}]
opensearch-node1 | [2023-11-14T22:48:03,230][INFO ][o.o.d.PeerFinder ] [opensearch-node1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-node1}{0vFCdHRxTiOGrFalS5m2Lw}{wuN6NZ72Sp-NHIlQs95A4w}{172.21.0.3}{172.21.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}]
opensearch-node2 | [2023-11-14T22:48:03,235][WARN ][o.o.c.r.a.DiskThresholdMonitor] [opensearch-node2] high disk watermark [90%] exceeded on [0vFCdHRxTiOGrFalS5m2Lw][opensearch-node1][/usr/share/opensearch/data/nodes/0] free: 4.7gb[8.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
opensearch-node2 | [2023-11-14T22:48:03,235][WARN ][o.o.c.r.a.DiskThresholdMonitor] [opensearch-node2] high disk watermark [90%] exceeded on [PiGFR-S1R564VXiElofAnQ][opensearch-node2][/usr/share/opensearch/data/nodes/0] free: 4.7gb[8.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
opensearch-node2 | [2023-11-14T22:48:03,244][INFO ][o.o.c.r.a.AllocationService] [opensearch-node2] Cluster health status changed from [YELLOW] to [RED] (reason: [state recovered]).
opensearch-node1 | [2023-11-14T22:48:03,265][INFO ][o.o.c.s.ClusterSettings ] [opensearch-node1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
opensearch-node1 | [2023-11-14T22:48:03,267][INFO ][o.o.a.c.HashRing ] [opensearch-node1] Node added: [0vFCdHRxTiOGrFalS5m2Lw, PiGFR-S1R564VXiElofAnQ]
opensearch-node1 | [2023-11-14T22:48:03,268][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node1 | [2023-11-14T22:48:03,272][INFO ][o.o.a.c.HashRing ] [opensearch-node1] Add data node to AD version hash ring: 0vFCdHRxTiOGrFalS5m2Lw
opensearch-node1 | [2023-11-14T22:48:03,273][INFO ][o.o.a.c.HashRing ] [opensearch-node1] Add data node to AD version hash ring: PiGFR-S1R564VXiElofAnQ
opensearch-node1 | [2023-11-14T22:48:03,273][INFO ][o.o.a.c.HashRing ] [opensearch-node1] All nodes with known AD version: {0vFCdHRxTiOGrFalS5m2Lw=ADNodeInfo{version=2.11.0, isEligibleDataNode=true}, PiGFR-S1R564VXiElofAnQ=ADNodeInfo{version=2.11.0, isEligibleDataNode=true}}
opensearch-node1 | [2023-11-14T22:48:03,273][INFO ][o.o.a.c.HashRing ] [opensearch-node1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
opensearch-node1 | [2023-11-14T22:48:03,273][INFO ][o.o.a.c.HashRing ] [opensearch-node1] Build AD version hash ring successfully
opensearch-node1 | [2023-11-14T22:48:03,274][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node1] Init AD version hash ring successfully
opensearch-node2 | [2023-11-14T22:48:03,274][INFO ][o.o.c.s.ClusterSettings ] [opensearch-node2] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
opensearch-node2 | [2023-11-14T22:48:03,277][INFO ][o.o.a.c.HashRing ] [opensearch-node2] Node added: [0vFCdHRxTiOGrFalS5m2Lw, PiGFR-S1R564VXiElofAnQ]
opensearch-node2 | [2023-11-14T22:48:03,278][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,281][INFO ][o.o.a.c.HashRing ] [opensearch-node2] Add data node to AD version hash ring: 0vFCdHRxTiOGrFalS5m2Lw
opensearch-node2 | [2023-11-14T22:48:03,282][INFO ][o.o.g.GatewayService ] [opensearch-node2] recovered [3] indices into cluster_state
opensearch-node2 | [2023-11-14T22:48:03,283][INFO ][o.o.a.c.HashRing ] [opensearch-node2] Add data node to AD version hash ring: PiGFR-S1R564VXiElofAnQ
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.HashRing ] [opensearch-node2] All nodes with known AD version: {0vFCdHRxTiOGrFalS5m2Lw=ADNodeInfo{version=2.11.0, isEligibleDataNode=true}, PiGFR-S1R564VXiElofAnQ=ADNodeInfo{version=2.11.0, isEligibleDataNode=true}}
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.HashRing ] [opensearch-node2] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.HashRing ] [opensearch-node2] Build AD version hash ring successfully
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.ADDataMigrator ] [opensearch-node2] Start migrating AD data
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.ADDataMigrator ] [opensearch-node2] AD job index doesn't exist, no need to migrate
opensearch-node2 | [2023-11-14T22:48:03,284][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node2] Init AD version hash ring successfully
opensearch-node2 | [2023-11-14T22:48:03,292][WARN ][o.o.o.i.ObservabilityIndex] [opensearch-node2] message: index [.opensearch-observability/oKGgi304RfWqPkZXgf-RTQ] already exists
opensearch-node2 | [2023-11-14T22:48:03,292][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Loading builtin types!
opensearch-node2 | [2023-11-14T22:48:03,292][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node2] observability:createMappingTemplate ss4o_metrics_template API called
opensearch-node2 | [2023-11-14T22:48:03,295][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-node1 | [2023-11-14T22:48:03,312][INFO ][o.o.s.l.LogTypeService ] [opensearch-node1] Loading builtin types!
opensearch-node1 | [2023-11-14T22:48:03,313][WARN ][o.o.o.i.ObservabilityIndex] [opensearch-node1] message: index [.opensearch-observability/oKGgi304RfWqPkZXgf-RTQ] already exists
opensearch-node1 | [2023-11-14T22:48:03,313][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node1] observability:createMappingTemplate ss4o_metrics_template API called
opensearch-node1 | [2023-11-14T22:48:03,315][INFO ][o.o.s.l.LogTypeService ] [opensearch-node1] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-node2 | [2023-11-14T22:48:03,318][WARN ][o.o.s.SecurityAnalyticsPlugin] [opensearch-node2] Failed to initialize LogType config index and builtin log types
opensearch-node2 | [2023-11-14T22:48:03,325][INFO ][o.o.p.PluginsService ] [opensearch-node2] PluginService:onIndexModule index:[validate-template-1px65qq4rmewhfrigll6ca/Z3E_NeQoR924rgrXM6u4pA]
opensearch-node1 | [2023-11-14T22:48:03,332][WARN ][o.o.s.SecurityAnalyticsPlugin] [opensearch-node1] Failed to initialize LogType config index and builtin log types
opensearch-node2 | [2023-11-14T22:48:03,410][INFO ][o.o.c.m.MetadataIndexTemplateService] [opensearch-node2] updating index template [ss4o_metrics_template] for index patterns [ss4o_metrics-*-*]
opensearch-node1 | [2023-11-14T22:48:03,439][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,440][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,440][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node2] observability:Mapping Template ss4o_metrics_template creation Acknowledged
opensearch-node2 | [2023-11-14T22:48:03,441][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node2] observability:createMappingTemplate ss4o_traces_template API called
opensearch-node1 | [2023-11-14T22:48:03,442][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node1] observability:Mapping Template ss4o_metrics_template creation Acknowledged
opensearch-node1 | [2023-11-14T22:48:03,443][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node1] observability:createMappingTemplate ss4o_traces_template API called
opensearch-node2 | [2023-11-14T22:48:03,443][INFO ][o.o.p.PluginsService ] [opensearch-node2] PluginService:onIndexModule index:[validate-template-yyxoplzmtzuoq01ou__kog/Zq8X05WARxSnNOzEgTOHsg]
opensearch-node2 | [2023-11-14T22:48:03,450][INFO ][o.o.c.m.MetadataIndexTemplateService] [opensearch-node2] updating index template [ss4o_traces_template] for index patterns [ss4o_traces-*-*]
opensearch-node1 | [2023-11-14T22:48:03,472][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,473][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:03,474][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node2] observability:Mapping Template ss4o_traces_template creation Acknowledged
opensearch-node1 | [2023-11-14T22:48:03,475][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-node1] observability:Mapping Template ss4o_traces_template creation Acknowledged
opensearch-node2 | [2023-11-14T22:48:04,120][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Loading builtin types!
opensearch-node2 | [2023-11-14T22:48:04,120][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-node2 | [2023-11-14T22:48:04,122][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Loading builtin types!
opensearch-node2 | [2023-11-14T22:48:04,122][INFO ][o.o.s.l.LogTypeService ] [opensearch-node2] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:04Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:04Z","tags":["info","savedobjects-service"],"pid":1,"message":"Creating index .kibana_1."}
opensearch-node2 | [2023-11-14T22:48:04,878][DEPRECATION][o.o.d.c.m.MetadataCreateIndexService] [opensearch-node2] index name [.kibana_1] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices
opensearch-node2 | [2023-11-14T22:48:04,880][INFO ][o.o.p.PluginsService ] [opensearch-node2] PluginService:onIndexModule index:[.kibana_1/lNMDgOqPR9KdIhtbasrKvQ]
opensearch-node2 | [2023-11-14T22:48:04,887][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch-node2] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1]
opensearch-node1 | [2023-11-14T22:48:04,961][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:04,962][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration
opensearch-node2 | [2023-11-14T22:48:33,098][WARN ][o.o.c.r.a.DiskThresholdMonitor] [opensearch-node2] Putting index create block on cluster as all nodes are breaching high disk watermark. Number of nodes above high watermark: 2.
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:34Z","tags":["error","opensearch","data"],"pid":1,"message":"[TimeoutError]: Request timed out"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:34Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: Request timed out"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:37Z","tags":["error","opensearch","data"],"pid":1,"message":"[resource_already_exists_exception]: index [.kibana_1/lNMDgOqPR9KdIhtbasrKvQ] already exists"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:37Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_1/lNMDgOqPR9KdIhtbasrKvQ] already exists"}
opensearch-dashboards | {"type":"log","@timestamp":"2023-11-14T22:48:37Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting OpenSearchDashboards."}
opensearch-node2 | [2023-11-14T22:49:03,162][WARN ][o.o.c.r.a.DiskThresholdMonitor] [opensearch-node2] Putting index create block on cluster as all nodes are breaching high disk watermark. Number of nodes above high watermark: 2.
opensearch-node2 | [2023-11-14T22:49:03,178][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch-node2] Performing move cluster state metadata.
opensearch-node2 | [2023-11-14T22:49:03,196][INFO ][o.o.i.i.MetadataService ] [opensearch-node2] ISM config index not exist, so we cancel the metadata migration job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment