Last active
March 2, 2024 01:33
-
-
Save Phate334/9fd90ddce0eaea74086c1a2e5d419402 to your computer and use it in GitHub Desktop.
JanusGraph and OpenSearch
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| version: "3" | |
| services: | |
| janusgraph: | |
| image: janusgraph/janusgraph:0.6.2 | |
| container_name: jce-janusgraph | |
| env_file: | |
| - janusgraph.env | |
| ports: | |
| - "8182:8182" | |
| networks: | |
| - jce-network | |
| depends_on: | |
| - cassandra | |
| - opensearch | |
| cassandra: | |
| image: cassandra:3 | |
| container_name: jce-cassandra | |
| ports: | |
| - "9042:9042" | |
| - "9160:9160" | |
| networks: | |
| - jce-network | |
| volumes: | |
| - janusgraph-data:/var/lib/cassandra | |
| opensearch: | |
| image: docker.io/opensearchproject/opensearch:2.4.0 | |
| container_name: jce-elastic | |
| hostname: jce-elastic | |
| env_file: | |
| - opensearch.env | |
| ports: | |
| - 9200:9200 # REST API | |
| - 9600:9600 # Performance Analyzer | |
| networks: | |
| - jce-network | |
| volumes: | |
| - janusgraph-index:/usr/share/opensearch/data | |
| dashboard: | |
| image: opensearchproject/opensearch-dashboards:2.4.0 | |
| container_name: opensearch-dashboards | |
| ports: | |
| - 5601:5601 | |
| environment: | |
| OPENSEARCH_HOSTS: '["https://opensearch:9200"]' | |
| networks: | |
| - jce-network | |
| networks: | |
| jce-network: | |
| volumes: | |
| janusgraph-index: | |
| janusgraph-data: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| JANUS_PROPS_TEMPLATE=cql-es | |
| janusgraph.storage.hostname=jce-cassandra | |
| janusgraph.graph.replace-instance-if-exists=true | |
| janusgraph.query.batch=true | |
| index.search.backend=elasticsearch | |
| janusgraph.index.search.hostname=jce-elastic | |
| janusgraph.index.search.elasticsearch.http.auth.type=BASIC | |
| janusgraph.index.search.elasticsearch.http.auth.basic.username=admin | |
| janusgraph.index.search.elasticsearch.http.auth.basic.password=admin | |
| janusgraph.index.search.elasticsearch.ssl.enabled=true | |
| janusgraph.index.search.elasticsearch.ssl.disable-hostname-verification=true | |
| janusgraph.index.search.elasticsearch.ssl.allow-self-signed-certificates=true | |
| janusgraph.metrics.enabled=true | |
| janusgraph.metrics.csvReporter.enabled=false | |
| janusgraph.metrics.slf4jReporter.enabled=false | |
| janusgraph.metrics.jmx.enabled=false | |
| gremlinserver.channelizer=org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer | |
| gremlinserver.metrics.csvReporter.enabled=false | |
| gremlinserver.metrics.jmxReporter.enabled=false | |
| gremlinserver.metrics.slf4jReporter.enabled=false | |
| gremlinserver.threadPoolWorker=2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| http.host=0.0.0.0 | |
| network.host=0.0.0.0 | |
| transport.host=127.0.0.1 | |
| cluster.name=opensearch-cluster # Name the cluster | |
| node.name=jce-elastic # Name the node that will run in this container | |
| discovery.type=single-node | |
| compatibility.override_main_response_version=true | |
| bootstrap.memory_lock=true # Disable JVM heap memory swapping | |
| OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m # Set min and max JVM heap sizes to at least 50% of system RAM |
Hello Phate,
Thank you for getting back to me. I tried swapping all three images with
the near latest version, which I have to do due to many security issues of
the older ones. I think I was able to start Janusgraph successfully, though
there are still warnings like out of memory or [36mjce-cassandra | [0m
WARN [main] 2023-03-17 03:28:05,221 StartupChecks.java:187 - JMX is not
enabled to receive remote connections. Please see cassandra-env.sh for more
information.
I did try to remove the dashboard but it gave me the error to start the
Janusgraph, soI added it back and from Janusgraph point of view, it looks
good.
By the way, do you know whether "JMX is not enabled to receive remote
connections. Please see cassandra-env.sh for more information. " is an
issue? All these three should be considered running with home network or
local right?
I have been using Janusgraph all in one package (with cassandra and
elasticsearch embedded) happily, and quite some learning to do to switch
to opensearch and split into three containers. Any suggestion you have is
highly appreciated.
Thanks
Yingjie
…On Fri, Mar 17, 2023 at 5:31 AM Phate ***@***.***> wrote:
***@***.**** commented on this gist.
------------------------------
hi @yingjie668 <https://github.com/yingjie668>
there are many OutOfMemoryError in your log. Are you sure your machine
has enough memory to execute all containers?
You can remove dashboard container first, JanusGraph doesn't need it.
Maybe you need to see Docker Compose documentation to limit memory for
Opensearch or Cassandra.
—
Reply to this email directly, view it on GitHub
<https://gist.github.com/9fd90ddce0eaea74086c1a2e5d419402#gistcomment-4506227>
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQWKICLFYOXJZDCLU4KOMODW4QVPHBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCNBZG4YTONZQU52HE2LHM5SXFJTDOJSWC5DF>
.
You are receiving this email because you were mentioned.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>
.
Hello Phate,
I think I am able to start all four with little bit of resource cleaning.
here is the output.
Looks all good, right?
On Fri, Mar 17, 2023 at 5:31 AM Phate ***@***.***> wrote:
***@***.**** commented on this gist.
------------------------------
hi @yingjie668 <https://github.com/yingjie668>
there are many OutOfMemoryError in your log. Are you sure your machine
has enough memory to execute all containers?
You can remove dashboard container first, JanusGraph doesn't need it.
Maybe you need to see Docker Compose documentation to limit memory for
Opensearch or Cassandra.
—
Reply to this email directly, view it on GitHub
<https://gist.github.com/9fd90ddce0eaea74086c1a2e5d419402#gistcomment-4506227>
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQWKICLFYOXJZDCLU4KOMODW4QVPHBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCNBZG4YTONZQU52HE2LHM5SXFJTDOJSWC5DF>
.
You are receiving this email because you were mentioned.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>
.
Attaching to opensearch-dashboards, jce-cassandra, jce-elastic, jce-janusgraph
[36mjce-cassandra |[0m OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[33mjce-elastic |[0m Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
[33mjce-elastic |[0m **************************************************************************
[33mjce-elastic |[0m ** This tool will be deprecated in the next major release of OpenSearch **
[33mjce-elastic |[0m ** opensearch-project/security#1755 **
[33mjce-elastic |[0m **************************************************************************
[33mjce-elastic |[0m OpenSearch Security Demo Installer
[33mjce-elastic |[0m ** Warning: Do not use on production or public reachable systems **
[33mjce-elastic |[0m Basedir: /usr/share/opensearch
[33mjce-elastic |[0m OpenSearch install type: rpm/deb on NAME="Amazon Linux"
[33mjce-elastic |[0m OpenSearch config dir: /usr/share/opensearch/config
[33mjce-elastic |[0m OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
[33mjce-elastic |[0m OpenSearch bin dir: /usr/share/opensearch/bin
[33mjce-elastic |[0m OpenSearch plugins dir: /usr/share/opensearch/plugins
[33mjce-elastic |[0m OpenSearch lib dir: /usr/share/opensearch/lib
[33mjce-elastic |[0m Detected OpenSearch Version: x-content-2.6.0
[33mjce-elastic |[0m Detected OpenSearch Security Version: 2.6.0.0
[33mjce-elastic |[0m
[33mjce-elastic |[0m ### Success
[33mjce-elastic |[0m ### Execute this script now on all your nodes and then start all nodes
[33mjce-elastic |[0m ### OpenSearch Security will be automatically initialized.
[33mjce-elastic |[0m ### If you like to change the runtime configuration
[33mjce-elastic |[0m ### change the files in ../../../config/opensearch-security and execute:
[33mjce-elastic |[0m "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
[33mjce-elastic |[0m ### or run ./securityadmin_demo.sh
[33mjce-elastic |[0m ### To use the Security Plugin ConfigurationGUI
[33mjce-elastic |[0m ### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
[33mjce-elastic |[0m ### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
[33mjce-elastic |[0m Enabling OpenSearch Security Plugin
[33mjce-elastic |[0m Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom(Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents()Z
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop()V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition()V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush(I)V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow(JI)V
[36mjce-cassandra |[0m CompileCommand: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly(I)J
[36mjce-cassandra |[0m CompileCommand: exclude org/apache/cassandra/utils/JVMStabilityInspector.forceHeapSpaceOomMaybe(Ljava/lang/OutOfMemoryError;)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/rows/SerializationHelper;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/io/util/Memory.checkBounds(JJ)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/io/util/SafeMemory.checkBounds(JJ)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/net/FrameDecoderWith8bHeader.decode(Ljava/util/Collection;Lorg/apache/cassandra/net/ShareableBytes;I)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/service/reads/repair/RowIteratorMergeListener.applyToPartition(ILjava/util/function/Consumer;)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/BloomFilter.indexes(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/BloomFilter.setIndexes(JJIJ[J)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare(Ljava/nio/ByteBuffer;[B)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare([BLjava/nio/ByteBuffer;)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/lang/Object;JI)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/memory/BufferPool$LocalPool.tryGetInternal(IZ)Ljava/nio/ByteBuffer;
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI)[B
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI[B)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/io/DataOutput;)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/nio/ByteBuffer;)V
[36mjce-cassandra |[0m CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLjava/io/DataOutput;)V
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:17,263 YamlConfigurationLoader.java:93 - Configuration location: file:/etc/cassandra/cassandra.yaml
[33mjce-elastic |[0m WARNING: A terminally deprecated method in java.lang.System has been called
[33mjce-elastic |[0m WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
[33mjce-elastic |[0m WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
[33mjce-elastic |[0m WARNING: System::setSecurityManager will be removed in a future release
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,242 Config.java:691 - Node configuration:[allocate_tokens_for_keyspace=null; allocate_tokens_for_local_replication_factor=3; audit_logging_options=AuditLogOptions{enabled=false, logger='BinAuditLogger', included_keyspaces='', excluded_keyspaces='system,system_schema,system_virtual_schema', included_categories='', excluded_categories='', included_users='', excluded_users='', audit_logs_dir='/opt/cassandra/logs/audit/', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_optimise_full_repair_streams=false; auto_optimise_inc_repair_streams=false; auto_optimise_preview_repair_streams=false; auto_snapshot=true; autocompaction_on_startup_enabled=true; automatic_sstable_upgrade=false; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; block_for_peers_in_remote_dcs=false; block_for_peers_timeout_in_secs=10; broadcast_address=172.30.0.4; broadcast_rpc_address=172.30.0.4; buffer_pool_use_heap_if_exhausted=false; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_group_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=64; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_builders=1; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_validations=0; concurrent_writes=32; consecutive_message_errors_threshold=1; corrupted_tombstone_strategy=disabled; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=true; ***@***.***; diagnostic_events_enabled=false; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=1.0; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=false; enable_sasi_indexes=false; enable_scripted_user_defined_functions=false; enable_transient_replication=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; endpoint_snitch=SimpleSnitch; file_cache_enabled=false; file_cache_round_up=null; file_cache_size_in_mb=null; flush_compression=fast; full_query_logging_options=FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; ideal_consistency_level=null; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_range_tombstone_list_allocation_size=1; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_application_receive_queue_capacity_in_bytes=4194304; internode_application_receive_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_receive_queue_reserve_global_capacity_in_bytes=536870912; internode_application_send_queue_capacity_in_bytes=4194304; internode_application_send_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_send_queue_reserve_global_capacity_in_bytes=536870912; internode_authenticator=null; internode_compression=dc; internode_max_message_size_in_bytes=null; internode_socket_receive_buffer_size_in_bytes=0; internode_socket_send_buffer_size_in_bytes=0; internode_streaming_tcp_user_timeout_in_ms=300000; internode_tcp_connect_timeout_in_ms=2000; internode_tcp_user_timeout_in_ms=30000; key_cache_keys_to_save=2147483647; key_cache_migrate_during_compaction=true; key_cache_save_period=14400; key_cache_size_in_mb=null; keyspace_count_warn_threshold=40; listen_address=172.30.0.4; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; local_system_data_file_directory=null; max_concurrent_automatic_sstable_upgrades=1; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_allow_older_protocols=true; native_transport_flush_in_batches_legacy=false; native_transport_idle_timeout_in_ms=0; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=null; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; native_transport_receive_queue_capacity_in_bytes=1048576; network_authorizer=AllowAllNetworkAuthorizer; networking_cache_size_in_mb=null; num_tokens=16; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; periodic_commitlog_sync_lag_block_in_ms=null; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; range_tombstone_list_growth_factor=1.5; read_request_timeout_in_ms=5000; reject_repair_compaction_threshold=2147483647; repair_command_pool_full_strategy=queue; repair_command_pool_size=0; repair_session_max_tree_depth=null; repair_session_space_in_mb=null; repaired_data_tracking_for_partition_reads_enabled=false; repaired_data_tracking_for_range_reads_enabled=false; report_unconfirmed_repaired_data_mismatches=false; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.30.0.4}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_links_per_second=0; snapshot_on_duplicate_row_detection=false; snapshot_on_repaired_data_mismatch=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; storage_port=7000; stream_entire_sstables=true; stream_throughput_outbound_megabits_per_sec=200; streaming_connections_per_host=1; streaming_keep_alive_period_in_secs=300; table_count_warn_threshold=150; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; ***@***.***; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; use_offheap_merkle_trees=true; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; validation_preview_purge_head_start_in_sec=3600; windows_timer_interval=1; write_request_timeout_in_ms=2000]
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,248 DatabaseDescriptor.java:416 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,249 DatabaseDescriptor.java:477 - Global memtable on-heap threshold is enabled at 980MB
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,249 DatabaseDescriptor.java:481 - Global memtable off-heap threshold is enabled at 980MB
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,480 JMXServerUtils.java:271 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,495 CassandraDaemon.java:623 - Hostname: 346dd3f99158:7000:7001
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,496 CassandraDaemon.java:630 - JVM vendor/version: OpenJDK 64-Bit Server VM/11.0.11
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,509 CassandraDaemon.java:631 - Heap size: 3.830GiB/3.830GiB
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,511 CassandraDaemon.java:636 - CodeHeap 'non-nmethods' Non-heap memory: init = 2555904(2496K) used = 1201408(1173K) committed = 2555904(2496K) max = 5836800(5700K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,556 CassandraDaemon.java:636 - Metaspace Non-heap memory: init = 0(0K) used = 27070648(26436K) committed = 27824128(27172K) max = -1(-1K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,556 CassandraDaemon.java:636 - CodeHeap 'profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 3678336(3592K) committed = 3735552(3648K) max = 122908672(120028K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,559 CassandraDaemon.java:636 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3107416(3034K) committed = 3379200(3300K) max = 1073741824(1048576K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,560 CassandraDaemon.java:636 - Par Eden Space Heap memory: init = 671088640(655360K) used = 161067968(157292K) committed = 671088640(655360K) max = 671088640(655360K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,561 CassandraDaemon.java:636 - Par Survivor Space Heap memory: init = 83886080(81920K) used = 0(0K) committed = 83886080(81920K) max = 83886080(81920K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,574 CassandraDaemon.java:636 - CodeHeap 'non-profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 726144(709K) committed = 2555904(2496K) max = 122912768(120032K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,574 CassandraDaemon.java:636 - CMS Old Gen Heap memory: init = 3357540352(3278848K) used = 0(0K) committed = 3357540352(3278848K) max = 3357540352(3278848K)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,575 CassandraDaemon.java:638 - Classpath: /etc/cassandra:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.8.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-4.0.0.jar:/opt/cassandra/lib/asm-7.1.jar:/opt/cassandra/lib/caffeine-2.3.5.jar:/opt/cassandra/lib/cassandra-driver-core-3.11.0-shaded.jar:/opt/cassandra/lib/chronicle-bytes-2.20.111.jar:/opt/cassandra/lib/chronicle-core-2.20.126.jar:/opt/cassandra/lib/chronicle-queue-5.20.123.jar:/opt/cassandra/lib/chronicle-threads-2.20.111.jar:/opt/cassandra/lib/chronicle-wire-2.20.117.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.11.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/ecj-4.6.1.jar:/opt/cassandra/lib/guava-27.0-jre.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.8.1.jar:/opt/cassandra/lib/j2objc-annotations-1.3.jar:/opt/cassandra/lib/jackson-annotations-2.9.10.jar:/opt/cassandra/lib/jackson-core-2.9.10.jar:/opt/cassandra/lib/jackson-databind-2.9.10.8.jar:/opt/cassandra/lib/jamm-0.3.2.jar:/opt/cassandra/lib/java-cup-runtime-11b-20160615.jar:/opt/cassandra/lib/javax.inject-1.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.25.jar:/opt/cassandra/lib/jcommander-1.30.jar:/opt/cassandra/lib/jctools-core-3.1.0.jar:/opt/cassandra/lib/jflex-1.8.2.jar:/opt/cassandra/lib/jna-5.6.0.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jvm-attach-api-1.5.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.25.jar:/opt/cassandra/lib/logback-classic-1.2.3.jar:/opt/cassandra/lib/logback-core-1.2.3.jar:/opt/cassandra/lib/lz4-java-1.7.1.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/mxdump-0.14.jar:/opt/cassandra/lib/netty-all-4.1.58.Final.jar:/opt/cassandra/lib/netty-tcnative-boringssl-static-2.0.36.Final.jar:/opt/cassandra/lib/ohc-core-0.5.1.jar:/opt/cassandra/lib/ohc-core-j8-0.5.1.jar:/opt/cassandra/lib/psjava-0.1.19.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/sjk-cli-0.14.jar:/opt/cassandra/lib/sjk-core-0.14.jar:/opt/cassandra/lib/sjk-json-0.14.jar:/opt/cassandra/lib/sjk-stacktrace-0.14.jar:/opt/cassandra/lib/slf4j-api-1.7.25.jar:/opt/cassandra/lib/snakeyaml-1.26.jar:/opt/cassandra/lib/snappy-java-1.1.2.6.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/zstd-jni-1.3.8-5.jar:/opt/cassandra/lib/jsr223/*/*.jar:
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,576 CassandraDaemon.java:640 - JVM Arguments: [-ea, -da:net.openhft..., -XX:+UseThreadPriorities, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -Djdk.attach.allowAttachSelf=true, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED, --add-exports=java.sql/java.sql=ALL-UNNAMED, --add-opens=java.base/java.lang.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED, --add-opens=java.base/jdk.internal.math=ALL-UNNAMED, --add-opens=java.base/jdk.internal.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED, --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED, -Dio.netty.tryReflectionSetAccessible=true, -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/opt/cassandra/logs/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760, -Xms4002M, -Xmx4002M, -Xmn800M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.2.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes]
[36mjce-cassandra |[0m WARN [main] 2023-03-17 14:01:18,745 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK.
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,858 MonotonicClock.java:199 - Scheduling approximate time conversion task with an interval of 10000 milliseconds
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,868 MonotonicClock.java:335 - Scheduling approximate time-check task with a precision of 2 milliseconds
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,873 StartupChecks.java:147 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so
[36mjce-cassandra |[0m WARN [main] 2023-03-17 14:01:18,874 StartupChecks.java:187 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:18,881 SigarLibrary.java:44 - Initializing SIGAR library
[36mjce-cassandra |[0m WARN [main] 2023-03-17 14:01:18,897 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
[36mjce-cassandra |[0m WARN [main] 2023-03-17 14:01:18,898 StartupChecks.java:329 - Maximum number of memory map areas per process (vm.max_map_count) 262144 is too low, recommended value: 1048575, you can change it with sysctl.
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:20,009 Keyspace.java:386 - Creating replication strategy system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:20,142 ColumnFamilyStore.java:385 - Initializing system.IndexInfo
[33mjce-elastic |[0m [2023-03-17T14:01:20,683][WARN ][o.o.b.JNANatives ] [jce-elastic] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[33mjce-elastic |[0m [2023-03-17T14:01:20,699][WARN ][o.o.b.JNANatives ] [jce-elastic] This can result in part of the JVM being swapped out.
[33mjce-elastic |[0m [2023-03-17T14:01:20,701][WARN ][o.o.b.JNANatives ] [jce-elastic] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[33mjce-elastic |[0m [2023-03-17T14:01:20,701][WARN ][o.o.b.JNANatives ] [jce-elastic] These can be adjusted by modifying /etc/security/limits.conf, for example:
[33mjce-elastic |[0m # allow user 'opensearch' mlockall
[33mjce-elastic |[0m opensearch soft memlock unlimited
[33mjce-elastic |[0m opensearch hard memlock unlimited
[33mjce-elastic |[0m [2023-03-17T14:01:20,702][WARN ][o.o.b.JNANatives ] [jce-elastic] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"mlCommonsDashboards\" is disabled."}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSource\" is disabled."}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""}
[33mjce-elastic |[0m WARNING: A terminally deprecated method in java.lang.System has been called
[33mjce-elastic |[0m WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
[33mjce-elastic |[0m WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
[33mjce-elastic |[0m WARNING: System::setSecurityManager will be removed in a future release
[33mjce-elastic |[0m [2023-03-17T14:01:21,315][INFO ][o.o.n.Node ] [jce-elastic] version[2.6.0], pid[104], build[tar/7203a5af21a8a009aece1474446b437a3c674db6/2023-02-24T18:57:04.388618985Z], OS[Linux/5.10.76-linuxkit/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[33mjce-elastic |[0m [2023-03-17T14:01:21,315][INFO ][o.o.n.Node ] [jce-elastic] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[33mjce-elastic |[0m [2023-03-17T14:01:21,316][INFO ][o.o.n.Node ] [jce-elastic] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-686125586547143388, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx10240m, -XX:MaxDirectMemorySize=5368709120, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:21Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [49] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,alertingDashboards,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,securityAnalyticsDashboards,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,visualize,ganttChartDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,searchRelevanceDashboards,notificationsDashboards,queryWorkbenchDashboards,charts,visTypeTimeseries,visTypeVislib,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,observabilityDashboards,bfetch]"}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:21,966 ColumnFamilyStore.java:385 - Initializing system.batches
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:21,993 ColumnFamilyStore.java:385 - Initializing system.paxos
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:22,097 BufferPools.java:49 - Global buffer pool limit is 512.000MiB for chunk-cache and 128.000MiB for networking
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:22,155 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/paxos-b7b7f0c2fd0a34108c053ef614bb7c2d/nb-1-big (0.148KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:22,249 CacheService.java:100 - Initializing key cache with capacity of 100 MBs.
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:22Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:22Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.30.0.3:9200"}
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:22Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:22,853 CacheService.java:122 - Initializing row cache with capacity of 0 MBs
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:22,857 CacheService.java:151 - Initializing counter cache with capacity of 50 MBs
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:22,860 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:22,978 ColumnFamilyStore.java:385 - Initializing system.local
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:22,985 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-27-big (0.051KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:22,986 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-26-big (0.679KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,017 ColumnFamilyStore.java:385 - Initializing system.peers_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,036 ColumnFamilyStore.java:385 - Initializing system.peers
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,052 ColumnFamilyStore.java:385 - Initializing system.peer_events_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,069 ColumnFamilyStore.java:385 - Initializing system.peer_events
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,087 ColumnFamilyStore.java:385 - Initializing system.compaction_history
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:23,093 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/nb-5-big (0.673KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:23,106 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/nb-6-big (0.708KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,130 ColumnFamilyStore.java:385 - Initializing system.sstable_activity
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:23,233 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/nb-6-big (1.104KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:23,260 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/nb-5-big (0.227KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,285 ColumnFamilyStore.java:385 - Initializing system.size_estimates
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:23,289 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/nb-5-big (4.815KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,311 ColumnFamilyStore.java:385 - Initializing system.table_estimates
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:23,316 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-5-big (9.792KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,338 ColumnFamilyStore.java:385 - Initializing system.available_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,351 ColumnFamilyStore.java:385 - Initializing system.available_ranges
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,366 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,381 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,398 ColumnFamilyStore.java:385 - Initializing system.view_builds_in_progress
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,417 ColumnFamilyStore.java:385 - Initializing system.built_views
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,438 ColumnFamilyStore.java:385 - Initializing system.prepared_statements
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:23,450 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-5-big (2.138KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:23,454 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-6-big (2.108KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,502 ColumnFamilyStore.java:385 - Initializing system.repairs
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,565 QueryProcessor.java:106 - Initialized prepared statement caches with 15 MB
[33mjce-elastic |[0m [2023-03-17T14:01:23,678][WARN ][stderr ] [jce-elastic] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[33mjce-elastic |[0m [2023-03-17T14:01:23,678][WARN ][stderr ] [jce-elastic] SLF4J: Defaulting to no-operation (NOP) logger implementation
[33mjce-elastic |[0m [2023-03-17T14:01:23,679][WARN ][stderr ] [jce-elastic] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[33mjce-elastic |[0m [2023-03-17T14:01:23,720][INFO ][o.o.s.s.t.SSLConfig ] [jce-elastic] SSL dual mode is disabled
[33mjce-elastic |[0m [2023-03-17T14:01:23,721][INFO ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] OpenSearch Config path is /usr/share/opensearch/config
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:23,993 Keyspace.java:386 - Creating replication strategy system_schema params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,007 ColumnFamilyStore.java:385 - Initializing system_schema.keyspaces
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,029 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-18-big (0.192KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,036 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-17-big (0.276KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:3] 2023-03-17 14:01:24,038 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-19-big (0.116KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,052 ColumnFamilyStore.java:385 - Initializing system_schema.tables
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,063 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-18-big (0.353KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,063 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-17-big (3.565KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,082 ColumnFamilyStore.java:385 - Initializing system_schema.columns
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,108 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-17-big (7.390KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,118 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-18-big (0.117KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,140 ColumnFamilyStore.java:385 - Initializing system_schema.triggers
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,153 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,167 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb-6-big (0.050KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,183 ColumnFamilyStore.java:385 - Initializing system_schema.dropped_columns
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,193 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb-6-big (0.096KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,204 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb-5-big (0.109KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,222 ColumnFamilyStore.java:385 - Initializing system_schema.views
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,233 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,236 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb-6-big (0.050KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,279 ColumnFamilyStore.java:385 - Initializing system_schema.types
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,291 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,293 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb-6-big (0.050KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,316 ColumnFamilyStore.java:385 - Initializing system_schema.functions
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,337 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb-6-big (0.050KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,340 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,363 ColumnFamilyStore.java:385 - Initializing system_schema.aggregates
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,374 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb-6-big (0.050KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,376 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,403 ColumnFamilyStore.java:385 - Initializing system_schema.indexes
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:24,419 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb-5-big (0.059KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:24,421 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb-6-big (0.050KiB)
[33mjce-elastic |[0m [2023-03-17T14:01:24,494][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] JVM supports TLSv1.3
[33mjce-elastic |[0m [2023-03-17T14:01:24,510][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,734 SystemKeyspaceMigrator40.java:76 - system.peers_v2 table was empty, migrating legacy system.peers, if this fails you should fix the issue and then truncate system.peers_v2 to have it try again.
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,753 SystemKeyspaceMigrator40.java:100 - Migrating rows from legacy system.peers to system.peers_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,759 SystemKeyspaceMigrator40.java:119 - Migrated 0 rows from legacy system.peers to system.peers_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,759 SystemKeyspaceMigrator40.java:129 - system.peer_events_v2 table was empty, migrating legacy system.peer_events to system.peer_events_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,761 SystemKeyspaceMigrator40.java:152 - Migrated 0 rows from legacy system.peer_events to system.peer_events_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,762 SystemKeyspaceMigrator40.java:162 - system.transferred_ranges_v2 table was empty, migrating legacy system.transferred_ranges to system.transferred_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,765 SystemKeyspaceMigrator40.java:190 - Migrated 0 rows from legacy system.transferred_ranges to system.transferred_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,766 SystemKeyspaceMigrator40.java:200 - system.available_ranges_v2 table was empty, migrating legacy system.available_ranges to system.available_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,769 SystemKeyspaceMigrator40.java:226 - Migrated 0 rows from legacy system.available_ranges to system.available_ranges_v2
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,770 StorageService.java:836 - Populating token metadata from system tables
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:24,824 StorageService.java:830 - Token metadata: Normal Tokens:
[36mjce-cassandra |[0m /172.30.0.4:7000:[-8522239254575085352, -6743604953751498223, -5647563053516559934, -4556265942824347674, -3711889295333185918, -2271621065430602678, -1200120449090569866, -182269044348142850, 635402888264207943, 1711158769489787482, 2447074462200681123, 3526447269648282781, 4769967275328874753, 5737331518407410252, 7327336070921308890, 8946359366618352187]
[36mjce-cassandra |[0m
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:24Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.30.0.3:9200"}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,426 Keyspace.java:386 - Creating replication strategy k1 params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,433 ColumnFamilyStore.java:385 - Initializing k1.fruit
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:25,440 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/k1/fruit-931ef910c4c711ed95b9b3186b5c54af/nb-1-big (0.048KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,491 Keyspace.java:386 - Creating replication strategy janusgraph params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,494 ColumnFamilyStore.java:385 - Initializing janusgraph.edgestore
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,519 ColumnFamilyStore.java:385 - Initializing janusgraph.edgestore_lock_
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,542 ColumnFamilyStore.java:385 - Initializing janusgraph.graphindex
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,558 ColumnFamilyStore.java:385 - Initializing janusgraph.graphindex_lock_
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,577 ColumnFamilyStore.java:385 - Initializing janusgraph.janusgraph_ids
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,594 ColumnFamilyStore.java:385 - Initializing janusgraph.system_properties
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:25,622 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/janusgraph/system_properties-b46a73d0c47011ed9ae6414e35eb446a/nb-2-big (0.120KiB)
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:2] 2023-03-17 14:01:25,623 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/janusgraph/system_properties-b46a73d0c47011ed9ae6414e35eb446a/nb-1-big (0.387KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,649 ColumnFamilyStore.java:385 - Initializing janusgraph.system_properties_lock_
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,676 ColumnFamilyStore.java:385 - Initializing janusgraph.systemlog
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:25,704 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/janusgraph/systemlog-cb224fa0c47311ed84c1d12645e7285d/nb-1-big (0.092KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,746 ColumnFamilyStore.java:385 - Initializing janusgraph.txlog
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:25,759 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/janusgraph/txlog-cb066330c47311ed84c1d12645e7285d/nb-1-big (0.097KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,780 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,789 ColumnFamilyStore.java:385 - Initializing system_auth.network_permissions
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,812 ColumnFamilyStore.java:385 - Initializing system_auth.resource_role_permissons_index
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,826 ColumnFamilyStore.java:385 - Initializing system_auth.role_members
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,843 ColumnFamilyStore.java:385 - Initializing system_auth.role_permissions
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,885 ColumnFamilyStore.java:385 - Initializing system_auth.roles
[36mjce-cassandra |[0m INFO [SSTableBatchOpen:1] 2023-03-17 14:01:25,913 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/nb-1-big (0.100KiB)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,934 Keyspace.java:386 - Creating replication strategy system_distributed params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,938 ColumnFamilyStore.java:385 - Initializing system_distributed.parent_repair_history
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,960 ColumnFamilyStore.java:385 - Initializing system_distributed.repair_history
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,977 ColumnFamilyStore.java:385 - Initializing system_distributed.view_build_status
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:25,998 Keyspace.java:386 - Creating replication strategy system_traces params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,002 ColumnFamilyStore.java:385 - Initializing system_traces.events
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,026 ColumnFamilyStore.java:385 - Initializing system_traces.sessions
[36mjce-cassandra |[0m INFO [pool-3-thread-1] 2023-03-17 14:01:26,055 AutoSavingCache.java:177 - Completed loading (14 ms; 22 keys) KeyCache cache
[33mjce-elastic |[0m [2023-03-17T14:01:26,061][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] TLS Transport Client Provider : JDK
[33mjce-elastic |[0m [2023-03-17T14:01:26,061][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] TLS Transport Server Provider : JDK
[33mjce-elastic |[0m [2023-03-17T14:01:26,061][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] TLS HTTP Provider : JDK
[33mjce-elastic |[0m [2023-03-17T14:01:26,061][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[33mjce-elastic |[0m [2023-03-17T14:01:26,062][INFO ][o.o.s.s.DefaultSecurityKeyStore] [jce-elastic] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[33mjce-elastic |[0m [2023-03-17T14:01:26,087][INFO ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] Clustername: opensearch-cluster
[33mjce-elastic |[0m [2023-03-17T14:01:26,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[33mjce-elastic |[0m [2023-03-17T14:01:26,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[33mjce-elastic |[0m [2023-03-17T14:01:26,104][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[33mjce-elastic |[0m [2023-03-17T14:01:26,104][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[33mjce-elastic |[0m [2023-03-17T14:01:26,104][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[33mjce-elastic |[0m [2023-03-17T14:01:26,104][WARN ][o.o.s.OpenSearchSecurityPlugin] [jce-elastic] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,116 CommitLog.java:173 - Replaying /opt/cassandra/data/commitlog/CommitLog-7-1679059092483.log
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,183 CommitLogReader.java:256 - Finished reading /opt/cassandra/data/commitlog/CommitLog-7-1679059092483.log
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,195 CommitLog.java:175 - Log replay complete, 0 replayed mutations
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,196 StorageService.java:836 - Populating token metadata from system tables
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,205 StorageService.java:830 - Token metadata: Normal Tokens:
[36mjce-cassandra |[0m /172.30.0.4:7000:[-8522239254575085352, -6743604953751498223, -5647563053516559934, -4556265942824347674, -3711889295333185918, -2271621065430602678, -1200120449090569866, -182269044348142850, 635402888264207943, 1711158769489787482, 2447074462200681123, 3526447269648282781, 4769967275328874753, 5737331518407410252, 7327336070921308890, 8946359366618352187]
[36mjce-cassandra |[0m
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,407 ColumnFamilyStore.java:2242 - Truncating system.size_estimates
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,468 ColumnFamilyStore.java:2279 - Truncating system.size_estimates with truncatedAt=1679061686418
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,483 ColumnFamilyStore.java:2731 - Truncation is dropping 1 sstables and keeping 0 due to sstable.maxDataAge > truncatedAt
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,528 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/nb-5-big-Data.db
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,529 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/nb-5-big
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,530 FileUtils.java:545 - Deleting file during startup: /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/nb-5-big-Summary.db
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,538 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.199KiB (0%) on-heap, 0.000KiB (0%) off-heap
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,578 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/nb_txn_unknowncompactiontype_354b4050-c4cc-11ed-919f-19e783708468.log
[36mjce-cassandra |[0m INFO [PerDiskMemtableFlushWriter_0:1] 2023-03-17 14:01:26,611 Memtable.java:469 - Writing ***@***.***(0.429KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
[36mjce-cassandra |[0m INFO [PerDiskMemtableFlushWriter_0:1] 2023-03-17 14:01:26,619 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-28-big-Data.db (0.224KiB) for commitlog position CommitLogPosition(segmentId=1679061678840, position=31288)
[36mjce-cassandra |[0m INFO [MemtableFlushWriter:1] 2023-03-17 14:01:26,680 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_35546810-c4cc-11ed-919f-19e783708468.log
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,712 ColumnFamilyStore.java:2306 - Truncate of system.size_estimates is complete
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,713 ColumnFamilyStore.java:2242 - Truncating system.table_estimates
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,718 ColumnFamilyStore.java:2279 - Truncating system.table_estimates with truncatedAt=1679061686718
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,726 ColumnFamilyStore.java:2731 - Truncation is dropping 1 sstables and keeping 0 due to sstable.maxDataAge > truncatedAt
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,740 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-5-big-Data.db
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,740 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-5-big
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,740 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.576KiB (0%) on-heap, 0.000KiB (0%) off-heap
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,741 FileUtils.java:545 - Deleting file during startup: /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-5-big-Summary.db
[36mjce-cassandra |[0m INFO [NonPeriodicTasks:1] 2023-03-17 14:01:26,747 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb_txn_unknowncompactiontype_35700660-c4cc-11ed-919f-19e783708468.log
[36mjce-cassandra |[0m INFO [PerDiskMemtableFlushWriter_0:2] 2023-03-17 14:01:26,749 Memtable.java:469 - Writing ***@***.***(0.091KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
[36mjce-cassandra |[0m INFO [PerDiskMemtableFlushWriter_0:2] 2023-03-17 14:01:26,750 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-29-big-Data.db (0.063KiB) for commitlog position CommitLogPosition(segmentId=1679061678840, position=31481)
[36mjce-cassandra |[0m INFO [MemtableFlushWriter:2] 2023-03-17 14:01:26,773 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_35725050-c4cc-11ed-919f-19e783708468.log
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:26,776 ColumnFamilyStore.java:2306 - Truncate of system.table_estimates is complete
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,005 QueryProcessor.java:150 - Preloaded 45 prepared statements
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,006 StorageService.java:734 - Cassandra version: 4.0.0
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,006 StorageService.java:735 - CQL version: 3.4.5
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,006 StorageService.java:736 - Native protocol supported versions: 3/v3, 4/v4, 5/v5, 6/v6-beta (default: 5/v5)
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,086 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 196 MB and a resize interval of 60 minutes
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,090 StorageService.java:753 - Loading persisted ring state
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,090 StorageService.java:836 - Populating token metadata from system tables
[33mjce-elastic |[0m [2023-03-17T14:01:27,163][INFO ][o.o.p.c.PluginSettings ] [jce-elastic] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14:01:27,391 InboundConnectionInitiator.java:127 - Listening on address: (/172.30.0.4:7000), nic: eth0, encryption: unencrypted
[35mopensearch-dashboards |[0m ***@***.***":"2023-03-17T14:01:27Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.30.0.3:9200"}
[36mjce-cassandra |[0m INFO [main] 2023-03-17 14
Hello Phate,
Do you have some process to check whether the mixedIndex created from Janusgraph hit the OpenSearch backend? In OpenSearch, is there a way to query the index?
Thanks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hi @yingjie668
there are many
OutOfMemoryErrorin your log. Are you sure your machine has enough memory to execute all containers?You can remove dashboard container first, JanusGraph doesn't need it.
Maybe you need to see Docker Compose documentation to limit memory for Opensearch or Cassandra.