cdahlqvist / commands.txt

Last active June 4, 2024 09:20

Sequence of commands to test forced shard allocation awareness

	# Check if the allocation aareness settings can be seen through _cluster/settings
	GET /_cluster/settings

	# Verify that the specified node attributes are present
	GET /_cat/nodeattrs

	# Create 4 test indices with default settings - 1 primary shard and 1 replica shard
	PUT /testindex1

	PUT /testindex2

cdahlqvist / docker-compose.yml

Created June 4, 2024 09:03

Docker compose file with forced allocation awareness enabled

	version: "2.2"

	services:
	setup:
	image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
	volumes:
	- certs:/usr/share/elasticsearch/config/certs
	user: "0"
	command: >
	bash -c '

cdahlqvist / restore_snapshot.sh

Created January 24, 2019 20:10

cdahlqvist / create_repositories.sh

Created November 26, 2018 11:48

Frozen indices benchmark

	#/bin/bash

	echo $(date) "Create snapshot repositories"

	curl -X PUT "localhost:9200/_snapshot/elasticlogs-nofm" -H 'Content-Type: application/json' -d'
	{
	"type": "fs",
	"settings": {
	"location": "/data/snapshots/elasticlogs-nofm"
	}

cdahlqvist / ccr_watch

Created November 7, 2018 20:25

cdahlqvist / filter_logs.conf

Created October 23, 2018 20:34

HTTP log replayer

	input {
	stdin {}
	}

	filter {
	grok {
	match => { "message" => [ '%{IP:ip}" %{GREEDYDATA:a}',
	'%{IP:ip1}, %{IP:ip}" %{GREEDYDATA:a}' ] }
	}

cdahlqvist / epoch_prefixed_md5_identifier.conf

Last active April 22, 2025 03:34

Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date

cdahlqvist / rally_split_indexing_challenge.json

Created September 16, 2018 08:41

Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput

	{% set p_bulk_indexing_clients = (bulk_indexing_clients \| default(20)) %}
	{% set p_duration = bulk_indexing_duration \| default(900) %}

	{
	"name": "split_indexing",
	"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
	"meta": {
	"client_count": {{ p_bulk_indexing_clients }},
	"benchmark_type": "split-indexing"
	},

cdahlqvist / recover_error_message.txt

Created July 20, 2018 12:23

	[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
	org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner

cdahlqvist / cold_index_stats.md

Last active July 20, 2018 11:56

250GB shards

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_all" : {

Christian Dahlqvist cdahlqvist