Rally error with custom track

rally_out.log

[elastic@localhost rally]$ rally --pipeline=benchmark-only --target-hosts=localhost:9200 --track=elasticlogs --challenge=test
Auto-updating Rally

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Writing logs to /home/elastic/.rally/benchmarks/races/2016-12-05-18-55-21/local/logs/rally_out.log
[INFO] Racing on track [elasticlogs], challenge [test] and car [external]


************************************************************************
************** WARNING: A dark dungeon lies ahead of you  **************
************************************************************************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

************************************************************************
****** Use this pipeline only if you are aware of the tradeoffs.  ******
*************************** Watch your step! ***************************
************************************************************************

[WARNING] None of the types for index [elasticlogs] defines documents. Please check that you either don't want to index data or parameter sources are defined for indexing.
[ERROR] Cannot race. ('Could not execute benchmark', RequestError(400, 'mapper_parsing_exception', {'status': 400, 'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters:  [mappings : {logs={properties={@timestamp={type=date}, useragent={properties={name={type=keyword, ignore_above=256}, os={type=keyword, ignore_above=256}, os_name={type=keyword, ignore_above=256}}}, agent={type=keyword, ignore_above=256}, clientip={type=ip}, response={type=short}, geoip={properties={country_name={type=keyword}, location={type=geo_point}}}, verb={type=keyword, ignore_above=256}, bytes={type=integer}, referrer={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}, @message={index=false, type=text}, httpversion={type=keyword, ignore_above=256}, request={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}}, _all={enabled=false}}}] [settings : {index.refresh_interval=5s, index.codec=best_compression}] [template : elasticlogs*] [aliases : {}]'}], 'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters:  [mappings : {logs={properties={@timestamp={type=date}, useragent={properties={name={type=keyword, ignore_above=256}, os={type=keyword, ignore_above=256}, os_name={type=keyword, ignore_above=256}}}, agent={type=keyword, ignore_above=256}, clientip={type=ip}, response={type=short}, geoip={properties={country_name={type=keyword}, location={type=geo_point}}}, verb={type=keyword, ignore_above=256}, bytes={type=integer}, referrer={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}, @message={index=false, type=text}, httpversion={type=keyword, ignore_above=256}, request={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}}, _all={enabled=false}}}] [settings : {index.refresh_interval=5s, index.codec=best_compression}] [template : elasticlogs*] [aliases : {}]'}})).

Getting further help:
*********************
* Check the log file at /home/elastic/.rally/benchmarks/races/2016-12-05-18-55-21/local/logs/rally_out.log for errors.
* Read the documentation at https://esrally.readthedocs.io/en/latest/
* Ask a question in the forum at https://discuss.elastic.co/c/rally
* Raise an issue at https://github.com/elastic/rally/issues and include the log file in /home/elastic/.rally/benchmarks/races/2016-12-05-18-55-21/local/logs/rally_out.log.

-------------------------------
[INFO] FAILURE (took 1 seconds)
-------------------------------
[elastic@localhost rally]$ 


[elastic@localhost rally]$ cat /home/elastic/.rally/benchmarks/races/2016-12-05-18-55-21/local/logs/rally_out.log
2016-12-05 18:55:21,887 rally.main INFO Rally version [0.4.6.dev0 (git revision: 40dd4fc)]
2016-12-05 18:55:21,887 rally.main INFO Command line arguments: Namespace(advanced_config=False, assume_defaults=False, car='defaults', challenge='test', client_options='timeout:60000,request_timeout:60000', cluster_health='green', configuration_name=None, data_paths=None, distribution_repository='release', distribution_version='', effective_start_date=datetime.datetime(2016, 12, 5, 18, 55, 21, 880865), laps=1, logging='file', offline=False, override_src_dir=None, pipeline='benchmark-only', preserve_install='False', quiet=False, report_file='', report_format='markdown', revision='current', subcommand=None, target_hosts='localhost:9200', telemetry='', test_mode=False, track='elasticlogs', track_repository='default', user_tag='')
2016-12-05 18:55:21,887 rally.net INFO Rally connects directly to the Internet (no proxy support).
2016-12-05 18:55:22,26 rally.main INFO Detected a working Internet connection.
2016-12-05 18:55:22,78 rally.process INFO Skipping myself (PID [14505]).
2016-12-05 18:55:22,173 rally.racecontrol INFO User specified pipeline [benchmark-only].
2016-12-05 18:55:22,173 rally.metrics INFO Creating in-memory metrics store
2016-12-05 18:55:22,174 rally.metrics INFO Opening metrics store for invocation=[20161205T185521Z], track=[elasticlogs], challenge=[test], car=[external]
2016-12-05 18:55:22,174 rally.mechanic INFO Starting engine.
2016-12-05 18:55:22,174 rally.client INFO Creating ES client connected to [{'port': '9200', 'host': 'localhost'}] with options [{'request_timeout': 60000, 'timeout': 60000}]
2016-12-05 18:55:22,178 rally.launcher INFO Distribution version was not specified by user. Rally-determined version is [5.0.1]
2016-12-05 18:55:22,709 rally.track INFO Checking out [master] in [/home/elastic/.rally/benchmarks/tracks/default] for distribution version [5.0.1].
2016-12-05 18:55:22,714 rally.track INFO Rebasing on [master] in [/home/elastic/.rally/benchmarks/tracks/default] for distribution version [5.0.1].
2016-12-05 18:55:22,758 rally.track INFO Reading track specification file [/home/elastic/.rally/benchmarks/tracks/default/elasticlogs/track.json].
2016-12-05 18:55:22,763 rally.track INFO Final rendered track for '/home/elastic/.rally/benchmarks/tracks/default/elasticlogs/track.json': {
  "meta": {
    "short-description": "Data generator for long running bulk indexing benchmarks",
    "description": "This track generates data based on a selected subset of traffic to the elastic.co website, which has been anonymized and post-processed."
  },
  "indices": [
    {
      "name": "elasticlogs",
      "types": [
        {
          "name": "logs",
          "mapping": "mappings.json"
        }
      ]
    }
  ],
  "#COMMENT": "'operations' just define all possible operations but this is not the actual execution schedule. The execution is defined in the 'challenges' block and it just refers to the defined operations. The intention between this separation is to allow reuse of operations",
  "operations": [
    {
      "name": "index-append-1000",
      "operation-type": "index",
      "param-source": "elasticlogs",
      "bulk-size": 1000
    },
    {
      "name": "index-append-500",
      "operation-type": "index",
      "param-source": "elasticlogs",
      "bulk-size": 500
    },
    {
      "name": "index-append-250",
      "operation-type": "index",
      "param-source": "elasticlogs",
      "bulk-size": 250
    },
    {
      "name": "index-append-125",
      "operation-type": "index",
      "param-source": "elasticlogs",
      "bulk-size": 125
    },
    {
      "name": "index-append-1000-elasticlogs_write",
      "operation-type": "index",
      "param-source": "elasticlogs",
      "index": "elasticlogs_write",
      "bulk-size": 1000
    },
    {
      "name": "create_elasticlogs_write",
      "operation-type": "createindex",
      "index_name": "elasticlogs-000001",
      "alias": "elasticlogs_write",
      "index_template_body": {
        "template": "elasticlogs*",
        "settings": {
          "index.refresh_interval": "5s",
          "index.codec": "best_compression"
        },
        "mappings": {
          "logs": {
            "properties": {
              "@timestamp": { "type": "date" },
              "@message": { "type": "text", "index": false },
              "agent": { "type": "keyword", "ignore_above": 256 },
              "bytes": { "type": "integer" },
              "clientip": { "type": "ip" },
              "httpversion": { "type": "keyword", "ignore_above": 256 },
              "response": { "type": "short" },
              "verb": { "type": "keyword", "ignore_above": 256 },
              "geoip" : {
                "properties" : {
                  "country_name" : { "type": "keyword" },
                  "location" : { "type": "geo_point" }
                }
              },
              "useragent": {
                "properties": {
                  "name": { "type": "keyword", "ignore_above": 256 },
                  "os": { "type": "keyword", "ignore_above": 256 },
                  "os_name": { "type": "keyword", "ignore_above": 256 }
                }
              },
              "request": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": { "ignore_above": 256, "type": "keyword" }
                }
              },
              "referrer": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": { "ignore_above": 256, "type": "keyword" }
                }
              }
            },
            "_all": { "enabled": false }
          }
        },
        "aliases": {}
      },
      "index_template_name": "elasticlogs"
    },
    {
      "name": "rollover_elasticlogs_write",
      "operation-type": "rollover",
      "alias": "elasticlogs_write",
      "body": {
        "conditions": {
          "max_age":   "1d",
          "max_docs":  1000000
        }
      }
    },
    {
      "name": "scanscroll_new_elasticlogs_1m",
      "operation-type": "scanscroll",
      "index": "elasticlogs-*",
      "query": {
        "query": {
          "range" : {
            "@timestamp" : {
              "gte" : "now-1m",
              "lt" :  "now"
            }
          }
        }
      }
    },
    {
      "name": "scanscroll_new_elasticlogs_5m",
      "operation-type": "scanscroll",
      "index": "elasticlogs-*",
      "query": {
        "query": {
          "range" : {
            "@timestamp" : {
              "gte" : "now-5m",
              "lt" :  "now"
            }
          }
        }
      }
    }
  ],
  "challenges": [
    {
      "name": "append-no-conflicts",
      "description": "Indexes using 8 clients and runs indexing with a batch size of 1000 for 10 minutes. IDs are autogenerated by Elasticsearcg, meaning there are no conflicts.",
      "index-settings": {
        "index.number_of_replicas": 0
      },
      "schedule": [
        {
          "operation": "index-append-1000",
          "warmup-time-period": 0,
          "time-period": 600,
          "clients": 8
        }
      ]
    },
    {
      "name": "append-no-conflicts-varying-bulk-size",
      "description": "Indexes using 8 clients and runs 4 different batchsizes for 5 minutes each. IDs are autogenerated by Elasticsearcg, meaning there are no conflicts.",
      "index-settings": {
        "index.number_of_replicas": 0
      },
      "schedule": [
        {
          "operation": "index-append-125",
          "warmup-time-period": 0,
          "time-period": 300,
          "clients": 8
        },
        {
          "operation": "index-append-250",
          "warmup-time-period": 0,
          "time-period": 300,
          "clients": 8
        },
        {
          "operation": "index-append-500",
          "warmup-time-period": 0,
          "time-period": 300,
          "clients": 8
        },
        {
          "operation": "index-append-1000",
          "warmup-time-period": 0,
          "time-period": 300,
          "clients": 8
        }
      ]
    },
    {
      "name": "append-no-conflicts-index-only",
      "description": "Indexes using 8 threads during 30 minutes. IDs are autogenerated by Elasticsearcg, meaning there are no conflicts.",
      "index-settings": {
        "index.number_of_replicas": 0
      },
      "schedule": [
        {
          "operation": "index-append-1000",
          "warmup-time-period": 0,
          "time-period": 1800,
          "clients": 8
        }
      ]
    },
    {
      "name": "test",
      "description": "Runs through and tests all parameter sources and runners.",
      "index-settings": {
        "index.number_of_replicas": 0
      },
      "schedule": [
        {
          "operation": "create_elasticlogs_write",
          "clients": 1,
          "warmup-iterations": 0,
          "iterations": 1
        },
        {
          "parallel": {
            "warmup-time-period": 0,
            "time-period": 600,
            "tasks": [
              {
                "operation": "index-append-1000-elasticlogs_write",
                "target-throughput": 10,
                "clients": 8
              },
              {
                "operation": "rollover_elasticlogs_write",
                "target-throughput": 0.01666667
              },
              {
                "operation": "scanscroll_new_elasticlogs_1m",
                "target-throughput": 0.01666667
              }
            ]
          }
        }
      ]
    }
  ]
}

2016-12-05 18:55:22,769 rally.track WARNING None of the types for index [elasticlogs] defines documents. Please check that you either don't want to index data or parameter sources are defined for indexing.
2016-12-05 18:55:22,769 rally.track INFO Using user-provided operation type [createindex] for operation [create_elasticlogs_write].
2016-12-05 18:55:22,769 rally.track INFO Using user-provided operation type [rollover] for operation [rollover_elasticlogs_write].
2016-12-05 18:55:22,769 rally.track INFO Using user-provided operation type [scanscroll] for operation [scanscroll_new_elasticlogs_1m].
2016-12-05 18:55:22,769 rally.track INFO Using user-provided operation type [scanscroll] for operation [scanscroll_new_elasticlogs_5m].
2016-12-05 18:55:22,770 rally.metrics INFO Creating in-memory race store
2016-12-05 18:55:22,812 rally.mechanic INFO Notifying cluster of benchmark start.
2016-12-05 18:55:22,812 rally.telemetry INFO Benchmark start
2016-12-05 18:55:22,862 rally.driver INFO Preparing track
2016-12-05 18:55:22,862 rally.track INFO Track [elasticlogs] does not specify a source root URL. Assuming data are available locally.
2016-12-05 18:55:22,862 rally.track INFO Type [logs] in index [elasticlogs] does not define a document archive. No data are indexed from a file for this type.
2016-12-05 18:55:22,862 rally.driver INFO Benchmark is about to start.
2016-12-05 18:55:22,863 rally.client INFO Creating ES client connected to [{'port': '9200', 'host': 'localhost'}] with options [{'request_timeout': 60000, 'timeout': 60000}]
2016-12-05 18:55:22,863 rally.metrics INFO Opening metrics store for invocation=[20161205T185521Z], track=[elasticlogs], challenge=[test], car=[external]
2016-12-05 18:55:22,867 rally.driver WARNING Index [elasticlogs] already exists. Deleting it.
2016-12-05 18:55:22,889 rally.driver INFO Creating index [elasticlogs]
2016-12-05 18:55:22,942 rally.driver INFO create mapping for type [logs] in index [elasticlogs] with content:
{
  "template": "elasticlogs*",
  "settings": {
    "index.refresh_interval": "5s",
    "index.codec": "best_compression"
  },
  "mappings": {
    "logs": {
      "properties": {
        "@timestamp": { "type": "date" },
        "@message": { "type": "text", "index": false },
        "agent": { "type": "keyword", "ignore_above": 256 },
        "bytes": { "type": "integer" },
        "clientip": { "type": "ip" },
        "httpversion": { "type": "keyword", "ignore_above": 256 },
        "response": { "type": "short" },
        "verb": { "type": "keyword", "ignore_above": 256 },
        "geoip" : {
          "properties" : {
            "country_name" : { "type": "keyword" },
            "location" : { "type": "geo_point" }
          }
        },
        "useragent": {
          "properties": {
            "name": { "type": "keyword", "ignore_above": 256 },
            "os": { "type": "keyword", "ignore_above": 256 },
            "os_name": { "type": "keyword", "ignore_above": 256 }
          }
        },
        "request": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": { "ignore_above": 256, "type": "keyword" }
          }
        },
        "referrer": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": { "ignore_above": 256, "type": "keyword" }
          }
        }
      },
      "_all": { "enabled": false }
    }
  },
  "aliases": {}
}
2016-12-05 18:55:22,945 elasticsearch WARNING PUT /elasticlogs/_mapping/logs [status:400 request:0.003s]
2016-12-05 18:55:22,945 rally.driver ERROR Main driver encountered a fatal exception. Shutting down.
Traceback (most recent call last):
  File "/home/elastic/rally/esrally/driver/driver.py", line 138, in receiveMessage
    self.start_benchmark(msg, sender)
  File "/home/elastic/rally/esrally/driver/driver.py", line 185, in start_benchmark
    setup_index(self.es, index, challenge.index_settings)
  File "/home/elastic/rally/esrally/driver/driver.py", line 462, in setup_index
    body=json.loads(mappings))
  File "/usr/lib/python3.4/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python3.4/site-packages/elasticsearch/client/indices.py", line 291, in put_mapping
    '_mapping', doc_type), params=params, body=body)
  File "/usr/lib/python3.4/site-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/lib/python3.4/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/lib/python3.4/site-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [mappings : {logs={properties={@timestamp={type=date}, useragent={properties={name={type=keyword, ignore_above=256}, os={type=keyword, ignore_above=256}, os_name={type=keyword, ignore_above=256}}}, agent={type=keyword, ignore_above=256}, clientip={type=ip}, response={type=short}, geoip={properties={country_name={type=keyword}, location={type=geo_point}}}, verb={type=keyword, ignore_above=256}, bytes={type=integer}, referrer={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}, @message={index=false, type=text}, httpversion={type=keyword, ignore_above=256}, request={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}}, _all={enabled=false}}}] [settings : {index.refresh_interval=5s, index.codec=best_compression}] [template : elasticlogs*] [aliases : {}]')
2016-12-05 18:55:22,947 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/home/elastic/rally/esrally/rally.py", line 459, in dispatch_sub_command
    racecontrol.run(cfg)
  File "/home/elastic/rally/esrally/racecontrol.py", line 238, in run
    raise e
  File "/home/elastic/rally/esrally/racecontrol.py", line 235, in run
    pipeline(cfg)
  File "/home/elastic/rally/esrally/racecontrol.py", line 48, in __call__
    self.target(cfg)
  File "/home/elastic/rally/esrally/racecontrol.py", line 183, in benchmark_only
    return race(Benchmark(cfg, mechanic.create(cfg, metrics_store, external=True), metrics_store), cfg)
  File "/home/elastic/rally/esrally/racecontrol.py", line 157, in race
    benchmark.run(lap)
  File "/home/elastic/rally/esrally/racecontrol.py", line 81, in run
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ('Could not execute benchmark', RequestError(400, 'mapper_parsing_exception', {'status': 400, 'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters:  [mappings : {logs={properties={@timestamp={type=date}, useragent={properties={name={type=keyword, ignore_above=256}, os={type=keyword, ignore_above=256}, os_name={type=keyword, ignore_above=256}}}, agent={type=keyword, ignore_above=256}, clientip={type=ip}, response={type=short}, geoip={properties={country_name={type=keyword}, location={type=geo_point}}}, verb={type=keyword, ignore_above=256}, bytes={type=integer}, referrer={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}, @message={index=false, type=text}, httpversion={type=keyword, ignore_above=256}, request={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}}, _all={enabled=false}}}] [settings : {index.refresh_interval=5s, index.codec=best_compression}] [template : elasticlogs*] [aliases : {}]'}], 'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters:  [mappings : {logs={properties={@timestamp={type=date}, useragent={properties={name={type=keyword, ignore_above=256}, os={type=keyword, ignore_above=256}, os_name={type=keyword, ignore_above=256}}}, agent={type=keyword, ignore_above=256}, clientip={type=ip}, response={type=short}, geoip={properties={country_name={type=keyword}, location={type=geo_point}}}, verb={type=keyword, ignore_above=256}, bytes={type=integer}, referrer={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}, @message={index=false, type=text}, httpversion={type=keyword, ignore_above=256}, request={norms=false, fields={keyword={type=keyword, ignore_above=256}}, type=text}}, _all={enabled=false}}}] [settings : {index.refresh_interval=5s, index.codec=best_compression}] [template : elasticlogs*] [aliases : {}]'}}))
2016-12-05 18:55:22,948 rally.main INFO Attempting to shutdown internal actor system.
2016-12-05 18:55:23,122 rally.main INFO Shutdown completed.