Skip to content

Instantly share code, notes, and snippets.

@nmattam
Last active April 4, 2022 12:09
Show Gist options
  • Save nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f to your computer and use it in GitHub Desktop.
Save nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f to your computer and use it in GitHub Desktop.
[Splunk] Automate "Apply Changes" for Monitoring Console
# Create DMC groups
# This is basically updating the system/local/distsearch.conf
# I got all the IPs used below from /servicesNS/admin/-/search/distributed/peers. I used DNS names for search head (I had only one SH) and cluster master. So it was easy to separate the indexer IPs.
# 8089 is my API PORT
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_search_head -d member=<search_head_member1:8089> -d member=<search_head_member2:8089> -d default=false
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_cluster_master -d member=<cluster_master:8089> -d default=false
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_indexer -d member=<indexer1:8089> -d member=<indexer2:8089> -d default=true # not idea why default=true for indexers
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_indexerclustergroup_<CLUSTER_LABEL> -d member=<peer1:8089> -d member=<peer2:8089> -d default=false # Configure all peers in the cluster.
# Second time you run the above commands, you will run into an ERROR that says that the group already exists.
# So if the group exists, hit the "edit" endpoint to update the list of peers in a group.
# curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups/<dmc_group_name>/edit -d member=<member1:8089> -d member=<member2:8089> -d default=<true/false> # true for indexers, false for everything else
# Now configure Monitoring Console.
# Update the splunk_monitoring_console_assets.conf
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/splunk_monitoring_console/configs/conf-splunk_monitoring_console_assets/settings -d configuredPeers=<comma separated lists of all the peers> -d output_mode=json -d eai:appName=splunk_monitoring_console -d eai:userName=nobody
# Kicks off the search to build the assets.
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC+Asset+-+Build+Full/dispatch -d output_mode=json -d trigger_actions=true -d dispatch.auto_cancel=30 -d dispatch.buckets=300 -d dispatch.enablePreview=true
# Finally update the app.
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/system/apps/local/splunk_monitoring_console -d output_mode=json -d author=Splunk -d check_for_updates=1 -d configured=1 -d description=The+Splunk+Monitoring+Console+application+gives+you+insight+into+your+Splunk+deployment. -d label=Monitoring+Console -d version=<Splunk Version> -d visible=1
# There was another endpoint that was hit to update the nav/default.xml to reveal new distributed dashboards. The request body
# had the entire xml, hence I landed it via Chef using the cerner_splunk cookbook apps hash (https://github.com/cerner/cerner_splunk/blob/stable/docs/databags.md#apps-hash). But this was the content
<nav color='#65A637'>
<view name='monitoringconsole_overview' default='true' />
<view name='monitoringconsole_check' />
<view name='monitoringconsole_instances' />
<collection label='Indexing'>
<collection label='Performance'>
<view name='indexing_performance_instance' />
<view name='indexing_performance_deployment' />
</collection>
<collection label='Indexer Clustering'>
<!--<a href='Clustering'>Indexer Clustering:Status</a>-->
<view name='indexer_clustering_status' />
<view name='indexer_clustering_service_activity' />
</collection>
<collection label='Indexes and Volumes'>
<view name='indexes_and_volumes_instance' />
<view name='indexes_and_volumes_deployment' />
<view name='index_detail_instance' />
<view name='index_detail_deployment' />
<view name='volume_detail_instance' />
<view name='volume_detail_deployment' />
</collection>
<collection label='Inputs'>
<view name='http_event_collector_instance' />
<view name='http_event_collector_deployment' />
<view name='splunk_tcpin_performance_instance' />
<view name='splunk_tcpin_performance_deployment' />
<view name='data_quality' />
</collection>
<collection label='License Usage'>
<view name='license_usage_today' />
<view name='license_usage_30days' />
</collection>
</collection>
<collection label='Search'>
<collection label='Activity'>
<view name='search_activity_instance' />
<view name='search_activity_deployment' />
<view name='search_usage_statistics_instance' />
<view name='search_usage_statistics_deployment' />
</collection>
<collection label='Distributed Search'>
<view name='distributed_search_instance' />
<view name='distributed_search_deployment' />
</collection>
<collection label='Search Head Clustering'>
<view name='shc_status_and_conf' />
<view name='shc_conf_rep' />
<view name='shc_artifact_replication' />
<view name='shc_scheduler_delegation_statistics' />
<view name='shc_app_deployment' />
</collection>
<collection label='Scheduler Activity'>
<view name='scheduler_activity_instance' />
<view name='scheduler_activity_deployment' />
</collection>
<collection label='KV Store'>
<view name='kv_store_instance' />
<view name='kv_store_deployment' />
</collection>
</collection>
<collection label='Resource Usage'>
<view name='resource_usage_instance' />
<view name='resource_usage_machine' />
<view name='resource_usage_deployment' />
</collection>
<collection label='Forwarders'>
<view name='forwarder_instance' />
<view name='forwarder_deployment' />
</collection>
<collection label='Settings'>
<view name='monitoringconsole_configure' />
<view name='monitoringconsole_forwarder_setup' />
<view name='monitoringconsole_alerts_setup' />
<view name='monitoringconsole_overview_preferences'/>
<view name='monitoringconsole_check_list' />
</collection>
<a href='search'>Run a Search</a>
</nav>
@Rohlik
Copy link

Rohlik commented Jun 28, 2021

It doesn't work anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment