multi datacenter best practices

multidc-best-practices.md

• application considerations
  • application servers usually in each DC talking to local C* nodes
  • DCAwareRoundRobin load balancing policy (Java, but each language driver has analog)
    • along with LOCAL_* consistency levels, keeps you within a DC
    • can enable using remote host(s) for fault tolerance
• when adding a dc
  • don't forget
    • open relevant ports across data centers
    • C* nodes in cluster fully connected
    • inter dc bandwidth can take ingest rate
  • if it’s the second dc
    • make sure SimpleStrategy->NetworkTopologyStrategy before adding
  • launch the nodes with auto_bootstrap=false
  • update the replication factor
  • double check settings for
    • streaming socket timeout (verify that it's > 0)
    • inter dc outgoing streaming throttle (also useful for repairs)
  • nodetool rebuild
• cross region/cloud or on-prem/cloud mix
  • may need to use broadcast address, broadcast rpc
• Relevant options
  • internode_encryption: dc (esp over shared links)
  • internode_compression: dc
  • inter_dc_tcp_nodelay
  • inter_dc_stream_throughput_outbound_megabits_per_sec
  • streaming_socket_timeout_in_ms
  • phi_convict_threshold, consider increasing from default 8 to 12