Skip to content

Instantly share code, notes, and snippets.

@bruno-garcia

bruno-garcia/Reproducing MongoDB C# driver issue

Created August 25, 2016 12:52
Show Gist options
  • Save bruno-garcia/d4509174d030b0a4db538e0b10c83388 to your computer and use it in GitHub Desktop.
Save bruno-garcia/d4509174d030b0a4db538e0b10c83388 to your computer and use it in GitHub Desktop.
Download ZIP
Reproducing MongoDB C# driver issue
Raw
Reproducing MongoDB C# driver issue
> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:30:34Z"),
"myState" : 2,
"syncingTo" : "abc-mdb-node1:27017",
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 112,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:30:32Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:30:32Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1471973280, 1),
"electionDate" : ISODate("2016-08-23T17:28:00Z")
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 110,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:30:32Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:30:33Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 112,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"infoMessage" : "syncing to: abc-mdb-node1:27017",
"self" : true
}
],
"ok" : 1
}
>
Console output:
abc-mdb-node1:27017:PRIMARY, Self: true
abc-mdb-node2:27017:SECONDARY, Ping: 0
abc-mdb-node3:27017:SECONDARY, Ping: 0
Stop node 3 (with systemd):
ReplicaSet:PRIMARY> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:32:31Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 372,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"electionTime" : Timestamp(1471973280, 1),
"electionDate" : ISODate("2016-08-23T17:28:00Z"),
"self" : true
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 322,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:32:31Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:32:31Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:32:30Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:31:26Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
}
],
"ok" : 1
}
ReplicaSet:PRIMARY>
Console output:
abc-mdb-node1:27017:PRIMARY, Self: true
abc-mdb-node2:27017:SECONDARY, Ping: 0
abc-mdb-node3:27017:(not reachable/healthy), Ping: 0
Stop node 2:
ReplicaSet:SECONDARY> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:33:28Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 429,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"self" : true
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:33:27Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:33:21Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:33:25Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:31:26Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
}
],
"ok" : 1
}
ReplicaSet:SECONDARY>
Console output after timeout (30 seconds):
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node1:27017" }", EndPoint: "Unspecified/abc-mdb-node1:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 2]" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node2:27017" }", EndPoint: "Unspecified/abc-mdb-node2:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.12:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node3:27017" }", EndPoint: "Unspecified/abc-mdb-node3:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.13:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }] }.
Stop node 1
No console to run rs.status anymore
App:
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node1:27017" }", EndPoint: "Unspecified/abc-mdb-node1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.11:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node2:27017" }", EndPoint: "Unspecified/abc-mdb-node2:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.12:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node3:27017" }", EndPoint: "Unspecified/abc-mdb-node3:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.13:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }] }.
Start node 1.
Console:
ReplicaSet:SECONDARY> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:36:07Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 18,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"self" : true
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:36:06Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:36:06Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0
}
],
"ok" : 1
}
ReplicaSet:SECONDARY>
App output:
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node1:27017" }", EndPoint: "Unspecified/abc-mdb-node1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.11:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node2:27017" }", EndPoint: "Unspecified/abc-mdb-node2:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.12:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node3:27017" }", EndPoint: "Unspecified/abc-mdb-node3:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.13:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }] }.
Bring back node 2 (At which point the replica set is healthy again after a couple seconds):
Shell:
ReplicaSet:SECONDARY> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:38:03Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 134,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"electionTime" : Timestamp(1471973871, 1),
"electionDate" : ISODate("2016-08-23T17:37:51Z"),
"self" : true
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 17,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:38:02Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:38:02Z"),
"pingMs" : 166,
"lastHeartbeatMessage" : "syncing to: abc-mdb-node1:27017",
"syncingTo" : "abc-mdb-node1:27017"
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:38:01Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0
}
],
"ok" : 1
}
ReplicaSet:PRIMARY>
The app, still off, although showing 1 less node in the stacktrace:
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node1:27017" }", EndPoint: "Unspecified/abc-mdb-node1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.11:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node2:27017" }", EndPoint: "Unspecified/abc-mdb-node2:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 2]" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node3:27017" }", EndPoint: "Unspecified/abc-mdb-node3:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.13:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }] }.
Start node 3. Now all nodes are up.
Shell:
ReplicaSet:PRIMARY> rs.status()
{
"set" : "ReplicaSet",
"date" : ISODate("2016-08-23T17:40:01Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "abc-mdb-node1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 252,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"electionTime" : Timestamp(1471973871, 1),
"electionDate" : ISODate("2016-08-23T17:37:51Z"),
"self" : true
},
{
"_id" : 1,
"name" : "abc-mdb-node2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 135,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:40:00Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:40:00Z"),
"pingMs" : 0,
"syncingTo" : "abc-mdb-node1:27017"
},
{
"_id" : 2,
"name" : "abc-mdb-node3:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 15,
"optime" : Timestamp(1471966340, 1),
"optimeDate" : ISODate("2016-08-23T15:32:20Z"),
"lastHeartbeat" : ISODate("2016-08-23T17:40:00Z"),
"lastHeartbeatRecv" : ISODate("2016-08-23T17:40:00Z"),
"pingMs" : 2,
"lastHeartbeatMessage" : "syncing to: abc-mdb-node1:27017",
"syncingTo" : "abc-mdb-node1:27017"
}
],
"ok" : 1
}
ReplicaSet:PRIMARY>
App:
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node1:27017" }", EndPoint: "Unspecified/abc-mdb-node1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.11:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node2:27017" }", EndPoint: "Unspecified/abc-mdb-node2:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 2]" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/abc-mdb-node3:27017" }", EndPoint: "Unspecified/abc-mdb-node3:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.61.13:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__42.MoveNext()" }] }.
Now the replicaset is up. The app won’t recover.
@bruno-garcia
Copy link
Author

bruno-garcia commented Nov 9, 2016 •
edited
Loading

JIRA ticket with MongoDB: https://jira.mongodb.org/browse/CSHARP-1752

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment