lukebakken/stackoverflow-75574227.md

Last active March 2, 2023 18:27

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/lukebakken/621e19cdd117599011c0e905ece59650.js"></script>
Save lukebakken/621e19cdd117599011c0e905ece59650 to your computer and use it in GitHub Desktop.

Download ZIP

https://stackoverflow.com/q/75574227/1466825

Raw

stackoverflow-75574227.md

Hello -

Thanks for using RabbitMQ. I'm a member of the core engineering team.

Carefully look at the first 16 octets in this error log message:

2023-02-26 16:43:12.635470+00:00 [error] <0.1056.0>  operation none caused a connection exception frame_error: "type 3, first 16 octets = <<\"{\\\"payload\\\":{\\\"res\">>: {invalid_frame_end_marker,\n                                                      
99}"
2023-02-26 16:43:15.638860+00:00 [error] <0.1056.0> closing AMQP connection <0.1056.0> (10.244.0.18:60608 -> 10.244.0.21:5672):
2023-02-26 16:43:15.638860+00:00 [error] <0.1056.0> fatal_frame_error

It looks as though your client application tried to publish a message containing a RabbitMQ error log message (invalid_frame_end_marker) BACK to RabbitMQ and then another error happened.

What you should do at this point is share a git repository containing a complete, runnable set of code that I can use to reproduce this issue. Basically I should just clone the repo, and execute a command. Assume there is a 3-node RabbitMQ cluster available, or, provide your complete minikube configuration.

Thanks - Luke

Author

lukebakken commented Mar 1, 2023

Can I invite you to a private repository?

Of course! @lukebakken is my GH handle.

I've never heard of devspace but I'm willing to get it running if that's the only way to reliably reproduce this issue.

Does your test demonstrate the issue every time?

andrinheusser commented Mar 1, 2023 •

edited

Loading

I sent you an invite for the repository.

Does your test demonstrate the issue every time?

Yes, reliably within 30 seconds or so.

Also please note the README in the repo, it contains information about the setup you might find helpful.

Author

lukebakken commented Mar 1, 2023

Thanks, I'll get to this when I can find time.

andrinheusser commented Mar 1, 2023

No worries, if I'm lucky I'll find some time to reduce the test case even further until the end of the week.

andrinheusser commented Mar 2, 2023

Hey Luke,

I managed to reduce the sample down further and can now share a public repository. I suggest we disregard the privately shared repository and use this one as basis for discussion.

Public Repository

Author

lukebakken commented Mar 2, 2023

Ok, got it!

andrinheusser commented Mar 2, 2023 •

edited

Loading

Progress: I've published a branch deno-amqp-only which uses the bare deno-amqp library. It creates the same error, but I believe it was previously caught by my code. Edit: I think I saw the message just below because I didn't expose the error which caused the connection to close. This was just when trying to send the next message on the now closed connection.

error: Uncaught (in promise) BadResource: Bad resource ID
    await this.#conn.write(encodeFrame(frame));
    ^
    at async write (deno:ext/net/01_net.js:33:12)
    at async AmqpSocket.write (https://deno.land/x/[email protected]/src/amqp_socket.ts:147:5)
    at async Promise.all (index 0)
    at async AmqpChannel.publish (https://deno.land/x/[email protected]/src/amqp_channel.ts:323:5)
    at async Promise.all (index 0)

Seems the underlying connection gets closed/dropped for some reason.

RabbitMQ logs show our error:

2023-03-02 16:21:16.063377+00:00 [info] <0.608.0> accepting AMQP connection <0.608.0> (10.244.0.4:48082 -> 10.244.0.5:5672)
2023-03-02 16:21:16.112253+00:00 [info] <0.608.0> connection <0.608.0> (10.244.0.4:48082 -> 10.244.0.5:5672): user 'rabbitmq-debug' authenticated and granted access to vhost '/'
2023-03-02 16:21:26.311125+00:00 [error] <0.608.0> Error on AMQP connection <0.608.0> (10.244.0.4:48082 -> 10.244.0.5:5672, vhost: '/', user: 'rabbitmq-debug', state: running), channel 1:
2023-03-02 16:21:26.311125+00:00 [error] <0.608.0>  operation none caused a connection exception frame_error: "type 3, first 16 octets = <<\"\\\"aaaaaaaaaaaaaaa\">>: {invalid_frame_end_marker,\n                                                    97}"
2023-03-02 16:21:29.313974+00:00 [error] <0.608.0> closing AMQP connection <0.608.0> (10.244.0.4:48082 -> 10.244.0.5:5672):
2023-03-02 16:21:29.313974+00:00 [error] <0.608.0> fatal_frame_error