Skip to content

Instantly share code, notes, and snippets.

@reetp
Last active February 22, 2025 15:02
Show Gist options
  • Save reetp/a66149d5f060f260643a353ca7067a98 to your computer and use it in GitHub Desktop.
Save reetp/a66149d5f060f260643a353ca7067a98 to your computer and use it in GitHub Desktop.
Debugging

Rocket.Chat - Asking for help

For users with issues with Rocket.Chat please do yourselves a few favours before asking for help.

First thing is read, comprehensively. Most of your questions have already been answered.

Docs

Dev Docs

Also try a search engine.

Contents

How to ask for help

To pay or not to pay

If you think your issue is urgent, then you have two choices.

a) Pay for support - Pricing

Email: [email protected]

b) Patiently wait for a volunteer to consider helping you.

You are entitled to exactly what you paid for

No more, and no less.

Please remember:

Abuse

Abusing any community member will get you banned completely.

  • YOU are the one with the problem. You would not be here otherwise
  • Being rude to them will not help you at all
  • Most people in the chat are volunteers who will try and help for free
  • Volunteers do this in their spare time and are not getting paid for this. Respect them
  • Just because it is urgent for you does mean that it is urgent for them
  • Your time might be valuable. Theirs is invaluable
  • They have jobs to do and mouths to feed. Don't waste their time
  • Answer their questions. They are not asking for fun but to try and understand
  • Repeating your question 10 times in one channel will not help you.
  • Posting the same question in several channels will not help you
  • Do not DM anyone unless you know them, or they DM you first
  • Do not randomly @ people or @all hoping to get attention

Abuse of these things you will just get you muted, blocked or banned and you will not get any help

Before you start

First, you must ALWAYS test on the latest version of Rocket.Chat and use the latest browser or app clients - you can try https://open.rocket.chat which always runs development versions. A large amount of issues are fixed with an update.

You are going to need to provide some basic information if you want a chance of resolving your issue.

If you haven't got this information to hand please go and get it now.

Information that support or devs will require

At a minimum you will need this:

  • Server Setup Information:

    • Licence type eg CE/Starter/Pro :
    • Number of users:
    • Server hardware: VPS/hypervisor/bare metal
    • Version of Rocket.Chat Server:
    • Operating System: Ubuntu/Redhat etc
    • Deployment Method: snap/docker/tar/etc
    • Number of Running Instances:
    • DB Replicaset Oplog:
    • NodeJS Version:
    • MongoDB Version:
  • Proxy information:

    • Web server: Apache/Nginx/Other
    • Reverse proxy setup: Yes
    • Type of SSL/https certificates: Letsencrypt/other
  • Client information:

    • Client type: Electron app/React-Native app/Browser and version
  • A list of the steps required to replicate the issue.

    • If we can't replicate it we can't debug it.

Don't open a bug yet

Do NOT open a new issue in github until you have read the rest of this document. Most 'issues' are configuration issues, not bugs.

Do NOT ask a question, wait for 30 seconds, and then decide you are way too important to wait around and your client is nagging and you'll open a bug to get it fixed faster. It will probably just get closed unless it is well documented and really is a bug.

If you ask in the channels or forums then devs or other community helpers will tell you if you really need to open an issue.

Frequently if there really IS a problem you won't be the first to experience it, so always check github carefully for duplicates. Use lots of different search terms and make sure you check closed Issues as well (see below)

Do not ask the same question in 5 different channels. Just one channel please.

Common channels are:

Support

React-Native

Ubuntu Snap

Raspberry Pi

Omnichannel

Desktop

Federation

Is it a feature request?

If Rocket works but doesn't do something the way you want it to then this is a feature request NOT a bug. Open a request in Feature Requests

RTFM

Help yourself. Have you read ALL the documentation? At least twice?

Those who have bothered will get a more sympathetic ear. You have chosen to become an administrator. With great power comes great responsibility.

A lot of the answers can be found in the documentation. Read through slowly and carefully.

It is easy to miss the obvious because you think you know the answer and you are just looking for confirmation of this, rather than reading to actually understand (and discovering you might just have been wrong).

Search

Have you looked through the bug reports? At least 2 different searches? Opened AND Closed bugs?

A lot of answers can be found there.

Check the various repos - Rocket itself, the mobile apps, electron etc.

Rocket.Chat server

Rocket.Chat server feature requests

React Native Mobile Client

Electron desktop application

Rocket.Chat documentation

When searching change the filter box to both open and closed issues like this:

is:issue is open

to

is:issue my search word

e.g.

is: issue theme

Try some variations - you won't hit the sweet spot first time usually. And the more often you do this the better you will get at learning what to search for and how. It is an art.

Still stuck?

Don't open an Issue yet.

How to ask for help

The next step is to ask your well documented question in the forums or Rocket.Chat

How to Report Bugs Effectively

It is impossible to diagnose something when you just say 'it's broken'. The helpers are not mind readers. They cannot see your screen or your your set-up. They have not watched you set this up, and the mistakes you made along the way. They are effectively blind. So you have got to describe things clearly and concisely.

Your problem may appear complex but it is often the basics that catch you out. It is hard to tell at first if a problem is simple or complex so please be patient if you get asked 'the stupid questions'. We are just trying to get a baseline and making sure you have done the basics, and trying to understand how you got to this position. It is extremely common for people to miss steps in their set-up.

Don't hide things away because you are embarrassed. They are most likely going to find out, so you might as well be honest. We have ALL made mistakes.

This is a really good guide on how to report issues clearly and (more importantly) how to get the attention of developers and helpers - the people who have the solutions to your issue.

How to ask a smart question that will attract attention

This one is very relevant and it can waste a lot of time:

The XY Info problem

Tell us about the problem you were originally trying to solve, not the problem that you are experiencing right now. Frequently the one you are experiencing now is the result of not understanding or being able to fix the original issue.

Remember, if you phoned the garage about your car and said 'it runs funny sometimes' what would they ask you? Make, model, age... etc.

This is no different.

Use this as a template to ask your question. Get all the right information. It will help if you then have to submit a real Issue.

Complete a bug template

Server Set-up Information:

See the template above. Fill it all out. You might not know why - which is why you are asking, not answering - but you will get far more help.

Information that support or devs will require

Some background will help, and will avoid XY issues. Is it a fresh install? Is it an upgraded install? Has it always been broken? When it did the issue start? Did you make any changes that could have caused it?

Make sure you post some BRIEF log snippets or use https://pastebin.com/

Don't block up the channel with ridiculously long messages.

If your problem relates to a connection to say a database, or LDAP, that might be on another server then check the logs for those as well.

Here is a great example of a good bug report:

Good bug reporting example

Last, be patient. Answer the questions you are asked. They may appear odd, but they are probably for a good reason. Debugging is a logical process and you need to help establish the facts first.

Remember, you are the one with the problem. They can just go back to work or their wife and kids and ignore you.

A classic is:

I want to know how to do X

Why do you want to do X?

Because I do.

But X is not a very good thing to be doing.

But I want to.

Tell us why because there may be a different solution, even though it might not be the one you want to hear.

Shut up bozo. You know nothing (So why are you asking us then?? Confirmation bias...). Just tell me what I want to know

I can tell you that is not the correct solution and will cause you untold grief and in a few days you'll be back complaining your system is broken/hacked/otherwise shredded and you have thousands of users shouting at you.

And you'll probably starting asking with "HELP URGENT HELP REQUIRED NOW"


Remember it is you asking for help. It is you with the problem. So please don't bite the hand that feeds you.


Open an Issue

Got this far with no solution? If you haven't already been told to do so then open an Issue with all the right information in the right repo.

Please use the templates provided. Add all the relevant information.

Stick to facts, not opinions, assumptions or guesses.

Rocket.Chat server

Rocket.Chat server feature requests

React Native Mobile Client

Electron desktop application

Rocket.Chat documentation

FAQ

Rocket.Chat and open source

Rocket.Chat is open source. But please take time to understand what open source really means.

The source code is freely available and MIT licenced. You can fork the repo, remove the EE folders, and hack and build to your hearts content.

But the binaries - eg snap, docker come with some ties.

You will have to register your server with the Rocket.Chat cloud servers.

For all offline use you will need a paid licence.

Starter edition is free and pretty well fully functional for up to 50 users.

Community Edition can run about 200 users. Due to limitations with node you need some serious skills to go above this number. (EE has optimisations to handle this). It has limited capabilities eg push notifications, apps etc.

Don't shoot the messenger.

Check your proxy server

You are using a reverse proxy and https, and not exposing Rocket.Chat directly?

Slow connections. iOS connection error

95% or more of these are due to improperly configure reverse proxies where the websocket is not working properly

If a websocket cannot be correctly established, the client will try and fall-back to an XHR connection. It is extremely slow and flaky. The iOS app will not connect at all without a websocket.

Please use Chrome/ium and dev tools to check for the presence of a websocket.

Websockets

  • Right Click, Inspect

  • Network, F5 to refresh

  • Look for the WS filter

  • Should see websocket

    websocket - wss://open.rocket.chat/sockjs/123/pbiz45rm/websocket

Click it and look on the right for 'Headers'

Has this got a green light?

 Status Code: 101 Switching Protocols

What do the headers say?

Response headers
 Connection: Upgrade
 Sec-WebSocket-Accept: nXd/PfaHuA1Kx/ggJWAAy+96Ee4=
 Upgrade: websocket

Check for duplicate indexes

Have a look at your logs.

If you have duplcaite indexes you can delete them and let them rebuild correctly

GridFS/ FileStorage

You should NOT use GridFS in production. It is for development/testing only.

A large amount of files will clog your DB and slow our install to treacle.

https://github.com/RocketChat/filestore-migrator

https://github.com/arminfelder/gridfsmigrate

Notifications

A lot of issues with notifications are down to set-up/configuration

Please read this for the current notification logic:

Push Notification Logic

General debugging

It isn't a bug unless you can repeat it.

So, make a note of everything you do. I always use a note editor and paste in histories of commands, logs, files. changes, hacks or whatever.

I tidy up afterwards and then test that I can repeat it.

If I can I know I have a case.

The more you document, the easier it is for a dev to find, and fix.

Make sure you have all the correct versions of software used.

Start your little history. Be clear. Be concise. Don't waffle. Stick to the facts, not your guesses.

Add your documentation.

Sit back and feel smug :-)

Upgrading Rocket.Chat

Upgrading is a fraught business at any time. First you have Rocket itself, and then the Mongo database.

Snaps should auto-update so this is aimed at those who use other methods.

  • Rule One. Backup. Lots of them and lots of ways, and practice restores as well.
  • Rule Two. Use virtual machines for testing. They are so cheap.

Upgrading Mongo

  • Did I mention backups? Read how to do this. Practice it.
  • Use the Mongo documentation. There are leaps between 2.x and 3.x
  • Then there are more leaps as you proceed trough 3.x and then again to 4.x
  • As of writing 3.4 is going EoL so you need to be on 3.6
  • For now I would get to 3.6 and then start preparing for 4.x
  • To convert from mmapv1 -> wiredtiger you will need to backup, convert, restore

Upgrading Rocket

Did I mention backups? Yup - boring. Until you need them.

The problem with Rocket is not the actual code itself, but the fact that most updates also update the database schema - the core layout.

Now, theoretically apart from adding oplogs to Mongo for Rocket 1.x + you should be able to merrily jump from say 0.62 through to todays 2.1.x

Except..... those schema changes. If you are going to hit a problem, it will be one of those.

The devs test changes from one version to another, and maybe even over a few versions. But not every single version up to latest or say 0.74.3 -> 1.3.2

That's where it may go wrong.

So a recommend path is something like this:

-> 0.73.x -> 1.0.x -> 1.32 -> 2.0.x -> 2.1.x

Or possibly even more cautious:

 -> 073.x - 1.0.x -> 1.1.x -> 1.2.x -> 1.3.x -> 2.0.x -> 2.1.x

( 'x' means the last patched version e.g. 1.3.2 Check tags or releases )

I have seen people make some big jumps. And some disasters too. If in doubt, take you time.

And did I mention backups?

See: Admin on Rocket.Chat

Developers

Read Lots

Please, please, please READ. Lots.

Most of your questions have been answered multiple time in multiple places.

Answering them becomes extremely tedious so don't be surprised at some less than friendly responses if you clearly haven't bothered doing some research. And they will know.

Search has been around for 30+ years. Make use of it.

Github

  • Github issues NEVER get assigned so please don't ask.
  • Please don't @ people anywhere - it gets really irritating and you will get muted or ignored. Those who need to know will see.
  • Make sure you ask if you can work on an issue - don't waste your own time. If an issue has been triaged as a bug it will have been referred to the devs already for them to fix. Feature requests or apps are a much better area to work on.
  • Rocket.Chat is a large mature project. It has multiple departments hidden from view with github being the very last bit right at the end. Don't expect any fast answers.

Links

More developer notes

There are NO simple solutions.

Unsupported versions

Can you access this from within the container?

https://releases.rocket.chat/v2/server/supportedVersions

@reetp
Copy link
Author

reetp commented Mar 6, 2019

Some links.

Forums:
https://forums.rocket.chat

Documentation (not always easy to find stuff - take your time here):
https://rocket.chat/docs/

New feature requests:
https://github.com/RocketChat/feature-requests

Nginx reverse proxy:
https://rocket.chat/docs/installation/manual-installation/configuring-ssl-reverse-proxy/

Notification logic:
https://whimsical.co/Bngw3kot3wWPMAAJTebDUJ

@reetp
Copy link
Author

reetp commented Aug 13, 2019

Thanks to https://stackedit.io/app for helping with the Markdown :-)

@reetp
Copy link
Author

reetp commented Oct 16, 2019

ToC added using Atom editor with https://atom.io/packages/markdown-toc

@reetp
Copy link
Author

reetp commented May 10, 2021

Another editor I found:

https://jbt.github.io/markdown-editor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment