Skip to content

Instantly share code, notes, and snippets.

@scyto

scyto/docker-swarm-portainer.md

Last active April 1, 2025 07:29
Show Gist options
  • Save scyto/a57d63a3b905b24e9eb25618848c8e80 to your computer and use it in GitHub Desktop.
Save scyto/a57d63a3b905b24e9eb25618848c8e80 to your computer and use it in GitHub Desktop.
Download ZIP
Installing Portainer on a swarm
Raw
docker-swarm-portainer.md

This is how to install portainer on the swarm you created using this gist note this should work on non-swarm installs too!

Setup Portainer

Download the portain stack yaml that will do this for you

curl -L https://downloads.portainer.io/ce2-17/portainer-agent-stack.yml -o portainer-agent-stack.yml

now run it

sudo docker stack deploy -c portainer-agent-stack.yml portainer

this will create a single portainer container that runs on the management node and deploy the agent to all worker nodes, once complete you can now access portainer at http://docker-host-ip:9000 and define your admin users etc

You now have a fully function docker swarm with portainer and can deploy containers, stacks or templates.

How to use portainer is beyond the scope of this gist

This is the portainer install doc at time of writing for reference but the instructions above will work just fine

@Jens-Wymeersch
Copy link

I got the following from the Portainer folks which should be in line with what i wrote
Hello! That’s correct. If the node running Portainer goes down, you’ll lose access to Portainer as well. We typically recommend, if possible, running the Portainer server outside of your swarm on a dedicated Docker standalone host and connecting the swarm via an agent. If you decide to keep Portainer within the swarm, ensure that, in the event it goes down, another node can access the Portainer data—usually through a network share. However, please note that storing Portainer’s data on a network share can have issues as databases and network shares can be hit and miss

@Jens-Wymeersch
Copy link

So I tried and again failure. Once I drained the manager node, Portainer is dead and doesn't start up anymore. I do see that the portainer server is on all nodes.

@scyto
Copy link
Author

scyto commented Nov 12, 2024 via email

@scyto
Copy link
Author

scyto commented Nov 12, 2024 via email

@Jens-Wymeersch
Copy link

Jens-Wymeersch commented Nov 12, 2024 •
edited
Loading

I am happy to try anything but need some more guidance on what to try.

The only difference i see btw the standard script of Portainer and your Portception is the following isn't included

    networks:
      - agent_network

not sure this was intended or not ? And what would be the impact ?

If I look at the prep, this is all done. Besides the agents installation. I intend to use the following https://docs.portainer.io/2.17/start/agent/swarm/linux Okay ?

The deployment however, I have a lot of questions

  1. created the stack and made sure I had different service names - what do you mean with different service names? Can’t I just use the stack as described ?
  2. i think i started the stack but then stopped it just after creation - why ?
  3. stopped the existing portainer service via the command line - what is the difference btw what you wrote in previous point and this one ?
  4. then restarted the new service from the command line - which new service ?

@Jens-Wymeersch
Copy link

Jens-Wymeersch commented Nov 12, 2024 •
edited
Loading

One step forward two steps back. So I installed a BE portainer outside the 3-node swarm. It works, now the problem is that I'm stuck with the license limit of 3 nodes that I now surpassed. They are asking 150 dollars per year. What do you think ?
Potential workaround, install a standalone agent on one of the nodes. → doesn't work

@scyto
Copy link
Author

scyto commented Nov 13, 2024

@Jens-Wymeersch i paid for EE and never been happier, i use it to manage software on all machines - even things like zwave-js install to a raspberrypi

image

@Jens-Wymeersch
Copy link

Looks like your environment is a bit more complex than mine. What is Syn01 and Syn02 ?

@scyto
Copy link
Author

scyto commented Nov 13, 2024 via email

@Jens-Wymeersch
Copy link

Got it: Just to give you an update where i'm at. It looks like the microceph/cephfs works only when all machines are on. Doing backsup and stuff breaks the link unrepairable somehow.
Hence, I'm thinking of going as you did to Gluster. Any advice or guide you followed ?

@scyto
Copy link
Author

scyto commented Nov 13, 2024

I don't know anything about microceph - i just used what proxmox gave me so can't express an opinion on that.

For gluster you might want to just mount the gluster volume rather than use the gluster volume plugin i used - it works great until i reboot the switch the promox host nodes are connected to (remeber my swarm in VMs on top of proxmox) and then the gluster plugin fails and so any container relying on it fails.

its simple to SSH in and reenable the plugin docker plugin enable <id> (get the id from docker plugin ls but i find i have to do this on all 3 nodes in sequence and if one fails with mysterious error do the others and come back to it.

This is why i plan to move to either mounted ceph across network using fuse, or dabble with one of the ceph volume plugins, but they are all so old.... which is the FUD that has made me do nothing to resolve this.

@scyto
Copy link
Author

scyto commented Nov 13, 2024

changed my mind about commenting on microceph - if you have 3 ceph nodes with 3 managers and an OSD on each node then the ceph should stay running quite happily when one node goes down, if two nodes go down then yeah you won't have a ceph volume until it become quorom.

@Jens-Wymeersch
Copy link

well... the problem I'm facing is that it somehow the servers forget they have a FS. Just finished with GlusterFS and I'm facing the same thing. I really don't know why....

@scyto
Copy link
Author

scyto commented Nov 13, 2024

IF you mounting the filesystem at boot if so you need to make sure fstab doesn't try and mount them utill the network AND the ceph/glsuter services are full up and operational (not just started)

this will require you mess with service dependencies...

consider running a scripted mount command instead of relying on fstab

this was one reason I used the volume driver, can't recall if i modified docker service to start only once gluster was fully up

If thats not the issue maybe you have an inherent networking issue, do all your nodes have names registered in DNS and it might be worth putting all nodes in /etc/hosts - sometimes found gluster would rever to wanting to use nodename for no reason i could fathom

@Jens-Wymeersch
Copy link

I used the following guide https://www.youtube.com/watch?v=Has6lUPdzzY
I've already added the nodes in /etc/hosts
How you do this ? IF you mounting the filesystem at boot if so you need to make sure fstab doesn't try and mount them utill the network AND the ceph/glsuter services are full up and operational (not just started)

Secondly, I have a 32GB disk on each of the nodes - 15GB goes to gluster - any idea how you can increase the amount for gluster ? And potential tweaks ?

@Jens-Wymeersch
Copy link

Jens-Wymeersch commented Nov 13, 2024 •
edited
Loading

I killed all the machines (which can happen after a long power surge). Then bringing them up and nothing works anymore. The scripted mount command, anything you documented ?

I can get it to work again when I run the mount -a command manually.
Strange thing is that last server booting up - it mounts correctly.

If only 1 server goes down, all work as expected. If 2 go down, the last one will mount correctly, but the first one doesn't.

Please note that I added the following to /etc/fstab
localhost:/staging-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0

@scyto
Copy link
Author

scyto commented Nov 13, 2024

oh i don't use a scripted command, just suggesting things for you to try

all i can tell you is what I did works and with the exception of that one annoyance where i have to re-enable the plugin after router reboot (not node reboots), replacing gluster with a connection to my ceph is a project for the holidays. my docker VMs won't be the ceph nodes, the host those docker VMs sit on are the ceph nodes.

@scyto
Copy link
Author

scyto commented Nov 13, 2024 •
edited
Loading

dont send me videos I am a grumpy early GenX' took me minutes to scrub through video, still couldn't figure out what the fuck he was doing, luckily he had good docs, took me seconds to read and undestand ;-)

yeah he fucked up and didn't include the automount, tl;dr don't rely on videos most of the time they don't really know what they are doing (or rather taking them as sign post, not a how to), alsogo to stack exchange, there are usually answers there from years ago

like this one
https://serverfault.com/questions/800494/glusterfs-mount-on-boot-on-clustered-servers-rhel-7

scroll down, you will see a script example

you will see the use of automount at the end of the glsuter fastab line.... he missed that.... this should mean the system won't try and mount until gluster is running

if that still fails you will need to start editing unit files, not fun - but folks seem to have posted those approaches too, always start with the most upvoted answer (in this case the automount directives), on linux its hard to know which is right approach as each distro works differently, i only use debian for my docker hosts for a good reason

@scyto
Copy link
Author

scyto commented Nov 13, 2024

if it helps this is what i have in my fstab on all 3 nodes

#for glusterfs
UUID=8a5eeebb-a268-45ac-a2c7-e2cef9b2d8d2 /mnt/glusterfs xfs  defaults  0  0

#for gluster vol1
localhost:/gluster-vol1  /mnt/gluster-vol1 glusterfs  defaults,_netdev,noauto,x-systemd.automount        0 0

@Jens-Wymeersch
Copy link

Hi mate,

I followed the following guide https://www.howtoforge.com/how-to-install-and-configure-glusterfs-on-ubuntu-22-04/#google_vignette together with your last post - and we got lift off !

@Jens-Wymeersch
Copy link

Another question : how do you get that you can use

volumes:
  data:
    driver: gluster-vol1

@scyto
Copy link
Author

scyto commented Nov 14, 2024

Hi mate,

I followed the following guide https://www.howtoforge.com/how-to-install-and-configure-glusterfs-on-ubuntu-22-04/#google_vignette together with your last post - and we got lift off !

fabulous, but i am starting to think you didn't follow my documentation... the automount has always been in my docs, and you question about glutser-vol1 means i thnk you didn't read my gluster documentation of what i did?

https://gist.github.com/scyto/f4624361c4e8c3be2aad9b3f0073c7f9

@Jens-Wymeersch
Copy link

There is a wealth of information in your git. I'm sorry. I just found what you meant (https://gist.github.com/scyto/7e9d471c70f92c8b983f38cdc838998d) and going to test it directly. You basically created the bible I'm following now.
I'll keep you posted... I hope you still be willing to respond to my stupid questions -;)

@scyto
Copy link
Author

scyto commented Nov 14, 2024

heheh, i am still on the fence if i do or don't like the glusterFS volume plugin as it works well until it doesn't - but the fix is easy, just renaable it on each node, also see some of my comments under each articles, even i am not sure if i did things the 'best way

next up is to move all the storage from gluster to ceph

@Jens-Wymeersch
Copy link

I was just looking at the plugin and got already stuck that somehow I need to use make build command. Not sure what this means ?

@scyto
Copy link
Author

scyto commented Nov 14, 2024 via email

@Jens-Wymeersch
Copy link

Jens-Wymeersch commented Nov 14, 2024 •
edited
Loading

@scyto So I need to run the command on each node in the docker swarm ?
I assume I do but I want to make sure before I screw things up

@scyto
Copy link
Author

scyto commented Nov 16, 2024 via email

@Jens-Wymeersch
Copy link

I looked at the docs, it doesn't specify if you need to run it in the master node or in all nodes.

@pderuiter
Copy link

pderuiter commented Apr 1, 2025 •
edited
Loading

IF you mounting the filesystem at boot if so you need to make sure fstab doesn't try and mount them utill the network AND the ceph/glsuter services are full up and operational (not just started)

this will require you mess with service dependencies...

consider running a scripted mount command instead of relying on fstab

this was one reason I used the volume driver, can't recall if i modified docker service to start only once gluster was fully up

If thats not the issue maybe you have an inherent networking issue, do all your nodes have names registered in DNS and it might be worth putting all nodes in /etc/hosts - sometimes found gluster would rever to wanting to use nodename for no reason i could fathom

A bit late comment, but by adding the "_netdev" to your mount options in the /etc/fstab file will solve the problem of mounting before the network is online.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment