-
-
Save Alex3917/9b09e67069c88361d4e0187e1968bc64 to your computer and use it in GitHub Desktop.
# In .ebextensions/01_celery.config | |
files: | |
"/etc/systemd/system/celery.service": | |
mode: "000644" | |
owner: celery | |
group: celery | |
content: | | |
[Unit] | |
Description=Celery Service | |
After=network.target | |
[Service] | |
# I saw some other tutorials suggesting using Type=simple, but that didn't work for me. Type=forking works | |
# as long as you're using an instance with at least 2.0 Gigs of RAM, but on a t2.micro instance it was running out | |
# of memory and crashing. | |
Type=forking | |
Restart=on-failure | |
RestartSec=10 | |
User=celery | |
Group=celery | |
# You can have multiple EnvironmentFile= variables declared if you have files with variables. | |
# The celery docs on daemonizing celery with systemd put their environment variables in a file called | |
# /etc/conf.d/celery, but I'm choosing to instead set the celery variables as environment variables so that | |
# celery can also access the necessary variables for interacting with Django. | |
EnvironmentFile=/opt/elasticbeanstalk/deployment/env | |
WorkingDirectory=/var/app/current | |
ExecStart=/bin/sh -c '${CELERY_BIN} multi start worker \ | |
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ | |
--logfile=${CELERYD_LOG_FILE} --loglevel=INFO --time-limit=300 --concurrency=2' | |
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait worker \ | |
--pidfile=${CELERYD_PID_FILE}' | |
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart worker \ | |
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ | |
--logfile=${CELERYD_LOG_FILE} --loglevel=INFO --time-limit=300 --concurrency=2' | |
[Install] | |
WantedBy=multi-user.target | |
"/etc/tmpfiles.d/celery.conf": | |
mode: "000755" | |
owner: celery | |
group: celery | |
content: | | |
d /var/run/celery 0755 celery celery - | |
d /var/log/celery 0755 celery celery - | |
container_commands: | |
01_create_celery_log_file_directories: | |
command: mkdir -p /var/log/celery /var/run/celery | |
02_give_celery_user_ownership_of_directories: | |
command: chown -R celery:celery /var/log/celery /var/run/celery | |
03_change_mode_of_celery_directories: | |
command: chmod -R 755 /var/log/celery /var/run/celery | |
04_reload_settings: | |
command: systemctl daemon-reload | |
# In .platform/hooks/postdeploy/01_start_celery.sh | |
#!/bin/bash | |
(cd /var/app/current; systemctl stop celery) | |
(cd /var/app/current; systemctl start celery) | |
(cd /var/app/current; systemctl enable celery.service) |
Many thanks! Was really useful! Some updates from that point:
- very important to have *.sh scripts used in hooks with LF instead of CRLF run (after replace <file_name.sh> with you filename:
sed -i 's/\r$//' <file_name.sh>
- another important thing is to make sure git uses LF - on windows core.autocrlf = true => LF is converted to CRLF so to make it false run:
git config --global core.autocrlf false
On ExecStart command I highly recommend based on celery latest docs use dynamic options
multi start worker
->multi start ${CELERYD_NODES}
. Have in elb config attributesCELERYD_NODES
:worker1 worker2 ...
easier to control.For celery beat I am not quite sure how to get it work - I already used:
https://docs.celeryproject.org/en/latest/userguide/daemonizing.html
Deployment works without any problem;
Celery beat starts;
Error:
- Permission denied: 'celerybeat-schedule'
In case you already have a solution for it please let me know
Versions that I am using:
Django 3.2
Celery 5.0.5
Hi @sorin-sabo, were you able to get this working? Can you share your configuration?
@edchelstephens No I have not gotten beat to start on deployment. I can start it manually after SSH into the instance.
@Alex3917 sorry to bother you on this, but I tried to add a file to start beat after starting celery - and after restarting the instance I'm somehow getting the same error as typefox09 'celery is not a valid user name'. Should I add this to my 01_celery.config file?
commands:
00_add_user_celery:
test: test ! "id -u celery 2> /dev/null"
command: useradd -d /opt/python/celery -g celery -u 1502 celery
ignoreErrors: false
Is this a permissions issue somehow? Any help you can provide would be very much appreciated.
@Alex3917 I've added a 01_python.config file like you mentioned in your first response to appli-intramuros, however should the path change from /opt/python/celery to something else in AL2?
I'm getting 'Created group celery successfully' but then 'Command 00_add_user_celery (useradd -d /opt/python/celery -g celery -u 1501 celery) failed' error..
When I SSH in and try to run the command 'useradd -d /opt/python/celery -g celery -u 1501 celery'
I get '-bash: /usr/sbin/useradd: Permission denied'
Thanks in advance.
@edchelstephens I was able to get beat to start on deployment. I added another file in .ebextensions/01_celery.config; similar to "/etc/systemd/system/celery.service":, but instead called "/etc/systemd/system/celerybeat.service". This is the file (underneath celery.service file):
"/etc/systemd/system/celerybeat.service":
mode: "000644"
owner: celery
group: celery
content: |
[Unit]
Description=Celery Beat Service
After=network.target
[Service]
Type=forking
Restart=on-failure
RestartSec=10
User=celery
Group=celery
EnvironmentFile=/opt/elasticbeanstalk/deployment/env
WorkingDirectory=/var/app/current
ExecStart=/bin/sh -c '/var/app/venv/staging-LQM1lest/bin/celery beat -A video.celery \
--pidfile=/tmp/celerybeat.pid \
--logfile=/var/log/celery/celerybeat.log \
--loglevel=INFO -s /tmp/celerybeat-schedule'
[Install]
WantedBy=multi-user.target
After that, I added:
(cd /var/app/current; systemctl stop celerybeat)
(cd /var/app/current; systemctl start celerybeat)
(cd /var/app/current; systemctl enable celerybeat.service)
to the 01_start_celery.sh file that had the same language for celery. Hope this helps.
Take a look at this gist. I have successfully deployed Django Rest App with celery, AWS sqs in Elastic Beanstalk with Amazon Linux 2
https://gist.github.com/MahirMahbub/f9c226dbc0a01da22c8c539cf9c9bcc9
Hi @Alex3917 , I followed the instructions and manually exported the django settings module to the environment when i ssh
But when I run:
celery inspect active
I get the following error:Do you know how can I fix this?