worker pattern using rabbit MQ, celery and MongoDB
- Install RabbitMQ
- setup rabbitMQ for celery
http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html
-
create user and set permissions for virtual host
-
create virtual host
abbitmqctl add_user puneet welcome1
rabbitmqctl add_vhost feedvhost
rabbitmqctl set_user_tags puneet dev
rabbitmqctl set_permissions -p feedvhost puneet "." "." ".*" -
start rabbit mq server : rabbitmq-server
-
to monitor http://localhost:15672 , can create a user with monitor role to monitor queues
-
create virtual env for your project and install celery,pymongo,PyYAML
virtualenv vworkpatt
source source vworkpatt/bin/activate
pip install celery pymongo PyYAML -
If you are using pyCharm use the virtual env just created
-
run rabbit mq : rabbitmq-server
-
rn mongodb : mongod
-
start celery :
celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info -
start flask app
python3 app.py
Post Feeds : http://0.0.0.0:4001/Usagefeed
Get job status : http://0.0.0.0:4001/Usagefeed/261d6625-d18e-4945-85da-8f89ff388cf6
9 . freeze env to generate requirements.txt
pip freeze > requirements.txt
-
To run celery with concurrency option
celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info --concurrency=20 -
for scaling
celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info --autoscale=10,3
References:
http://flask.pocoo.org/docs/0.11/patterns/celery/
http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html