Python Multiprocessing — Celery
In this blog, I’m going to record down the experience I went through recently — finding a suitable tool to run my python program in multiprocess.
At first, I use a simple threading library to boost up my process. But after then, the running time is not ideal since all the threads run on the same core. This is because of the limitation of python’s global interpreter lock (GIL).
Then I found another option — Coroutines and Tasks, which also have the same problem as the threading library.
What is GIL?
GIL shows its purpose in its name. Only one thread gets to control the interpreter at a time. To do so, it blocks Python threads to work on a single CPU, or else it will not be thread-safe.
Why python needs GIL?
- Avoid memory race condition — python objects count will never end if race condition happens. Moreover, our program might be stuck by problems like a memory leak or worse.
- Accepting programs to lock objects might cause multiple locks, which results in deadlock.
Eventually, I got two options to choose, which both are real tools to let me achieve multiprocess — Multiprocess library & Celery plus Broker
It works around the interpreter by creating multiple subprocesses. If you are looking for explanations and examples, I think this article will solve your problems — Using Multiprocessing to Make Python Code Faster. And don’t forget to check the library tutorial above.
2. Celery & Broker
Multiprocess programming is achieved by using celery workers (subprocesses). Each of them will execute the task (a function, series of jobs …) you have given and send the result back to the creator.
I would say it is a little bit similar to the Coroutines and Tasks, but with real multiprocessing.
Why I choose Celery & Broker?
- Able to route tasks with priority — tasks can be level by different priorities.
- Unfinished task can be continued easily.
- Simple way to scale up the workers & given them memory limitations.
- Using celery flower can watch each task status in real-time.
Celery & Broker Architecture
In order to learn the structure of a celery server, try to read this superb article — Celery Architecture [2] — which contains great details and fabulous pictures.
Practical Experiences and Observations
- Great scalability on workers.
- Able to limit the memory usage of each worker.
celery worker -A ... --autoscale=4,2 --max-memory-per-child=5120003. Easily to see task status (success or fail) by using celery flower.
4. Subprocesses set on different cores. (K8s environment)
That’s it! Thank you for reading.
Any discussion about building a software system is welcome!
Reference
[1] What is the Python Global Interpreter Lock (GIL)? — Real Python
[2] Celery Architecture — Durai Pandian
