PyCon Nigeria Annual Conference

Concurrency: How to use "ThreadPool", How "ThreadPool" saved me once.

speaker-foto

Joshua Adewole

Joshua is a Systems Engineering student at the University of Lagos. Fell in love with python in 2016. He is a software engineering professional with some years of experience working in software development teams, particularly in startups. These teams include mobile and web application teams, where he participates in all phases of the software development lifecycle, from conception to deployment; he is currently a technical partner manager at Victoria Bros. Outside of technology, he is enthralled with mathematics in all of its forms and at his leisure enjoys playing chess.

Description

Concurrency is a very interesting concept, which efficiently solves some type of problems. What is concurrency about? Let's dig in together and see what it is like on the surface level, it just might be what you need in some cases where you feel stuck (haven made several attempts) and your application feels slow.

Abstract

With the ever increasing demand for a fast application, sometimes the traditional procedural approaches just might not work. I was faced with a feature, which required handling lots of data, I did iteratively try so many procedural approaches ensuring I sorted efficiently, but that wasn't enough, for weeks or months this went on, until I had this thought in my head of seeing each query as a Needle's thread, this brought about Concurrency. Over the course of this tutorial, I will be providing more context on my thought process leading right until the Concurrency solution and right after, the time saved and several other details. Majorly, a breakdown of what we'll be looking at is - What is Concurrency (in simple terms)? - What is Parallelism (in simple terms), and how they are differentiated from Concurrency? - Background Story, providing context on the feature I was working on and the solutions I used - Brief deep-dive into a code showing how to implement one or rather How I implemented one e.g

 # importing the Library
 from multiprocessing.pool import ThreadPool as Pool

 # a class utilizing the pool library
 class ObjectUtilizingThreadPool():

     # one of the processes to be used
     def one_process(self):
         pass
     # one of the processes to be used
     def two_process(self):
         pass
     # the method implementing concurrency
     def pool_process(self):

         with Pool(processes=3) as pool:
             proc_one = pool.apply_async(self.one_process())
             res1 = proc_one.get()
             proc_two = pool.apply_async(self.two_process())
             res2 = pool.apply_async(self.two_process())
         res = res1 + res2
         return res
  • Use-cases of the concept
  • Beyond the basics
Audience level: Novice