Multiprocessing with Python

Originally published at https://www.niit.com/india/

Multiprocessing is a combination of spawning processes that uses an API similar to the threading module. The multiprocessing package deal provides both local and distant concurrency by using sub-processes instead of threads, thereby avoiding the Global interpreter lock (GDL). If all the cores are needed to be used in the machine, they must fork processes for higher and increased speed. Proceeding with a group of methods can be difficult because if communication among processes is required, it can frequently get complicated to coordinate.

The Multiprocessing package in Python contains an innovative and clear API for dividing work between many processes.

Start Methods:

Multiprocessing supports different ways to start a process. Those ways are as follows:

  • Spawn: It’s a method for users to quickly create and perform many jobs with complex parameter changes. When compared to the other two techniques, Fork and Forkserver, this method takes longer to start a function.
  • Fork: Fork is a process or a method that uses os.fork() in Python to create an interpreter process i.e., a child process. When a new interpreter process is created, both processes will carry out the next instruction.

Basic fork in Python:

Let us have a look at a basic fork in Python

Output :

The next example compliments the original fork code and sets an environmental variable that is then copied into the child process.

Example 1: Fork in Python:

Output

In the above output, you can see that the “changed” environment variable fork is stuck with the child processes in addition to the parent process. This example can be tested by further changing the environmental variable in the parent process, and it can be observed that the child is separated now. Though less complicated than the multiprocessing module, the subprocess module can be used to handle fork processes.

Multiprocessing:

Simple Multiprocessing:

Now that you have hands-on the basics with the forking in Python let us look at an example of how a higher-level multiprocessing library works.

Example 2:

Output :

It is seen that the main process forks an interpreter process and sleeps for 5 seconds. The instructions of this process appear when p.start() gets called.

Building An asynchronous Net-SNMP engine:

Until now, nothing useful has been built. The following example will solve realistic and practical problems by formulating Python bindings for Net-SNMP asynchronous.

Before starting the process, check whether few programs are being installed in your system or not for using both multiprocessing library with Python 2.6 and Net SNMP bindings:

  1. Firstly, download Python from Python Download :
  2. Adjust the shell path with the aim that Python launches automatically when you type “Python”. Example: If Python is compiled in /usr/local/bin/, there will be a need to prepend the $PATH variable to make sure it comes before an older version of Python.
  3. Install Setuptools: Setuptools
  4. Lastly, download the Net-SNMP and configure it, including the “–with-python-modules” flag and in addition to other flags as required by your operating system.

./configure — with-python-modules

Check the code of the following module and then run it.

Example 3:

Multiprocessing SNMP:

Here, the SNMP session carries a method representing the query with the help of the SNMP library, Net-SNMP. Since the call will block automatically, you can then import the multiprocessing library and run it in your operating system easily. This process is similar to that of threading API.

Special attention to the hosts list should be paid in section #clients. Now, you’d be able to potentially run asynchronous SNMP queries to 50 or 100 hosts, or more, depending on what hardware you are running on. Lastly, the two sections take the results from the queue and then put a “STOP” message into the queue.

Configuration of OS X’s SNMPD:

To configure OS X ‘s SNMP Daemon for testing, you need to re-write the configuration file with the help of following three commands on the shell:

This command will back up your configuration, making a new configuration followed by restarting the SNMP Daemon.

For the permanent run of OS X SNMP Daemon, the following edit in the list file is needed:

/System/Library/Launch/org.net-snmp.snmpd.plist

Conclusion

It can be concluded that there are specific notes in the official documentation that are required to be kept in mind: try and avoid shared state, it is best to distinctly join processes that you create, try to eliminate processes with shared states, and finally, ensure that all the queue items are cleared or removed before you join since deadlock can occur.

Multiprocessing is authoritarian and a new extension in the Python Programming language. While there are a few limitations of Global Interpreter lock (GDL) with threading which can become a weakness, Python has more than that made up for it, which is powerful yet a new addition to Python language. To understand the intricacies and concepts of Python you can check out the Professional Program in Full Stack Software Engineering course that provides in-depth knowledge.

It can be concluded that there are specific notes in the official documentation that are required to be kept in mind: try and avoid shared state, it is best to distinctly join processes that you create, try to eliminate processes with shared states and finally, ensure that all the queue items are cleared or removed before you join since deadlock can occur.

Multiprocessing is authoritarian and a new extension in the Python Programming language. While there are a few limitations of Global Interpreter lock (GDL) with threading which can become a weakness, Python has more than that made up for it, which is powerful yet a new addition to Python language. To understand the intricacies and concepts of Python you can check out the Professional Program in Full Stack Software Engineering course that provides in-depth knowledge.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
NIIT Digital

NIIT Digital

We are a skill development hub for learners worldwide sharing ideas on technology. Explore insightful reads at https://www.niit.com/india/knowledge-center