在python中控制用于调用外部命令的子进程的数量

5t7ly7z5  于 2022-12-10  发布在  Python
关注(0)|答案(3)|浏览(156)

我知道使用subprocess是调用外部命令的首选方式。
但是,如果我想并行运行几个命令,但限制产生的进程数量,该怎么办?

subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile)

然后进程将继续,而不等待cmd完成。因此,我不能将其 Package 在multiprocessing库的工作进程中。
例如,如果我这样做:

def worker(cmd): 
    subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile);

pool = Pool( processes = 10 );
results =[pool.apply_async(worker, [cmd]) for cmd in cmd_list];
ans = [res.get() for res in results];

那么每个工作线程将在生成一个子进程后完成并返回。
限制子进程数量的正确方法是什么?

9gm1akwq

9gm1akwq1#

不需要多个Python进程甚至线程来限制并行子进程的最大数量:

from itertools import izip_longest
from subprocess import Popen, STDOUT

groups = [(Popen(cmd, stdout=outputfile, stderr=STDOUT)
          for cmd in commands)] * limit # itertools' grouper recipe
for processes in izip_longest(*groups): # run len(processes) == limit at a time
    for p in filter(None, processes):
        p.wait()

请参阅Iterate an iterator by chunks (of n) in Python?
如果您希望限制并行子进程的最大和最小数量,可以使用线程池:

from multiprocessing.pool import ThreadPool
from subprocess import STDOUT, call

def run(cmd):
    return cmd, call(cmd, stdout=outputfile, stderr=STDOUT)

for cmd, rc in ThreadPool(limit).imap_unordered(run, commands):
    if rc != 0:
        print('{cmd} failed with exit status: {rc}'.format(**vars()))

只要limit子进程中的任何一个结束,就会启动一个新的子进程,以始终保持limit数量的子进程。
或者使用ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor # pip install futures
from subprocess import STDOUT, call

with ThreadPoolExecutor(max_workers=limit) as executor:
    for cmd in commands:
        executor.submit(call, cmd, stdout=outputfile, stderr=STDOUT)

下面是一个简单的线程池实现:

import subprocess
from threading import Thread

try: from queue import Queue
except ImportError:
    from Queue import Queue # Python 2.x

def worker(queue):
    for cmd in iter(queue.get, None):
        subprocess.check_call(cmd, stdout=outputfile, stderr=subprocess.STDOUT)

q = Queue()
threads = [Thread(target=worker, args=(q,)) for _ in range(limit)]
for t in threads: # start workers
    t.daemon = True
    t.start()

for cmd in commands:  # feed commands to threads
    q.put_nowait(cmd)

for _ in threads: q.put(None) # signal no more commands
for t in threads: t.join()    # wait for completion

若要避免过早退出,请添加异常处理。
如果你想在字符串中捕获subprocess的输出,请参见Python: execute cat subprocess in parallel

hmtdttj4

hmtdttj42#

如果要等待命令完成,可以使用subprocess.call。有关详细信息,请参阅pydoc subprocess
您也可以在工作线程中调用Popen.wait方法:

def worker(cmd): 
    p = subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile);
    p.wait()

因为这个答案似乎有些混乱,下面是一个完整的例子:

import concurrent.futures
import multiprocessing
import random
import subprocess

def worker(workerid):
    print(f"start {workerid}")
    p = subprocess.Popen(["sleep", f"{random.randint(1,30)}"])
    p.wait()
    print(f"stop {workerid}")
    return workerid

def main():
    tasks = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as pool:
        for i in range(20):
            tasks.append(pool.submit(worker, i))

        print("waiting for tasks...", flush=True)
        for task in concurrent.futures.as_completed(tasks):
            print(f"completed {task.result()}", flush=True)
        print("done.")

if __name__ == "__main__":
    main()

如果运行上面的代码,您将看到所有worker进程 * 并行 * 启动,并且我们能够在它们完成时收集值。

e4eetjau

e4eetjau3#

我的解决方案是创建一个进程列表并为其设置最大可用长度。
每次在将新进程追加到列表中之前,我都会检查列表。
编码:

import subprocess
import time

max_num_process = 16
check_gap = .1. # sec

def check_processes(processes):
    # check & wait until available
    while True:
        if len(processes) < max_num_process:
                break

        for process in processes:
            if process.poll() is not None:  # not running
                processes.remove(process)
        time.sleep(check_gap)
    return processes

processes = []
for command in commands:
    processes = check_processes(processes)
    processes.append(subprocess.Popen(command, shell=True))

相关问题