3

Python 的多线程和并行

 3 years ago
source link: https://zhiqiang.org/coding/python-thread-and-multi-processing.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

1. 另起线程

核心就是threading.Thread：

from threading import Thread

def square(x):
    print(x * x)

def mul(x, y):
    print(x * y)

if __name__ == "__main__":
    thread1 = Thread(target=square, args=(10, ))
    thread2 = Thread(target=mul, args=(10, 10))

    thread1.start()
    thread2.start()

    thread1.join()
    thread2.join()
  • 必须调用start，线程才会被启动。
  • join会将线程合并到当前线程，当前线程被阻塞，直到合并的线程执行完毕。若不合并，主线程退出后，新起的线程将自动退出。

2. 多进程并行任务

核心就是multiprocessing.Pool.map和multiprocessing.Pool.starmap。

2.1. 单参数函数

Pool.map用起来简单顺手，两行代码就可以并行执行多任务：

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool() as pool:
        res = pool.map(square, range(10))

        print(res)

2.2. 绑定多参数函数的多余参数

如果函数有多个参数，然后只遍历其中一个参数，一个直观的想法是：

from multiprocessing import Pool

def mul(x, y):
    return x * y

if __name__ == "__main__":
    double = lambda x: mul(x, 2)

    with Pool() as pool:
        print(pool.map(double, range(10)))

可惜这个方法行不通，会报pickle序列化错误：

_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f9d3e447f70>: attribute lookup <lambda> on __main__ failed

一个解决办法是使用functools.partial，也就多一行代码：

from multiprocessing import Pool
from functools import partial

def mul(x, y):
    return x * y


if __name__ == "__main__":
    double = partial(mul, y=2)

    with Pool() as pool:
        print(pool.map(double, range(10)))

2.3. 多参数函数

并行执行多参数函数需要联合使用starmap和zip：

from multiprocessing import Pool

def mul(x, y):
    return x * y

if __name__ == "__main__":
    with Pool() as pool:
        print(pool.starmap(mul, zip(range(10), range(10)))

2.4. 并行计算任意任务

上面三种都是并行执行同样的函数（作用于多个参数）。如果要并行执行任意任务，参考下面例子，也就几行代码：

from multiprocessing import Pool

def mul(x, y):
    return x * y

if __name__ == "__main__":
    with Pool() as pool:
        res = []
        for i in range(10):
            res.append(pool.async_apply(mul, (i, i)))
        print([r.get() for r in res])

Q. E. D.


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK