python 列表解析和函数式函数比“for循环”更快吗?

zed5wv10  于 2023-01-08  发布在  Python
关注(0)|答案(8)|浏览(201)

就Python的性能而言,列表解析或者像map()filter()reduce()这样的函数比for循环快吗?为什么从技术上讲,它们以C的速度运行,而for循环以Python虚拟机的速度运行
假设在我开发的一个游戏中,我需要使用for循环来绘制复杂而巨大的Map,这个问题肯定是相关的,因为如果列表理解确实更快,那么为了避免延迟,它将是一个更好的选择(尽管代码在视觉上很复杂)。

vh0rcniy

vh0rcniy1#

下面是一些基于经验的粗略指导和有根据的猜测,您应该timeit或剖析您的具体用例以获得确切的数字,这些数字可能偶尔与下面的数字不一致。
列表解析通常比完全等价的for循环(实际构建列表)快一点点,很可能是因为它不必在每次迭代时查找列表及其append方法。

>>> dis.dis(<the code object for `[x for x in range(10)]`>)
 1           0 BUILD_LIST               0
             3 LOAD_FAST                0 (.0)
       >>    6 FOR_ITER                12 (to 21)
             9 STORE_FAST               1 (x)
            12 LOAD_FAST                1 (x)
            15 LIST_APPEND              2
            18 JUMP_ABSOLUTE            6
       >>   21 RETURN_VALUE

使用列表解析代替一个循环,它 * 不 * 构建列表,毫无意义地积累一个无意义值的列表,然后丢弃这个列表,通常会 * 慢 *,因为创建和扩展列表的开销很大。列表解析并不是魔术,它本质上比一个好的旧循环快。
对于功能列表处理函数:虽然这些都是用C编写的,并且可能比用Python编写的等效函数性能更好,但是它们 * 不 * 一定是最快的选择。如果函数也是用C编写的,那么速度会有所提高。但是大多数情况下使用lambda(或其他Python函数),重复设置Python堆栈帧等的开销会消耗掉所有的节省。而没有函数调用(例如,列表解析而不是mapfilter)通常稍快。
假设在我开发的一个游戏中,我需要使用for循环来绘制复杂而巨大的Map,这个问题肯定是相关的,因为如果列表理解确实更快,那么为了避免延迟,它将是一个更好的选择(尽管代码在视觉上很复杂)。
很有可能,如果用良好的非"优化" Python编写这样的代码还不够快,那么再多Python级别的微优化也不会使它足够快,您应该开始考虑降到C。虽然广泛的微优化通常可以显著提高Python代码的速度,但仍有很低的此外,即使在你达到这个上限之前,咬紧牙关写一些C语言也会变得更加经济高效(15%的加速比同样的努力300%的加速)。

whlutmcx

whlutmcx2#

如果您查看www.example.com上的信息python.org,可以看到以下摘要:

Version Time (seconds)
Basic loop 3.47
Eliminate dots 2.45
Local variable & no dots 1.79
Using map function 0.54

但是您确实应该详细阅读上面的文章,以了解性能差异的原因。
我还强烈建议您使用timeit来计算代码的时间。在一天结束时,可能会出现这样的情况,例如,当满足某个条件时,您可能需要中断for循环。这可能比通过调用map来查找结果更快。

0g0grzrc

0g0grzrc3#

您特别询问了map()filter()reduce(),但我认为您想了解函数式编程的一般知识,我自己在计算一组点中所有点之间的距离的问题上测试了这个问题,函数式程序设计(使用内置itertools模块中的starmap函数)比for循环稍慢(实际上是for循环的1.25倍)。

import itertools, time, math, random

class Point:
    def __init__(self,x,y):
        self.x, self.y = x, y

point_set = (Point(0, 0), Point(0, 1), Point(0, 2), Point(0, 3))
n_points = 100
pick_val = lambda : 10 * random.random() - 5
large_set = [Point(pick_val(), pick_val()) for _ in range(n_points)]
    # the distance function
f_dist = lambda x0, x1, y0, y1: math.sqrt((x0 - x1) ** 2 + (y0 - y1) ** 2)
    # go through each point, get its distance from all remaining points 
f_pos = lambda p1, p2: (p1.x, p2.x, p1.y, p2.y)

extract_dists = lambda x: itertools.starmap(f_dist, 
                          itertools.starmap(f_pos, 
                          itertools.combinations(x, 2)))

print('Distances:', list(extract_dists(point_set)))

t0_f = time.time()
list(extract_dists(large_set))
dt_f = time.time() - t0_f

功能性版本比程序性版本快吗?

def extract_dists_procedural(pts):
    n_pts = len(pts)
    l = []    
    for k_p1 in range(n_pts - 1):
        for k_p2 in range(k_p1, n_pts):
            l.append((pts[k_p1].x - pts[k_p2].x) ** 2 +
                     (pts[k_p1].y - pts[k_p2].y) ** 2)
    return l

t0_p = time.time()
list(extract_dists_procedural(large_set)) 
    # using list() on the assumption that
    # it eats up as much time as in the functional version

dt_p = time.time() - t0_p

f_vs_p = dt_p / dt_f
if f_vs_p >= 1.0:
    print('Time benefit of functional progamming:', f_vs_p, 
          'times as fast for', n_points, 'points')
else:
    print('Time penalty of functional programming:', 1 / f_vs_p, 
          'times as slow for', n_points, 'points')
bwitn5fc

bwitn5fc4#

我修改了@Alisa的代码,并使用cProfile来说明为什么列表理解更快:

from functools import reduce
import datetime

def reduce_(numbers):
    return reduce(lambda sum, next: sum + next * next, numbers, 0)

def for_loop(numbers):
    a = []
    for i in numbers:
        a.append(i*2)
    a = sum(a)
    return a

def map_(numbers):
    sqrt = lambda x: x*x
    return sum(map(sqrt, numbers))

def list_comp(numbers):
    return(sum([i*i for i in numbers]))

funcs = [
        reduce_,
        for_loop,
        map_,
        list_comp
        ]

if __name__ == "__main__":
    # [1, 2, 5, 3, 1, 2, 5, 3]
    import cProfile
    for f in funcs:
        print('=' * 25)
        print("Profiling:", f.__name__)
        print('=' * 25)
        pr = cProfile.Profile()
        for i in range(10**6):
            pr.runcall(f, [1, 2, 5, 3, 1, 2, 5, 3])
        pr.create_stats()
        pr.print_stats()

以下是结果:

=========================
Profiling: reduce_
=========================
         11000000 function calls in 1.501 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.162    0.000    1.473    0.000 profiling.py:4(reduce_)
  8000000    0.461    0.000    0.461    0.000 profiling.py:5(<lambda>)
  1000000    0.850    0.000    1.311    0.000 {built-in method _functools.reduce}
  1000000    0.028    0.000    0.028    0.000 {method 'disable' of '_lsprof.Profiler' objects}

=========================
Profiling: for_loop
=========================
         11000000 function calls in 1.372 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.879    0.000    1.344    0.000 profiling.py:7(for_loop)
  1000000    0.145    0.000    0.145    0.000 {built-in method builtins.sum}
  8000000    0.320    0.000    0.320    0.000 {method 'append' of 'list' objects}
  1000000    0.027    0.000    0.027    0.000 {method 'disable' of '_lsprof.Profiler' objects}

=========================
Profiling: map_
=========================
         11000000 function calls in 1.470 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.264    0.000    1.442    0.000 profiling.py:14(map_)
  8000000    0.387    0.000    0.387    0.000 profiling.py:15(<lambda>)
  1000000    0.791    0.000    1.178    0.000 {built-in method builtins.sum}
  1000000    0.028    0.000    0.028    0.000 {method 'disable' of '_lsprof.Profiler' objects}

=========================
Profiling: list_comp
=========================
         4000000 function calls in 0.737 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.318    0.000    0.709    0.000 profiling.py:18(list_comp)
  1000000    0.261    0.000    0.261    0.000 profiling.py:19(<listcomp>)
  1000000    0.131    0.000    0.131    0.000 {built-in method builtins.sum}
  1000000    0.027    0.000    0.027    0.000 {method 'disable' of '_lsprof.Profiler' objects}

恕我直言:

  • reducemap通常非常慢,不仅如此,与sum处理列表相比,在map返回的迭代器上使用sum也很慢
  • for_loop使用append,这在某种程度上当然比较慢
  • map相比,list-comprehensive不仅构建列表花费的时间最少,而且使sum的速度更快
8zzbczxx

8zzbczxx5#

我写了一个简单的脚本来测试速度,这是我发现的。实际上for循环在我的例子中是最快的。这真的让我很惊讶,看看bellow(正在计算平方和)。

from functools import reduce
import datetime

def time_it(func, numbers, *args):
    start_t = datetime.datetime.now()
    for i in range(numbers):
        func(args[0])
    print (datetime.datetime.now()-start_t)

def square_sum1(numbers):
    return reduce(lambda sum, next: sum+next**2, numbers, 0)

def square_sum2(numbers):
    a = 0
    for i in numbers:
        i = i**2
        a += i
    return a

def square_sum3(numbers):
    sqrt = lambda x: x**2
    return sum(map(sqrt, numbers))

def square_sum4(numbers):
    return(sum([int(i)**2 for i in numbers]))

time_it(square_sum1, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum2, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum3, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum4, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
0:00:00.302000 #Reduce
0:00:00.144000 #For loop
0:00:00.318000 #Map
0:00:00.390000 #List comprehension
xsuvu9jc

xsuvu9jc6#

我修改了@alpii的一些代码,发现列表解析比for循环快一些,这可能是int()造成的,列表解析和for循环之间不公平。

from functools import reduce
import datetime

def time_it(func, numbers, *args):
    start_t = datetime.datetime.now()
    for i in range(numbers):
        func(args[0])
    print (datetime.datetime.now()-start_t)

def square_sum1(numbers):
    return reduce(lambda sum, next: sum+next*next, numbers, 0)

def square_sum2(numbers):
    a = []
    for i in numbers:
        a.append(i*2)
    a = sum(a)
    return a

def square_sum3(numbers):
    sqrt = lambda x: x*x
    return sum(map(sqrt, numbers))

def square_sum4(numbers):
    return(sum([i*i for i in numbers]))

time_it(square_sum1, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum2, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum3, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum4, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
0:00:00.101122 #Reduce

0:00:00.089216 #For loop

0:00:00.101532 #Map

0:00:00.068916 #List comprehension
fkaflof6

fkaflof67#

Alphii answer做一点改动,实际上for循环是第二好的,比map慢6倍左右

from functools import reduce
import datetime

def time_it(func, numbers, *args):
    start_t = datetime.datetime.now()
    for i in range(numbers):
        func(args[0])
    print (datetime.datetime.now()-start_t)

def square_sum1(numbers):
    return reduce(lambda sum, next: sum+next**2, numbers, 0)

def square_sum2(numbers):
    a = 0
    for i in numbers:
        a += i**2
    return a

def square_sum3(numbers):
    a = 0
    map(lambda x: a+x**2, numbers)
    return a

def square_sum4(numbers):
    a = 0
    return [a+i**2 for i in numbers]

time_it(square_sum1, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum2, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum3, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
time_it(square_sum4, 100000, [1, 2, 5, 3, 1, 2, 5, 3])

主要的变化是消除了缓慢的sum调用,以及在最后一种情况下可能不必要的int()。实际上,将for循环和map放在同一个术语中使其变得非常真实。请记住,lambda是函数概念,理论上不应该有副作用,但是,它们 * 可能 * 会有副作用,比如添加到a。本例中的结果使用Python 3.6.1、Ubuntu 14.04、英特尔(R)酷睿(TM)i7-4770 CPU@3.40GHz

0:00:00.257703 #Reduce
0:00:00.184898 #For loop
0:00:00.031718 #Map
0:00:00.212699 #List comprehension
bd1hkmkf

bd1hkmkf8#

我正在寻找一些关于“for”循环和“list comprehension”的性能信息,偶然发现了这个主题。Python 3.11发布(2022年10月)已经有几个月了,Python 3.11的主要特性之一就是速度改进。https://www.python.org/downloads/release/python-3110/
Faster CPython项目已经产生了一些令人兴奋的结果。Python 3.11比Python 3.10快了10-60%。平均而言,我们测量了标准基准测试套件的1.22倍加速。参见Faster CPython了解详细信息。
我运行了最初由Alphi发布,然后由jjmerelo“扭曲”的相同代码。Python3.10和Python3.11的结果如下:

from functools import reduce
    import datetime
    
    def time_it(func, numbers, *args):
        start_t = datetime.datetime.now()
        for i in range(numbers):
            func(args[0])
        print(datetime.datetime.now()-start_t)
    
    def square_sum1(numbers):
        return reduce(lambda sum, next: sum+next**2, numbers, 0)
    
    
    def square_sum2(numbers):
        a = 0
        for i in numbers:
            a += i**2
        return a
    
    
    def square_sum3(numbers):
        a = 0
        map(lambda x: a+x**2, numbers)
        return a
    
    
    def square_sum4(numbers):
        a = 0
        return [a+i**2 for i in numbers]
    
    
    time_it(square_sum1, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
    time_it(square_sum2, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
    time_it(square_sum3, 100000, [1, 2, 5, 3, 1, 2, 5, 3])
    time_it(square_sum4, 100000, [1, 2, 5, 3, 1, 2, 5, 3])

我还没有计算出确切的百分比改进,但很明显,性能增益-至少在这个特定的例子-似乎是令人印象深刻的(3至4倍的速度),除了'Map',它有微不足道的性能改进。

#Python 3.10
0:00:00.221134  #Reduce
0:00:00.186307  #For
0:00:00.024311  #Map
0:00:00.206454  #List comprehension

#python3.11
0:00:00.072550  #Reduce
0:00:00.037168  #For
0:00:00.021702  #Map
0:00:00.058655  #List Comprehension

注:我在Windows 11下使用WSL运行的Kali Linux虚拟机上运行了这段代码。我不确定如果在Linux示例上原生运行(裸机),这段代码是否会执行得更好。
我的Kali Linux VM规格如下:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   39 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          8
On-line CPU(s) list:             0-7
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Core(TM) i7-6700T CPU @ 2.80GHz
CPU family:                      6
Model:                           94
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
Stepping:                        3
BogoMIPS:                        5615.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
Virtualization:                  VT-x
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       128 KiB (4 instances)
L1i cache:                       128 KiB (4 instances)
L2 cache:                        1 MiB (4 instances)
L3 cache:                        8 MiB (1 instance)
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Unknown: Dependent on hypervisor status
Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

相关问题