Counter to the [[PythonSpeed/PerformanceTips|PythonSpeed/PerformanceTips]], on python 2.4 the following string concatenation is almost twice as fast:




{{{#!python
from time import time
t = time()

s = 'lksdajflakjdsflku09uweoir'
for x in range(40):
    s += s[len(s)/2:]

print 'duration:', time()-t
}}}



as:







{{{#!python
from time import time
t = time()

s = 'lksdajflakjdsflku09uweoir'
for x in range(40):
    s = "".join((s, s[len(s)/2:]))

print 'duration:', time()-t
}}}



----

On the win32 Python 2.4 I'm seeing the join sample above complete in less than half the time of the concatenating sample.


 . -db


Usually the join() is located ''outside'' the loop, that code makes this extremely hard though (becuase of the self-referencing of the generated string). But that situation is not the norm. -- [[JürgenHermann|JürgenHermann]] 2005-08-01 06:07:51


Are you guys kidding? The whole page is contrieved. Correct implementation of "join" is:


{{{
from time import time
t = time()

s = 'lksdajflakjdsflku09uweoir'
r = [s]
for x in range(40):
    r.append(s[len(s)/2:])
s = "".join(r)

print 'duration:', time()-t
}}}


which gives on [[PythonWin|PythonWin]] 2.4 (#60, Nov 30 2004, 09:34:21) [MSC v.1310 32 bit (Intel)] on win32 execution times:


{{{
1st  duration: 54.4060001373
Last duration: 0.0160000324249
}}}


-- -- [[MikeRovner|MikeRovner]] 2005-08-02 10:19:06


 .
 Mike, that code generates a very different (and much shorter) s. Note how the original code takes the half of the ''preconcatenated'' s, making the size grow exponentially (which generates megabytes of data). -- [[JürgenHermann|JürgenHermann]] 2005-08-30 18:44:05



-- -- [[DavidFord|DavidFord]] 2005-10-18 10:19:06 A few notes (your mileage may vary - this is a 4Mb file being stripped of unprintable characters)


 * Regex replacement rather than creating a list and joining it is 2.5x faster than the tooling above
 * This is far slower than the equivalent Java code (around 4x slower) using String.charAt() and [[StringBuffers|StringBuffers]]


----

I tend to create a stream of strings in a loop that should be concatenated. I generated the script to test the join vs += performance for some randomly generated data and found that for 100,000 strings of length up to ten characters, join is maybe 20% faster than using +=. It certainly was not an order of magnitude faster. The results tended to vary each time through the outer loop, even though I attempted to control the garbage collection and ensured my Windows XP machine was 95% idle apart from running the script.




{{{#!python
from time import time
import random, gc

'''
Check speed of string concatenation vs joining in different versions of Python



Python 2.6.1 (r261:67517, Dec  4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.

    ****************************************************************
    Personal firewall software may warn about the connection IDLE
    makes to its subprocess using this computer's internal loopback
    interface.  This connection is not visible on any external
    interface and no data is sent to or received from the Internet.
    ****************************************************************

IDLE 2.6.1
>>> ================================ RESTART ================================
>>>
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =      0.062  concattime =  0.0780001  join/concat =  79.49%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =      0.062  concattime =  0.0780001  join/concat =  79.49%
 jointime =      0.062  concattime =  0.0780001  join/concat =  79.49%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =      0.062  concattime =  0.0779998  join/concat =  79.49%
 jointime =      0.062  concattime =  0.0780001  join/concat =  79.49%
 jointime =      0.062  concattime =  0.0940001  join/concat =  65.96%
 jointime =  0.0469999  concattime =  0.0780001  join/concat =  60.26%



Python 2.5.3 (r253:67855, Dec 19 2008, 16:58:30) [MSC v.1310 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.

    ****************************************************************
    Personal firewall software may warn about the connection IDLE
    makes to its subprocess using this computer's internal loopback
    interface.  This connection is not visible on any external
    interface and no data is sent to or received from the Internet.
    ****************************************************************

IDLE 1.2.3
>>> ================================ RESTART ================================
>>>
 jointime =  0.0779998  concattime =      0.063  join/concat = 123.81%
 jointime =  0.0780001  concattime =  0.0780001  join/concat = 100.00%
 jointime =      0.109  concattime =  0.0939999  join/concat = 115.96%
 jointime =  0.0780001  concattime =      0.062  join/concat = 125.81%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =  0.0780001  concattime =  0.0779998  join/concat = 100.00%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =      0.062  concattime =      0.172  join/concat =  36.05%
 jointime =      0.079  concattime =  0.0779998  join/concat = 101.28%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
>>>


PythonWin 2.4.3 - Enthought Edition 1.0.0 (#69, Aug  2 2006, 12:09:59) [MSC v.1310 32 bit (Intel)] on win32.
Portions Copyright 1994-2004 Mark Hammond (mhammond@skippinet.com.au) - see 'Help/About PythonWin' for further copyright information.
>>>  jointime =      0.062  concattime =  0.0940001  join/concat =  65.96%
 jointime =  0.0929999  concattime =  0.0780001  join/concat = 119.23%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =      0.062  concattime =  0.0780001  join/concat =  79.49%
 jointime =      0.062  concattime =      0.062  join/concat = 100.00%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =  0.0780001  concattime =  0.0780001  join/concat = 100.00%
 jointime =  0.0940001  concattime =  0.0940001  join/concat = 100.00%
 jointime =      0.063  concattime =  0.0780001  join/concat =  80.77%
 jointime =  0.0780001  concattime =  0.0780001  join/concat = 100.00%


'''

def stringstotest(n=100000, rmin=0, rmax=10):
    ' Returns a list of random strings of between rmin to rmax characters in length'
    allchars = 'qwertyuiopasdfghjklzxcvbnm'
    allchars += allchars.upper()

    return [ "".join( random.choice(allchars)
                      for i in xrange(random.randint(rmin, rmax)) )
             for j in xrange(n) ]

strings = stringstotest()

for i in xrange(10):
    gc.collect()
    gc.disable()
    # JOIN
    t0 = time()
    l = []  # list to "".join()
    for string in strings:
        l.append(string)
    joined = "".join(l)
    jointime = time() - t0

    gc.enable()
    del l, joined
    gc.collect()
    gc.disable()

    # CONCATENATION
    t0 = time()
    s = ''  # string to +=, concatenate
    for string in strings:
        s += string
    concattime = time() - t0

    del s

    print " jointime = %10g  concattime = %10g  join/concat = %6.2f%%" % (
        jointime, concattime, jointime/float(concattime)*100 )

gc.enable()
}}}



-- Paddy3118 2009-01-01 09:48:00


Thanks for this. I've run a modified version of above examples on Windows for Python 2.7.1 and Python 3.2. And strangely '''Python 3.2 is 2x slower'''.


{{{
Z:\Code\Python>"c:\Program Files\Python3.2\python.exe" string_concatenation.py
Loops 100
concatenate1 duration: 0.0s
Size: 173285
concatenate2 duration: 0.015000104904174805s
Size: 173285
concatenate3 duration: 0.0s
Size: 173285

Loops 1000
concatenate1 duration: 10.375s
Size: 17482535
concatenate2 duration: 10.17199993133545s
Size: 17482535
concatenate3 duration: 0.06200003623962402s
Size: 17482535

Z:\Code\Python>"c:\Program Files\Python2.7.1\python.exe" string_concatenation.py
Loops 100
concatenate1 duration: 0.0160000324249s
Size: 173285
concatenate2 duration: 0.0s
Size: 173285
concatenate3 duration: 0.0s
Size: 173285

Loops 1000
concatenate1 duration: 5.17199993134s
Size: 17482535
concatenate2 duration: 5.07800006866s
Size: 17482535
concatenate3 duration: 0.0320000648499s
Size: 17482535
}}}







{{{#!python
from time import time
s = 'qwertyuiopasdfghjklzxcvbnm123456789'
LOOPS = 10

def timeit(f, *args):
    def new_f(*args):
        t = time()
        result = f(*args)
        print('{0} duration: {1}s'.format(f.__name__,time()-t))
        print('Size: {result}'.format(result=result))
    return new_f

@timeit
def concatenate1(s, loops):
    """slow and memory consuming"""
    r = s
    for x in range(loops):
        r += x*s
    return len(r)

@timeit
def concatenate2(s, loops):
    """faster using join in loop"""
    r = s
    for x in range(loops):
        r = "".join((r, x*s))
    return len(r)

@timeit
def concatenate3(s, loops):
    """much faster using list"""
    r = [s]
    for x in range(loops):
        r.append(x*s)
    r = "".join(r)
    return len(r)

if __name__ == '__main__':
    for loops in (100, 1000):
        print("\nLoops {}".format(loops))
        concatenate1(s, loops)
        concatenate2(s, loops)
        concatenate3(s, loops)
}}}



-- llakomy 2011-03-10 12:03:00


Not surprising at all. Every character in the original string and all generated strings is 2-4x the size in Python 3 compared to Python 2 (depending on whether you compiled Python 3 for UTF-16 or UTF-32). Most of the work is copying characters around, so your results would be expected. You might try running the same test with your 's' variable and the "" used with join defined as bytes literals (b'qwertyuiopasdfghjklzxcvbnm123456789' and b"".join(...) respectively).


-- Josh 2011-04-20 16:07:00


As Josh hints at above, the format of the string matters a lot for this. With byte-code strings, concatenating with += is as fast as a .join outside of the for loop, and several times faster than a .join inside the for loop. With Unicode, the concatenation is roughly 300x slower than a .join outside the for loop (with Unicode, the for inside the loop is faster than concatenation, but not by much). Used llakomy's code to test on 2.7.3.


-- Different Josh