## page was renamed from How do I use gzip module with pickle?
== How to use the gzip module with the pickle module ==
I have a python program which creates a list of prime numbers [2L,3L,5L,7L,13L]. This program save the list to a file on the hard disk.

=== example of my code ===
{{{
  fp=open('primes.data','r')
  # ListOfPrimes=[2L,3L,5L,7L]
  ListOfPrimes=cPickle.load(fp)
  fp.close()

  generate_more_primes()

  fp=open('primes.data','w')
  cPickle.dump(ListOfPrimes,fp)
  fp.close()
}}}
=== What I want ===
My problem is that the primes.data file is now 9 megabytes in size
and I wanted to reduce the size taken by the file by first compressing
the data with the gzip module before saving it. Hence I would also need
to decompress the data after I read it from the file.

Any idea how I can do this?

=== Solutions ===
[[lwickjr]]: I'd have to research for details, but you`d have to pickle the data to a string, then save the string to a file through gzip, and read the file from gzip into a string which is then unpickled.

MarcChr: There is no need for an temporary string. Just `import gzip` and use `gzip.open` instead of `open`:
{{{
import gzip, cPickle

fp=gzip.open('primes.data','rb') # This assumes that primes.data is already packed with gzip
ListOfPrimes=cPickle.load(fp)
fp.close()

generate_more_primes()

fp=gzip.open('primes.data','wb')
cPickle.dump(ListOfPrimes,fp)
fp.close()
}}}

You could also use the binary pickle format which is more compact: `cPickle.dump(ListOfPrimes,fp,1)`. 

==== use this module gzippickle.py ====
{{{
"""Generic object pickler and compressor

This module saves and reloads compressed representations of generic Python
objects to and from the disk. Added Protocol field.
"""

__author__ = "Bill McNeill <billmcn@speakeasy.net>"
__version__ = "1.1"

import pickle
import gzip


def save(object, filename, protocol = 0):
        """Saves a compressed object to disk
        """
        file = gzip.GzipFile(filename, 'wb')
        file.write(pickle.dumps(object, protocol))
        file.close()

def load(filename):
        """Loads a compressed object from disk
        """
        file = gzip.GzipFile(filename, 'rb')
        buffer = ""
        while True:
                data = file.read()
                if data == "":
                        break
                buffer += data
        object = pickle.loads(buffer)
        file.close()
        return object

if __name__ == "__main__":
        import sys
        import os.path
        
        class Object:
                x = 7
                y = "This is an object."
        
        filename = sys.argv[1]
        if os.path.isfile(filename):
                data = load(filename)
                z,l = data[0],data[1]
                print "Loaded %s" % data
                print "z.x = %d z.y = %s" % (z.x,z.y)
                print "list = %s" % l
        else:
                z = Object()
                z.x = 666
                l = [2,4,9]
                data = (z,l)
                save(data, filename)
                print "Saved %s" % data



}}}
----
CategoryAskingForHelp CategoryAskingForHelpAnswered