Marco de discusión

and shelve modules

8.10.1

Pickling

Consider a situation where information is stored in a file, and you’ve written a Python program to read that file and perform some operation on it. If the contents of the file change regularly, there is usually no recourse but to read the file each time you run your program. But if the contents of the file don’t change very often, it may be worthwhile to store the data in a form which is easier for Python to read than plain text. The pickle module takes any Python object and writes it to a file in a format which Python can later read back into memory in an efficient way. On systems that support it, there will also be a cPickle module which is implemented in the C programming language and will generally much faster than the Python implementaion; I’ll refer to the cPickle module in the examples that follow. Especially in the case where a large amount of processing needs to be done on a data set to create the Python objects you need, pickling can often make your programs run much faster. In the larger scheme of things, using a database may be a more appropriate solution to achieve persistence. But the pickling approach is very simple, and is adequate for many problems.

8.10. OBJECT PERSISTENCE: THE PICKLE/CPICKLE AND SHELVE MODULES129

created through a call to the Pickler function. You pass this function a file (or file-like) object, such as that returned by the built-in open function. (See Section 5.4). If you pass Pickler a second argument of 1, it will write the pickled object in a more efficient binary format, instead of the default human-readable format. Having created a Pickler object, you can now invoke the dump method to pickle the object of your choice. At this point, you can invoke the close method on the file object passed to Pickler, or you can let Python close the file when your program terminates.

As a simple example, let’s consider a dictionary whose elements are dic- tionaries containing information about employees in a company. Of course, the real benefit of pickling comes when you are creating objects from some large, external data source, but this program will show the basic use of the module: import cPickle,sys employees = { ’smith’:{’firstname’:’fred’,’office’:201,’id’:’0001’,’phone’:’x232’}, ’jones’:{’firstname’:’sue’,’office’:207,’id’:’0003’,’phone’:’x225’}, ’williams’:{’firstname’:’bill’,’office’:215,’id’:’0004’,’phone’:’x219’}} try: f = open(’employees.dump’,’w’) except IOError:

print >>sys.stderr, ’Error opening employees.dump for write’ sys.exit(1)

pkl = cPickle.Pickler(f,1) pkl.dump(employees)

When the employee dictionary is needed in another program, all we need to do is open the file containing the pickled object, and use the load function of the cPickle module:

>>> import cPickle

>>> f = open(’employees.dump’,’r’) >>> employees = cPickle.load(f) >>> employees[’jones’][’office’] 207

>>> employees[’smith’][’firstname’] ’fred’

Pickling can offer an especially attractive alternative to databases when you’ve created a class which represents a complex data structure. (Sec- tion 10.4).

8.10.2

The shelve module

Notice that to create a pickled object the entire object must be in memory; similarly, when you unpickle an object, you must read the entire object into memory. When you’re dealing with very large amounts of data, the shelve module may provide a better alternative. (It should be noted that interfacing to a relational database may be a more useful solution in some cases. There are a number of Python modules available to access many different databases; see http://www.python.org/topics/database/modules.html for more in- formation.) The shelve module creates a persistent, file-based version of an object very similar to a Python dictionary, but data is only read from or written to the file when necessary, for example when you need to access a value stored in the file, or add a value to the file. One limitation of shelve objects is that the keys to the objects must be strings, but the values stored in a shelve object can be any Python object, as long as it can be written with the pickle module.

To create a shelve object, use the open function from the shelve module, providing the name of a file to be used to hold the shelve object. The shelve module may call another program which will create more than one file, but the open method will always find your shelved object when you refer to the filename that was used when you first open the shelve object. If the file you specify already exists, but is not a shelve object created in a previous program, you’ll get a anydbm.error exception; otherwise the open function will recognize a previously shelved object and open it. Since information is written to the shelved object only when necessary, it’s important to invoke the close method on any shelved objects you use to insure that changes that you make during your program are properly stored.

Here’s the employee example revisited, using a shelve object. While there’s not much advantage in shelving over pickling for such a small data set, it will illustrate the basic ideas of using a shelve object.

8.11. CGI (COMMON GATEWAY INTERFACE): THE CGI MODULE131