Reading and Writing files in Python Using the Pickle Library.

Photo by Jonathan Pielmayer on Unsplash

Storing data in files can be an important tool for data scientists. It is essential to be able to store information in files of data so that you can access them and so that others can update them as needed. In this article I will show you how to use the pickle library in Python to dump and load data to and from a file.

First you want to use the import pickle and import os commands in your favorite Python editor like this (McGrath, 2016):

import pickle
import os

If you want to see what methods are available in each library try entering these commands:

print(dir(pickle))
print(dir(os))

Next, I populate a dictionary with values. I am a Tolkien fan so here is a dictionary of some of the mythical creatures in Middle Earth.

Middle_Earth_Dictionary = {1:"Hobbits", 2:"Dwarves", 3:"Elves", 4:"Wizzards", 5:"People", 6:"Skin Changers", 7:"Orcs", 8:"Easterlings", 9:"Uruk Hai", 10:"Eagles"}

Please note that this is not an exhaustive list of the creatures in middle earth.

Then I can open a file to get it ready to receive information like so:

file = open(‘Tolkien.txt’,’wb’)

As you can see, open is a method that takes two arguments. The first is the file name as a string, while the second is the type of connection to that file as a string. The connection type ‘wb’ is used to write to the file, while ‘rb’ is used to read the data from the file, as you will see later.

Then I store the data using the pickle.dump method like this:

pickle.dump(Middle_Earth_Dictionary, file)

The pickle.dump method also takes two arguments. The first is the name of the variable containing the data to be stored in the file and the second is the name of the file in which to store the data. And finally I close the file for writing like so:

file.close()

To write to a file with Pickle, you need to do these three steps (McGrath 2016).

When you are ready to read the data from the file you must execute a similar set of commands. These three commands will get the job done (McGrath, 2016):

Middle_Earth_file = open(“Tolkien.txt”, “rb”)
Middle_Earth_dict = pickle.load(Middle_Earth_file)
Middle_Earth_file.close()

This time we used the ‘rb’ string when opening the file to indicate that we want to read the file instead of write to it. The pickle.load method uses just one argument that is the name of the open file where we have stored the data. And don’t forget to close the file when you are done.

Then I print the variable where the pickled dictionary was saved using the following code:

print(Middle_Earth_Dict)

This gives me the following:

{1: 'Hobbits',
 2: 'Dwarves',
 3: 'Elves',
 4: 'Wizzards',
 5: 'People',
 6: 'Skin Changers',
 7: 'Orcs',
 8: 'Easterlings',
 9: 'Uruk Hai',
 10: 'Eagles'}

This is a good way to access stored data, but what if we want to update the file with more information? I can first create another dictionary with different keys. This is to store the different battles in middle earth.

Battles = {11: “Battle of Five Armies”, 12:”Battle of Dale”,
           13:”Battle for Helms Deep”, 14:”Seige of Gondor”,
           15:”Battle of the Pelenor Fields”, 
           16:”Battle at the Black Gate”}

Again this is not a complete list of the battles in middle earth. But, when I dump this dictionary to the file, and retrieve it using the following code(McGrath, 2016):

Middle_Earth_file = open(“Tolkien.txt”, “wb”)
pickle.dump(Battles, Middle_Earth_file)
Middle_Earth_file.close()

And then load it back in using a different variable(McGrath, 2016):

Middle_Earth_file = open(“Tolkien.txt”, “rb”)
Middle_Earth_dict = pickle.load(Middle_Earth_file)
Middle_Earth_file.close()

I find that only the battles are present in the Middle_Earth_dict variable as follows.

{11: 'Battle of Five Armies',
12: 'Battle of Dale',
13: 'Battle for Helms Deep', 
14: 'Seige of Gondor', 
15: 'Battle of the Pelenor Fields', 
16: 'Battle at the Black Gate'}

This is due to the fact that dump replaces the entire file with a new file. To fix this I have to combine the two dictionaries using the following code (Lenka, 2022):

Middle_Earth_Dictionary.update(Battles)

This adds the contents of the Battles dictionary to the Middle_Earth_Dictionary. Now Middle_Earth_Dictionary has all of the data:

{1: 'Hobbits',
 2: 'Dwarves',
 3: 'Elves',
 4: 'Wizzards',
 5: 'People',
 6: 'Skin Changers',
 7: 'Orcs',
 8: 'Easterlings',
 9: 'Uruk Hai',
 10: 'Eagles',
 11: 'Battle of Five Armies',
 12: 'Battle of Dale',
 13: 'Battle for Helms Deep',
 14: 'Seige of Gondor',
 15: 'Battle of the Pelenor Fields',
 16: 'Battle at the Black Gate'}

Now I can dump the dictionary into the file just like I did above using the following code:

ME_file = open(“Tokien.txt”, “wb”)
pickle.dump(Middle_Earth_Dictionary,ME_file)
ME_file.close()

And then load it back up into a different variable like this:

ME_file = open(“Tokien.txt”, “rb”)
ME_D = pickle.load(ME_file)
ME_file.close()

And bingo! When I print the ME_D dictionary variable I get the following output.

{1: 'Hobbits',
 2: 'Dwarves',
 3: 'Elves',
 4: 'Wizzards',
 5: 'People',
 6: 'Skin Changers',
 7: 'Orcs',
 8: 'Easterlings',
 9: 'Uruk Hai',
 10: 'Eagles',
 11: 'Battle of Five Armies',
 12: 'Battle of Dale',
 13: 'Battle for Helms Deep',
 14: 'Seige of Gondor',
 15: 'Battle of the Pelenor Fields',
 16: 'Battle at the Black Gate'}.

This is the same dictionary that we printed above when we printed the original data.

This article has given you a glimpse into the realm of storing data with pickle. If you want to learn more please visit the sites and book below:

I hope you have enjoyed this article and that it was helpful to you in learning how to store data in Python. Please consider following me.

BIBILIOGRAPHY

Python Pickling

pickle - Python object serialization - Python 3.10.4 documentation

The Python pickle Module: How to Persist Objects in Python - Real Python

McGrath, Mike. Python in Easy Steps. Easy Steps Limited. United Kingdom. 2016.

pickle - Python object serialization - GeeksforGeeks

Python | Merging two Dictionaries - GeeksforGeeks

Lenka, Chinmoy. Python|Merging two Dictionaries. Geeks for Geeks. https://www.geeksforgeeks.org/python-merging-two-dictionaries/ Accessed on May 16, 2022 at about 4:00 pm.