How To Structure a Python Project

Writing programs isn’t all about the code.

Sure when it comes down to it, the code is what makes the functioning program, but in order for that program to run as it does it needs structure. Most programs are made up of multiple files and these files need to be organised in such a way that they all interact seamlessly with one another. This is partly done with the code, by using class inheritance and other methods, but it is also achieved through structuring the project correctly.

A Python project structure sounds fancy but at its core, it is simply the arrangement of files. When you are writing programs, you should always aim to to keep the code clean and well organised and to do this you need a solid, well thought out structure. Not only does it benefit you as a developer but it also benefits anyone else who reads or works on that code, either alongside you or after you.

Python is already a fairly clean and simple language so structuring a project isn’t too complicated. This article will cover the most important requirements of a project structure and should give you a good understanding as to why you should always include them.

What should a project include?

Fortunately for developers, there are a set of guidelines that can be followed in order to ensure a good structure is formed for each project. This creates a somewhat universal method generally accepted by most programmers. The benefit of this is that everyone learns to work according to these guidelines, thus eliminating a lot of confusion that could otherwise occur when looking through a persons project.

Here are the main files a project structure should always include.

README - You should always include a README.rst file for each project. This is a simple text file that contains useful information about the software it relates to. It may seem unnecessary to some, but it’s important to remember that not everyone will interpret the software in the same way. Having a piece of text to go with it can really benefit other developers and potential users. It also gives the developer a way to convey something specific about the program, such as a certain feature.

License - This is the second most important part of the project after the code. This file includes the full text about licensing and copyright, so it’s imperative this file contains accurate information. Specifying a license gives users a legal right to download the project, modify it and then distribute it themselves. There are a number of different licenses to choose from and they are categorised as either ‘More Permissive’ or ‘Less Permissive’.

More permissive licenses focus on the users freedom and usually let them do as they please with the software. Less permissive licenses are more concerned with making sure the code itself remains free, regardless of any changes a user makes to it. This is to prevent someone making a few changes to free software and then selling it to others for profit.

requirements.txt - This file lists any third-party dependencies that are needed for the project. It lets others know what libraries are needed in order to work on the project. In simple terms, it is a list of pip install requirements.

This file is extremely important if you are working with other developers. Thanks to this file, when another person pulls the project to work on, they know exactly what packages need to be installed on their Python environment.

To create a requirements.txt file you simply need to list of all the necessary libraries for the project. Here is an example to show you how one should look.

Flask==0.12
Jinja==2.9.5
MarkupSafe==0.23
Werkzeug==0.11.15
argparse==1.2.1
click==6.7
wsgiref==0.1.2

setup.py - In terms of project structure, this file tells the user that the package they are about to install has been packaged and distributed with Distutils. This is the standard for distributing Python modules.

Here is a simple example of how a setup.py file is structured using Distutils.

from distutils import setup
setup(
name=='nameOfProject',
version=='1.0',
description=='awesomeProject',
author=='Joe Bloggs',
author email='joe@awesomeproject.com',
url='hhtp://joebloggsblog.com',
packages ['awesomeProject'],
install_requires=['coolProject', 'greatProject',]
scripts=['path/to/your/script',],
...
)

As you can see the file lists all the information about the package. The main purpose of the ‘scripts’ is to tell the Distutils about the module distribution. This ensures that various commands on the modules do the right things. 

setup.cfg - Sometimes in order to write down everything that’s needed to build a distribution you need some information from the user, or the user’s system. If that information is simple, then a setup.cfg file can be used to get such information. An example of something simple would be needing to search a list of directories for certain types of files.

In other words, you can override parts of the setup.py file by editing the setup.cfg file. You can also override anything in the setup.cfg file using the command-line options to setup.py.

Modules - Python files are saved with the .py extension. These files on their own are called modules. Within these modules, you can define classes, variables and functions under the same name as the module itself. Modules are to programming libraries what books are to regular libraries. You use a library to find a book (module) that has the information you need and then you borrow it for your own use.

Packages - If you put one or more modules inside an __init__.py file then you have a package. In short, a package is essentially a folder containing your code. A package is really just a directory, however all packages must include an __init__.py file. This file tells Python to treat the directories as containing packages.

An example of a complex package is NumPy. This is a library used for scientific computing so as you can imagine the library is made up of many different modules all coming together to form the package that is NumPy. This shows the benefit of using packages as a method of organisation.

How these pieces come together

Here is an example of how to structure a simple Python project.

README.rst
LICENSE
setup.py
requirements.txt
example/__init__.py
example/mainprog.py
example/addlprog.py
tests/test.py

From this example you should be able to get an idea of how each file plays its part in the structure. We start with the text based aspects of the project, listing information about the software, copyright and package requirements. That is then followed by the core elements i.e. the code and then finally we have a test file.

Test files as you have probably guessed, are used to test the program works as it should do. Tests need to be kept simple and consistent while also being thorough and fast. No developer wants to be held up testing the program, so a blend of efficiency and reliability is key here.

This is a very basic example however and most projects contain a lot more files and sub-directories. If you are new to programming, it is worth getting familiar with this structure even if you aren’t working with many files. Good habits go a long way in programming, not just in terms of writing code but also when working with others. If you get into the habit of structuring each project correctly it will become just another part of the process and you and your peers will have a much more enjoyable programming experience.