Month 20 — Slow and steady with TDD, Pytest and Django

Data Quality checking with Pandas

In June, We started with some utility functions on top of Pandas to speed up some of the upfront data quality (DQ) checking work I do as part of my consulting work. When I connect to tables from ERP systems especially from big vendors, there are a lot of empty columns, columns with just single-values — I wanted to write a script that would take a table name and strip out all these unnecessary columns and give me a frequency table for the rest of the columns. When I go through the frequency table output with the client, we are able to identify many other DQ issues.

Many little moments of joy and satisfaction from TDD

So, I set up pytest and started writing the function using TDD. It has been such a joy to use TDD, especially the small atomic steps that it allows us to take and the ability to see how things unfold, especially when the output is something complex. The ability to see the green in the Terminal as I press the Save key is satisfying — thanks a lot to the creator of pytest and pytest-watch. What tremendous tools. I could feel the freedom to refactor freely since the tests were there to cover for anything I could miss out — this is such a satisfactory feeling.

Jupyter or VSCode?

Since I was new to TDD, I had to figure out a lot of things and make them up as I went along. I did not know whether to use VScode along with pytest as the main place to code or Jupyter. Given the user-friendliness of Jupyter, I started off with Jupyter. But getting TDD to work in Jupyter was a bit cumbersome. But after some rounds, I really missed seeing the red-to-green change as I write each line of code. TDD really connects each line of code I write with the test it is supposed to make it green — I was missing it.

Discovered YAML

As part of this project, I was writing some config files in JSON format. But I hate the need to explicitly put double-quotes around every piece of text in JSON. So, I was quite pleased to see that YAML just allows me to do just that. When I digged into YAML, I realized how simple and beautiful it was. Like markdown, the syntax is unobstrusive — a human could write it and a program could understand. So, I started using YAML in my project also.

Quran SRS Project v2

It has been almost a year since I have touched this personal project. I have been using the app every day and so was my family and a few friends. I have run into a few bugs in the tool and my family also have seen some of these bugs. But I decided not to go on fixing the bugs or introducing new features unless I had tests for all functionality. But this was such an enormous task that I kept pushing it away. But finally around mid-June, on a weekend, we got into it.

describe-style tests in Python

As I was writing more tests in python, I was longing for the beautiful describe-it syntax in JS that I was using in Cypress. So, after a google search and a few minutes of exploration, I landed onto pytest-describe. Though it is not exactly like the JS flavour, it is very cool as it allows me to write test case name which read like a typical English sentence.

  • Cypress
  • pytest
  • pytest watch
  • pytest describe
  • Test Runner/Explorer in VSCode
  • Django native debugging in VSCode
  • Django pytest extensions

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Siraj Samsudeen

Siraj Samsudeen

26 Followers

An entrepreneur who is coming back to coding after a gap of 16 years due to love of coding.