First major project in Python — Quran Revision using SRS is Live
Yesterday on March 4, 2020, I turned on the system in production. It was such a good feeeling and I did throw a small party to celebrate this milestone. I stand in awe of the Almighty for enabling me to materialize this dream of mine in such a short time — Abundant praise and thanks be to God.
So, what is this system about and what problems is this intended to address — I can give a technical intro to the system. But the best way to understand the benefit of the system is to come with me as I take you on a quick tour of my journey of memorizing the Quran and the challenges I faced along the way.
I have started Quran memorization in 2007 with a few surahs that I really loved (Surah Maryam, Taha, Yaseen, etc.). Then I went on to study Arabic grammar through Understand Quran Academy and Shariah Program courses for a few years. Then in 2014 after having a good understanding of Quranic arabic, I started my first serious attempt to memorize the Quran. After doing the last few Juz from the back, I gave up as I was not able to do my revisions regularly.
The biggest roadblocks were travel, vacation, sickness, etc. which puts a brake on the routine — after the break, when I go back to the Quran, I have forgotten a lot of it — it is quite demotivating and a big challenge to rememorize those pages. I have been stuck in the Memorize-Forget-Rememorize cycle for many years.
Then on 17 Jan 2018, I made a resolve to start again on the memorization project and not stop till I complete it. This time, I decided to see whether I can apply a few ideas that I have used successfully to learn Arabic and French (during my stay in Geneva, Swiss). The most beneficial idea was a concept called Spaced Repetition System (SRS). I have used a number of other ideas from memory science, which have helped me a lot like Memory Palaces, but I would not discuss them here.
Here is a great video about SRS — the key idea in the video is the forgetting curve — we forget quite a lot quite quickly, unless we revise it after a certain interval. The solution to the forgetting curve is the spaced repetition system.
And if you prefer reading — here is a great write-up by the same author :
So I wanted to apply SRS to my Quran Revision to address the following challenges:
- Rather than revising a fixed number of pages every day (say 20 pages), can I revise less number of pages and still not forget the pages memorized earlier?
- When I do cycle through the pages in a fixed order from beginning to end (a style used by a lot of quran schools), the strong pages become stronger and the weak pages become weaker. Can I focus my time more on the weaker pages and take it easy on the stronger pages?
- When I travel (which I often do for work), can I just revise the weakest pages in the system — the pages that I am likely to forget?
- Is there a way to break the Memorize-Forget-Rememorize cycle?
The most famous software for SRS is Anki. However, when I started using SRS for Arabic and French in 2011, Anki did not have a mobile app and I found FlashCard Deluxe which was great in terms of its usability. I love the software after using it a few years and hence I wanted to see whether I can use it to memorize the Quran.
That was my phase 1 — After about 9–10 months of using Flashcard Deluxe, I decided to stop as I realized that memorizing individual words and entire pages are so different. The system made it very stressful for me — No matter how much time I allocate, the number of due pages kept increasing. Also, the algorithm used multiplication which produced too many repetitions when the page was new and too few when the page was old — both adding to the Memorize-Forget-Rememorize cycle.
In Phase 2, I decided to create a system using Excel — since I was an advanced user of Excel, it was reasonably easy for me to create the system. I was able to build a system that would help me to solve a number of the above challenges. Here is a snapshot from the system on Day 47 (11 Dec 2018) of using the system.
The biggest problem I had with this system was that I had to be in front of my computer to enter data into Excel and to refresh the pivot tables to produce the output dashboard shown above — so, I used to carry a visiting card and would note down the number of mistakes in each page when I revise. Then, once i am on the computer, I would enter the data in Excel so that the output dashboard can be updated. Then, I would copy the pages for revision in the visiting card. A lot of manual work, but still the system was benefiting me a lot to make me stick with it. When I was traveling or sick, the system started breaking down as I could not reach my computer. Some funny events also happened like misplacing the visiting card and searching for it.
Despite the disadvantages, the system gave me a big confident boost — that SRS can work for Quran memorization and I was beginning to see that I could recollect certain pages almost after 100 days with few mistakes. When I had these experiences, they were so wonderful that I wanted to repeat these experiences.
Then came Phase 3 after last Ramadan — May 2019. I decided to go back to a paper system (notebook) like my children and see whether I can muster the strength to do regular revisions like my children do at their school. So, I devised a system on paper so that my wife and children could use it too. When the system started moving out of me, I had to get out of all the technical jargons and simplify the idea so that my wife could apply it — this led to a lot of time spent in simplifying and improving the algorithm. A number of iterations went on paper along with applying other helpful techniques like 10X method mentioned in this video:
By this time, my first son has completed Hifz and that came as a big motivation boost for me — Upon reflecting, I realized that the key difference between his success and my lack of it was the consistency in revision. When I was preparing for Taraweeh with him during last Ramadan, I realized the power of rote memorization that he has achieved through consistent revision as part of his school system. But I also saw the key weakness of the system — even after doing about 10 cycles of full revision, there were certain pages which were very weak for him — I felt that SRS could help him achieve this. So, my wife tried the paper version of the SRS system and my son reported significant improvement using the system in one of the weakest surahs for him (chronologically, this event occured a lot before last Ramadan but I don’t remember when).
During all this time, I was looking at building a mobile app that would cater to my needs and studied technologies like Flutter, talked to a lot of techies to see whether anyone could help me with building the app. But somehow, nothing concrete came through and I decided to DIY.
Then came the unexpected breakthrough in Phase 4 — once I started studying python, in my first month, an idea came that I could build something quick using Google Forms that would serve me well — if i can use the system on the phone, it would take away the pain of carrying a business card and pen and doing offline data entry. This system has worked very well for me in the last 4 months, despite its key handicap — I had to run the python script to see the output updated. Once again like my Excel system, I had to be in front of the computer. I did some automation so that if my system is open, it would run it automatically every hour. But I could not automate it fully.
Now for Phase 5 from yesterday — Now when I finish revision of a page, I can enter the data in my phone and instantly, the due pages output would be updated as all the data is stored in the cloud. I am so happy and relieved as it addressed a number of pain-points .
Here are some screenshots from the system:
- This shows the the list of pages due as of today — 5 March 2020. It lists some statistics for each page.
If you take the first page 29, it became due today. Interval of 6 indicates that is coming for revision after 6 days. Last score of 0 means zero mistakes. Revision count is the number of revisions done. Page Strength is a calculation that the system uses to determine the weakest pages — so, if I have too many pages to review or too little time, if I focus on the pages with low strength, then I am safe — this calculation is far from perfect. But All praise to God, this is definitely a step in the right direction and I am sure that God will guide me towards discovering the right formula.
2. This shows the data entry page — you capture 2 types of mistakes — word mistakes and line mistakes.
Word mistakes are forgetting a word, changing the grammatical ending, etc. Line mistakes are forgetting an entire ayah or part of it, or mixing it up with another similar verse — in this case, we have to calculate the number of lines affected by this mistakes and enter it. Each word mistake is given a score of 1 and line mistake a score of 4. Then, the system will schedule the revision of the page based on the score.
It is such a great feeling and an amazing journey in the last 5 months. When I started my journey to learn Python on Oct 28, 2019, little did I realize that I could make so much progress so quickly. The biggest blessing has been the freedom to tweak the system a little by little as I felt the need. Since I was both the developer and the end user of the system, it is a rare blessing.
As of now, the system is in an alpha state — meaning, it is not really meant for general users. I have invited my family members and some friends to try the system and give me feedback. But, if you are serious about Quran memorization and want to try the system, please let me know.
If you are interested in knowing more about memory palaces, here are a few resources — it is a great concept and it has helped me tremendously in quran memorization:
For the techies reading this — The system uses Django web framework in Python as the back-end and uses some simple HTML, CSS (with Bootstrap) and Javascript on the front-end. I have hosted it on pythonanywhere.com using a free account. I have used axios for AJAX queries to the REST APIs exposed by Django. I plan to use Vue.JS to improve the usability of the system for general users — that is where I am now — studying Javascript.
I have set up a GitHub repository with all the source code — I am gonna use this for tracking the feature request and bugs using GitHub issues. So, if you want to check out the code to improve your Python or Javascript skills or if you want to contribute, please check it out here.
Here is the release for the first alpha release:
You can file any issues you find or request for new features or improvements to the system in the issues section: