Memorize terminal Commands project

How does it work?

Memorizing/Training

After the user registers successfully, the training platform appears. At the right side, the user can choose if he/she would like to learn based on the time or based on the number of commands (questions). The user can choose the time duration (default 5 minutes) or the number of the commands (default 20) for the current training session.

There are two topics which the user can currently select. After clicking the "Start" button, the questions start appearing, and the user should answer the question by writing a command inside the input text. The user should submit his answer either by clicking the "Submit" button or pressing "Enter". If the answer of the user is correct, then the border's color of the question card turns to green, and then the next question appears. If the user's answer is wrong, then the border's color of the question card turns to red, and the right answer appears in the bottom.

Tracing performance

The user can stop the training any time by pressing the "Stop training" button. To keep tracking the training progress, the user can save the training data anytime by clicking the "Save results". To trace the training progress, the user can click on the "Trace performance" button. A graph appears on the bottom of the page and shows the scores of the user for each training session for both topics. At the bottom of the graph, the user can click on the "History detail" to move to the historical data page and browse the previous training sessions.

Table of content

Introduction

Learning is my passion. In the last few years, I spent most of my free time learning something new. I focus on technology topics such as AI (Artificial Intelligence), Data Engineering, DevOps, and so on. I come across that most of the hands-on technologies require fundamental knowledge in some necessary tools such as the Linux terminal-command line and Git.

Last year I wrote a small program in Python that helps to memorize the new stuff that I am learning. I gathered about 300+ Linux, and Git commands. I kept using the program every day for a while. After a few months, I started noticing that I began to see a positive effect on my daily life processes:

  • I learn new technologies faster than I usually do;
  • I become more efficient than before when it comes to closing coding tickets in my daily job as data scientists;
  • I have a KPI ("Key Performance Indicator") that helps me to set my goals and monitor my learning progress;
  • Of course, better memory than before;
  • The most important thing: learning with fun;

My pain is that if I don't use the newly gained knowledge soon in a project, I lose it. Therefore, I decided this year to bring my great small programs out to the world as side projects. I started with "Memorize terminal commands".

Goal

The goal of the project is to provide a platform for memorizing the terminal-command line commands of some essential tools such as Linux and Git. My motivation for this project is:

  • As I mentioned above, I want to apply my knowledge and sharpen my skills. I want to bring my projects out into the world
  • I want to have the more and more production-based experience, especially in the case of data science projects

As you may know, data scientists work mostly offline. Outside my daily job, I have been engaging in several Kaggle competitions during the last two years. I have been organizing Kaggle Munich meetup events for more than one and a half year. Such activities are an excellent opportunity to have fun together with improving as a data scientist, but they don't close the real gap between education and industry.

In work, data in production is sensitive for the business and testing new technologies with such data for learning is not always a good idea. In the "Memorize terminal commands" project, I am not collecting any sensitive data. I am using the Emails for the login purpose, but I don't perform any Email address validation. You can use a fake Email address when you register.

In the second part of the project, I am going to build a machine learning model to predict the score of the user. The model should react near the real-time. I will provide the expected score to the user on the training page.

Technology

Front-End

I implemented the Front-End almost from scratch. I used pure HTML, CSS, and Javascript in the beginning. Later, I added Bootstrap and some JQuery to the code.

First time I wrote an HTML code was between 2000 and 2003 when I was at high school. At that time, I created my first website, where I published my "visual basic 6" programs that I wrote to solve Math problems and visualize some functions. I used a free service from Yahoo to host the website. Today, I noticed that the world changed a lot regarding writing web services. I have to learn a lot of new stuff, and I am happy that I did.

There are five HTML pages: Register, Training, History, About, and all of them are an extension from the base HTML page. I used the "jinja2" template to replace most of the variable inside the HTML pages. I used Bootstrap. When I needed more advanced CSS scripts, I embedded them inside the HTML page since there were not that much. I tried to split the Javascript files based on the functionality and the page that they affect. Since I am willing to maintain the website in the future, I am going to refactor the code frequently.

In the server side, I started using "Django" first. Later I noticed that "Django" is too heavy for my time plan and my current knowledge level , so I switched to "flask". I don't use the Cookies storage in the current version of the project, but I am expecting to use it later.

Back-End

When I started the implementation, I use SQLite to store the registration information of the user. I used my Laptop's local storage to save the probability array, which I update after each training session and the training results as a JSON file. I hash the Email of the user and use this hash as the name of the directory. I used this setup most of the time since I was not sure about where I am going to host the website.

Currently, I save the questions as CSV files in the file system. Later, when I increase the number of the topics and the questions, I am going to replace the CSV files with a database.

I was confused between hosting my website using a pre-managed service from a cloud provider or using a VM (Virtual Machine) server and take care of all heavy deployment work by myself. Although the second option is more attractive, I decided to go to the first option because I want to bring the website online as soon as possible. I narrowed my choices to AWS Beanstalk, Google App Engine, and Heroku. Honestly, I could find sufficient arguments about why I should use one of them against the others. Finally, I decided to host the project on Google since I still have more than 250 Euro credit for enough time.

It was relatively easy to integrate Google services inside my code. So I switched from SQLite to MySQL and from saving the results in my local system to the Google Storage. I am using the standard environment inside the Google App Engine for Python37. The great thing about this setup is that it scales in seconds up and down based on traffic and I pay as I go. I am using the smallest instance of MySQL available in Google Cloud Platform. For development, it is an OK choice. Later I am going to switch to a standard instance for better availability.

I limit my budget to 10 Euros/day for the Google App Engine and almost 10 Euro/Month for the MySQL instance. Together with the Google Storage cost, I am expecting that the maximum cost of the current back-end should not exceed 320 Euro/month. I am assuming that the average cost will be about 15-20 Euro/month. However, this number may increase to 70-80 Euro/month when I switch to the standard MySQL instance.

Architecture

Memorize Terminal Commands by Maher Deeb

Future Features

The next steps will be:

  • I want to improve the UI and provide a better Responsive Web Design for the project.
  • I want to improve the backend of the project.
  • The user should be able to feed the database with new commands.
  • Although I collected about 300+ questions, I need to polish those questions, so that the user has no problem to understand them. Now I have 50 questions in the database for each topic. I am planning to add more questions weakly. It will take a while until I have all the questions in place.
  • I am going to include more hints and learning resources which will be associated with the questions and the topics.
  • I am planning to add more topics such as Docker, Kubernetes, Python, and so on.
  • I am planning to include a machine learning model that predicts the score of the user before the user starts training.
  • I am going to announce other features in the future.

Issues

I am aware that there are still some issues which I am going to take care of as soon as possible. Most of the features work as expected. However, in different browsers, I observed some unexpected behavior. The website works in Firefox, Chrome, latest version of Opera, Safari, and Edge. Internet Explorer is NOT supported.

Your feedback is highly appreciated. You can send me your feedback anytime via Email.

Memorize Commands
Maher Deeb
Data Scientist at KI labs
Contact me:
maherdeebcv@gmail.com
my Github my LinkedIN
my photo