Penesive Random Experiments

Creating a URL Shortener with Flask - Part 5

In part 5 of this series, i will discuss how i added login support for the url shortener. If the user has logged in we will provide him with a set of links that he created and statistics regarding all the links he created. The first step is creating the user account so we can track all the user login.

  1. As usual lets create a new branch for handling of user account

         git branch -b addLoginSupport
    
  2. I will be using a flask package called the flask-login that handles most of the login handling.

         pip install flask-login
    
  3. Lets first add the backend storage , our database model is going to be very simple to start with , we will have a ‘_id’ field which is the users email address and a ‘password’ stored in the ‘password’ field.

  4. The code for storing is going to be straight forward similar to that of the short url insertion we saw in part 1. The code is as follows

     import pymongo
     from MongodbHandler import MongoDatabaseHandler
    
    
     class user_database(object):
         def __init__(self):
             """ save the username and password """
             databaseHandler = MongoDatabaseHandler()
             self.collection = databaseHandler.get_users_collection()
    
         def get_password(self, username):
             """ get password from a given username """
             doc = self.collection.find_one({'_id': username}, {'_id': 0, 'password': 1})
             if doc is None:
                 return None
             return doc['password']
    
         def save_user(self, username, password):
             """save the username and password to the database
                 return false if trying to add the same user again
             """
             try:
                 self.collection.insert_one({'_id': username, 'password': password})
             except pymongo.errors.DuplicateKeyError:
                 return False
    
             return True
    
         def delete_user(self, username):
             """  delete the user given an username  """
    
             result = self.collection.delete_one({'_id': username})
             if result.deleted_count:
                 return True
             else:
                 return False
    
    
  5. The corresponding test coverage for the same is as follows. We as usual duct type to make sure that the database writes are to the test database

     import unittest
     import pymongo
    
     from user_database import user_database
    
     class TestUser(unittest.TestCase):
         def get_user_collection(self):
             return self.collection
    
         def setUp(self):
             connection = pymongo.MongoClient('mongodb://localhost:27017/')
             self.database = connection.test
             self.collection = self.database.user
             self.collection.drop()
             self.user = user_database()
    
             # monkey patch the mongodb handler to use the test database of user
             from MongodbHandler import MongoDatabaseHandler
    
             MongoDatabaseHandler.get_users_collection = self.get_user_collection
    
         def test_insert_user(self):
             """save a valid user to the database """
             user = 'testuser@gmail.com'
             password = 'testpassword'
    
             result = self.user.save_user(user, password)
    
             self.assertEqual(result, True)
             # read the user from database
             doc = self.collection.find_one({'_id': user})
             self.assertEqual(doc, {'_id': user, 'password': password})
    
             self.collection.delete_one({'_id': user})
    
         def test_insert_duplicateuser(self):
             """" save the user twice and that we return a false"""
    
             user = 'duplicatetestuser'
             password = 'duplicatetestuserpassword'
             password2 = 'duplicatetestuserpassword2'
             result = self.user.save_user(user, password)
    
             result = self.user.save_user(user, password)
    
             self.assertEqual(result, False)
    
             self.collection.delete_one({'_id': user})
    
         def test_remove_user(self):
    
             user = 'removeuser'
             password = 'removeuserpassword'
             self.user.save_user(user, password)
    
             result = self.user.delete_user(user)
    
             self.assertEqual(result, True)
             doc = self.collection.find_one({'_id':user})
             self.assertEqual(doc, None)
    
    
         def test_remove_non_existing_user(self):
    
             user = 'dummyuser'
    
             result = self.user.delete_user(user)
    
             self.assertEqual(result, False)
    
    
         def test_get_user(self):
    
             user = 'myuser'
             password = 'mypassword'
             self.user.save_user(user, password)
    
             result = self.user.get_password(user)
    
             self.assertEqual(result, password)
    
             self.collection.delete_one({'_id':user})
    
         def test_get_user_non_existing(self):
    
             result = self.user.get_password('nonExistingUser')
    
             self.assertEqual(result, None)
    
    

6.We will add a LoginForm which will handle all our login related text boxes. The code of app/login_form.py is as follows

```
    from flask_wtf import Form
    from wtforms import StringField, validators, PasswordField, SubmitField
    from wtforms.validators import email, data_required


    class LoginForm(Form):

        email = StringField('email', validators = [email(), data_required()])
        password = PasswordField('Password', validators = [data_required()])
        submit = SubmitField('Login')
        register = SubmitField('register')
```
  1. We will add the login at the top of our index page, so i will be adding the code to a separate login.html as follows

        
        
     <div>
         <form action= "{{ url_for('login') }}" method="post" >
             {{ login_form.hidden_tag() }} 
    
             {{ login_form.email }}
             {{ login_form.password }}
    
             {{ login_form.submit }}
         </form>
         <form action= "{{ url_for('register') }}" method="get" ></form>
             {{ login_form.register }}
         </form>
     </div>
    
        
    
  2. The next part is the using the ‘flask_login’ initialization code in app/__init__.py.This will initialize the login manager of flask login which we will use shortly.

         from flask import Flask
         from flask_wtf.csrf import CsrfProtect
         from flask_login import LoginManager
    
         app = Flask(__name__)
    
         CsrfProtect(app)
    
         login_manager = LoginManager()
         login_manager.init_app(app)
    
         from app import views
    

Reference

Many of the things in this series is inspired/learnt by studying and looking into several sites.The login is inspired by the code from this blog

Creating a URL Shortener with Flask - Part 4

In this part 4 , i will discuss more on additional 2 tools which will help us keep the code consistent and also figure out ares of improvement.

Code coverage

Code coverage is one of the oldest tools available for measuring the amount of test coverage. Earlier that code coverage was performed with manual test cases but with automation of unit test becoming a norm now a days , the test coverage can be run regularly.

  1. Python support of code coverage is done with a tool called coverage. It can be installed as below

         pip install coverage
    
  2. Now that we have all the tests that are automated we can easily run the code coverage as below

     $coverage run --source app -m unittest discover 
     -------------------------------
     Ran 11 tests in 0.173s
     OK
    

    The above code runs all the unit tests and stores the coverage in a file called .coverage in the directory you ran the command.

  3. To see the report , we should see the coverage report command as

        $ coverage report -m
        Name                              Stmts   Miss  Cover   Missing
        ---------------------------------------------------------------
        app/__init__.py                       3      0   100%   
        app/models/MongodbHandler.py          9      0   100%   
        app/models/__init__.py                0      0   100%   
        app/models/test_urlShortener.py      55      1    98%   90
        app/models/urlshortener.py           32      4    88%   32-33, 43-45
        app/views.py                         27      1    96%   48
        ---------------------------------------------------------------
        TOTAL                               126      6    95%   
    
    

    The output clearly points out the missing lines in the coverage .

  4. Lets make it easy to run the coverage by putting all the coverage in a make file . Lets first create a new branch called addCoverageSupport to track all our coverage related changes.

         git branch addCoverageSupport
         git checkout addCoverageSupport
    
  5. Though we can run it manually every time, it would be better if we could make the coverage also run automatically as part of our automatic build process(travis).To do this we have to use a online tool called coveralls.

  6. Sign up and create a new account for you and enable the repo on the coverall. Doing all this should be straight forward.

  7. Install coverall for doing the code coverage as below

         pip install coveralls
    
  8. Add the coveralls to .travis.yml for running code coverage as below

         # command to run tests
         script:
             coverage run --source=app -m unittest discover
    
         after_success:
             coveralls
    
  9. Now update the requirement.txt and push the changes to the github so that it runs on travis

         pip freeze > requirement.txt
         git add .
         git commit -m "Added coveralls support"
         git push origin addCoverageSupport
    
  10. When travis runs and is successful, it pushes all the code coverage report and the output should be somthing similar to below.

    travis

    From the above, its clear that we have a code coverage of 94%, we will look more into how to get to a code coverage of 100%

  11. Merge the changes into the master and we will create a new branch for making the code coverage to 100%

        git checkout master
        git merge  addCoverageSupport
    
  12. Push the changes to github using

        git push master origin
    

Improve code Coverage

Now lets work on to improve the code coverage to make sure we cover all the cases, by looking at the profile output from the coveralls.

  1. Lets first create a branch to do all the changes related to code coverage improvement.

         git checkout -b improveCode
    
  2. The code has no coverage for the case where we try to access a shortURL that is not available. In this case , the url shortener must return 404 not found. So lets add a functional test to improve code coverage in testing/test_basicsite.py

         # try to access a invalid shorturl and the code
         # must return error code 404 not found
         def test_get_invalidShortUrl(self):
    
             # invalid shortUrl
             shorturl = self.baseURL + '/' + '112111111'
    
             rv = self.client.get(shorturl)
    
             self.assertEqual(rv.status_code, 404)
    
  3. The code has no coverage for case where the model fails to perform operation like save. In case of error trying to create a shortURL ,we should return a back to the index page and with possibly a flash messsage. For time being we will only check if the page returns back to the index.html

        # the case where we send a request to shorten the url and for
        # whatever reason , the code shortened url is not created
        def test_post_to_urlShortener_fail_in_model(self):
    
            # monkey patch the code to make sure that the saveUrl returns
            # false so we can check the return value.
            from app.models import urlshortener
            beforepatch = urlshortener.urlShortener.saveUrl
            urlshortener.urlShortener.saveUrl = self.stub_saveURL_returns_false
            post_data = {'url': 'http://www.google.com/'}
    
            rv = self.client.post('/urlshorten',
                                  data=post_data,
                                  follow_redirects=False)
    
            self.assertEqual(rv.status_code, 200)
    
            #cleanup
            urlshortener.urlShortener.saveUrl = beforepatch
    

With this we improve the code coverage to 96%. We will stop at this as we have some more code at model level to perform and we will look at improving the coverage more when we make changes.

Creating a URL Shortener with Flask - Part 3

In this section we dive a bit more deep into looking into how we are going to have some basic functional testing for the application and then we look look into making a config file and finally merge all the changes and make them into our release 0.1

Integration tests

Integration tests are integral part of system development. They make sure that the code works correctly, when we make changes and also validating code merge from several branches. The integration tests in this case will be written our well known unit test framework.

  1. Lets make a testing folder for functional testing

         mkdir testing
         cd testing
    
  2. We will add a new file called test_basicsite.py in this folder and write the basic setup code

     import unittest
     import sys
     import urllib
    
     sys.path.append('..')
    
    
     class TestBasicUrlShortener(unittest.TestCase):
         def setUp(self):
             self.client = app.test_client()
             self.baseURL = 'http://localhost:5000'
    
         # this is one of the functions that must be
         # implemented for flask testing.
         def create_app(self):
             app = Flask(__name__)
             app.config['TESTING'] = True
             app.debug = True
             self.baseURL = 'http://localhost:5000'
             return app
    
  3. To test index we must first send a Get request with the flask client application and check for the return value and the presence of input field and url field in the page that is received

      # Make sure that we have the index page working
     def test_get_to_index(self):
         rv = self.client.get('/')
    
         assert rv.status_code == 200
         assert 'name=\"url\"' in str(rv.data)
         assert 'input' in str(rv.data)
    
     # when we send a Get , we need to make sure that
     # it redirects to index
     def test_get_to_urlshortener(self):
    
         rv =self.client.get('urlshorten')
    
         self.assertEqual(rv.status_code, 302)
         assert 'localhost'in rv.location
    
    
  4. In order to test the post request, we monkey patch the ‘generate_shortURL’ to make sure that it returns a fixed string so that we know the exact string to look for in the response

         # When we send a post we expect it to return a output
     # containing the baseURL and short url
     def test_post_to_urlshortener(self):
    
         # monkeypatch the generate shortURL so that we know
         # the correct value to expect and perform validation
         # accordingly
         from app.models import urlshortener
         urlshortener.urlShortener.generateShortUrl = self.generate_shortURL
         post_data = {'url': 'http://www.google.com/'}
    
         rv = self.client.post('/urlshorten',
                               data=post_data,
                               follow_redirects=False)
    
         self.assertEqual(rv.status_code, 200)
         shorturl = self.baseURL + '/' + self.generate_shortURL()
         assert shorturl in str(rv.data)
    
         #cleanup so next time it works
         urlshort = urlshortener.urlShortener()
         urlshort.removeUrl(self.generate_shortURL())
    
    
  5. Now to check the URL redirect is working correctly we store a short url and then send a get request to make sure that we get a redirect response from our application.

        def test_get_shorturl(self):
    
             # monkey patch to a particular short url
             # store it in database and then
             # do a get with short url
             from app.models import urlshortener
             urlshortener.urlShortener.generateShortUrl = self.generate_shortURL_for_redirect
             post_data = {'url': 'http://www.google.com/'}
             self.client.post('/urlshorten',
                                   data=post_data,
                                   follow_redirects=False)
    
             shorturl = self.baseURL + '/' + self.generate_shortURL_for_redirect()
             rv = self.client.get(shorturl)
    
             self.assertEqual(rv.status_code, 302)
             self.assertEqual(rv.location, 'http://www.google.com/')
    
             #cleanup so next time it works
             urlshort = urlshortener.urlShortener()
             urlshort.removeUrl(self.generate_shortURL())
    

With these, we have covered all the basic integration test cases of our application.

Making a config file for our application

Now all our configuration , we are getting it from our environment variables, but it would be better if we had a uniform configuration file for all our configuration.

  1. we create a new file called config.py which will hold all our configuration in the top level directory.

  2. The code structure looks like below

         ├── LICENSE
         ├── Procfile
         ├── app
         │   ├── __init__.py
         │   ├── __init__.pyc
         │   ├── models
         │   ├── static
         │   ├── templates
         │   ├── views.py
         │   └── views.pyc
         ├── config.py
         ├── config.pyc
         ├── requirements.txt
         ├── run.py
         ├── testing
         │   ├── __init__.py
         │   ├── __init__.pyc
         │   ├── test_basicsite.py
         │   └── test_basicsite.pyc
         ├── tmp
    
    
  3. In config.py we add all configuration to be read from the os environment variable and if not available it would not set a default value as below

         import os
    
         CONNECTION_URI = os.getenv('CONNECTION_STRING','mongodb://localhost:27017/')
         SITE_URL = os.getenv('SITE_URL', 'http://localhost:5000')
         PORT = os.getenv('PORT', 5000)
    
    
  4. We import the configuration from all places where we would want to use the configuration as below

         from config import PORT
         app.run(host='0.0.0.0', port=PORT,debug=True)
    

    This would import the PORT value from the configuration file.

After we have changed the configuration in all the other files where we were doing an os.getenv

  1. Finally we run all the tests again to make sure that nothing is broken

         python -m unittest discover
         ...........
         ----------------------------------------------------------------------
         Ran 11 tests in 0.077s
         OK
    

Merging code to main

  1. Merge the code from basicURLShortener to master

         $ git checkout master
         Switched to branch 'master'
            
         $ git merge basicUrlShorterner 
         Updating 5c968d2..0f66d0b
         Fast-forward
          .travis.yml                     |  25 +++++++++++++++++++++++++
          app/__init__.py                 |   1 +
          app/models/Makefile             |   7 +++++++
          app/models/MongodbHandler.py    |  15 +++++++++++++++
          app/models/__init__.py          |   0
          app/models/test_urlShortener.py |  90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
          app/models/urlshortener.py      |  51 +++++++++++++++++++++++++++++++++++++++++++++++++++
          app/templates/base.html         |  14 ++++++++++++++
          app/templates/index.html        |  23 +++++++++++++++++++++++
          app/views.py                    |  49 ++++++++++++++++++++++++++++++++++++++++++++++++-
          config.py                       |   6 ++++++
          requirements.txt                |   5 +++++
          run.py                          |   4 +++-
          testing/__init__.py             |   0
          testing/test_basicsite.py       | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
          15 files changed, 390 insertions(+), 2 deletions(-)
          create mode 100644 .travis.yml
          create mode 100644 app/models/Makefile
          create mode 100644 app/models/MongodbHandler.py
          create mode 100644 app/models/__init__.py
          create mode 100644 app/models/test_urlShortener.py
          create mode 100644 app/models/urlshortener.py
          create mode 100644 app/templates/base.html
          create mode 100644 app/templates/index.html
          create mode 100644 config.py
          create mode 100644 testing/__init__.py
          create mode 100644 testing/test_basicsite.py
         (venv)Pradheep (master) UrlShortener $