22 Dec 2015
In part 5 of this series, i will discuss how i added login support for the url shortener. If the user has logged in we will provide him with a set of links that he created and statistics regarding all the links he created. The first step is creating the user account so we can track all the user login.
-
As usual lets create a new branch for handling of user account
git branch -b addLoginSupport
-
I will be using a flask package called the flask-login that handles most of the login handling.
-
Lets first add the backend storage , our database model is going to be very simple to start with , we will have a ‘_id’ field which is the users email address and a ‘password’ stored in the ‘password’ field.
-
The code for storing is going to be straight forward similar to that of the short url insertion we saw in part 1. The code is as follows
import pymongo
from MongodbHandler import MongoDatabaseHandler
class user_database(object):
def __init__(self):
""" save the username and password """
databaseHandler = MongoDatabaseHandler()
self.collection = databaseHandler.get_users_collection()
def get_password(self, username):
""" get password from a given username """
doc = self.collection.find_one({'_id': username}, {'_id': 0, 'password': 1})
if doc is None:
return None
return doc['password']
def save_user(self, username, password):
"""save the username and password to the database
return false if trying to add the same user again
"""
try:
self.collection.insert_one({'_id': username, 'password': password})
except pymongo.errors.DuplicateKeyError:
return False
return True
def delete_user(self, username):
""" delete the user given an username """
result = self.collection.delete_one({'_id': username})
if result.deleted_count:
return True
else:
return False
-
The corresponding test coverage for the same is as follows. We as usual duct type to make sure that the database writes are to the test database
import unittest
import pymongo
from user_database import user_database
class TestUser(unittest.TestCase):
def get_user_collection(self):
return self.collection
def setUp(self):
connection = pymongo.MongoClient('mongodb://localhost:27017/')
self.database = connection.test
self.collection = self.database.user
self.collection.drop()
self.user = user_database()
# monkey patch the mongodb handler to use the test database of user
from MongodbHandler import MongoDatabaseHandler
MongoDatabaseHandler.get_users_collection = self.get_user_collection
def test_insert_user(self):
"""save a valid user to the database """
user = 'testuser@gmail.com'
password = 'testpassword'
result = self.user.save_user(user, password)
self.assertEqual(result, True)
# read the user from database
doc = self.collection.find_one({'_id': user})
self.assertEqual(doc, {'_id': user, 'password': password})
self.collection.delete_one({'_id': user})
def test_insert_duplicateuser(self):
"""" save the user twice and that we return a false"""
user = 'duplicatetestuser'
password = 'duplicatetestuserpassword'
password2 = 'duplicatetestuserpassword2'
result = self.user.save_user(user, password)
result = self.user.save_user(user, password)
self.assertEqual(result, False)
self.collection.delete_one({'_id': user})
def test_remove_user(self):
user = 'removeuser'
password = 'removeuserpassword'
self.user.save_user(user, password)
result = self.user.delete_user(user)
self.assertEqual(result, True)
doc = self.collection.find_one({'_id':user})
self.assertEqual(doc, None)
def test_remove_non_existing_user(self):
user = 'dummyuser'
result = self.user.delete_user(user)
self.assertEqual(result, False)
def test_get_user(self):
user = 'myuser'
password = 'mypassword'
self.user.save_user(user, password)
result = self.user.get_password(user)
self.assertEqual(result, password)
self.collection.delete_one({'_id':user})
def test_get_user_non_existing(self):
result = self.user.get_password('nonExistingUser')
self.assertEqual(result, None)
6.We will add a LoginForm which will handle all our login related text boxes. The code of app/login_form.py is as follows
```
from flask_wtf import Form
from wtforms import StringField, validators, PasswordField, SubmitField
from wtforms.validators import email, data_required
class LoginForm(Form):
email = StringField('email', validators = [email(), data_required()])
password = PasswordField('Password', validators = [data_required()])
submit = SubmitField('Login')
register = SubmitField('register')
```
-
We will add the login at the top of our index page, so i will be adding the code to a separate login.html as follows
<div>
<form action= "{{ url_for('login') }}" method="post" >
{{ login_form.hidden_tag() }}
{{ login_form.email }}
{{ login_form.password }}
{{ login_form.submit }}
</form>
<form action= "{{ url_for('register') }}" method="get" ></form>
{{ login_form.register }}
</form>
</div>
-
The next part is the using the ‘flask_login’ initialization code in app/__init__.py.This will initialize the login manager of flask login which we will use shortly.
from flask import Flask
from flask_wtf.csrf import CsrfProtect
from flask_login import LoginManager
app = Flask(__name__)
CsrfProtect(app)
login_manager = LoginManager()
login_manager.init_app(app)
from app import views
Reference
Many of the things in this series is inspired/learnt by studying and looking into several sites.The login is inspired by the code from this blog
17 Dec 2015
In this part 4 , i will discuss more on additional 2 tools which will help us keep the code consistent and also figure out ares of improvement.
Code coverage
Code coverage is one of the oldest tools available for measuring the amount of test coverage. Earlier that code coverage was performed with manual test cases but with automation of unit test becoming a norm now a days , the test coverage can be run regularly.
-
Python support of code coverage is done with a tool called coverage. It can be installed as below
-
Now that we have all the tests that are automated we can easily run the code coverage as below
$coverage run --source app -m unittest discover
-------------------------------
Ran 11 tests in 0.173s
OK
The above code runs all the unit tests and stores the coverage in a file called .coverage in the directory you ran the command.
-
To see the report , we should see the coverage report command as
$ coverage report -m
Name Stmts Miss Cover Missing
---------------------------------------------------------------
app/__init__.py 3 0 100%
app/models/MongodbHandler.py 9 0 100%
app/models/__init__.py 0 0 100%
app/models/test_urlShortener.py 55 1 98% 90
app/models/urlshortener.py 32 4 88% 32-33, 43-45
app/views.py 27 1 96% 48
---------------------------------------------------------------
TOTAL 126 6 95%
The output clearly points out the missing lines in the coverage .
-
Lets make it easy to run the coverage by putting all the coverage in a make file . Lets first create a new branch called addCoverageSupport to track all our coverage related changes.
git branch addCoverageSupport
git checkout addCoverageSupport
-
Though we can run it manually every time, it would be better if we could make the coverage also run automatically as part of our automatic build process(travis).To do this we have to use a online tool called coveralls.
-
Sign up and create a new account for you and enable the repo on the coverall. Doing all this should be straight forward.
-
Install coverall for doing the code coverage as below
-
Add the coveralls to .travis.yml for running code coverage as below
# command to run tests
script:
coverage run --source=app -m unittest discover
after_success:
coveralls
-
Now update the requirement.txt and push the changes to the github so that it runs on travis
pip freeze > requirement.txt
git add .
git commit -m "Added coveralls support"
git push origin addCoverageSupport
-
When travis runs and is successful, it pushes all the code coverage report and the output should be somthing similar to below.
From the above, its clear that we have a code coverage of 94%, we will look more into how to get to a code coverage of 100%
-
Merge the changes into the master and we will create a new branch for making the code coverage to 100%
git checkout master
git merge addCoverageSupport
-
Push the changes to github using
Improve code Coverage
Now lets work on to improve the code coverage to make sure we cover all the cases, by looking at the profile output from the coveralls.
-
Lets first create a branch to do all the changes related to code coverage improvement.
git checkout -b improveCode
-
The code has no coverage for the case where we try to access a shortURL that is not available. In this case , the url shortener must return 404 not found. So lets add a functional test to improve code coverage in testing/test_basicsite.py
# try to access a invalid shorturl and the code
# must return error code 404 not found
def test_get_invalidShortUrl(self):
# invalid shortUrl
shorturl = self.baseURL + '/' + '112111111'
rv = self.client.get(shorturl)
self.assertEqual(rv.status_code, 404)
-
The code has no coverage for case where the model fails to perform operation like save. In case of error trying to create a shortURL ,we should return a back to the index page and with possibly a flash messsage. For time being we will only check if the page returns back to the index.html
# the case where we send a request to shorten the url and for
# whatever reason , the code shortened url is not created
def test_post_to_urlShortener_fail_in_model(self):
# monkey patch the code to make sure that the saveUrl returns
# false so we can check the return value.
from app.models import urlshortener
beforepatch = urlshortener.urlShortener.saveUrl
urlshortener.urlShortener.saveUrl = self.stub_saveURL_returns_false
post_data = {'url': 'http://www.google.com/'}
rv = self.client.post('/urlshorten',
data=post_data,
follow_redirects=False)
self.assertEqual(rv.status_code, 200)
#cleanup
urlshortener.urlShortener.saveUrl = beforepatch
With this we improve the code coverage to 96%. We will stop at this as we have some more code at model level to perform and we will look at improving the coverage more when we make changes.
06 Dec 2015
In this section we dive a bit more deep into looking into how we are going to have some basic functional testing for the application and then we look look into making a config file and finally merge all the changes and make them into our release 0.1
Integration tests
Integration tests are integral part of system development. They make sure that the code works correctly, when we make changes and also validating code merge from several branches. The integration tests in this case will be written our well known unit test framework.
-
Lets make a testing folder for functional testing
-
We will add a new file called test_basicsite.py in this folder and write the basic setup code
import unittest
import sys
import urllib
sys.path.append('..')
class TestBasicUrlShortener(unittest.TestCase):
def setUp(self):
self.client = app.test_client()
self.baseURL = 'http://localhost:5000'
# this is one of the functions that must be
# implemented for flask testing.
def create_app(self):
app = Flask(__name__)
app.config['TESTING'] = True
app.debug = True
self.baseURL = 'http://localhost:5000'
return app
-
To test index we must first send a Get request with the flask client application and check for the return value and the presence of input field and url field in the page that is received
# Make sure that we have the index page working
def test_get_to_index(self):
rv = self.client.get('/')
assert rv.status_code == 200
assert 'name=\"url\"' in str(rv.data)
assert 'input' in str(rv.data)
# when we send a Get , we need to make sure that
# it redirects to index
def test_get_to_urlshortener(self):
rv =self.client.get('urlshorten')
self.assertEqual(rv.status_code, 302)
assert 'localhost'in rv.location
-
In order to test the post request, we monkey patch the ‘generate_shortURL’ to make sure that it returns a fixed string so that we know the exact string to look for in the response
# When we send a post we expect it to return a output
# containing the baseURL and short url
def test_post_to_urlshortener(self):
# monkeypatch the generate shortURL so that we know
# the correct value to expect and perform validation
# accordingly
from app.models import urlshortener
urlshortener.urlShortener.generateShortUrl = self.generate_shortURL
post_data = {'url': 'http://www.google.com/'}
rv = self.client.post('/urlshorten',
data=post_data,
follow_redirects=False)
self.assertEqual(rv.status_code, 200)
shorturl = self.baseURL + '/' + self.generate_shortURL()
assert shorturl in str(rv.data)
#cleanup so next time it works
urlshort = urlshortener.urlShortener()
urlshort.removeUrl(self.generate_shortURL())
-
Now to check the URL redirect is working correctly we store a short url and then send a get request to make sure that we get a redirect response from our application.
def test_get_shorturl(self):
# monkey patch to a particular short url
# store it in database and then
# do a get with short url
from app.models import urlshortener
urlshortener.urlShortener.generateShortUrl = self.generate_shortURL_for_redirect
post_data = {'url': 'http://www.google.com/'}
self.client.post('/urlshorten',
data=post_data,
follow_redirects=False)
shorturl = self.baseURL + '/' + self.generate_shortURL_for_redirect()
rv = self.client.get(shorturl)
self.assertEqual(rv.status_code, 302)
self.assertEqual(rv.location, 'http://www.google.com/')
#cleanup so next time it works
urlshort = urlshortener.urlShortener()
urlshort.removeUrl(self.generate_shortURL())
With these, we have covered all the basic integration test cases of our application.
Making a config file for our application
Now all our configuration , we are getting it from our environment variables, but it would be better if we had a uniform configuration file for all our configuration.
-
we create a new file called config.py which will hold all our configuration in the top level directory.
-
The code structure looks like below
├── LICENSE
├── Procfile
├── app
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── models
│ ├── static
│ ├── templates
│ ├── views.py
│ └── views.pyc
├── config.py
├── config.pyc
├── requirements.txt
├── run.py
├── testing
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── test_basicsite.py
│ └── test_basicsite.pyc
├── tmp
-
In config.py we add all configuration to be read from the os environment variable and if not available it would not set a default value as below
import os
CONNECTION_URI = os.getenv('CONNECTION_STRING','mongodb://localhost:27017/')
SITE_URL = os.getenv('SITE_URL', 'http://localhost:5000')
PORT = os.getenv('PORT', 5000)
-
We import the configuration from all places where we would want to use the configuration as below
from config import PORT
app.run(host='0.0.0.0', port=PORT,debug=True)
This would import the PORT value from the configuration file.
After we have changed the configuration in all the other files where we were doing an os.getenv
-
Finally we run all the tests again to make sure that nothing is broken
python -m unittest discover
...........
----------------------------------------------------------------------
Ran 11 tests in 0.077s
OK
Merging code to main
-
Merge the code from basicURLShortener to master
$ git checkout master
Switched to branch 'master'
$ git merge basicUrlShorterner
Updating 5c968d2..0f66d0b
Fast-forward
.travis.yml | 25 +++++++++++++++++++++++++
app/__init__.py | 1 +
app/models/Makefile | 7 +++++++
app/models/MongodbHandler.py | 15 +++++++++++++++
app/models/__init__.py | 0
app/models/test_urlShortener.py | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
app/models/urlshortener.py | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
app/templates/base.html | 14 ++++++++++++++
app/templates/index.html | 23 +++++++++++++++++++++++
app/views.py | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
config.py | 6 ++++++
requirements.txt | 5 +++++
run.py | 4 +++-
testing/__init__.py | 0
testing/test_basicsite.py | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
15 files changed, 390 insertions(+), 2 deletions(-)
create mode 100644 .travis.yml
create mode 100644 app/models/Makefile
create mode 100644 app/models/MongodbHandler.py
create mode 100644 app/models/__init__.py
create mode 100644 app/models/test_urlShortener.py
create mode 100644 app/models/urlshortener.py
create mode 100644 app/templates/base.html
create mode 100644 app/templates/index.html
create mode 100644 config.py
create mode 100644 testing/__init__.py
create mode 100644 testing/test_basicsite.py
(venv)Pradheep (master) UrlShortener $