Introduction
In web applications, there is usually a need to display large datasets. Displaying the whole dataset in a single web page at a go may easily choke/misuse computer resources such as memory. Again, no one loves sluggish and slow web pages. Some users may stay after throwing a few curses while others may angrily turn away.
There are two common ways of solving this problem; pagination and infinite scroll.
Both hinge on the idea of loading a small chunk of the data at
a go, a form of lazy loading. Both methods help in keeping
the content well structured and gives a better
user experience. They can also help in efficient use of computer resources if used properly.
Pagination is a mechanism which provides users with additional
navigation options for browsing through a long list of data or a large dataset.
For example, given a list of 1000 objects, we split(paginate) it into
10 pages each with 100 objects and provide a way for the user to arbitrarily navigate through
these pages back and forth in a simple and predictable way.
Instead of splitting the dataset into several pages, infinite scroll auto
loads a small chunk of data whenever the user scrolls to nears the end
of the page. This technique is very common in social sites.
Front end zealots have already picked a fight between these two methods(pun intended). We leave that to them.
However, an opinion on how to decide between these two methods may be welcome.
Infinite scroll is better suited for discovery interfaces- where the user is not searching
for something specific but scrolling through a page discovering new content as they go, eg in social media sites.
It is also to be preferred where the primary user interface is a mobile device. Scrolling down a single page is more
intuitive in mobile devices compared to clicking through several pages.
Pagination is better suited for information interfaces- where the user has a good idea of the information
in a page and may want to refer to specific past data in an arbitrary manner. An example is a
page displaying a list of invoices. The accountant may want to see invoices of
a given past date, a case which is does do not do well with infinite scrolling.
In this tutorial we’ll look at pagination.
Django has a built in independent pagination system which can be used in views and even in pdf generation. It is a basic implementation which assumes nothing on the navigational structure in the front end. All it provides is splitting(pagination) of the data and some navigational helper attributes. It is up to the developer to decide how the user will navigate through the paginated objects. Several css frameworks have support for pagination which can be easily integrated to work well with Django’s pagination.
In this tutorial we will integrate django’s pagination system with Bootstrap 4’s pagination functionality. The same can be easily done with other css frameworks which have pagination support.
Set up
To begin with, lets create a new Django project called students and within this
project a Django app which we’ll call main for simplicity.
$ django-admin startproject students
$ cd students
$ python manage.py startapp mainNext lets add the main app to the installed apps list in settings.py
# students/settings.py
# ...
INSTALLED_APPS = [
'main',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]Static files setup
Lets add Boostrap 4 files to our static files. Download Bootstrap 4 and add the appropriate files to the project. Bootstrap 4’s pagination functionality has no javascript dependencies so the css file alone will be enough.
Also create a templates dir in main and add a base.html file which will be inherited by all other templates.
Your contents should look like this:
├── main
│ ├── admin.py
│ ├── apps.py
│ ---- # more files
│ ├── static
│ │ └── main
│ │ ├── css
│ │ │ └── bootstrap.min.css
│ ├── templates
│ │ └── main
│ │ └── base.html
├── ---- # more filesWe’ll load our static files in base.html and create title and content blocks which child templates will override appropriately.
Our base.html will look something like this:
{% raw %}
{% load static %}
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<link rel="stylesheet" href="{% static 'main/css/bootstrap.min.css' %}">
<title>{% block title %} | Super Students{% endblock title %}</title>
</head>
<body>
{% block content %}{% endblock content %}
</body>
</html>
{% endraw %}Student model
In the main app’s models.py lets create a simple model which will record
student data.
# main/models.py
from django.db import models
class Student(models.Model):
GENDER = (
('M', 'Male'),
('F', 'Female'),
)
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
dob = models.DateField() # date of birth
gender = models.CharField(max_length=6, choices=GENDER)
def __str__(self):
return '{} {}'.format(self.first_name, self.last_name)
class Meta:
ordering = ('first_name', )
def full_name(self):
return str(self)Next lets run makemigrations and migrate to persist our changes in the
database.
$ python manage.py makemigrations main
$ python manage.py migrateTo make things simple, download this json fixture which contains 1000 students records and load it into the database:
# replace 'path-to' with the path where you've downloaded the file
$ python manage.py loaddata path-to/students.jsonYou can as well generate the data instead of using the fixtures above. What matters most is that you should have a few hundreds of rows in the students table.
Pagination in function based views
Lets create a function based view which will display the students.
# main/views.py
from django.shortcuts import render
from .models import Student
def students(request):
context = {
'students': Student.objects.all()
}
return render(request, 'main/students.html', context)The template used in this view, main/templates/main/students.html:
# main/templates/main/students.html
{% raw %}
{% extends "main/base.html" %}
{% block title %}All {{ block.super }}{% endblock title %}
{% block content %}
<div class="container">
<h3 class="text-center my-5">List of students </h3>
<table class="table table-striped table-bordered">
<thead class="thead-inverse">
<tr>
<th>#</th>
<th>Name</th>
<th>Gender</th>
<th>Date of birth</th>
</tr>
</thead>
<tbody>
{% for student in students %}
<tr>
<td>{{ forloop.counter }}</td>
<td>{{ student.full_name }}</td>
<td>{{ student.get_gender_display }}</td>
<td>{{ student.dob }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock content %}
{% endraw %}Lets add this view in the app’s url configuration:
# main/urls.py
from django.conf.urls import url
from main import views
urlpatterns = [
url(r'^students$', views.students, name='students'),
]Next lets add main.urls to the root(project) url configuration:
# students/urls.py
from django.conf.urls import include, url
from django.contrib import admin
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^', include('main.urls'))
]You can now access the students view in the browser. As you can see, scrolling down a table with over a hundred rows is not that interesting. This is where pagination comes in.
Let’s add pagination to display 100 students per page.
# main/views.py
from django.core.paginator import InvalidPage, Paginator
from django.http import Http404
from django.shortcuts import render
from .models import Student
def students(request):
students = Student.objects.all()
paginator = Paginator(students, 100, orphans=5)
is_paginated = True if paginator.num_pages > 1 else False
page = request.GET.get('page') or 1
try:
current_page = paginator.page(page)
except InvalidPage as e:
raise Http404(str(e))
context = {
'current_page': current_page,
'is_paginated': is_paginated
}
return render(request, 'main/students.html', context)The Paginator class we’ve used in the view above is at the heart of pagination in Django. Give it a collection of objects(list, tuple, queryset- essentially any object with a count() or __len__() method), plus the number of items you’d like to have on each page, and it splits the collection into pages each consisting the number of items you’ve given. Additionaly, the Paginator gives you methods for accessing the items for each page.
The Paginator class also accepts two optional arguments: orphans and allow_empty_first_page.
orpans is an integer specifying the number of “overflow” objects the last page can contain. This extends the paginate_by limit on the last page by up to orphans, in order to keep the last page from having a very small number of objects. For example, with 23 items, per_page=10, and orphans=3, there will be two pages; the first page with 10 items and the second (and last) page with 13 items. orphans defaults to zero, which means pages are never combined and the last page may have one item. We’ve specifed 5 orphans in our view.
allow_empty_first_page: a boolean deciding whether or not the first page is allowed to be empty. If False and the collection passed is empty, then an EmptyPage error will be raised. Defaults to True.
In our case we’ve given the Paginator a queryset and asked it to split it into pages each consisting of 100 items and the last page to contain up to 5 orphans. The generated pages can be accessed using an index, 1-indexed.
For example if we give the paginator a list of 30 items and tell it to paginate it by 10 items per page, we end up with three pages, which can be accessed like so:
page1 = paginator.page(1)
page2 = paginator.page(2)
page3 = paginator.page(3)Accesing a page with a non existing index throws an EmptyPage exception:
page5 = paginator.page(5)
Traceback (most recent call last):
...
EmptyPage: That page contains no resultsAgain if we access a page with an index that does not make sense as far as integers are concerned, a PageNotAnInteger exception is raised.
page6 = paginator.page('dy')
Traceback (most recent call last):
...
PageNotAnInteger: That page number is not an integerIn our view above, we’ve taken great advantage of this idea of accesing pages with an index and raising an exception when the index is out of bounds or does not make sense.
We’ve used the index as a request query param known as page.
When a request comes, we get the query param from the request.GET object. If its not found or is None, we set it to a default of 1(first page.) We then try to use this value to find a page using paginator.page(). If a page is found(the value refers to a valid page number), we add the found page in the context dict as current_page. Otherwise, either PageNotAnInteger or EmptyException is raised.
If an exceptions is raised, we return a Http404 Not Found with the appropriate error message. Both exceptions are subclasses of InvalidPage, so
we use InvalidPage instead of (EmptyPage, PageNotAnInteger) in the catch phrase.
Instead of returning a Http404 when an exception is raised, you can as well return the last or first page as explained in the docs. Either way is OK.
Now, we can iterate over the current_page object in the template as though its a queryset to get the students. Good enough, we now only have the students in the current_page alone.
We also need to update our template to add the navigation support for the pagination as shown below:
{% raw %}
{% extends "main/base.html" %}
{% block title %}All {{ block.super }}{% endblock title %}
{% block content %}
<div class="container">
<h3 class="text-center my-5">List of students </h3>
<table class="table table-striped table-bordered">
<thead class="thead-inverse">
<tr>
<th>#</th>
<th>Name</th>
<th>Gender</th>
<th>Date of birth</th>
</tr>
</thead>
<tbody>
{% for student in current_page %}
<tr>
<td>{{ current_page.start_index|add:forloop.counter0 }}</td>
<td>{{ student.full_name }}</td>
<td>{{ student.get_gender_display }}</td>
<td>{{ student.dob }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% if is_paginated %}
<div>
<span>
{% if current_page.has_previous %}
<a href="?page={{ current_page.previous_page_number }}">Previous</a>
{% endif %}
<span>
Page {{ current_page.number }} of {{ current_page.paginator.num_pages }}.
</span>
{% if current_page.has_next %}
<a href="?page={{ current_page.next_page_number }}">Next</a>
{% endif %}
</span>
</div>
{% endif %}
</div>
{% endblock content %}
{% endraw %}The page object current_page returned in the context has several navigation helper methods which we’ve used above to implement the Next and Previous links.
Note thow we’re using the is_paginated boolean we added in the context dict to control the display of the navigational options. There is no need to display them if we only have a single page. It even looks weird.
However, the # counter column in the table starts from 1 in each page instead of continuing from the last item of the previous page.
If this is not what you want you can change it to keep a continous count in all the pages.
We change the simplistic
{% raw %}
<td>{{ forloop.counter }}</td>
{% endraw %}To:
{% raw %}
<td>{{ current_page.start_index|add:forloop.counter0 }}</td>
{% endraw %}Page.start_index() returns the 1-based index of the first object on the page, relative to all of the objects in the paginator’s list. For example, when paginating a list of 5 objects with 2 objects per page, the second page’s start_index() would return 3.
We then add the 0-indexed forloop counter to the start_index to
increment it in each row. Otherwise all rows in each page will have the same counter value.
So far so good except that our navigational links are ugly and not that intuitive. Lets use Bootstrap’s 4 pagination support to turn them into fancy looking stuff.
Integrating with Bootstrap 4’s pagination support
So as to list the pages using bootstrap’s pagination support, we need a way to loop over them in the context. Luckily, the Paginator class has a page_range attribute which serves exactly this purpose. Its a 1-based range iterator of page numbers. To make use of it, lets add the
paginator object we’ve declared in the view to the contex dict:
context = {
'students': students,
'is_paginated': is_paginated,
'paginator': paginator
}Now replace the div containing the page navigation elements with Bootstrap 4’s pagination elements:
{% raw %}
{% extends "main/base.html" %}
{% block title %}All {{ block.super }}{% endblock title %}
{% block content %}
<div class="container">
<h3 class="text-center my-5">List of students </h3>
<table class="table table-striped table-bordered">
# ... same as before
</table>
{% if is_paginated %}
<nav>
<ul class="pagination">
{% if current_page.has_previous %}
<li class="page-item"><a class="page-link" href="?page={{ current_page.previous_page_number }}">Previous</a></li>
{% endif %}
{% for page in paginator.page_range %}
<li class="page-item {% if page == current_page.number %}active{% endif %}">
<a class="page-link" href="?page={{ page }}">{{ page }}</a>
</li>
{% endfor %}
{% if current_page.has_next %}
<li class="page-item"><a class="page-link" href="?page={{ current_page.next_page_number }}">Next</a></li>
{% endif %}
</ul>
</nav>
{% endif %}
</div>
{% endblock content %}
{% endraw %}We’ve used the Paginator.page_range attribute to loop over the pages. We’ve also checked for the current_page in the loop and set it to active so
that it can have the active state in the display.
We’ve also used the current page's navigational helper methods to control the display of Next and Previous links.
Pagination in class based views
The MultipleObjectMixin has inbuilt support for pagination.
Class based views which include this mixin such as ListView have this functionality baked in. To paginate a list of objects in a class based view which does not include this mixin, just include(inherit) it in the view, as long as the said view has a get_queryset method. This is because internally the result of get_queryset is what is paginated by MultipleObjectMixin. If the class does not have a get_queryset method, implement it. It can return any iterable of items, not just a queryset.
Lets use a ListView to display the students. Instead of replacing the function based view lets create a new view altogether. However we’ll use the same template.
# main/views.py
from django.core.paginator import InvalidPage, Paginator
from django.shortcuts import render
from django.views.generic import ListView
# ... code for the function based view
class StudentListView(ListView):
model = Student
template_name = 'main/students.html'
paginate_by = 100
paginated_orphans = 5
def get_context_data(self):
ctx = super().get_context_data()
current_page = ctx.pop('page_obj', None)
ctx['current_page'] = current_page
return ctxRegister the view in main/urls.py:
url(r'^all-students$', views.StudentListView.as_view(), name='student_list'),Django’s implementation of pagination in MultipleObjectMixin is similar to our own. First, a view inheriting the MultipleObjectMixin is considered paginated if paginate_by is specified
or get_paginate_by is implemented. paginate_by is an integer specifying how many objects should be displayed per page. We’ve specified 100.
paginated_orphans is the orphans argument passed to the Paginator class.
By default the page query attribute in request.GET is called page. You can specify page_kwarg to change this.
Internally, django paginates the queryset based on paginate_by. Upon pagination, the following are added to the context dict for use in templates:
page_obj- same ascurrent_pagein our function based view. Thats why we’ve overridenget_context_data()and changedpage_objtocurrent_pageso as to leave the template code untouched. We’re using the same template for both views.paginator- an instance of the paginator to used in this view. Its possible to change this by specifying thepaginator_classattribute or by overridingget_paginator. Defaults todjango.core.paginator.Paginator, the one we’ve used in the function based view.is_paginated- a boolean serving the same purpose like the one we’ve used in the function based view.
When this mixin is used, its allowed to access the last page using last as the query param value. Eg:
http://localhost:8000/all-students?page=lastAs an exercise, you can try to implement this in our other views. Hint: make use of paginator.num_pages
In a situation where pagination is used in several views, it is good to move the template pagination code to a separate template and include it as appropriate to avoid code duplication.
Paginating JSON views
With a few changes, we can easily make our paginated views return json. Instead of changing the two views, lets add two new views which have the same functionality but return json instead.
An equivalent of the function based view, this time returning json:
# ... previous imports
from django.http import JsonResponse, Http404
# ...code for the other views
def studentsjson(request):
students = Student.objects.values('id', 'first_name', 'last_name', 'gender', 'dob')
paginator = Paginator(students, 100, orphans=5)
page = request.GET.get('page') or 1
try:
current_page = paginator.page(page)
except InvalidPage as e:
page = int(page) if page.isdigit() else page
context = {
'page_number': page,
'message': str(e)
}
return JsonResponse(context, status=404)
context = {
'students': list(current_page)
}
return JsonResponse(context)Add it to main/urls.py:
url(r'^api/students$', views.studentjson, name='api_students'),In the json view above, we have removed context data except the current_page which we’ve also converted to a list. This is because pagination in the json response can be used without the nagivational helpers. We’ve also had to convert the Page object current_page to a list to make it json serializable. For the same reason we’ve used
Student.objects.values(...) instead of Student.objects.all(). This is because Student.objects.values(...) returns a queryset of dictionaries(json serializable) while Student.objects.all() returns a queryset of model instances which are not json serializable. In case of an InvalidPage, we return a JsonResponse with a status of 404.
Instead of trying to make the ListView return json, it is much easier to use the base generic View with essentially the same code as the function based view above:
# ... previous imports
from django.views.generic import ListView, View
# ...code for the other views
class StudentJsonView(View):
def get(self, *args, **kwargs):
students = Student.objects.values('id', 'first_name', 'last_name', 'gender', 'dob')
paginator = Paginator(students, 100, orphans=5)
page = self.request.GET.get('page') or 1
try:
current_page = paginator.page(page)
except InvalidPage as e:
page = int(page) if page.isdigit() else page
context = {
'page_number': page,
'message': str(e)
}
return JsonResponse(context, status=404)
context = {
'students': list(current_page)
}
return JsonResponse(context)Add it to main/urls.py:
url(r'^api/all-students$', views.StudentJsonView.as_view(), name='api_student_list'),Using HTTPie or curl if you prefer, you can access the json views in the command line, something like:
$ http http://localhost:8000/api/all-students?page=1
$ curl http://localhost:8000/api/students?page=4
$ ... and so onInstead of rolling out your own json views, you should really use the excellent DJango Rest Framework which has inbuilt support for pagination, offering several pagination styles. The PageNumberPagination style is ideally the same with what we’ve been doing.
The code for this tutorial is on github. Happy coding!