How We Hide Files from Public Access

Today, we will unveil the secret ingredients that make our technologies unique. Our CTO, Yevgeniy Khatsko, will tell you how to prepare Django Nginx Secure Link. Enjoy reading.

Purpose of the Module

We have released the first version of our package, which you can find here. The motivation was the problem of hiding files from public access for unauthorized users or, more deeply, hiding the file from everyone by creating temporary links for each specific file.

Some might see echoes of object storage in this and say «here's AWS S3, Azure Blobs, etc.», but there are some issues and complexities that hinder quick integration of such storage. It is not always feasible to connect an object storage to an existing project, here are a few examples from our experience:

Projects that do not require scaling can often be realized with the simplest set of tools and technologies without needing to connect additional services distributed across different nodes (object storage). This inherently reduces the complexity of the project and the number of interdependent components.
Projects where working with files is limited to exchanging some report data within a single node. In such cases, generating report files and subsequently uploading them to object storage seems redundant (not always, of course), as it is much easier to manage files within one node, closed off from public access, thus reducing the number of errors during their transfer.
A simple MVP project, where it is important to quickly release the product with minimal costs, without unnecessary blockers and dependencies. It is much easier to take one powerful server and set up all the necessary components on it, such as a DBMS, queues, web application server, etc. Practically, this is the fastest and most workable option in terms of setup and development time, considering expected load and the inability for horizontal scaling.
Legacy projects, where integrating object storage seems labor-intensive and complicated due to old framework versions and their dependencies, and often even impossible to integrate.
And of course, projects for which object storage would only complicate file management. For example, processing a large file in the background entails loading the file back onto the server from the object storage to unpack it or perform any other processing.

This is just a small list of examples where it is much simpler to take an Nginx module, closing public access to files, adding a component to generate private links for file distribution, thus not changing the entire logic of the code base working with files. Our package allows on simple and more complex projects where Nginx acts as the front-end server to delegate file protection directly to the web server, offloading our backend.

This solution is also suitable for fairly complex architecture solutions, where files can reside on separate servers, accessed through Nginx.

Our package allows taking over the generation of temporary file links, restricting public access, while Nginx verifies these links and decides whether to serve the file. Nginx has an excellent module ngx_http_secure_link_module for this task, we based our solution on it and wrote a middleware between Django and this module. The solution turned out to be simple and convenient, we also accounted for specific cases where only particular directories from the media storage need to be hidden, or vice versa, made public. All this is configured at the package level and should be set up at the Nginx level by specifying particular locations for media/...

The package also includes auxiliary manage.py commands aimed at helping configure the necessary block for location.

Work is still ongoing and improvements and fixes will be made. There are several open tasks that will be included in the next release soon. If you are interested in this package and wish to improve it, please write and participate. We welcome any involvement, even constructive criticism to improve the package.

Let's delve into the technical part and look at an example setup for a Django project and Nginx virtual host configuration.

Step 1: Module Installation

pip install django-nginx-secure-links

Step 2: Django Project Configuration

Depending on the Django version, you need to specify the following settings:

General settings for all Django versions

INSTALLED_APPS = (
    ...,

    'nginx_secure_links',
)
SECURE_LINK_EXPIRATION_SECONDS = 100
SECURE_LINK_SECRET_KEY = '8SypVsPwf3PypUfdVmos9NdmQNCsMG'
SECURE_LINK_TOKEN_FIELD = 'token'
SECURE_LINK_EXPIRES_FIELD = 'exp'
SECURE_LINK_PRIVATE_PREFIXES = [
    'private',
]
SECURE_LINK_PUBLIC_PREFIXES = []

For Django versions < 4.2:

DEFAULT_FILE_STORAGE = 'nginx_secure_links.storages.FileStorage'

For Django versions >= 4.2:

STORAGES = {
    "default": {
        "BACKEND": "nginx_secure_links.storages.FileStorage",
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

Also, as an example, we’ll configure MEDIA_URL and MEDIA_ROOT:

MEDIA_URL = '/media/'
MEDIA_ROOT = '/var/www/html/media/'

Step 3: Nginx Virtual Host File Configuration

We have added an auxiliary command secure_links_nginx_location for manage.py, which is convenient for automatically generating the block of necessary locations, considering the current Django project settings:

python manage.py secure_links_nginx_location

The console will display an example configuration block, which needs to be inserted into the site.conf file:

nks_nginx_location
В консоль будет выведен пример блока конфигурации, который и нужно вставить в файл site.conf:

location /media/private/ {
    secure_link $arg_token,$arg_exp;
    secure_link_md5 "$secure_link_expires$uri 8SypVsPwf3PypUfdVmos9NdmQNCsMG";
    if ($secure_link = "") {
        return 403;
    }
    if ($secure_link = "0") {
        return 410;
    }
    alias /var/www/html/media/private/;
}

location /media/ {
    alias /var/www/html/media/;
}

Step 4: Verification and Testing

1. We will consider an example data model that saves files in a private directory specified in the SECURE_LINK_PRIVATE_PREFIXES setting:

from django.db import models


class PrivateDocument(models.Model):
    file = models.FileField(upload_to='private/documents')

2. Create a new record in the database by uploading a file through the admin panel for PrivateDocument.

3. To generate a protected link, just access the url property of the file -> obj.file.url:

from django.http import HttpResponse
from django.shortcuts import get_object_or_404

from models import PrivateDocument


def private_document_view(request, pk):
    obj = get_object_or_404(PrivateDocument, pk=pk)
    private_url = obj.file.url
    return HttpResponse(private_url)

The variable private_url will store a private link containing two additional GET parameters:

Example link: /media/private/documents/file1.txt?token=S9tBKKkXcAq77g8MrVe6LQ&exp=1721519555
GET parameter token (see SECURE_LINK_TOKEN_FIELD setting)
GET parameter exp (see SECURE_LINK_EXPIRES_FIELD setting)

Note that if the files are not in the private directories listed in the SECURE_LINK_PRIVATE_PREFIXES setting, the generation of a private link will be skipped. As a result, we’ll get a public link without additional GET parameters token and exp.

The full working example for different Django versions can be found at this link. The example features the simplest installation of the Nginx module via the nginx-extras package, but it’s worth noting that if you build Nginx from source, you can also add this module at the ./configure step, avoiding the installation of nginx-extras in the system. We have documented the repository with the example, creating branches for specific Django framework versions, we hope this will simplify the integration of the package into your projects.