Hacking Django Channels for Fun (and Profit)

LoudSwarm by Six Feet Up, a Django-based virtual event platform, has always supported Slack for instant, in-app text communication. In addition to Slack, clients were requesting Discord integration, so Six Feet Up’s expert team of developers found a solution.  

How did Six Feet Up make this integration a reality? It began with the Django channels package. Let’s dive in. 

TL;DR

There are the three critical steps required to create a long-lived client websocket connection within a Django/Channels application:

1. Start a Python asyncio task.
2. Place the task in the handle method of a custom subclass of the Channels Worker class.
3. Append it to the list of listeners that the worker is managing, before that list is awaited. 

The Discord client library —  discord.py — is well-suited to this method. It has self-contained websocket handling and can be started as a single Task. However, to communicate with Discord from tasks connected to Channels’ “channel layer,” it’s essential to create a wrapper around the Discord client to bridge the channel layer and the client library.

Understand the Basics

Django is a high-level, server-side Python Web framework which was released publicly in 2005. This open source framework — maintained by the Django Software Foundation — has an active and friendly developer community that consistently creates add-on packages for further improvement. 

Django also supports the classic model of HTTP request→response by:

• taking URL or POST parameters in the request,
• retrieving/storing some data, 
• assembling a complete HTML page within its template system to return a response.  

There is also support available for REST APIs, GraphQL queries and countless other features.

In 2015, Andrew Godwin, a Django core developer, released an add-on package named channels. This package provided a way for Django to use websockets — a departure from the classic HTTP response cycle. 

How are websockets a superset of HTTP? 

• A client makes an HTTP request to a given endpoint.
• Connections are upgraded via a standard negotiation protocol.
• The TCP connection between the two peers is held open as long as both sides deem necessary.
  • This occurs rather than only supporting one HTTP request and response (ignoring the concept of pipelining for the moment).
• Data is passed back and forth asynchronously as a bidirectional, full duplex pipe.
  • This uses standard protocols defined in RFC 6455.

The channels package has been through nearly a full rewrite since its initial release, and it is now at version 3.0.3 (as of March 2021). The channels package supports a lightweight message passing system (called a channel layer), which allows messages to be supported/communicated among multiple server hosts running the same Django application. 

The package also provides separate application worker support to offload and distribute processing away from the main Django application — which is primarily concerned about the incoming client traffic. Both of these features are prominently highlighted in the technique used to integrate Discord into LoudSwarm by Six Feet Up.

Apply Outgoing WebSockets

The channels package supports incoming websocket requests from clients. These requests are typically human-directed web browsers, but can be external server-to-server connections using a defined API. However, there are times when it would be easier for the Django application to maintain a websocket connection as a client, instead of as a server. Note: this concept is not currently supported by either standard Django or the channels package.

 Django works in a “request loop” due its origins as a server framework serving ephemeral, stateless HTTP requests. This doesn’t allow indirect, long-running tasks to serve requests from clients. Several packages — such as celery and rq — provide task worker queues to handle offline processing. However, these are single-task-oriented systems that deal with queues of well-defined and discrete chunks of work, rather than real-time messages streamed from an external peer.

Assign Worker Tasks

The channels package does provide a lightweight external worker task capability by listening on a specified channel — within the channel layer — and running the corresponding function that is pre-configured from the Django application. While running this worker as a separate process, it’s vital to specify the channel names this worker process should handle. Now, many different workers can be started — on multiple hosts — while handling different segments of background processing:

Within the Worker class — defined by Channels to run these workers — the list of channel names is mapped to the appropriate functions. These functions are started as Python asyncio tasks, and then the Worker class waits for the list of tasks to finish:

async def handle(self):
    """
    Listens on all the provided channels and handles the messages.
    """
    # For each channel, launch its own listening coroutine
    listeners = []
    for channel in self.channels:
        listeners.append(asyncio.ensure_future(self.listener(channel)))
    # Wait for them all to exit
    await asyncio.wait(listeners)
    # See if any of the listeners had an error (e.g. channel layer error)
    [listener.result() for listener in listeners]

Since the Worker class can be subclassed or cloned, its behavior can be altered to run an additional Task in addition to the channels passed on the command line — which is the function that runs the client websocket instance:

class WebsocketWorker(Worker):
    async def handle(self):
        """
        Listens on all the provided channels and handles the messages.
        """
        # For each channel, launch its own listening coroutine
        listeners = []
        for channel in self.channels:
            listeners.append(asyncio.ensure_future(self.listener(channel)))

        listeners.append(asyncio.ensure_future(self.run_websocket()))

        # Wait for them all to exit
        await asyncio.wait(listeners)
        # See if any of the listeners had an error (e.g. channel layer error)
        [listener.result() for listener in listeners]

To be useful, the websocket handler function must create and manage the lifecycle of the websocket and its connection. It also needs somewhere for the incoming websocket data to go, and some way to receive that data from other parts of the application to transmit to the websocket peer. 

Build the Function

There are many ways this function could be built — many of which utilize the websockets library created by Aymeric Augustin. In this example, the function should communicate with the Discord API. Conveniently, Discord provides a prebuilt library exactly for this purpose (although confusingly named discord.py), which:

• manages the websocket on its own,
• provides customization hooks to manage the flow of data to and from the remote API,
• maintains excellent documentation to get it setup. 

The easiest method of integrating the library with Django and Channels (using the technique described above) is to write a subclass of the discord.Client class, and sprinkle some magic in its event handler methods:

class ChatClient(discord.Client):
    async def on_ready(self):
        logger.info(f"Initialized Discord connection as {self.user.name}, {self.user.id}")

    async def on_message(self, message):
        incoming_data = {
            "channel": str(message.channel.id),
            "text": message.content,
            "user": message.author.id,
            "discord_display_name": message.author.display_name,
            "discord_username": message.author.username,
            "discord_avatar_url": str(message.author.avatar_url),
            "event_id": str(message.id),
            "event_time": int(message.created_at.timestamp()),
            "team_id": str(message.guild.id),
        }

        await do_something_with_incoming_data(incoming_data)

Next, build the code that will bridge the gap between the Channels message system (the channel layer) and the Discord library. Don’t forget to include:

• a channel listener (so that other parts of the application can send messages), and
• a do_something_with_incoming_data function that’s in the block of code above.

How to Integrate Discord into a Django-Based Platform

Now let’s get specific about how Six Feet Up integrated Discord into the Django-based virtual event platform, LoudSwarm by Six Feet Up. The task: find a way to connect the various parts described above to the Discord client. In the case of LoudSwarm, Six Feet Up chose to put a reference to the client in a module-level dictionary called discord_client. This is a hack developers can use to access a “global-ish” object without using the dreaded global keyword.  

The team created an AsyncConsumer which listens on a defined channel and will be passed into the management command when starting the worker process. This enables the listener to have access to the in-memory client object, as it passes the data to Discord:

class DiscordSender(AsyncConsumer):
    async def send_to_discord(self, message):

        # Yes, imports within code are usually frowned upon, but we do it
        # here in order to avoid import loops
        from loudswarm.chat.discordworker import discord_client

        text_msg = message["message"]
        chat_chan = int(message["discord_channel"])  # This is the Discord channel, not the Django one
        channel = discord_client.client.get_channel(chat_chan)
        await channel.send(text_msg)

In the implementation, the do_something_with_incoming_data function — mentioned above — becomes a call to a synchronous function wrapped with the Channels database_sync_to_async decorator, which enabled the safe use of the Django ORM from async tasks. This function passes the incoming message to the existing chat application — originally built for talking to Slack — so the team translated the message parameters into the Slack equivalents. 

The full source of the implemented DiscordWorker is below:

import asyncio
import logging

from addict import Dict
import sys
import discord
from channels.worker import Worker
from loudswarm.chat.tasks import create_slack_message
from django.conf import settings
from loudswarm.chat.consumers import DiscordSender
from loudswarm.webhooks.models import WebhookTransaction
from channels.db import database_sync_to_async

logger = logging.getLogger(__name__)

discord_client = Dict(client=None)


@database_sync_to_async
def make_webhook_transaction(data):
    webhook_transaction = WebhookTransaction.objects.create(
        body=data, integration="slack"
    )

    create_slack_message.apply_async([webhook_transaction.id], {}, countdown=1)


class ChatClient(discord.Client):
    async def on_message(self, message):
        data = {
            "channel": str(message.channel.id),
            "text": message.content,
            "user": message.author.id,
            "discord_display_name": message.author.display_name,
            "discord_username": message.author.username,
            "discord_avatar_url": str(message.author.avatar_url),
            "event_id": str(message.id),
            "event_time": int(message.created_at.timestamp()),
            "team_id": str(message.guild.id),
}        }

        await make_webhook_transaction(data)


class DiscordWorker(Worker):
    async def handle(self):
        """
        Listens on all the provided channels and handles the messages.
        """
        # For each channel, launch its own listening coroutine
        listeners = []
        for channel in self.channels:
            listeners.append(asyncio.ensure_future(self.listener(channel)))

        # Add coroutine for outgoing websocket connection to Discord API
        listeners.append(asyncio.ensure_future(self.run_websocket()))

        # Most of our time is spent here, waiting until all the lackeys exit
        await asyncio.wait(lackeys)
        # See if any of the listeners had an error (e.g. channel layer error)
        [listener.result() for listener in listeners]

    async def run_websocket(self):
        client = ChatClient()
        discord_client.client = client  # We must put this into a mutable object
        (await client.start(settings.DISCORD_TOKEN)).result()

Analyze the Results

Six Feet Up developers successfully created a system that communicates efficiently with Discord, while not disturbing the rest of the client-serving parts of the Django application. There are certainly parts of the implementation that can —  and will — continue to be improved. For starters, there are many points to branch off and provide larger functionality, which includes multiple outgoing websockets — not just to Discord, but through self-managed websockets as well.

I hope you have enjoyed the tour through Channels! This is just the tip of the iceberg. There are countless creative solutions you can develop to enhance its capabilities. Have fun (and make profit)!