博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
如何使用Django和Celery为Web构建进度栏
阅读量:2526 次
发布时间:2019-05-11

本文共 18184 字,大约阅读时间需要 60 分钟。

制作表面上非常简单的东西的惊人复杂性 (The surprising complexity of making something that is, on its surface, ridiculously simple)

Progress bars are one of the most common, familiar UI components in our lives. We see them every time we download a file, install software, or attach something to an email. They live in our browsers, on our phones, and even on our TVs.

进度栏是我们生活中最常见,最熟悉的UI组件之一。 每次下载文件,安装软件或将某些内容附加到电子邮件时,我们都会看到它们。 它们存在于我们的浏览器,我们的手机,甚至我们的电视中。

And yet — making a good progress bar is a surprisingly complex task!

但是-取得良好的进度条是一项令人惊讶的复杂任务!

In this post, I’ll describe all of the components of making a quality progress bar for the web, and hopefully by the end you’ll have a good understanding of everything you’d need to build your own.

在这篇文章中,我将描述制作Web质量进度条的所有组件,并希望到最后,您将对构建自己的一切有一个很好的了解。

This post describes everything I had to learn (and some things I didn’t!) to make , a library that hopefully makes it easy to drop in dependency-free progress bars to your Django/Celery applications.

这篇文章描述了制作所需要学习的所有知识(以及一些我没有学到的东西),该库可以使您轻松地将无依赖项进度条放到Django / Celery应用程序中。

That said, most of the concepts in this post should translate across all languages/environments, so even if you don’t use Python you probably can learn something new.

就是说,本文中的大多数概念都应在所有语言/环境中进行翻译,因此,即使您不使用Python,您也可能会学到新知识。

为什么选择进度条? (Why Progress Bars?)

This might be obvious, but just to get it out of the way — why do we use progress bars?

这可能很明显,但是只是为了避免干扰-为什么我们使用进度条?

The basic reason is to provide users feedback for something that takes longer than they are used to waiting. According to , 40% of people abandon a website that takes more than 3 seconds to load! And while you can use something like a spinner to help mitigate this wait, a tried and true way to communicate to your users while they’re waiting for something to happen is to use a progress bar.

根本原因是为用户提供的反馈时间要比等待时间长。 根据 ,有40%的人放弃了网站,而该网站的加载时间超过3秒! 尽管您可以使用微调器等功能来减轻这种等待时间,但在用户等待事件发生时与用户进行交流的一种切实可行的尝试方法是使用进度条。

Generally, progress bars are great whenever something takes longer than a few seconds and you can reasonably estimate its progress over time.

通常, 只要某事花费的时间超过几秒钟 ,进度条就会很棒 您可以合理地估计其随着时间的进度。

Some examples include:

一些示例包括:

  • When your application first loads (if it takes a long time to load)

    首次加载应用程序时(如果加载时间较长)
  • When processing a large data import

    处理大数据导入时
  • When preparing a file for download

    准备下载文件时
  • When the user is in a queue waiting for their request to get processed

    当用户在队列中等待其请求得到处理时

进度条的组成 (The Components of a Progress Bar)

Alright, with that out of the way lets get into how to actually build these things!

好了,让我们开始研究如何实际构建这些东西!

It’s just a little bar filling up across a screen. How complicated could it be?

屏幕上只是一个小条。 它有多复杂?

Actually, quite!

其实挺不错的!

The following components are typically a part of any progress bar implementation:

以下组件通常是任何进度条实现的一部分:

  1. A front-end, which typically includes a visual representation of progress and (optionally) a text-based status.

    前端 ,通常包括进度的可视表示和(可选)基于文本的状态。

  2. A backend that will actually do the work that you want to monitor.

    一个实际上可以完成您要监视的工作的后端

  3. One or more communication channels for the front end to hand off work to the backend.

    前端的一个或多个通信通道将工作移交给后端。
  4. One or more communication channels for the backend to communicate progress to the front-end.

    后端的一个或多个通信通道将进度传达到前端。

Immediately we can see one inherent source of complexity. We want to both do some work in the backend and show that work happening on the frontend. This immediately means we will be involving multiple processes that need to interact with each other asynchronously.

立即我们可以看到一种固有的复杂性来源。 我们既要在后端做一些工作 ,又要证明工作在前端进行。 这立即意味着我们将涉及需要异步进行交互的多个进程。

These communication channels are where much of the complexity lies. In a relatively standard Django project, the front-end browser might submit an AJAX HTTP request (JavaScript) to the backend web app (Django). This in turn might pass that request along to the task queue (Celery) via a message broker (RabbitMQ/Redis). Then the whole thing needs to happen in reverse to get information back to the front end!

这些通信渠道是许多复杂性所在。 在一个相对标准的Django项目中, 前端浏览器可能会向后端Web应用程序 (Django)提交AJAX HTTP请求(JavaScript)。 反过来,这可能会将请求通过消息代理 (RabbitMQ / Redis)传递到任务队列 (Celery)。 然后,整个过程需要反向进行,以使信息返回到前端!

The entire process might look something like this:

整个过程可能如下所示:

Let’s dive into all of these components and see how they work in a practical example.

让我们深入研究所有这些组件,并在一个实际示例中了解它们如何工作。

前端 (The Front End)

The front end is definitely the easiest part of the progress bar. With just a few small lines of HTML/CSS, you can quickly make a decent looking horizontal bar using the background color and width attributes. Splash in a little JavaScript to update it and you’re good to go!

前端绝对是进度条中最简单的部分。 仅需几行HTML / CSS,您就可以使用背景颜色和宽度属性快速制作出美观的水平条。 溅入一些JavaScript对其进行更新,您就可以开始了!

function updateProgress(progressBarElement, progressBarMessageElement, progress) {  progressBarElement.style.backgroundColor = '#68a9ef';  progressBarElement.style.width = progress.percent + "%";  progressBarMessageElement.innerHTML = progress.current + ' of ' + progress.total + ' processed.';}var trigger = document.getElementById('progress-bar-trigger');trigger.addEventListener('click', function(e) {  var barWrapper = document.getElementById('progress-wrapper');  barWrapper.style.display = 'inherit'; // show bar  var bar = document.getElementById("progress-bar");  var barMessage = document.getElementById("progress-bar-message");  for (var i = 0; i < 11; i++) {    setTimeout(updateProgress, 500 * i, bar, barMessage, {      percent: 10 * i,      current: 10 * i,      total: 100    })  }})

后端 (The Backend)

The backend is equally simple. This is essentially just some code that’s going to execute on your server to do the work you want to track. This would typically be written in whatever application stack you’re using (in this case Python and Django). Here’s an overly simplified version of what the backend might look like:

后端同样简单。 从本质上讲,这只是一些要在服务器上执行以完成您要跟踪的工作的代码。 通常,这可以用您正在使用的任何应用程序堆栈(在本例中为Python和Django)编写。 这是后端看起来过于简化的版本:

def do_work(self, list_of_work):     for work_item in list_of_work:         do_work_item(work_item)     return 'work is complete'

做工作 (Doing the Work)

Okay so we’ve got our front-end progress bar, and we’ve got our work doer. What’s next?

好的,我们有了前端进度栏,并且有了工作人员。 下一步是什么?

Well, we haven’t actually said anything about how this work will get kicked off. So let’s start there.

好吧,我们实际上还没有说过如何开展这项工作。 因此,让我们从那里开始。

错误的方式:在Web应用程序中进行操作 (The Wrong Way: Doing it in the Web Application)

In a typical ajax workflow this would work the following way:

在典型的ajax工作流程中,这将通过以下方式工作:

  1. Front-end initiates request to web application

    前端向Web应用程序发起请求
  2. Web application does work in the request

    Web应用程序确实可以在请求中工作
  3. Web application returns a response when done

    Web应用程序完成后返回响应

In a Django view, that would look something like this:

在Django视图中,看起来像这样:

def my_view(request):     do_work()     return HttpResponse('work done!')

错误的方式:从视图中调用函数 (The wrong way: calling the function from the view)

The problem here is that the do_work function might do a lot of work that takes a long time (if it didn't, it wouldn't make sense to add a progress bar for it).

这里的问题是do_work函数可能会执行很多耗时的工作(如果没有,则为其添加进度条是没有意义的)。

Doing a lot of work in a view is generally considered a bad practice for several reasons, including:

在视图中执行大量工作通常被认为是不好的做法,原因有几个,其中包括:

  • You create a poor user experience, since people have to wait for long requests to finish

    您创建的用户体验很差,因为人们必须等待很长时间才能完成请求
  • You open your site up to potential stability issues with lots of long-running, work-doing requests (which could be triggered either maliciously or accidentally)

    您会通过许多长期运行的工作请求(可能是恶意或意外触发)来打开站点,以解决潜在的稳定性问题

For these reasons, and others, we need a better approach for this.

由于这些原因以及其他原因,我们需要一种更好的方法。

更好的方法:异步任务队列(又名Celery) (The Better Way: Asynchronous Task Queues (aka Celery))

Most modern web frameworks have created asynchronous task queues to deal with this problem. In Python, the most common one is . In Rails, there is ().

大多数现代Web框架都创建了异步任务队列来处理此问题。 在Python中,最常见的是 。 在Rails中,有 ( )。

The details between these vary, but the fundamental principles of them are the same. Basically, instead of doing work in an HTTP request that could take arbitrarily long — and be triggered with arbitrary frequency — you stick that work in a queue and you have background processes — often referred to as workers — that pick the jobs up and execute them.

它们之间的细节各不相同,但是它们的基本原理是相同的。 基本上,不是在HTTP请求中执行可能会花费很长时间(并以任意频率触发)的工作,而是将工作放在队列中,并且有后台进程(通常称为工作进程)来接替工作并执行。

This asynchronous architecture has several benefits, including:

这种异步体系结构具有许多优点,包括:

  • Not doing long-running work in web processes

    不在Web流程中长时间运行
  • Enabling rate-limiting of the work done — work can be limited by the number of worker-processes available

    限制完成的工作的速度-可以通过可用的工作流程数量来限制工作
  • Enabling work to happen on machines that are optimized for it, for example, machines with high numbers of CPUs

    使工作能够在经过优化的机器上进行,例如,具有大量CPU的机器

异步任务的机制 (The Mechanics of Asynchronous Tasks)

The basic mechanics of an asynchronous architecture are relatively simple, and involve three main components: the client(s), the worker(s), and the message broker.

异步体系结构的基本机制相对简单,并且包含三个主要组件: 客户端工作线程消息代理

The client is primarily responsible for the creation of new tasks. In our example, the client is the Django application, which creates tasks on user input via a web request.

客户主要负责创建新任务。 在我们的示例中,客户端是Django应用程序,该应用程序通过Web请求在用户输入上创建任务。

The workers are the actual processes that do the work. These are our Celery workers. You can have an arbitrary number of workers running on however many machines, which allows for high availability and horizontal scaling of task processing.

工人是完成工作的实际流程。 这些是我们的芹菜工人。 您可以在任意多台计算机上运行任意数量的工作程序,这可以实现高可用性和水平扩展任务处理。

The client and task queue talk to each other via a message broker, which is responsible for accepting tasks from the client(s) and delivering them to the worker(s). The most common message broker for Celery is RabbitMQ, although Redis is also a commonly used and feature complete message broker.

客户端和任务队列通过消息代理相互对话,该消息代理负责从客户端接受任务并将其传递给工作人员。 尽管Redis也是常用且功能齐全的消息代理,但Celery上最常见的消息代理是RabbitMQ。

When building a standard celery application, you will typically do development of the client and worker code, but the message broker will be a piece of infrastructure that you just have to stand up (and beyond that can [mostly] ignore).

在构建标准的celery应用程序时,通常将开发客户端代码和工作程序代码,但是消息代理将是您仅需站起来的一部分基础结构(超出它的范围(通常可以忽略))。

一个例子 (An Example)

While this all sounds rather complicated, Celery does a good job making it quite easy for us via nice programming abstractions.

尽管这一切听起来都很复杂,但Celery出色地完成了工作,通过良好的编程抽象使我们很容易做到。

To convert our work-doing function to something that can be executed asynchronously, all we have to do is add a special decorator:

要将工作功能转换为可以异步执行的功能,我们要做的就是添加一个特殊的装饰器:

from celery import task # this decorator is all that's needed to tell celery this is a# worker task@task def do_work(self, list_of_work):     for work_item in list_of_work:         do_work_item(work_item)     return 'work is complete'

注释要从Celery调用的工作函数 (Annotating a work function to be called from Celery)

Similarly, calling the function asynchronously from the Django client is similarly straightforward:

同样,从Django客户端异步调用该函数也很简单:

def my_view(request):     # the .delay() call here is all that's needed    # to convert the function to be called asynchronously         do_work.delay()     # we can't say 'work done' here anymore     # because all we did was kick it off     return HttpResponse('work kicked off!')

异步调用工作函数 (Calling the work function asynchronously)

With just a few extra lines of code, we’ve converted our work to an asynchronous architecture! As long as you’ve got your worker and broker processes configured and running, this should just work.

仅需几行代码,我们就将工作转换为异步架构! 只要您已配置并运行了工作进程和代理进程, 它就可以正常工作

追踪进度 (Tracking the Progress)

Alrighty, so we’ve finally got our task running in the background. But now we want to track progress on it. So how does that work, exactly?

好了,所以我们终于在后台运行了任务。 但是现在我们要跟踪它的进度。 那到底是怎么工作的呢?

We’ll again need to do a few things. First we’ll need a way of tracking progress within the worker job. Then we’ll need to communicate that progress all the way back to our front-end so we can update the progress bar on the page. Once again, this ends up being quite a bit more complicated than you might think!

我们将再次需要做一些事情。 首先,我们需要一种跟踪工作者工作进度的方法。 然后,我们需要将该进度一直传递到前端,以便我们可以更新页面上的进度栏。 再一次,这最终变得比您想象的要复杂得多!

使用观察者对象跟踪工作进程 (Using an Observer Object to Track Progress in the Worker)

Readers of the seminal might be familiar with the . The typical observer pattern includes a subject which tracks state, as well as one or more observers that do something in response to state. In our progress scenario, the subject is the worker process/function that is doing the work, and the observer is the thing that is going to track the progress.

具有开创性读者可能熟悉 。 典型的观察者模式包括跟踪状态的主题 ,以及一个或多个响应状态而做某事的观察者 。 在我们的进度方案中,主题是执行工作的工作人员流程/职能,而观察者是要跟踪进度的事物。

There are many ways to link the subject and the observer, but the simplest is to just pass the observer in as an argument to the function doing the work.

有许多方法可以链接主题和观察者,但最简单的方法是将观察者作为传递给执行工作的函数的参数。

That looks something like this:

看起来像这样:

@task def do_work(self, list_of_work, progress_observer):         total_work_to_do = len(list_of_work)         for i, work_item in enumerate(list_of_work):                     do_work_item(work_item)                 # tell the progress observer how many out of the total items         # we have processed        progress_observer.set_progress(i, total_work_to_do)            return 'work is complete'

使用观察员监控工作进度 (Using an observer to monitor work progress)

Now all we have to do is pass in a valid progress_observer and voilà, our progress will be tracked!

现在我们要做的就是传递一个有效的progress_observer和voilà,我们的进度将被跟踪!

取得进展回客户 (Getting Progress Back to the Client)

You might be thinking “wait a minute… you just called a function called set_progress, you didn’t actually do anything!”

您可能会想“等一下……您刚刚调用了一个名为set_progress的函数,您实际上什么都没做!”

True! So how does this actually work?

真正! 那么,这实际上如何工作?

Remember — our goal is to get this progress information all the way up to the webpage so we can show our users what’s going on. But the progress tracking is happening all the way in the worker process! We are now facing a similar problem we had with handing off the asynchronous task earlier.

请记住-我们的目标是一直将此进度信息一直带到网页,以便我们向用户显示发生了什么情况。 但是进度跟踪在工作者过程中一直在发生! 现在,我们面临着一个类似的问题,该问题与我们之前完成异步任务有关。

Thankfully, Celery also provides a mechanism for passing messages back to the client. This is done via a mechanism called , and, like , you have the option of several different backends. Both RabbitMQ and Redis can be used as brokers and result backends and are reasonable choices, though there is technically no coupling between the broker and the result backend.

值得庆幸的是,Celery还提供了一种将消息传递客户端的机制。 这是通过一种称为的机制来完成的,与一样,您可以选择多个不同的后端。 RabbitMQ和Redis都可以用作代理和结果后端,并且是合理的选择,尽管从技术上讲,代理与结果后端之间没有耦合。

Anyway, like brokers, the details typically don’t come up unless you’re doing something pretty advanced. But the point is that you stick the result from the task somewhere (with the task’s unique ID), and then other processes can get information about tasks by ID by asking the backend for it.

无论如何,像经纪人一样,除非您正在做相当高级的事情,否则通常不会提供详细信息。 但是关键是您将任务的结果粘贴到某个位置 (具有任务的唯一ID),然后其他进程可以通过向后端索取ID来获取有关任务的信息。

In Celery, this is abstracted quite well via the state associated with the task. The state allows us to set an overall status, as well as attach arbitrary metadata to the task. This is a perfect place to store our current and total progress.

在Celery中,这通过与任务相关的state很好地抽象了。 state允许我们设置总体状态,以及将任意元数据附加到任务。 这是存储我们当前和全部进度的理想场所。

设定状态 (Setting the state)

task.update_state(     state=PROGRESS_STATE,     meta={'current': current, 'total': total} )

读状态 (Reading the state)

from celery.result import AsyncResult result = AsyncResult(task_id) print(result.state) # will be set to PROGRESS_STATE print(result.info) # metadata will be here

将进度更新获取到前端 (Getting Progress Updates to the Front End)

Now that we can get progress updates out of the workers / tasks and into any other client, the final step is to just get that information to the front end and display it to the user.

现在我们可以从工作人员/任务以及其他任何客户端获取进度更新,最后一步就是将这些信息发送到前端并将其显示给用户。

If you want to get fancy, you can use something like websockets to do this in real time. But the simplest version is to just poll a URL every so often to check on progress. We can just serve the progress information up as JSON via a Django view and process and render it client-side.

如果您想花哨的话,可以使用诸如websockets之类的方法实时进行操作。 但最简单的版本是每隔一段时间轮询一次URL以检查进度。 我们仅可以通过Django视图和进程将进度信息作为JSON提供,并在客户端进行渲染。

Django view:

Django视图:

def get_progress(request, task_id):     result = AsyncResult(task_id)     response_data = {         'state': result.state,         'details': self.result.info,    }     return HttpResponse(        json.dumps(response_data),         content_type='application/json'    )

Django view to return progress as JSON.

Django视图以JSON格式返回进度。

JavaScript code:

JavaScript代码:

function updateProgress (progressUrl) {    fetch(progressUrl).then(function(response) {         response.json().then(function(data) {             // update the appropriate UI components             setProgress(data.state, data.details);             // and do it again every half second            setTimeout(updateProgress, 500, progressUrl);         });     }); }

Javascript code to poll for progress and update the UI.

用于轮询进度并更新UI的Javascript代码。

放在一起 (Putting it All Together)

This has been quite a lot of detail on what is — on its face — a very simple and everyday part of our lives with computers! I hope you’ve learned something.

从表面上看,这是关于计算机生活中非常简单和日常的一部分的很多细节! 我希望你学到了一些东西。

If you need a simple way to make progress bars for you Django/celery applications you can check out — a library I wrote to help make all of this a bit easier. There is also .

如果您需要一种简单的方法来为Django / celery应用程序制作进度条,则可以签出 ,这是我编写的帮助简化所有步骤的库。 也 。

Thanks for reading! If you’d like to get notified whenever I publish content like this on building things with Python and Django, please sign up to receive updates below!

谢谢阅读! 如果您希望在我发布有关使用Python和Django构建内容的此类内容时得到通知,请注册以获取以下更新!

Originally published at .

最初发布于 。

翻译自:

转载地址:http://qikzd.baihongyu.com/

你可能感兴趣的文章
[CSS] Scale on Hover with Transition
查看>>
状压DP(挑战程序设计竞赛)
查看>>
POJ 2386
查看>>
腾讯云“动态加速”与“CDN”的区别——浅谈对“动态加速”的理解(可能有误)...
查看>>
Spring源码学习笔记(5)
查看>>
Objective-C 日记⑧ 对象初始化
查看>>
mybatis中#{}与${}的区别
查看>>
RTP/RTSP/RTCP的区别和应用
查看>>
Adaboost算法简介
查看>>
在【此电脑】隐藏【设备和驱动器】中不需要的图标
查看>>
【Leetcode】【Medium】Palindrome Partitioning
查看>>
51单片机 | 实现数码管动态显示
查看>>
十进制向十六进制的转换
查看>>
练习JsonJquery查找数据
查看>>
如何使用Goolge Timeline工具
查看>>
POJ3667 Hotel
查看>>
深入浅出 Java Concurrency (16): 并发容器 part 1 ConcurrentMap (1)[转]
查看>>
深入浅出 Java Concurrency (23): 并发容器 part 8 可阻塞的BlockingQueue (3)[转]
查看>>
将博客搬至CSDN
查看>>
HDOJ 2081
查看>>