Would it be possible to provide a github demo? about celery-progress HOT 25 CLOSED

@czue @formacube I quickly put together a demo of a fake download page (the task is actually running a 5 seconds timer, based on the celery_progress documentation):
https://github.com/eeintech/django-celery-progress-demo

It's functional but I haven't documented it yet, @formacube feel free to look around and ask if you have any question. You'll need redis-5.0.8 (have not tested 6.x) server to be installed and running.
Make sure to install all required packages: pip install -r requirements.txt (it will install celery and celery-progress packages, along with Django 3)
You can start celery worker with the included script: ./start_celery
Then run the Django project: .manage.py runserver (don't forget to migrate first)

Once I get more time, I'll try to improve it with something actually functional where it really downloads the file/url requested.

(and sorry for the lack of comments and terrible coding form 😛)

from celery-progress.

I'm going to close this. @formacube if you have any other issues feel free to post back. Thanks for the help @eeintech !

from celery-progress.

Hey - would love to put together a demo sometime.

Did you see this article? It's got a lot of background information that might help: https://buildwithdjango.com/blog/post/celery-progress-bars/

from celery-progress.

That's the one which led me to celery-progress actually

You don't know any public project on github which may use celery-progress and that I could get inspiration from, by chance?

Originally, I was trying to find (or expecting to find) the code behind the folllowing page https://buildwithdjango.com/projects/celery-progress/ in a github repository.

I don't know how easily it could be done, so... just a sugestion for a potential quick win

from celery-progress.

@formacube I also struggled the first time to implement the full circle between the front and backend. I do have it running with Django(+celery/redis) now, was thinking maybe of sharing a demo code. I can let you know once I get time to set it up.

Did you make any progress until now?

from celery-progress.

@eeintech that would be amazing! Keep me posted if you do make any progress so I can link to the examples in the docs.

from celery-progress.

@eeintech : I didn't progress much.
I went through some documentation,
Some videos,
And then I was wondering if celery was the right option for me based on all those comments claiming that celery was sometime very hard to parametrize.

I don't need anything complex.
I just need to launch some processes that are 10 minutes to 12h long and be able to get a feedback at what stage we are for the ongoing processes.

It's OK for me if each process is launched after the other completes, and I don't have the infrastructure to parallelize.

Therefore I'm looking for something lightweight.
I was maybe thinking about django-q.

But I put this on pause so far

from celery-progress.

Awesome! @eeintech whenever you feel it's ready for primetime can you submit a PR to the readme so I can link it up? Thanks so much for the help!

from celery-progress.

@czue Thanks! I've just released the first version of the demo 😄
I am writing an article on my website on the different pieces needed to get it up and running.
When I submit the PR, would you mind if I also add a link to my article?

from celery-progress.

Sure, go ahead. I'll review both the demo and the article and merge as long as I think they're helpful for users of the library!

from celery-progress.

@eeintech
Thanks a lot for the precious help.
I have other things currently requiring my attention, but I'll look at it as quickly as possible

from celery-progress.

I am writing an article on my website on the different pieces needed to get it up and running.

That's great, thanks @eeintech .
I went through the repository, and it really looks great.

I must say that as I'm not really a professional IT, the thing that worries me a bit at the moment is the configuration of Redis / Celery (based on configuration concerns expressed through my different readings).

Therefore I guess your article would be most welcome !

Another immediate question for me is: I'm coding on my Windows machine.
It's not really recommanded to install redis on windows.
So for development / testing purpose, can I install Redis on the production server and send commands from the dev environement?

Or what would you recommend?

from celery-progress.

@formacube I will explain this in my article, in my demo I'm using Redis as message broker for Celery because Redis is very versatile and I'm also using it as cache on my server. However, you can totally run this demo replacing Redis with any other message broker, Celery recommends RabbitMQ. I just checked if it's possible to install it on Windows, looks like it:
https://www.rabbitmq.com/install-windows.html

The Django and Celery settings will have to point to RabbitMQ instead, it should be pretty easy to adjust them. Celery itself can run in any Python environment so I believe it is OS agnostic.

I have not tried to connect Celery to a remote Redis instance, I think it is possible using a SSH tunnel and when using Redis as message broker only. I'll give this a shot soon. Is your server running Linux?

from celery-progress.

@formacube Good news, it is possible to use an SSH tunnel to connect to a Redis instance on a remote server! Have you already installed Redis and is it running on your remote server?
If not, my article will explain how to install and run it (almost there).

For the ssh tunnel, I'm using this command:
ssh -f -N -L 6379:localhost:6379 <username>@<hostname> -p <port>
This put the tunnel in the background (no prompt). It redirects traffic from your computer port 6379 to your remote server port 6379, creating a (virtual) Redis instance on your computer at the localhost address. Replace <username> and <hostname> with your server configuration and use the -p (port) option if needed.

from celery-progress.

@eeintech no I didn't install it yet

I will wait for your article

Thanks a lot for the information about the ssh tunnel!

from celery-progress.

@formacube Alright, I've just completed most of my article: https://eeinte.ch/stream/progress-bar-django-using-celery/
I tested all the code you can find there, I hope it works smoothly for you too.

Eventually, I will write the last section which is meant to explain the final GitHub demo code but I hope it helps you grasp on the concepts involved to get the progress bar up and running.

If you have any comments, suggestions, or questions please do let me know!

from celery-progress.

@eeintech that article looks epic! definitely will give it a link

from celery-progress.

@czue I finally sent the PR request, I hope it looks okay!

@formacube Any luck on your side? Would be interested to know what you think about my step-by-step guide :)
The last section is on its way, it's basically bridging the gap between a simple timer task and a download task (which is the scope of the github demo application).

from celery-progress.

@eeintech I can try to have a look between this evening and tomorrow, but I will not be able to try it before this weekend

from celery-progress.

@eeintech I just went through your article. That's great and very detailed. Very didactic!

I think it will help a lot of people finding their way through the ton of information which is available out there...but not always relevant for a non specialist like me.

The only aspect that I'm still missing to get my own solution working is the following :

Imagine I have several tasks that I can put in the queue, and that each tasks can last from 30 minutes to 14 hours.
My goal would be to have a page where I could log to, and see which are the tasks that I launched and what are their respective progression status.

In all the examples I've seen so far as in yours, tasks ids are passed on to the page which displays the progresses.

In my case, I would like that the view goes to retrieve the ids of the relevant tasks by itself. In my current understanding, I guess that this place where current tasks ids are stored could be redis.

Is that realistic?

from celery-progress.

PS: I saw your site is .ch. Where are you located if it's not too indiscreet ? I'm in Lausanne

from celery-progress.

@formacube Thanks! I was hoping for an approachable format because I did get confused too at first, it's a pretty high step for non-experts in this field (which I am not, my background is hardware engineering). After much reading though, I finally put a finger on each part and how they interact with each other, which I wanted to summarize in this guide (also for me to remember in the future).

This sounds like a cool project you're working on! I actually want to recommend you the django-celery-model library, I'm using it for my other website that I'm building (still work in progress). Using this library, you can attach multiple tasks to a specific model and instances of the model, store them as long as you need to and be able to use the "model instance <-> task ID" association to retrieve information about the task. I think this would fit your application: each time the "Tasks Statuses" page would be loaded by a logged user, you could pull each task status and update the progress bars accordingly.
On my website, I'm storing everything in a SQL database. I believe Redis could replace the SQL database to store the "model instance <-> task ID" associations, however, I have not tried it. If the rate you need to check all progresses and pull data from the database is low, I think you can just stick with SQL (for long-running tasks of few hours that's probably your case). If all tasks had to be executed quickly, maybe in this case Redis comes handy because, from what I've read, I think it has much faster access times compared to a SQL database.
Is this an open-source project? Sounds like a fun one to contribute to and I wouldn't mind getting my hands dirty on a new challenge 😛

I have friends in Switzerland but I'm not from there, I am from a small country near Switzerland called... France 😄
I guess you could say I've cheated but I've actually used the ".ch" extension because I wanted my username as my domain name, and ".tech" extension is twice the price or more for no extra reward. ".ch" does offer better privacy too, I do recommend!

from celery-progress.

@formacube Just checking in to see if you have made progress (literally 😃)

from celery-progress.

Sorry, no progress, and no time allocated to this project.

from celery-progress.

@formacube Ok, you may consider closing this issue for the time being 😉

from celery-progress.