Giter Club home page Giter Club logo

gofer.net's Introduction

Gofer.NET: Easy distributed tasks/jobs for .NET Core

Build Status Nuget Version Number Join the chat at https://gitter.im/Gofer-NET/Lobby

Documentation

Read the Docs

What is this?

This is a distributed job runner for .NET Standard 2.0 Applications.

Inspired by Celery for Python, it allows you to quickly queue code execution on a worker pool.

  • Use natural expression syntax to queue jobs for execution.

  • Queued jobs are persisted, and automatically run by the first available worker.

  • Scale your worker pool by simply adding new nodes.

  • Backed by Redis, all tasks are persistent.

Getting Started

Install the dotnet cli.

We recommend using the dotnet cli to get started, but it's not a necessity.

The dotnet cli is part of the .NET Core SDK.

Start a Redis instance.

We recommend using docker to start a local Redis instance for testing. Setting up a production-level Redis instance is out of the scope of this documentation.

$ docker run -d -p 127.0.0.1:6379:6379 redis:4-alpine

Create a project.

Open up a terminal and create a new console project to get started.

$ mkdir myProject && cd myProject
$ dotnet new console

Add the Gofer.NET NuGet package.

$ dotnet add package Gofer.NET --version 1.0.0-*

Queue up some jobs.

This example Program.cs shows how to queue jobs for the worker pool to process, then start a worker to go and run them.

Some important notes:

  • Workers would usually be on a separate machine from the code queueing the jobs, this is purely to give an example.

  • More workers can be added at any time, and will start picking up jobs off the queue immediately.

public class Program
{
    public static async Task Main(string[] args)
    {
        var redisConnectionString = "127.0.0.1:6379";
        
        // Create a Task Client connected to Redis
        var taskClient = new TaskClient(TaskQueue.Redis(redisConnectionString));
        
        // Queue up a Sample Job
        await taskClient.TaskQueue.Enqueue(() => SampleJobFunction("Hello World!"));
        
        // Start the task listener, effectively turning this process into a worker.
        // NOTE: This will loop endlessly waiting for new tasks.
        await taskClient.Listen();
    }
    
    private static void SampleJobFunction(object value)
    {
        Console.WriteLine(value.ToString());
    }
}

Read the Docs for more details.

gofer.net's People

Contributors

brthor avatar gitter-badger avatar patricker avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gofer.net's Issues

Priority tasks

Hello!

I just wanted to ask whether its possible to add priority to tasks. Simply adding a task to the start of the Queue (or close to it) would also be useful.

Thanks,
Mijago

Can't use connection strings with passwords

I'm struggling to connect to redis when the connection string has a password. How do I specify a password in the connection string?

I tried the following formats to no avail,

var connString = "redis://<user>:<password>@<host>:<port>";

var connString = "<host>:<port>,password=<password>";

Implement Tests for RedisQueue class

The RedisQueue class is a corner stone of the application, but is only tested indirectly.

After finding a bug just now, I'm convinced it should be tested directly.

TaskClient should run TaskScheduler in its own thread.

In its current implementation, the TaskClient runs the TaskScheduler in the same loop it uses to execute tasks.

Tasks of any duration more than a second will cause the TaskScheduler to start running behind, scheduled tasks to be moved to the execution queue late, and recurring tasks to be potentially missed.

As a part of fixing this issue, tests should be developed which deliberately use long-running tasks and prove that scheduled tasks are not moved to the execution queue late, and recurring tasks are not missed.

Add a configurable delay if there are no jobs in the queue

With just a few task clients it's no big deal, but with a very large number of TaskClient's running, the CPU hit on Redis of all the clients checking for jobs continuously with no delay, because the queue is empty, is quite noticeable. What are your thoughts if I added a configurable Thread.sleep(nodatasleep) if info==null?

I started up a ridiculous number of TaskClients (~1800 instances spread across 18 VM's) as a test, and saw Redis start consuming 50% of the CPU :D

       private async Task ExecuteQueuedTask()
        {
            var (json, info) = await TaskQueue.SafeDequeue();
            if (info != null)
            {
                LogTaskStarted(info);

                try
                {
                    var now = DateTime.Now;
                    
                    await info.ExecuteTask();
                    
                    var completionSeconds = (DateTime.Now - now).TotalSeconds;
                    LogTaskFinished(info, completionSeconds);
                }
                catch (Exception e)
                {
                    LogTaskException(info, e);
                }
            } else { // some kind of sleep code would go here I guess 
            }
        }

Add Windows Build

Leaving this issue to track adding a windows build to the current build matrix.

The primary issue here is running an updated instance of redis on windows with which to test against. Having tried multiple methods, I do not believe this is currently possible on travis.

  1. LCOW containers cannot be run on a travis instance. (Ultimately ends up with no hyper-v support).

  2. Redis for windows is very outdated.

Add TaskQueue.Enqueue(Expression<Func<T>>) to suppress warning CS4014

public static TaskInfo ToTaskInfo<T>(this Expression<Func<T>> expression)

Since ToTaskInfo(this Expression<Func<T>>) is available, why not add TaskQueue.Enqueue(Expression<Func<T>>)?
It's helpful to suppress warning CS4014 with async function.

CS4014: Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the await operator to the result of the call.

At least TaskQueue.Enqueue(Expression<Func<Task>>) should be added.

Test Gofer.NET with Redis Cluster

This should be able to be done in automated testing in travis, by starting multiple redis hosts locally.

Running the existing test suite against it may suffice but thought should be placed into new potential failure scenarios and test scenarios created to cover them.

The only potential issue I see is with the incoming overhaul to the TaskScheduler that now uses Lua Scripts to promote due scheduled tasks to running tasks. Specifically the SCRIPT LOAD call will likely need to be reworked. Additional research will need to be conducted into how lua scripts interact with Redis Cluster.

/cc @Efp95 from Gitter

Persist Scheduled Tasks to the Backing Store

Currently, when a Gofer.NET worker resets, it's log of scheduled tasks that need to be accounted for, and run on their specified schedules is also reset.

Scheduled tasks are, in fact, specific to the worker they are created on. Although logic exists to prevent scheduled tasks with the same key from having duplicate runs across multiple workers, no method exists to persist and share scheduled tasks across workers.

Multiple requirements:

  1. When a worker restarts, or is brought up, it should load the current set of scheduled tasks into it's memory, and account for them in it's scheduler loop.

  2. When a new scheduled task is added to any worker it should be persisted to the backing store.

  3. When a new scheduled task is added to any worker sharing the same backing store, that scheduled task should be loaded into each of the other workers memory as a part of their scheduler loop, although not necessarily in every iteration. Some small amount of lag between a task being added to the store and existing workers picking it up is acceptable.

Re-enable At-Least-Once Processing

At-Least-Once Processing was disabled due to incorrect behavior.

  1. Improve Test Coverage of this scenario including simulated worker failures.
  2. Re-enable At-least-once processing

Task execution with argument of type System.Type throws exception

Since version 1.0.0-beta-00110, our task execution code has stopped working. You can see how we do it, as well as the resulting exception, below.

await _taskClient.TaskScheduler.AddRecurringTask(() => Execute(command, _logger, _bus), TimeSpan.FromSeconds(schedule.Interval), taskName);

private static void Execute(Type command, IBumaLogger logger, ICommandSenderBus bus)
{
    logger.Debug($"Executing scheduled command {command.FullName}");
    try
    {
        var commandInstance = (IScheduledCommand)Activator.CreateInstance(command);
        bus.SendAsync(commandInstance);
    }
    catch (Exception e)
    {
        logger.Error($"Error executing scheduled command {command.FullName}", e);
    }
}

Task Error: Buma.Host, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.Execute threw an exception. (Task Id: 445f31a4-e1c7-4f26-8cf1-433577506137)
Sample.Host> System.ArgumentException: Object of type 'System.String' cannot be converted to type 'System.Type'.
Sample.Host>    at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
Sample.Host>    at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
Sample.Host>    at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Sample.Host>    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Sample.Host>    at Gofer.NET.Utils.TaskInfo.ExecuteTask()
Sample.Host>    at Gofer.NET.TaskClient.Listen()
Sample.Host> Debug: Forwarded 1 lines

Example of running multiple job listeners in a single process?

I've been using the library as a distributed job processor across over a dozen VM's, but scaling has not been the best experience.

Right now I have a Linux VM that runs a dotnet process as a systemd service. That process represents a single executor.
I templatized the service so I can spin up multiple copies, and run 20 instances of the dotnet service per server. I can then replicate this across more servers as needed.

But I find that I don't do a good job of estimating how much CPU usage my jobs actually take, and adding/removing systemd services is a bit of a nuisance across more than a dozen VM's. I'd prefer if I could define the number of executors in code and run a single process with multiple job executors.

So a single VM, with a single dotnet service, that contains multiple job listeners in the process.

Any ideas on how to implement this? I was thinking maybe a thread pool, but I wasn't sure what your experience had been.

Ensure ScheduledTasks are queued in order of timestamp.

When scheduled tasks are added they should be put on the task queue in the order of their timestamps.

While this won't necessarily guarantee they are run in order in a distributed environment, it at least provides some guarantee of ordering in a single-node environment.

Simple way to get Queue Depth?

I wanted to add an output showing how many jobs are currently queued, but it doesn't look like queue depth is something that is tracked internally?

Recommended strategy to split producer code and worker code ?

Hello
It is written on this page that the producer and the workers must share "tasks" code.
This seems counter-intuitive to me for two reasons:

  • Main Gofer examples show a producer with tasks code, and workers without (tasks are processed at client side!). By reading examples, we can think that tasks are executed by the producer (lambda evaluation).
  • Worker and producer generally have different library requirements. Ex: if we work on image processing tasks, the worker needs an image processing library but the producer doesn't.

What is the recommended strategy to split producer code and worker code so that the producer remains agnostic about how the worker processes tasks?

Dashboard for monitoring

Hi,
I want to use this lib for my project in production environment but i have a few question

  • Do you have any plan for the dashboard feature?
  • Will this library still be supported in the future?

Thanks.

Roadmap to 1.0 Stable Release

In order to reach a 1.0 stable release several work items need to be completed, that will be tracked in separate issues.

The purpose of this issue is for overall tracking of the items which are necessary to reach 1.0.

  • Scheduled & Recurring Tasks Persistence + TaskScheduler Overhaul #20

  • At-Least-Once Processing #18

  • Load Testing

  • TaskClient should run TaskScheduler in its own thread. #30

  • Adequate Documentation with examples. #17

  • Add OSX Build

  • Change LICENSE to LGPL 3.0

Improve Documentation

Supported scenarios and unsupported scenarios are not well explained outside the example on the README.

Improve Documentation with more examples and explanations of what will/won't work.

  • Set up Github Pages site with great documentation

  • Improve & Update README

  • XML <summary> tags on classes & methods.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.