brthor / gofer.net Goto Github PK
View Code? Open in Web Editor NEWEasy C# API for Distributed Background Tasks/Jobs for .NET Core.
License: MIT License
Easy C# API for Distributed Background Tasks/Jobs for .NET Core.
License: MIT License
JsonPrimitiveConverter causes issues with Deserialization.
In its current implementation, the TaskClient runs the TaskScheduler in the same loop it uses to execute tasks.
Tasks of any duration more than a second will cause the TaskScheduler to start running behind, scheduled tasks to be moved to the execution queue late, and recurring tasks to be potentially missed.
As a part of fixing this issue, tests should be developed which deliberately use long-running tasks and prove that scheduled tasks are not moved to the execution queue late, and recurring tasks are not missed.
Motivation:
We start Gofer task client from .net core hosted service.
We use the BackgroundService.ExecuteAsync() method that expects that underlying code will exit on cancellation. We've imitate cancellation support with
cancellation.Register(()=>taskClient.CancelListen())
but I think it will be good to have such feature out of the box.
With just a few task clients it's no big deal, but with a very large number of TaskClient's running, the CPU hit on Redis of all the clients checking for jobs continuously with no delay, because the queue is empty, is quite noticeable. What are your thoughts if I added a configurable Thread.sleep(nodatasleep)
if info==null
?
I started up a ridiculous number of TaskClients (~1800 instances spread across 18 VM's) as a test, and saw Redis start consuming 50% of the CPU :D
private async Task ExecuteQueuedTask()
{
var (json, info) = await TaskQueue.SafeDequeue();
if (info != null)
{
LogTaskStarted(info);
try
{
var now = DateTime.Now;
await info.ExecuteTask();
var completionSeconds = (DateTime.Now - now).TotalSeconds;
LogTaskFinished(info, completionSeconds);
}
catch (Exception e)
{
LogTaskException(info, e);
}
} else { // some kind of sleep code would go here I guess
}
}
Leaving this issue to track adding a windows build to the current build matrix.
The primary issue here is running an updated instance of redis on windows with which to test against. Having tried multiple methods, I do not believe this is currently possible on travis.
LCOW containers cannot be run on a travis instance. (Ultimately ends up with no hyper-v support).
Redis for windows is very outdated.
Hello
It is written on this page that the producer and the workers must share "tasks" code.
This seems counter-intuitive to me for two reasons:
What is the recommended strategy to split producer code and worker code so that the producer remains agnostic about how the worker processes tasks?
I wanted to add an output showing how many jobs are currently queued, but it doesn't look like queue depth is something that is tracked internally?
Title says it all.
Enqueuing an async method that returns a Task doesn't work.
Parameter types and positions are not currently examined when searching for the method to run after deserialization of the TaskInfo
on the worker.
Implement a search strategy that covers
https://github.com/brthor/Gofer.NET/blob/master/Gofer.NET.Utils/TaskInfo.cs#L30
When scheduled tasks are added they should be put on the task queue in the order of their timestamps.
While this won't necessarily guarantee they are run in order in a distributed environment, it at least provides some guarantee of ordering in a single-node environment.
The RedisQueue
class is a corner stone of the application, but is only tested indirectly.
After finding a bug just now, I'm convinced it should be tested directly.
Currently, when a Gofer.NET worker resets, it's log of scheduled tasks that need to be accounted for, and run on their specified schedules is also reset.
Scheduled tasks are, in fact, specific to the worker they are created on. Although logic exists to prevent scheduled tasks with the same key from having duplicate runs across multiple workers, no method exists to persist and share scheduled tasks across workers.
Multiple requirements:
When a worker restarts, or is brought up, it should load the current set of scheduled tasks into it's memory, and account for them in it's scheduler loop.
When a new scheduled task is added to any worker it should be persisted to the backing store.
When a new scheduled task is added to any worker sharing the same backing store, that scheduled task should be loaded into each of the other workers memory as a part of their scheduler loop, although not necessarily in every iteration. Some small amount of lag between a task being added to the store and existing workers picking it up is acceptable.
Supported scenarios and unsupported scenarios are not well explained outside the example on the README.
Improve Documentation with more examples and explanations of what will/won't work.
Set up Github Pages site with great documentation
Improve & Update README
XML <summary> tags on classes & methods.
Hi,
I want to use this lib for my project in production environment but i have a few question
Thanks.
I'm struggling to connect to redis when the connection string has a password. How do I specify a password in the connection string?
I tried the following formats to no avail,
var connString = "redis://<user>:<password>@<host>:<port>";
var connString = "<host>:<port>,password=<password>";
At-Least-Once Processing was disabled due to incorrect behavior.
ToTaskInfo(this Expression<Func<T>>)
is available, why not add TaskQueue.Enqueue(Expression<Func<T>>)
?CS4014: Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the await operator to the result of the call.
At least TaskQueue.Enqueue(Expression<Func<Task>>)
should be added.
Hello!
I just wanted to ask whether its possible to add priority to tasks. Simply adding a task to the start of the Queue (or close to it) would also be useful.
Thanks,
Mijago
Since version 1.0.0-beta-00110, our task execution code has stopped working. You can see how we do it, as well as the resulting exception, below.
await _taskClient.TaskScheduler.AddRecurringTask(() => Execute(command, _logger, _bus), TimeSpan.FromSeconds(schedule.Interval), taskName);
private static void Execute(Type command, IBumaLogger logger, ICommandSenderBus bus)
{
logger.Debug($"Executing scheduled command {command.FullName}");
try
{
var commandInstance = (IScheduledCommand)Activator.CreateInstance(command);
bus.SendAsync(commandInstance);
}
catch (Exception e)
{
logger.Error($"Error executing scheduled command {command.FullName}", e);
}
}
Task Error: Buma.Host, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.Execute threw an exception. (Task Id: 445f31a4-e1c7-4f26-8cf1-433577506137)
Sample.Host> System.ArgumentException: Object of type 'System.String' cannot be converted to type 'System.Type'.
Sample.Host> at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
Sample.Host> at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
Sample.Host> at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Sample.Host> at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Sample.Host> at Gofer.NET.Utils.TaskInfo.ExecuteTask()
Sample.Host> at Gofer.NET.TaskClient.Listen()
Sample.Host> Debug: Forwarded 1 lines
RedisTimeoutException occurs when using the StackExchange.Redis
Synchronous API.
StackExchange.Redis
Synchronous API.I've been using the library as a distributed job processor across over a dozen VM's, but scaling has not been the best experience.
Right now I have a Linux VM that runs a dotnet process as a systemd service. That process represents a single executor.
I templatized the service so I can spin up multiple copies, and run 20 instances of the dotnet service per server. I can then replicate this across more servers as needed.
But I find that I don't do a good job of estimating how much CPU usage my jobs actually take, and adding/removing systemd services is a bit of a nuisance across more than a dozen VM's. I'd prefer if I could define the number of executors in code and run a single process with multiple job executors.
So a single VM, with a single dotnet service, that contains multiple job listeners in the process.
Any ideas on how to implement this? I was thinking maybe a thread pool, but I wasn't sure what your experience had been.
This should be able to be done in automated testing in travis, by starting multiple redis hosts locally.
Running the existing test suite against it may suffice but thought should be placed into new potential failure scenarios and test scenarios created to cover them.
The only potential issue I see is with the incoming overhaul to the TaskScheduler that now uses Lua Scripts to promote due scheduled tasks to running tasks. Specifically the SCRIPT LOAD
call will likely need to be reworked. Additional research will need to be conducted into how lua scripts interact with Redis Cluster.
/cc @Efp95 from Gitter
In order to reach a 1.0 stable release several work items need to be completed, that will be tracked in separate issues.
The purpose of this issue is for overall tracking of the items which are necessary to reach 1.0.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.