betterment / delayed Goto Github PK
View Code? Open in Web Editor NEWa multi-threaded, SQL-driven ActiveJob backend used at Betterment to process millions of background jobs per day
License: MIT License
a multi-threaded, SQL-driven ActiveJob backend used at Betterment to process millions of background jobs per day
License: MIT License
I saw that Delayed::Command has been removed from this gem in favor of running through rake.
Could you provide an example of how to achieve the same (or the replacement) behavior?
I saw that the QUEUE env is used to specify the queue used by the workers. Is MAX_CLAIMS your way of specifying concurrency? delayed_job would use OS level processes -- delayed uses threads, is that correct?
I am opening this issue primarily to get the thoughts and feedbacks from others on whether or not there are glaring problems with this idea that I'm not seeing. So, to the idea:
What if the job queue were run against a dedicated database, separate from the primary app database? Rails now supports multiple databases, so the infrastructure is in place. I began an experimental branch in a project of mine (leading to PR #12) where I am using a SQLite database to manage my job queue independently of my primary PostgreSQL database. To my initial thinking, this gives the benefits of a Redis-backed queue of not blocking or throttling application database work, while also giving the benefits of having a SQL-backed queue.
However, one of the the primary benefits of a SQL-backed jobs queue is co-transactionality. In the README I note:
Important: the above assumes that the connection used by the transaction is the one provided by
ActiveRecord::Base
. (Support for enqueuing jobs via other database connections is possible, but is not yet exposed as a configuration.)
So, this gets me thinking, what all do you know that I have yet to learn? And, how can I help, if possible, to bring co-transactional, but separate DB jobs into the world?
Hi there, I've been running delayed
as my job backend for a while now and just started noticing that some concurrent jobs don't seem to run.
I only have 1 worker, so this might be expected but in https://github.com/Betterment/delayed#running-a-worker-process it says
By default, a worker process will pick up 2 jobs at a time (ordered by priority) and run each in a separate thread.
(That seems to be 5 by default now)
My jobs are very long running and I have:
Job B will never actually run - I think technically it does, but it just no-ops as it is past it's finished time.
So for my understanding, each worker will run multiple jobs in parallel but it does not check for additional jobs while 1 job is running. Even for less long running/scheduled tasks, having 100 jobs of 1 min duration, sprinkled in with some 5 min duration jobs we would sit idle every once in a while until the 5min job is completed?
https://github.com/Betterment/delayed#migrating-from-delayedjob states "that some configurations, like queue_attributes, exit_on_complete, backend, and raise_signal_exceptions have been removed entirely." I think the lack of raise_signal_exceptions (and the reliance on the behaviour described in https://github.com/Betterment/delayed#running-a-worker-process) could prevent me from suggesting switching over from delayed_job to delayed. Would it be difficult to support raise_signal_exceptions and are there any concerns with the idea of supporting it?
We plan to use 'delayed' as an asynchronous processing worker on Kubernetes pods.
So we need a way to do healthcheck of the worker process and we have the following method to do the healthcheck now.
livenessProbe:
exec:
command:
- bash
- -lc
- '[[ $(ps aux | grep delayed | grep -v grep | wc -l) > 0 ]]'
However, if there is a better method, we would like to adopt it.
So, do you have any other methods?
I'd like to know it for reference because I saw that you use kubertenes in other issues.
Working with @emmaoberstein on setting up a new queue, we discovered that there is a condition when the worker will get stuck in a spinloop, running the pickup query multiple times per second (effectively disobeying the sleep_delay
config).
The likeliest reproduction would be to enqueue a job that fails deserialization completely, or that otherwise prevents the usual job cleanup (e.g. it crashes the thread).
Here's where the work_off
method hits the spinloop:
def work_off(num = 100)
success = Concurrent::AtomicFixnum.new(0)
failure = Concurrent::AtomicFixnum.new(0)
num.times do
jobs = reserve_jobs
break if jobs.empty? # jobs is not empty
pool = Concurrent::FixedThreadPool.new(jobs.length)
jobs.each do |job|
pool.post do
# Exception encountered when `payload_object` is first called, e.g. job fails to deserialize
# - success and failure are never incremented
# - job remains in queue and is immediately returned by the next `reserve_jobs` without waiting
end
end
pool.shutdown
pool.wait_for_termination
break if stop?
end
[success, failure].map(&:value)
end
There a few ways I could think to fix this:
num.times do
.reserve_jobs
won't immediately return the same job again (due to exponential backoff).All of these feel fairly reasonable, though I'd be inclined to explore the second and third. (Adding a new delay would require more tuning & testing and would not actually address the underlying failure mode for the job.) So, actually, I'd want to start with the third option, since it would likely also address the remaining issue in #23.
Hello!
We're receiving the following error after upgrading to Rails 7.1.1
Job failed to load: undefined class/module Delayed::JobWrapper.
Handler: \"--- !ruby/object:Delayed::JobWrapper
job_data:
job_class: TestJob
job_id: af7d22a1-35c9-42d8-9f57-8af524bffcc5
provider_job_id:
queue_name: default
priority:
arguments: []
executions: 0
exception_executions: {}
locale: en
timezone: UTC
enqueued_at: '2023-10-19T16:03:27.456958896Z'
scheduled_at:
\"
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:84:in `rescue in payload_object'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:81:in `payload_object'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/backend/base.rb:133:in `max_run_time'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:194:in `max_run_time'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:137:in `block in run'
/usr/local/lib/ruby/3.2.0/benchmark.rb:311:in `realtime'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:136:in `run'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:210:in `block in run_job'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:210:in `run_job'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:102:in `block (4 levels) in work_off'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/plugins/connection.rb:7:in `block (3 levels) in <class:Connection>'
/usr/local/bundle/ruby/3.2.0/gems/activerecord-7.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:229:in `with_connection'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/plugins/connection.rb:6:in `block (2 levels) in <class:Connection>'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:79:in `block in add'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:121:in `run_thread_callbacks'
/usr/local/bundle/ruby/3.2.0/gems/delayed-0.5.1/lib/delayed/worker.rb:101:in `block (3 levels) in work_off'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/local/bundle/ruby/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
# frozen_string_literal: true
class TestJob < ApplicationJob
queue_as :default
def perform
end
end
delayed_jobs
tableHello again!
Thanks again for this awesome library. I've ended up with one more question as I dig through our code to evaluate switching.
In what ways have you changed either the schema of the delayed_jobs
table, or the assumptions surrounding how it's used?
We have a number of places in our code which--whether or not this is something that should have been done--directly query the table.
Generally these do one of a few things:
last_error IS (NOT) NULL
for monitoring errorslocked_at IS (NOT) NULL
for monitoring current queue usagehandler LIKE '%SomeJobClass%'
to check for the number of instances of a particular job in the queue. This most often seems to be used to prevent enqueueing too many instances of expensive/side-effectful jobs at once.handler
column to get the name of each job in the queue, again for a reporting page.Would you expect any of those queries, or queries against the table in general, to break? Does Delayed provide new/better ways to do any of these things? Are there any DB migrations that need to be run to switch libraries?
Hi! We have migrated DelayedJob
to Delayed
and are running it.
While observing the worker, we came across an error and found a difference in behavior.
Here is the stack trace in the last_error
column.
Job failed to load: undefined class/module Sample. Handler: "--- !ruby/object:Delayed::JobWrapper
job_data:
job_class: Sample
job_id: 0f9f613a-682b-439c-8169-d45a4984e8eb
provider_job_id:
queue_name: v2
priority:
arguments: &2 []
executions: 0
exception_executions: &1 {}
locale: en
timezone: UTC
enqueued_at: '2023-01-20T11:51:44Z'
job: !ruby/object:Sample
arguments: []
job_id: 0f9f613a-682b-439c-8169-d45a4984e8eb
queue_name: v2
priority:
executions: 0
exception_executions: *1
timezone: UTC
provider_job_id:
serialized_arguments: *2
locale: en
enqueued_at: '2023-01-20T11:51:44Z'
"
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/backend/base.rb:84:inrescue in payload_object' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/backend/base.rb:81:in
payload_object'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/backend/base.rb:133:inmax_run_time' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:194:in
max_run_time'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:137:inblock in run' /usr/local/lib/ruby/3.0.0/benchmark.rb:308:in
realtime'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:136:inrun' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:210:in
block in run_job'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:61:inblock in initialize' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:66:in
execute'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:40:inrun_callbacks' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:210:in
run_job'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:102:inblock (4 levels) in work_off' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/plugins/connection.rb:7:in
block (3 levels) in class:Connection'
/usr/local/bundle/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:inwith_connection' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/plugins/connection.rb:6:in
block (2 levels) in class:Connection'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:79:inblock (2 levels) in add' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:61:in
block in initialize'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:79:inblock in add' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:66:in
execute'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/lifecycle.rb:40:inrun_callbacks' /usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:121:in
run_thread_callbacks'
/usr/local/bundle/gems/delayed-0.4.0/lib/delayed/worker.rb:101:inblock (3 levels) in work_off' /usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in
run_task'
/usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:inblock (3 levels) in create_worker' /usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in
loop'
/usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:inblock (2 levels) in create_worker' /usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in
catch'
/usr/local/bundle/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
This error occurs when a job is enqueued even though the class definition of the job does not exist in the worker.
(For example, this can occur when adding a new job definition and trying to synchronize the webapp, worker Deployment with ArgoCD.)
I noticed that some kind of error occurs when executing a job, whether it is a DelayedJob
or Delayed
, but the difference is that Delayed
does not retry and the failed_at column is updated.
The situation of enqueuing a non-existent job is not good, but we expect the worker to retry and complete the job execution once the code is up-to-date.
I am assuming that this difference in behavior exists because of the difference in serialization and deserialization of jobs.
In the case of Delayed
, the job
key is added to the serialization string of the job, and an error seems to occur when deserializing it.
DeserializationError
is handled so that it is not retried.
(In DelayedJob
, there is no job
key and no error occurs during deserialization.)
--- !ruby/object:Delayed::JobWrapper
job_data:
job_class: Sample
job_id: c5d50b73-c8c9-4df0-8ea3-d2f6ca45b85d
provider_job_id:
queue_name: v2
priority:
arguments: &2 []
executions: 0
exception_executions: &1 {}
locale: en
timezone: UTC
enqueued_at: '2023-01-20T04:17:09Z'
job: !ruby/object:Sample
arguments: []
job_id: c5d50b73-c8c9-4df0-8ea3-d2f6ca45b85d
queue_name: v2
priority:
executions: 0
exception_executions: *1
timezone: UTC
provider_job_id:
serialized_arguments: *2
locale: en
enqueued_at: '2023-01-20T04:17:09Z'
I assume that the job can be executed even if this job key does not exist.
Is there any intention behind this difference in specifications?
I have been testing delayed for a few days, seems rock solid!
I like the philosophy on sticking with ActiveJob and ActiveRecord database being used.
Got a quick question on the number of workers we can run safely at the same time?
I want to utilize all the cores on a big server. I have kicked 4 workers with 5 max_claims and
have not experienced any issues so far. Would there be any adverse effects starting up to 10 workers?
Thank you.
Hi there, I've raised several PRs to DelayedJob which aren't getting merged :(
I'm wondering if the owners of this repo are open to me porting some of the work here:
Let me know and I'll start raising PRs soon.
In the case of many jobs getting enqueued at about the same time, they might overwhelm the system, fail and be retried at some later point.
However if there's so many and they all get retried at the same (exponential backoff) interval, they might still end up causing trouble.
Adding a(n option to add a) random jitter to the retry interval might distribute the load a bit better and resolve the retry stampede.
Ruby 3 changed the way it handles keyword arguments, and is a roadblock with delayed_job. I checked the relevant bits in the source of this gem (at least the bits that used to be relevant in delayed_job) and didn't see any changes or fixes for Ruby 3 keyword argument handling.
delayed/lib/delayed/message_sending.rb
Lines 11 to 13 in 2d296b9
delayed/lib/delayed/performable_method.rb
Lines 25 to 27 in 2d296b9
While the gemspec suggests Ruby 3 should work, does it really?
Hello! Thanks for taking up the torch and improving DelayedJob with all the goodies. I'm particularly grateful that this supports Ruby3 kwargs, whereas DelayedJob said they won't for some reason.
We are in the midst of migrating from DJ to Delayed and have found a sticking point with respect to our handling of "Duplicate Jobs". Duplicate jobs are separate jobs created by different parts of the system that do the same thing. For instance, let's say I have an expensive calculation that updates any time a certain frequent activity occurs in our app. So, different actions can result in the same job being created. While the job itself is idempotent, I don't want to fill my job queue with "duplicates" of the same job.
A long time ago, I had created a gist to handle this and it was recently turned into a gem by some other devs, which validates the need for Duplicate checking.
The way this gem works is that it expands the ActiveRecord model to have a "signature" column which can be indexed. It will try to infer a signature for a given job or it can be explicitly defined with custom logic for each individual job. The signature is then utilized by a "standard" DJ plugin.
Now that we are migrating to Delayed gem, I'd like to build support for this functionality into Delayed. The issue I found is that the Delayed gem seems to only include the Delayed::Job
class (the ActiveRecord) when Rails::Engine is not defined:
Lines 18 to 23 in 66125d1
This prevents the gem from hooking into the ActiveRecord model without some fancy require logic. Is there a reason for this? Or is there another way to hook into the ActiveRecord class to extend functionality?
I see the above code was created via this commit and says that Delayed::Job
should autoload. However, it's not present during Duplicate checking gem code. I can see that the load paths doesn't have /app/models at the time of the next gem, but does at the time my Rails console loads. So, I'm wondering if/how its possible to get the load path updated earlier or before this next gem loads
Thanks!
When i run 'rake delayed:work' to start the worker process, i am getting this error.
D, [2022-10-07T17:38:17.245081 #20659] DEBUG -- : 2022-10-07T17:38:17+0530: [Worker(host:Vs-MacBook-Pro.local pid:20659)] Starting Delayed::Worker
D, [2022-10-07T17:38:17.372640 #20659] DEBUG -- : Delayed::Job Load (0.5ms) UPDATE "delayed_jobs" SET locked_at = '2022-10-07 12:08:17.367212', locked_by = 'host:Vs-MacBook-Pro.local pid:20659' WHERE ctid = ANY (ARRAY (SELECT ctid FROM "delayed_jobs" WHERE (((run_at <= '2022-10-07 12:08:17.340440' AND (locked_at IS NULL OR locked_at < '2022-10-07 12:02:47.340461')) OR locked_by = 'host:Vs-MacBook-Pro.local pid:20659') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5 FOR UPDATE SKIP LOCKED)) RETURNING *
D, [2022-10-07T17:38:17.373088 #20659] DEBUG -- : 2022-10-07T17:38:17+0530: [Worker(host:Vs-MacBook-Pro.local pid:20659)] Error while reserving job(s): PG::SyntaxError: ERROR: syntax error at or near "SKIP"
LINE 1: ...ER BY priority ASC, run_at ASC LIMIT 5 FOR UPDATE SKIP LOCKE...
My postgresql version is :
$ postgres -V
postgres (PostgreSQL) 14.3
Anyone have any idea why i am getting this error ?
^
Hi! This project looks very exciting for teams looking to migrate away from delayed_job
. However, I find myself having some lingering questions about the idempotency requirement of this gem:
delayed
not be used with these jobs?after_perform
would there still be a race condition?)Massive thanks to the team at Betterment for this project!
Our jobs are remaining locked after the worker process receives the SIGKILL
(in our case from docker stop after the grace period elapses).
I'm not sure if this is by design or not, the README states (may being the keyword):
the process and may result in long-running jobs remaining locked until Delayed::Worker.max_run_time has elapsed.
In my tests they always remains locked when a job is running and Delayed also doesn't terminate with a SIGTERM
.
class TestWorker
def start
trap('TERM') { quit! }
trap('INT') { quit! }
500.times do |i|
puts "Run #{i}"
sleep 1.second
end
ensure
on_exit!
end
def quit!
puts 'quit!'
end
def on_exit!
puts 'on_exit!'
end
end
namespace :test do
desc "Test signal interupts"
task work: :environment do
TestWorker.new.start
end
end
If you run that as as a rake task, it won't terminate on a SIGTERM
, only SIGKILL
which won't execute the ensure
block which in Delayed is what unlocks the jobs.
class TestWorker
def start
trap('TERM') { quit! }
trap('INT') { quit! }
500.times do |i|
puts "Run #{i}"
sleep 1.second
end
ensure
on_exit!
end
def quit!
puts 'quit!'
exit
end
def on_exit!
puts 'on_exit!'
end
end
namespace :test do
desc "Test signal interupts"
task work: :environment do
TestWorker.new.start
end
end
What's the recommended way to gracefully shut down a running job when the process is terminated? Should a job implement its own traps or is there a hook that Delayed offers?
I'm trying to configure basic exception notification, but can't seem to get the event to trigger. It triggers if I subscribe to "delayed.job.error", just not "delayed.job.failure". I've confirmed that the job actually fails according to the logs, does anyone see anything I may be doing wrong here?
ActiveSupport::Notifications.subscribe("delayed.job.failure") do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
job = event.job
ExceptionNotifier.notify_exception(job.last_error, data: { handler: job.handler })
end
I'm thinking of porting https://github.com/codez/delayed_cron_job to delayed, should I make a separate gem or is there any change a PR would be merged?
I am looking for a Sidekiq alternative and from what I can tell this is quite similar to good_job? - Could you please comment where you see the differences/advantages?
The lack of a dashboard in delayed is currently the reason that is making me hesitant, is that something on the roadmap or is there an immediate solution for that?
Any chance the private dashboard fork will be open-sourced?
If not, any good gems we can rely on for a decent UI similar to sidekiq?
Hi and thank you for open sourcing this job framework!
I'm just starting to try it out but it looks quite promising. My mine motivator is the SQLite support.
One feature I'm missing (or haven't found yet) are periodic aka cron jobs. I have many use cases for some daily/weekly etc sync or update jobs.
Not sure if you think that should be part of delayed
or are you maybe combining delayed with some kind of scheduler that runs alongside and simply queues jobs when their time has come? If so, any recommendations?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.