When displaying the UUID in the UI we should truncate to the first 8 chars. Two considerations:

• The UI should accept short UUIDs
• We should have a way to detect a duplicate truncation (pretty unlikely, but possible).

When starting nomad with the -config option and passing a path to a directory, Nomad fails and throws an error. Passing a path to a file works fine.

Error

$ nomad agent -config="/etc/nomad.d/"
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x8 pc=0x49120d]

goroutine 1 [running]:
github.com/hashicorp/nomad/command/agent.(*Config).Merge(0xc820077b00, 0x0, 0x0)
    /home/ubuntu/go/src/github.com/hashicorp/nomad/command/agent/config.go:251 +0x6d
github.com/hashicorp/nomad/command/agent.(*Command).readConfig(0xc820087a40, 0xb09e00)
    /home/ubuntu/go/src/github.com/hashicorp/nomad/command/agent/command.go:109 +0xf09
...

Successful

$ nomad agent -config="/etc/nomad.d/nomad.conf"
==> Starting Nomad agent...
==> Error starting agent: must have at least client or server mode enabled
ubuntu@ip-172-31-10-48:~$

Full error log and successful log at https://gist.github.com/clstokes/4cb766a9209dda8fff65.

The demo Java app is currently in Dropbox. We should put this in the git repo and serve via http test server or similar.

Most deployments will have only a single region and "global" sounds better than "region1"

Looks like make test assumes it's running on AWS with access to the docker daemon socket. Maybe the README should be updated to reflect environment requirements for development.

2015/09/27 18:29:05 [INFO] raft: Node at 127.0.0.1:17017 [Leader] entering Leader state
2015/09/27 18:29:05 [INFO] raft: Disabling EnableSingleNode (bootstrap)
2015/09/27 18:29:05 [INFO] serf: EventMemberJoin: Node 17017.global 127.0.0.1
2015/09/27 18:29:05 [DEBUG] raft: Node 127.0.0.1:17017 updated peer set (2): [127.0.0.1:17017]
2015/09/27 18:29:05 [INFO] nomad: starting 4 scheduling worker(s) for [service batch _core]
2015/09/27 18:29:05 [INFO] nomad: cluster leadership acquired
2015/09/27 18:29:05 [INFO] client: using alloc directory /var/folders/ds/jbxmdvmx0rj1z7vc4br6j97h0000gn/T/NomadClient631364695
2015/09/27 18:29:05 [INFO] nomad: adding server Node 17017.global (Addr: 127.0.0.1:17017) (DC: dc1)
2015/09/27 18:29:05 [WARN] fingerprint.network: Ethtool not found, checking /sys/net speed file
2015/09/27 18:29:05 [WARN] fingerprint.network: Error getting information about net speed
2015/09/27 18:29:05 [ERR] fingerprint.env_aws: Error querying AWS Metadata URL, skipping
2015/09/27 18:29:05 [DEBUG] client: applied fingerprints [arch cpu host memory storage network]
2015/09/27 18:29:05 [DEBUG] driver.java: must run as root user, disabling
2015/09/27 18:29:05 [DEBUG] driver.qemu: must run as root user, disabling
--- FAIL: TestAgent_RPCPing (0.04s)
    agent_test.go:67: err: client setup failed: driver setup failed: Get http://unix.sock/version: dial unix /var/run/docker.sock: connect: no such file or directory

The sidebar in the docs (and probably other) sections is a bit too tight. We end up line-breaking just about everything we put there. Can we make it wider? /cc @captainill

We have the ability to create "service" and "batch" jobs, but it'd be really nice to have an additional "cron" type. In this type, a task would be created whenever the cron triggers and once the task is complete (be it the process exits gracefully or not), the task is not recreated until the next cron trigger.

The type would not allocate a new task if the number of tasks are already at the maximum count for the group. This might be a nice configuration item to control how this works; performing a no-op, replacement or forced addition.

Assuming we think this a good idea, I'd be happy to jump into it.

Right now we use something like execContext.AllocDir but we actually want to use task.AllocDir

I am not sure if this is within the scope of the project, but I think a use-case like this is pretty common place.

I would like to run something like hdfs on a nomad cluster. Mesos uses schedulers and executors for their frameworks. They have a hdfs framework here: https://github.com/mesosphere/hdfs

Basically I would like to be able to do the following:

• Collocate a namenode, journalnode and zookeeper failover controller on 1 node in the cluster.
• Collocate a standby namenode, journalnode and zookeeper failover controller on another node in the cluster.
• Run a journalnode on another node in the cluster.
• Run datanodes on all other nodes in the cluster.
• Easily scale up and down the number of namenodes, journalnodes and datanodes (along with their collocated components).

There doesn't really appear to be a way to do this in a job specification at the moment, but if we can do something like this, it would be really awesome!

I think this is a good time to refactor this call to use the API instead of CLI. The CLI doesn't give me the data I want here.

POSIX conventions give us three means to stop a program:

1. SIGINT is actually a keyboard interrupt (Ctrl-C). Many traditional daemons (apache, mysql) ignore this.
2. SIGTERM means please cleanup and shutdown immediately. Apache, MySQL obey this as the termination command.
3. SIGKILL Aaaand you're done. In the application is stuck or you want to crash instead of waiting for cleanup, you can kill the process outright.

I think we should TERM and follow with KILL (vs. INT followed by KILL that we do currently).

These will all be uppercased for use in environment variables. As such they should be normalized and de-duped. We should do the same thing with meta tags.

Could you guys please fill in the docs for the task drivers as you are more familiar.
Feel free to copy/paste from exec for consistency.

• Exec
• Java
• Docker
• Qemu

/cc: @cbednarski @catsby @dadgar

Since some of the agent-* commands only pertain to servers, it would be more obvious if they were named accordingly.

agent
agent-info
agent-force-leave -> server-force-leave
agent-join        -> server-join
agent-members     -> server-members
node-drain
node-status
run
status
stop

Thoughts?

Similar in theme to #60 and #61, we want to share data between tasks in a group. We will need to mount the task group allocation directory as a docker volume so we can share data between these processes.

We have some spots (maybe even just docker) where we panic if the provided task resources is a typed nil. We should either handle this uniformly before invoking a specific driver, or remove the panic.

Example: https://github.com/hashicorp/nomad/blob/master/client/driver/docker.go#L65-L67

Client agents do not use Serf, and instead are provided a list of server hostnames or IPs. As a result, we need some way to query and manipulate that.

When running a task we would like to share data between other tasks in a task group. Also, if using something like ceph or EBS we would like to support mounting a volume into the executor's context (chroot, cgroup, etc.) so it can write persistent data.

Probably depends on #60, so mounts are mounted into a tasks's environment and have some level of isolation from other tasks on the system.

Ports are dynamically allocated in a range between 20000 and 60000. We should scan and detect ports in this range that are already bound so we don't collide with other services. The high port range is likely safe, but we should check.

Also, we should check for low ports in case a service tries to use a reserved port in that range.

Docker currently supports dynamic ports but not static ports

When nomad starts it should mark its own ports as reserved to prevent tasks from trying to use that port number.

It'd be fantastic if Nomad supported CoreOS's rkt container engine in addition to Docker.

Nomad itself will need root to access things like network resources and cgroups. However, processes that are started by nomad should be forked as nobody or other non-privileged access by default.

After stopping nomad for about a minute or two and attempting to restart it, I get the following error:

nomad agent -config /etc/nomad.hcl 

==> Starting Nomad agent...
==> Error starting agent: client setup failed: failed to restore state: 1 error(s) occurred:

* 1 error(s) occurred:

* Failed to find container ed401155afbf0686f372be304a23eae6b3b57555730ab42df34f04abe746dfa3: <nil>

Maybe nomad should start regardless of missing containers and just log an error.

I tried starting nomad on both CoreOS (local and GCE) and Debian (GCE) and running a cluster (both with -dev and with client/server. For installation, I follow the steps described in the Vagrantfile.

As soon as I try to run:

nomad init
nomad run example.nomad

I get a message that my resources are exhausted:

username@instance-2:~$ nomad run example.nomad
==> Monitoring evaluation "bac37878-9a39-bd3e-5ef2-acc51cde7981"
    Evaluation triggered by job "example"
    Scheduling error for group "cache" (failed to find a node for placement)
    Allocation "2263a54c-ea0e-ab68-adee-30f8d2e0827a" status "failed" (0/1 nodes filtered)
      * Resources exhausted on 1 nodes
      * Dimension "network: bandwidth exceeded" exhausted on 1 nodes
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "bac37878-9a39-bd3e-5ef2-acc51cde7981" finished with status "complete"

Or this variant:

core@core-01 ~ $ ./nomad run example.nomad 
==> Monitoring evaluation "ee52b439-49e1-f3e4-a3d2-0ad6a1fc48d2"
    Evaluation triggered by job "example"
    Scheduling error for group "cache" (failed to find a node for placement)
    Allocation "c23ca37e-2de5-b10d-d90a-d36bc468092c" status "failed" (0/1 nodes filtered)
      * Resources exhausted on 1 nodes
      * Dimension "network: no networks available" exhausted on 1 nodes
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "ee52b439-49e1-f3e4-a3d2-0ad6a1fc48d2" finished with status "complete"

This is the first job I am scheduling so it is unlikely that the network resources are actually exhausted. Am I missing some kind of configuration telling nomad about the network?

There are a few places in the docs or CLI usage where it would be nice to reference a comprehensive bootstrapping guide, like Consul has, rather than re-documenting bootstrap a couple of times.

When a driver or fingerprinted runs it should mark any related ports as reserved. For example, we should prevent a task from trying to allocate the same ports used by docker or consul, if we detect those on the nomad host.

It would be nice to remove docker images only after they are unused after X time. This allows minor gaps in time where the image isn't in use, but still cleans up stale images that aren't needed.

Chroot gives us disk isolation and also provides a foundation to mount shared data and persistent volumes.

@ryanuber Many of the "list" APIs use a Stub response instead of the full object. For example, job list returns the JobListStub instead of a list of Job structs. We are decoding into the full struct, which is fine as its a super set. I just wonder if its confusing since many fields will be blank. It may be more clear to also have ListStub versions in the API. Up for discussion, I'm not sure either way.

@armon documenting this here, but it's the same bug we were seeing earlier today.

We are seeing duplicate allocations when a job is rescheduled. For example, register a simple job with 1 task. You get 1 allocation. Then schedule the same thing again. You get 2 more new allocations. Scheduling yet again yields 4 new allocations.

Steps to reproduce:

$ cat ~/test.job 
job "job1" {
    region = "region1"
    datacenters = ["dc1"]
    type = "batch"
    group "group1" {
        count = 1
        task "task1" {
            driver = "exec"

            resources {
                cpu = 1
                memory = 1
            }

            config {
                command = "yes"
                args = "hello world"
            }
        }
    }
}

$ nomad run ~/test.job 
JobID  = job1
EvalID = 7c675f82-2493-f3c1-7adf-32cf5d699ee8

$ curl localhost:4646/v1/evaluation/7c675f82-2493-f3c1-7adf-32cf5d699ee8/allocations?pretty
[
    {
        "ID": "effb8e12-faaa-01d3-0e9b-75c5a290b3e1",
        "EvalID": "7c675f82-2493-f3c1-7adf-32cf5d699ee8",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 14,
        "ModifyIndex": 18
    }
]

$ nomad run ~/test.job 
JobID  = job1
EvalID = 56156e2a-99e1-11de-0d5f-efa448a442dd

$ curl localhost:4646/v1/evaluation/56156e2a-99e1-11de-0d5f-efa448a442dd/allocations?pretty
[
    {
        "ID": "af746b79-d35a-3c1f-e17d-d73221a20e60",
        "EvalID": "56156e2a-99e1-11de-0d5f-efa448a442dd",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 23,
        "ModifyIndex": 27
    },
    {
        "ID": "c9a1dcbc-a259-ad43-9645-49b9d110123e",
        "EvalID": "56156e2a-99e1-11de-0d5f-efa448a442dd",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 23,
        "ModifyIndex": 26
    }
]

$ nomad run ~/test.job 
JobID  = job1
EvalID = 2d64a70b-ef9a-51f1-99ab-a558b6f9e122

$ curl localhost:4646/v1/evaluation/2d64a70b-ef9a-51f1-99ab-a558b6f9e122/allocations?pretty
[
    {
        "ID": "59ccc408-c3f5-7785-ef4f-1fbb06e05f0b",
        "EvalID": "2d64a70b-ef9a-51f1-99ab-a558b6f9e122",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 35,
        "ModifyIndex": 40
    },
    {
        "ID": "6b49c24b-ac95-6028-9765-e5d10ac176df",
        "EvalID": "2d64a70b-ef9a-51f1-99ab-a558b6f9e122",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 35,
        "ModifyIndex": 42
    },
    {
        "ID": "c5b7b880-e044-4dc8-9ac1-40fdc8ead470",
        "EvalID": "2d64a70b-ef9a-51f1-99ab-a558b6f9e122",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 35,
        "ModifyIndex": 41
    },
    {
        "ID": "f783b3c7-8df0-1da1-fe71-90fedbb654f4",
        "EvalID": "2d64a70b-ef9a-51f1-99ab-a558b6f9e122",
        "Name": "job1.group1[0]",
        "NodeID": "3202242b-f871-d578-59a3-5e044d3780aa",
        "JobID": "job1",
        "TaskGroup": "group1",
        "DesiredStatus": "run",
        "DesiredDescription": "",
        "ClientStatus": "running",
        "ClientDescription": "{\"task1\":{\"Status\":\"running\",\"Description\":\"task started\"}}",
        "CreateIndex": 35,
        "ModifyIndex": 39
    }
]

I think this is also the reason behind the multiple "Allocation xxx created on node yyy" log messages. Looking at the CreateIndex for each allocation associated with a given evaluation, they are all always the same, and always greater than the evaluation's CreateIndex, which makes me think we might just be dup'ing something on the server side each time.

In my attempt to build nomad I had to update the following deps:

github.com/fsouza/go-dockerclient
github.com/mitchellh/mapstructure

Might be better to manage this with something like Godeps.

We need a common utility for fetching source artifacts over the internet. It should at least:

• support retries
• some authentication support

Possibly an interface that has more concrete implementations, like SSHFetcher or HTTPSFetcher, etc.

We should have a "fingerprint" that just reports our build information as a node attribute.

Please made sure you add yourselves to community page.
/cc: @cbednarski @catsby @dadgar @ryanuber

I run nomad run example.nomad and I see the eval all worked and is now complete.

vagrant@nomad:/opt/gopath/src/github.com/hashicorp/nomad$ nm run example.nomad
==> Monitoring evaluation "ac974418-715f-4349-9807-cc5c213697a3"
    Evaluation triggered by job "example"
    Allocation "c3d0fc1e-9b17-2ec3-4ba8-89270a7231d1" created: node "f89dcb8c-f0f3-68c9-8cfd-04ef1f09def9", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "ac974418-715f-4349-9807-cc5c213697a3" finished with status "complete"

However, I do not see if my thing is actually running. In my example, it's not..

vagrant@nomad:/opt/gopath/src/github.com/hashicorp/nomad$ nm status example
ID          = example
Name        = example
Type        = service
Priority    = 50
Datacenters = dc1
Status      =

==> Evaluations
ID                                    Priority  TriggeredBy   Statusac974418-715f-4349-9807-cc5c213697a3  50        job-register  complete

==> Allocations
ID                                    EvalID                                NodeID                                TaskGroup  Desired  Status
c3d0fc1e-9b17-2ec3-4ba8-89270a7231d1  ac974418-715f-4349-9807-cc5c213697a3  f89dcb8c-f0f3-68c9-8cfd-04ef1f09def9  cache      run      failed
vagrant@nomad:/opt/gopath/src/github.com/hashicorp/nomad$

For operating or debugging, it would be helpful if I could run a command on a node and get some information on the results of the Fingerprints, both which ones ran, and what they recorded. This would be helpful for understanding why a job isn't placing where I thought it would, or debugging a Fingerprint.

Ex:

$ nomad node-status -fingerprint
Fingerprints ran:
- Java
  - version: 1.7...
- Docker
- CPU:
  - cores: 24
  - mhz: 9001
- Blah..
- Blah

Periodic failure (probably a race condition) while running tests on nomad/nomad; seen on linux with 2 cpus, running make test. Log below.

/cc @armon

On Linux we can use cgroups to constrain resource usage. There is initial support and documentation for this feature in the Docker driver, but we should extend it to other drivers.

Questions:

• Is there a good cgroups library for go? If not we can use the cg* tools but will need to note that they are an external dependency that must be installed on the system in advance.

Docs:

• Linux
• Darwin
• BSD
• Windows

/cc @catsby

When building #63 on OS X, I get the following errors:

echo "--> Building..."
--> Building...
gox \
        -osarch="darwin/amd64 freebsd/amd64 linux/amd64 linux/arm windows/amd64" \
        -output="bin/nomad_{{.OS}}_{{.Arch}}" \
        -verbose \
        .
Number of parallel builds: 8

-->   freebsd/amd64: github.com/hashicorp/nomad
-->    darwin/amd64: github.com/hashicorp/nomad
-->       linux/arm: github.com/hashicorp/nomad
-->     linux/amd64: github.com/hashicorp/nomad
-->   windows/amd64: github.com/hashicorp/nomad

3 errors occurred:
--> freebsd/amd64 error: exit status 2
Stderr: # github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrus
../../fsouza/go-dockerclient/external/github.com/Sirupsen/logrus/terminal_freebsd.go:10: ioctlReadTermios redeclared in this block
    previous declaration at ../../fsouza/go-dockerclient/external/github.com/Sirupsen/logrus/terminal_bsd.go:7
../../fsouza/go-dockerclient/external/github.com/Sirupsen/logrus/terminal_freebsd.go:12: Termios redeclared in this block
    previous declaration at ../../fsouza/go-dockerclient/external/github.com/Sirupsen/logrus/terminal_bsd.go:9

--> linux/arm error: exit status 2
Stderr: # github.com/shirou/gopsutil/host
../../shirou/gopsutil/host/host_linux.go:110: cannot use u.User[:] (type []byte) as type []int8 in argument to common.IntToString
../../shirou/gopsutil/host/host_linux.go:111: cannot use u.Line[:] (type []byte) as type []int8 in argument to common.IntToString
../../shirou/gopsutil/host/host_linux.go:112: cannot use u.Host[:] (type []byte) as type []int8 in argument to common.IntToString

--> windows/amd64 error: exit status 2
Stderr: # github.com/hashicorp/nomad/client/executor
client/executor/exec.go:125: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)
client/executor/exec.go:126: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)
client/executor/exec.go:126: undefined: syscall.Credential
client/executor/exec.go:128: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)
client/executor/exec.go:141: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)
client/executor/exec.go:142: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)
client/executor/exec.go:142: undefined: syscall.Credential
client/executor/exec.go:144: c.Cmd.SysProcAttr.Credential undefined (type *syscall.SysProcAttr has no field or method Credential)

make: *** [xc] Error 1

Suggested by @cbednarski in #33: It could be helpful to an operator if we had a debug-level log to show the names of the config files used when starting the agent, especially in cases where a config dir is used.

There are two sets of ports we are concerned with, those internal to the VM and those external which are running on the host. The internal ports are specific to the machine and application and we don't know what they are. Likely they are hard coded based on the application.

The external ports may either be statically reserved ("always give me 8080") or dynamically assigned. The task driver should take a list of internal ports to map to, and then use the list of ReservedPorts as provided by the scheduler and do the 1:1 mapping.

For example:

task "web" {
   driver = "qemu"
   config {
      image = "foo"

      # SSH on port 22 always, web app on 8000
      ports = [22, 8000]
   }
   resources {
       ....
       network {
           dynamic_ports = ["ssh", "web"] # Dynamically assign 2 ports from the host
       }
   }
}

/cc: @catsby

While testing the example redis job, I stopped the agent and the job was marked dead. When I restarted the agent, the job did not get rescheduled. This maybe by design, but it was unexpected, and I assumed since the job was a service nomad would just reschedule the job to an health node.

In kubernetes, there is the idea of services. Essentially, it assigns a virtual ip (from a range) to each task launched in the cluster.

This means, that if we launch task, we do not have to ask nomad to dynamically assign a port, or force it to assign a port and hope it exists. Since each task has a virtual ip, we can access all http services on 80, https on 443, etc.

#14 introduced a mock server for HTTP requests. It should be extracted into a generic thing that wraps net.httptestserver, and allows a given test library to append "endpoints" to it:

s := nomadTesting.NewTestHTTPServer()
s.AddEndpoint(&nomadTesting.HTTPTestEndpoint{
  Endpoint: "/v1/some/endpoint",
  ContentType: "text/json",
  Body: <some body>,
})

Or something like that...

@armon Right now we have mirrored structs in the API the same way we do in other projects. I think the point of this was to avoid the extra import of the structs package everywhere. Can we just import the structs package in the API, and use structs.Job and the stubs directly? Or are there other reasons I'm not thinking of?

Motivation is #46. We are going to need this signature to take a *structs.Job instead of *api.Job.

Right now processes have 5 seconds to clean up before a force kill.
We should explore ways to make this configurable, depending on the job(s)

Backstory:

• #15 (comment)

The Getting Started guide references a vagrant file to download,we need to post it and update the link:

• http://localhost:4567/intro/getting-started/install.html

@armon seeing this during some tests: https://gist.github.com/ryanuber/c358e95f2e3e468f1210

After tracing some code, it looks like it stems from this: https://github.com/hashicorp/nomad/blob/master/scheduler/generic_sched.go#L106 if Process() is called on an eval with its initial TriggeredBy != "job-register" && TriggeredBy != "node-update".