nunchistudio / hashibox Goto Github PK

View Code? Open in Web Editor NEW

81.0 5.0 17.0 12.49 MB

Simulate a highly-available Cloud Platform with Consul, Nomad, and Vault. OSS & Enterprise versions supported.

Home Page: https://nunchi.studio/hashibox

License: MIT License

Makefile 10.04% Shell 63.12% Puppet 5.37% HCL 21.47%

hashicorp consul vault nomad vagrant docker bolt hashistack localstack puppet-bolt

hashibox's People

Contributors

Stargazers

Watchers

Forkers

ahosam marcpires rchenzheng guybarros dhung-hashicorp powhapki wealthsimple bogdandragosvasile accesaacademy heartshare andykinghashicorp atomicjets alluvium-cloud dennislwm mvandermeulen tracphil mithril52

hashibox's Issues

Parallels: Consul clients encounter connection error

What OS are you using?

macOS

What architecture are you using?

arm64

What Vagrant provider are you using?

parallels

Unexpected behavior

When using Parallels Desktop, Consul clients can connect to Consul servers but a connection error occurs. This prevents both Consul and Nomad client services to be registered in Consul even though they run as expected.

What kind of specifics do we need to set in Parallels Desktop to make this work as expected?
This doesn't happen when using VirtualBox.

Related issues:

hashicorp/consul#5195

Steps to reproduce

Run HashiBox with Parallels Desktop.

Expected behavior

No connection errors. Both Consul and Nomad client services should be registered in Consul.

Logs

-- Logs begin at Sat 2022-06-25 10:41:00 PDT, end at Sat 2022-06-25 10:47:55 PDT. --
Jun 25 10:42:53 node-client-1 systemd[1]: Starting "HashiCorp Consul"...
Jun 25 10:42:53 node-client-1 consul[22223]: ==> Starting Consul agent...
Jun 25 10:42:53 node-client-1 consul[22223]:            Version: '1.12.0'
Jun 25 10:42:53 node-client-1 consul[22223]:            Node ID: '94d27eb5-17a4-a047-33e0-8adfde2750d2'
Jun 25 10:42:53 node-client-1 consul[22223]:          Node name: 'node-client-1'
Jun 25 10:42:53 node-client-1 consul[22223]:         Datacenter: 'us' (Segment: '')
Jun 25 10:42:53 node-client-1 consul[22223]:             Server: false (Bootstrap: false)
Jun 25 10:42:53 node-client-1 consul[22223]:        Client Addr: [192.168.61.10] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Jun 25 10:42:53 node-client-1 consul[22223]:       Cluster Addr: 192.168.61.10 (LAN: 8301, WAN: 8302)
Jun 25 10:42:53 node-client-1 consul[22223]:            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Jun 25 10:42:53 node-client-1 consul[22223]: ==> Log data will now stream in as it occurs:
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.373-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-client-1 192.168.61.10
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.373-0700 [INFO]  agent.router: Initializing LAN area manager
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.373-0700 [INFO]  agent: Started DNS server: address=192.168.61.10:8600 network=udp
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.373-0700 [INFO]  agent: Started DNS server: address=192.168.61.10:8600 network=tcp
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: Starting server: address=192.168.61.10:8500 network=tcp protocol=http
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: started state syncer
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: Consul agent running!
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: Joining cluster...: cluster=LAN
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [INFO]  agent: (LAN) joining: lan_addresses=[192.168.60.10, 192.168.60.20, 192.168.60.30]
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [WARN]  agent.router.manager: No servers available
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.374-0700 [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.375-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-server-1 192.168.60.10
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.375-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-client-3 192.168.61.30
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.375-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-client-2 192.168.61.20
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.375-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-server-3 192.168.60.30
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.376-0700 [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node-server-2 192.168.60.20
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.376-0700 [INFO]  agent.client: adding server: server="node-server-1 (Addr: tcp/192.168.60.10:8300) (DC: us)"
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.376-0700 [INFO]  agent.client: adding server: server="node-server-3 (Addr: tcp/192.168.60.30:8300) (DC: us)"
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.376-0700 [INFO]  agent.client: adding server: server="node-server-2 (Addr: tcp/192.168.60.20:8300) (DC: us)"
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.378-0700 [INFO]  agent: (LAN) joined: number_of_nodes=3
Jun 25 10:42:53 node-client-1 systemd[1]: Started "HashiCorp Consul".
Jun 25 10:42:53 node-client-1 consul[22223]: 2022-06-25T10:42:53.398-0700 [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
Jun 25 10:42:56 node-client-1 consul[22223]: 2022-06-25T10:42:56.378-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:42:57 node-client-1 consul[22223]: 2022-06-25T10:42:57.380-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:42:59 node-client-1 consul[22223]: 2022-06-25T10:42:59.375-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:01 node-client-1 consul[22223]: 2022-06-25T10:43:01.374-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:43:02 node-client-1 consul[22223]: 2022-06-25T10:43:02.375-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:03 node-client-1 consul[22223]: 2022-06-25T10:43:03.378-0700 [WARN]  agent: [core]grpc: addrConn.createTransport failed to connect to {us-192.168.60.30:8300 node-server-3 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.61.10:0->192.168.60.30:8300: i/o timeout". Reconnec>
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.376-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.779-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.NodeServiceList server=192.168.60.10:8300 error="rpc error getting client: failed to get conn: dial tcp 192.168.61.10:0->192.168.60.10:8300: i/o timeout"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.781-0700 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error getting client: failed to get conn: dial tcp 192.168.61.10:0->192.168.60.10:8300: i/o timeout"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.781-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.ListDatacenters server=192.168.60.10:8300 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.781-0700 [INFO]  agent.http: Request cancelled: method=GET url=/v1/catalog/datacenters from=192.168.61.10:33066 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.783-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=192.168.60.10:8300 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.784-0700 [WARN]  agent: Syncing node info failed.: error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:04 node-client-1 consul[22223]: 2022-06-25T10:43:04.784-0700 [ERROR] agent: failed to sync changes: error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:05 node-client-1 consul[22223]: 2022-06-25T10:43:05.377-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:43:07 node-client-1 consul[22223]: 2022-06-25T10:43:07.375-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:09 node-client-1 consul[22223]: 2022-06-25T10:43:09.375-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:43:12 node-client-1 consul[22223]: 2022-06-25T10:43:12.373-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:13 node-client-1 consul[22223]: 2022-06-25T10:43:13.376-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:43:13 node-client-1 consul[22223]: 2022-06-25T10:43:13.377-0700 [WARN]  agent: [core]grpc: addrConn.createTransport failed to connect to {us-192.168.60.20:8300 node-server-2 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.61.10:0->192.168.60.20:8300: i/o timeout". Reconnec>
Jun 25 10:43:13 node-client-1 consul[22223]: 2022-06-25T10:43:13.377-0700 [WARN]  agent: [core]grpc: addrConn.createTransport failed to connect to {us-192.168.60.10:8300 node-server-1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.61.10:0->192.168.60.10:8300: i/o timeout". Reconnec>
Jun 25 10:43:13 node-client-1 consul[22223]: 2022-06-25T10:43:13.545-0700 [WARN]  agent: Check is now critical: check=_nomad-check-2c3e0263793586b31542ff179fddaba87a4a9e0b
Jun 25 10:43:14 node-client-1 consul[22223]: 2022-06-25T10:43:14.379-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.164-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.NodeServiceList server=192.168.60.30:8300 error="rpc error getting client: failed to get conn: dial tcp 192.168.61.10:0->192.168.60.30:8300: i/o timeout"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.165-0700 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error getting client: failed to get conn: dial tcp 192.168.61.10:0->192.168.60.30:8300: i/o timeout"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.165-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=192.168.60.30:8300 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.166-0700 [WARN]  agent: Syncing node info failed.: error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.166-0700 [ERROR] agent: failed to sync changes: error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.166-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.ListDatacenters server=192.168.60.30:8300 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.166-0700 [ERROR] agent.http: Request error: method=GET url=/v1/catalog/datacenters from=192.168.61.10:33110 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.167-0700 [ERROR] agent.client: RPC failed to server: method=Catalog.ListDatacenters server=192.168.60.30:8300 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:15 node-client-1 consul[22223]: 2022-06-25T10:43:15.167-0700 [INFO]  agent.http: Request cancelled: method=GET url=/v1/catalog/datacenters from=192.168.61.10:33104 error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
Jun 25 10:43:16 node-client-1 consul[22223]: 2022-06-25T10:43:16.374-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:17 node-client-1 consul[22223]: 2022-06-25T10:43:17.376-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:43:19 node-client-1 consul[22223]: 2022-06-25T10:43:19.374-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:43:19 node-client-1 consul[22223]: 2022-06-25T10:43:19.424-0700 [INFO]  agent: Newer Consul version available: new_version=1.12.2 current_version=1.12.0
Jun 25 10:43:22 node-client-1 consul[22223]: 2022-06-25T10:43:22.374-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-2 but other probes failed, network may be misconfigured
Jun 25 10:43:23 node-client-1 consul[22223]: 2022-06-25T10:43:23.375-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-1 but other probes failed, network may be misconfigured
Jun 25 10:43:24 node-client-1 consul[22223]: 2022-06-25T10:43:24.377-0700 [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to node-server-3 but other probes failed, network may be misconfigured
Jun 25 10:43:24 node-client-1 consul[22223]: 2022-06-25T10:43:24.382-0700 [WARN]  agent: [core]grpc: addrConn.createTransport failed to connect to {us-192.168.60.30:8300 node-server-3 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.61.10:0->192.168.60.30:8300: i/o timeout". Reconnec>
Jun 25 10:43:25 node-client-1 consul[22223]: 2022-06-25T10:43:25.177-0700 [WARN]  agent: Check is now critical: check=_nomad-check-2c3e0263793586b31542ff179fddaba87a4a9e0b

Constraint "${attr.consul.version} semver >= 1.7.0": 1 nodes excluded by filter

What OS are you running?

Linux

What architecture are you running?

amd64

What Vagrant provider are you running?

parallels

Unexpected behavior

When trying to run the Consul service mesh dashboard example from the official Hashi docs,
I'm getting:

Steps to reproduce

Hashibox on commit: 0d3fc70

run make init
run the dashboard example job from Nomad UI

Expected behavior

No warning should appear

Logs

Apr 02 08:19:20 node-server-1 systemd[1]: Starting "HashiCorp Consul"...
░░ Subject: A start job for unit consul.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit consul.service has begun execution.
░░
░░ The job identifier is 3287.
Apr 02 08:19:20 node-server-1 consul[3692]: ==> Starting Consul agent...
Apr 02 08:19:20 node-server-1 consul[3692]: Version: '1.15.2'
Apr 02 08:19:20 node-server-1 consul[3692]: Build Date: '2023-03-30 17:51:19 +0000 UTC'
Apr 02 08:19:20 node-server-1 consul[3692]: Node ID: 'a2dbd0ce-3500-2a9b-4624-2ddf766082c9'
Apr 02 08:19:20 node-server-1 consul[3692]: Node name: 'node-server-1'
Apr 02 08:19:20 node-server-1 consul[3692]: Datacenter: 'us' (Segment: '')
Apr 02 08:19:20 node-server-1 consul[3692]: Server: true (Bootstrap: false)
Apr 02 08:19:20 node-server-1 consul[3692]: Client Addr: [192.168.60.10] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600)
Apr 02 08:19:20 node-server-1 consul[3692]: Cluster Addr: 192.168.60.10 (LAN: 8301, WAN: 8302)
Apr 02 08:19:20 node-server-1 consul[3692]: Gossip Encryption: false
Apr 02 08:19:20 node-server-1 consul[3692]: Auto-Encrypt-TLS: false
Apr 02 08:19:20 node-server-1 consul[3692]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
Apr 02 08:19:20 node-server-1 consul[3692]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Apr 02 08:19:20 node-server-1 consul[3692]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
Apr 02 08:19:20 node-server-1 consul[3692]: ==> Log data will now stream in as it occurs:
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.941-0500 [WARN] agent: bootstrap_expect > 0: expecting 3 servers
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.949-0500 [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.raft: initial configuration: index=0 servers=[]
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-1.us 192.168.60.10
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-1 192.168.60.10
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.router: Initializing LAN area manager
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.956-0500 [INFO] agent.server.autopilot: reconciliation now disabled
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.957-0500 [INFO] agent.server.raft: entering follower state: follower="Node at 192.168.60.10:8300 [Follower]" leader-address= leader-id=
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.959-0500 [INFO] agent.server: Adding LAN server: server="node-server-1 (Addr: tcp/192.168.60.10:8300) (DC: us)"
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.960-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-1.us area=wan
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.960-0500 [WARN] agent: [core][Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {
Apr 02 08:19:20 node-server-1 consul[3692]: "Addr": "us-192.168.60.10:8300",
Apr 02 08:19:20 node-server-1 consul[3692]: "ServerName": "node-server-1",
Apr 02 08:19:20 node-server-1 consul[3692]: "Attributes": null,
Apr 02 08:19:20 node-server-1 consul[3692]: "BalancerAttributes": null,
Apr 02 08:19:20 node-server-1 consul[3692]: "Type": 0,
Apr 02 08:19:20 node-server-1 consul[3692]: "Metadata": null
Apr 02 08:19:20 node-server-1 consul[3692]: }. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.60.10:0->192.168.60.10:8300: operation was canceled"
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.961-0500 [INFO] agent.server.cert-manager: initialized server certificate management
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=udp
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=tcp
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Starting server: address=192.168.60.10:8500 network=tcp protocol=http
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: started state syncer
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Consul agent running!
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Started gRPC listeners: port_name=grpc address=192.168.60.10:8502 network=tcp
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=192.168.60.10:8503 network=tcp
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Joining cluster...: cluster=LAN
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: (LAN) joining: lan_addresses=["192.168.60.10", "192.168.60.20", "192.168.60.30"]
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-3 192.168.60.30
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-2 192.168.60.20
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server: Adding LAN server: server="node-server-3 (Addr: tcp/192.168.60.30:8300) (DC: us)"
Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.994-0500 [INFO] agent: (LAN) joined: number_of_nodes=3
Apr 02 08:19:20 node-server-1 systemd[1]: Started "HashiCorp Consul".
░░ Subject: A start job for unit consul.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit consul.service has finished successfully.
░░
░░ The job identifier is 3287.
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.017-0500 [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.028-0500 [INFO] agent.server: Found expected number of peers, attempting bootstrap: peers="192.168.60.30:8300,192.168.60.20:8300,192.168.60.10:8300"
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-3.us 192.168.60.30
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-2.us 192.168.60.20
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-3.us area=wan
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-2.us area=wan
Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.061-0500 [INFO] agent.server: Adding LAN server: server="node-server-2 (Addr: tcp/192.168.60.20:8300) (DC: us)"
Apr 02 08:19:28 node-server-1 consul[3692]: 2023-04-02T08:19:28.623-0500 [INFO] agent.server: New leader elected: payload=node-server-3
Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.665-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-1 192.168.61.10
Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.790-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-3 192.168.61.30
Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.808-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-2 192.168.61.20
Apr 02 08:19:33 node-server-1 consul[3692]: 2023-04-02T08:19:33.957-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:34 node-server-1 consul[3692]: 2023-04-02T08:19:34.960-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:35 node-server-1 consul[3692]: 2023-04-02T08:19:35.940-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:36 node-server-1 consul[3692]: 2023-04-02T08:19:36.522-0500 [ERROR] agent: Coordinate update error: error="rpc error making call: ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:36 node-server-1 consul[3692]: 2023-04-02T08:19:36.959-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:37 node-server-1 consul[3692]: 2023-04-02T08:19:37.451-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:39 node-server-1 consul[3692]: 2023-04-02T08:19:39.959-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:40 node-server-1 consul[3692]: 2023-04-02T08:19:40.963-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:41 node-server-1 consul[3692]: 2023-04-02T08:19:41.963-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:42 node-server-1 consul[3692]: 2023-04-02T08:19:42.965-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:44 node-server-1 consul[3692]: 2023-04-02T08:19:44.958-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Caught: signal=interrupt
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Graceful shutdown disabled. Exiting
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Requesting shutdown
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent.server: shutting down server
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [WARN] agent.server.serf.lan: serf: Shutdown without a Leave
Apr 02 08:19:46 node-server-1 systemd[1]: Stopping "HashiCorp Consul"...
░░ Subject: A stop job for unit consul.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit consul.service has begun execution.
░░
░░ The job identifier is 4336.
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [WARN] agent.server.serf.wan: serf: Shutdown without a Leave
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [INFO] agent.router.manager: shutting down
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [INFO] agent.router.manager: shutting down
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: consul server down
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: shutdown complete
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: protocol=DNS address=192.168.60.10:8600 network=tcp
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: protocol=DNS address=192.168.60.10:8600 network=udp
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: address=192.168.60.10:8500 network=tcp protocol=http
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Waiting for endpoints to shut down
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Endpoints down
Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Exit code: code=1
Apr 02 08:19:46 node-server-1 systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit consul.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 02 08:19:46 node-server-1 systemd[1]: consul.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit consul.service has entered the 'failed' state with result 'exit-code'.
Apr 02 08:19:46 node-server-1 systemd[1]: Stopped "HashiCorp Consul".
░░ Subject: A stop job for unit consul.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit consul.service has finished.
░░
░░ The job identifier is 4336 and the job result is done.
Apr 02 08:19:50 node-server-1 systemd[1]: Starting "HashiCorp Consul"...
░░ Subject: A start job for unit consul.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit consul.service has begun execution.
░░
░░ The job identifier is 4336.
Apr 02 08:19:50 node-server-1 consul[3822]: ==> Starting Consul agent...
Apr 02 08:19:50 node-server-1 consul[3822]: Version: '1.15.2'
Apr 02 08:19:50 node-server-1 consul[3822]: Build Date: '2023-03-30 17:51:19 +0000 UTC'
Apr 02 08:19:50 node-server-1 consul[3822]: Node ID: 'a2dbd0ce-3500-2a9b-4624-2ddf766082c9'
Apr 02 08:19:50 node-server-1 consul[3822]: Node name: 'node-server-1'
Apr 02 08:19:50 node-server-1 consul[3822]: Datacenter: 'us' (Segment: '')
Apr 02 08:19:50 node-server-1 consul[3822]: Server: true (Bootstrap: false)
Apr 02 08:19:50 node-server-1 consul[3822]: Client Addr: [192.168.60.10] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600)
Apr 02 08:19:50 node-server-1 consul[3822]: Cluster Addr: 192.168.60.10 (LAN: 8301, WAN: 8302)
Apr 02 08:19:50 node-server-1 consul[3822]: Gossip Encryption: false
Apr 02 08:19:50 node-server-1 consul[3822]: Auto-Encrypt-TLS: false
Apr 02 08:19:50 node-server-1 consul[3822]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
Apr 02 08:19:50 node-server-1 consul[3822]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Apr 02 08:19:50 node-server-1 consul[3822]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
Apr 02 08:19:50 node-server-1 consul[3822]: ==> Log data will now stream in as it occurs:
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.548-0500 [WARN] agent: bootstrap_expect > 0: expecting 3 servers
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.551-0500 [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:247b43d4-680e-91a7-2559-ed5f7edf04f0 Address:192.168.60.30:8300} {Suffrage:Voter ID:0568fec4-01e3-982f-d971-9954e2fe12c6 Address:192.168.60.20:8300} {Suffrage:Voter ID:a2dbd0ce-3500-2a9b-4624-2ddf766082c9 Address:192.168.60.10:8300}]"
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-1.us 192.168.60.10
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-1 192.168.60.10
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.router: Initializing LAN area manager
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.autopilot: reconciliation now disabled
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.raft: entering follower state: follower="Node at 192.168.60.10:8300 [Follower]" leader-address= leader-id=
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: node-server-3.us: 192.168.60.30:8302
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: node-server-2: 192.168.60.20:8301
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server: Adding LAN server: server="node-server-1 (Addr: tcp/192.168.60.10:8300) (DC: us)"
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.556-0500 [INFO] agent.server: Raft data found, disabling bootstrap mode
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.556-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-1.us area=wan
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.cert-manager: initialized server certificate management
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=udp
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-2 192.168.61.20
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-3 192.168.60.30
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-2 192.168.60.20
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: node-server-2: 192.168.60.20:8301
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server: Adding LAN server: server="node-server-3 (Addr: tcp/192.168.60.30:8300) (DC: us)"
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server: Adding LAN server: server="node-server-2 (Addr: tcp/192.168.60.20:8300) (DC: us)"
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=tcp
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Starting server: address=192.168.60.10:8500 network=tcp protocol=http
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: started state syncer
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Consul agent running!
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Started gRPC listeners: port_name=grpc address=192.168.60.10:8502 network=tcp
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=192.168.60.10:8503 network=tcp
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Joining cluster...: cluster=LAN
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: (LAN) joining: lan_addresses=["192.168.60.10", "192.168.60.20", "192.168.60.30"]
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.559-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-2.us 192.168.60.20
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.559-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-2.us area=wan
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.564-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-3.us 192.168.60.30
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.564-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-3.us area=wan
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.566-0500 [INFO] agent: (LAN) joined: number_of_nodes=3
Apr 02 08:19:50 node-server-1 systemd[1]: Started "HashiCorp Consul".
░░ Subject: A start job for unit consul.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit consul.service has finished successfully.
░░
░░ The job identifier is 4336.
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.587-0500 [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: node-server-3.us: 192.168.60.30:8302
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.587-0500 [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.634-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-3 192.168.61.30
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.651-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-1 192.168.61.10
Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.663-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:51 node-server-1 consul[3822]: 2023-04-02T08:19:51.670-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:53 node-server-1 consul[3822]: 2023-04-02T08:19:53.569-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:54 node-server-1 consul[3822]: 2023-04-02T08:19:54.575-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:55 node-server-1 consul[3822]: 2023-04-02T08:19:55.679-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.559-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.583-0500 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.585-0500 [INFO] agent.server.raft: entering candidate state: node="Node at 192.168.60.10:8300 [Candidate]" term=3
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: election won: term=3 tally=2
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: entering leader state: leader="Node at 192.168.60.10:8300 [Leader]"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: added peer, starting replication: peer=247b43d4-680e-91a7-2559-ed5f7edf04f0
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server.raft: added peer, starting replication: peer=0568fec4-01e3-982f-d971-9954e2fe12c6
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server: cluster leadership acquired
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server: New leader elected: payload=node-server-1
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.613-0500 [INFO] agent.server.raft: pipelining replication: peer="{Voter 0568fec4-01e3-982f-d971-9954e2fe12c6 192.168.60.20:8300}"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.614-0500 [INFO] agent.server.raft: pipelining replication: peer="{Voter 247b43d4-680e-91a7-2559-ed5f7edf04f0 192.168.60.30:8300}"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.619-0500 [INFO] agent.server: initializing acls
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.leader: started routine: routine="acl token reaping"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.server.autopilot: reconciliation now enabled
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.leader: started routine: routine="federation state anti-entropy"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="federation state pruning"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="streaming peering resources"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="metrics for streaming peering resources"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="peering deferred deletion"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] connect.ca: initialized primary datacenter CA from existing CARoot with provider: provider=consul
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="intermediate cert renew watch"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="CA root pruning"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="CA root expiration metric"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="CA signing expiration metric"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="virtual IP version check"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="config entry controllers"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.626-0500 [INFO] agent.leader: stopping routine: routine="virtual IP version check"
Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.626-0500 [INFO] agent.leader: stopped routine: routine="virtual IP version check"
Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.300-0500 [WARN] agent.server.acl: failed to remove bootstrap file: error="remove /opt/consul/acl-bootstrap-reset: no such file or directory"
Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.329-0500 [INFO] agent.server.acl: ACL bootstrap completed
Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.365-0500 [INFO] agent.http: Request cancelled: method=GET url="/v1/catalog/service/nomad?dc=us&near=_agent&stale=&tag=serf&wait=2000ms" from=192.168.60.10:48842 error="ACL not found"
Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.560-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:58 node-server-1 consul[3822]: 2023-04-02T08:19:58.560-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:19:59 node-server-1 consul[3822]: 2023-04-02T08:19:59.563-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured
Apr 02 08:20:02 node-server-1 consul[3822]: 2023-04-02T08:20:02.425-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"
Apr 02 08:20:02 node-server-1 consul[3822]: 2023-04-02T08:20:02.711-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"

Missing Vault token in Nomad servers

What OS are you using?

macOS

Unexpected behavior

Nomad servers are not able to start due to missing Vault token

Steps to reproduce

$ make init

Navigate to a Nomad server (e.g. http://192.168.60.10:4646/)

Expected behavior

The Nomad UI is displayed.

Logs

$ vagrant ssh node-server-1
$ journalctl -u nomad
Feb 06 20:14:25 node-server-1 systemd[1]: Started "HashiCorp Nomad".
Feb 06 20:14:25 node-server-1 nomad[15561]: ==> Loaded configuration from /hashibox/defaults/nomad/config/defaults.hcl, /hashibox/overrides/nomad/config/overrides.hcl
Feb 06 20:14:25 node-server-1 nomad[15561]: ==> Starting Nomad agent...
Feb 06 20:14:25 node-server-1 nomad[15561]: ==> Error starting agent: server setup failed: Failed to setup Vault client: Vault token must be set
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.833Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/opt/nomad/plugins
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Feb 06 20:14:25 node-server-1 nomad[15561]:     2022-02-06T20:14:25.837Z [INFO]  nomad: shutting down server

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.