The log saved during kurtosis log collection is too large about kurtosis HOT 7 OPEN

Thanks for filing this @icesfeathers ! As mentioned in Discord, it seems like what would make sense as a solution here is:

the ability the ability to turn off the collection of logs (in which case you won't be able to view logs after services have died)

OR

a byte based retention mechanism where historical logs are removed after a certain threshold is passed so it logs fill up disk space.(For context, currently, we have time based log retention where old logs are removed after 4 weeks but no byte based log retention)

from kurtosis.

Here is the command information and configuration information for running kurtosis

kurtosis run --cli-log-level "panic" --enclave my-testnet github.com/kurtosis-tech/[email protected] "$(cat /data/fleet/kurtosis_network_params.yaml)"

participants:
    - validator_count: 512
      beacon_extra_params:
          - "--reconstruct-historic-states"
      el_extra_params:
          - "--gcmode=archive"
      el_max_mem: 8192
      bn_max_mem: 4096
      v_max_mem: 4096
    - validator_count: 1
      el_max_mem: 8192
      bn_max_mem: 4096
      v_max_mem: 4096
network_params:
    seconds_per_slot: 6
    eth1_follow_distance: 2
    deneb_fork_epoch: 9999999999999
additional_services:
    - tx_spammer
    - blob_spammer
    - el_forkmon
    - blockscout
    - beacon_metrics_gazer
    - dora
    - prometheus_grafana
mev_type: "full"

from kurtosis.

After I stopped the log container, the memory usage of the dockerd process soared and could not be released. Is this a bug?

from kurtosis.

from kurtosis.

@tedim52

from kurtosis.

Since I have not turned on the debug mode, I cannot explain in detail. Since this docker only runs kurtosis, I would like to give you some feedback.

from kurtosis.

Hey @icesfeathers ! I suspect what's causing the memory issues are the fact that logs are being sent to logs aggregator but aren't being processed/forwarded. Currently, we don't support a way to turn off logs collection safely in the product.

from kurtosis.