High memory usage with CRS about coraza-caddy HOT 6 CLOSED

It's cool, thank you for your discovery, I will close this issue once a solution is found, so you are kept posted.

from coraza-caddy.

Hey @abstractvector, I have added a new cache feature for transactions so the memory is reused instead of freed and reallocated, your feedback on the branch would be really appreciated: https://github.com/jptosso/coraza-caddy/tree/tx_clean (132f1f6)

My results are a 0,3% (96MB) variation of memory with 100.000 requests:

Concurrency Level:      100
Time taken for tests:   35.207 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      10400000 bytes
HTML transferred:       1300000 bytes
Requests per second:    2840.36 [#/sec] (mean)
Time per request:       35.207 [ms] (mean)
Time per request:       0.352 [ms] (mean, across all concurrent requests)
Transfer rate:          288.47 [Kbytes/sec] received

Thank you

from coraza-caddy.

Hey @abstractvector thank you very much for the information and the docker implementation, the memory footprint of the rules cannot be decreased a lot, but I will run some tests.

Regarding the transactions, maybe the GC is not fast enough to clear old transactions, I will run some experiments with a transaction.Close function to flush resources.

Can you provide me your hardware specs and the parameters used for AB? I will try to replicate the issue and create a detailed profiling.

from coraza-caddy.

My testing was crude and not particularly extensive, but the following command demonstrates the issue quite well:

$ ab -c 100 -n 10000 http://localhost:8000

Running this against either your Docker image with CRS compiled in or my Docker image linked above shows the situation quite clearly in contrast to the same test with the two CRS config file lines commented out. I'm just running this on a MacBook Pro - Docker has access to 4 CPUs and 2GB RAM.

Having now left it a while, it seems like the memory consumption does eventually drop - it looks like a minute or two after the ab run, the CPU usage for the process jumps to around 11-12% and the memory consumption begins dropping. I'm guessing that's the GC.

My hope is to turn this into a cross-platform build so it could run on a Raspberry Pi or similar, but that memory consumption is too high unfortunately.

from coraza-caddy.

You are right about the memory spike but the GC turns down after a few seconds. It should be a priority to reduce the effort for the GC and proposals are more than welcome. I think I should create a function after tx.ProcessLogging(), something like tx.Close()

func (tx *Transaction) Close() error {
  if tx.collections == nil {
    return fmt.Errorf("transaction already closed")
  tx.collections = nil
}

or manually clearing it:

for k := range tx.collections {
    delete(tx.collections, k)
}

tx.collections is the heavier pointer for a transaction, forcing it's deletion should be enough.

from coraza-caddy.

I would love to help out, but I know very little about Go so I'm unlikely to be much assistance I'm afraid.

from coraza-caddy.