curiefense / curiefense Goto Github PK
View Code? Open in Web Editor NEWCuriefense is a unified, open source platform protecting cloud native applications.
Home Page: https://linktr.ee/curiefense
License: Apache License 2.0
Curiefense is a unified, open source platform protecting cloud native applications.
Home Page: https://linktr.ee/curiefense
License: Apache License 2.0
First of all, great project!!
I was following the Quick start guide, creating the Header profiling list.
While testing this, in a new install with docker-compose, the requests doesn't get tagged when using the Header validation.
I've seen that the response from the server is always with the Header with the first character capitalized (Foo), so I've tested both with capitalized and not.
Strangely, when following exactly what's in the quick start, no log even appear when this rule is created.
So I've tested with my own IP in the profiling list, and it works fine (the query gets tagged, and further I can block it), so I'm wondering if this is something with the header parsing.
Thank you!
Describe the bug
Pulling curietasker (curiefense/curietasker:latest)...
ERROR: manifest for curiefense/curietasker:latest not found: manifest unknown: manifest unknown
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Go to curiefense/deploy/compose
docker-compose up
See error
Run on MacOS
we need to add passive challenge support.
Changes
Left/Right columns
Left column containing: name, description, threshold, ttl
Right column containing: count by, event, action, include, exclude
New entity
blaclist/Whitelist -> Change to better terms blocklist/allowlist
TTL
Add suffix with units (seconds)
Note: needs to be added in flow control as well
Reported by @tzuryby
Change source of list profiling
Clicked 'update now' button (next to last update)
nothing shows up in the UI
Add security disclosure policy
new document type named: flowcontrol
structure (draft):
{
"id": "d45gai67",
"name": "login",
"sequence":[
["GET /login", "HEAD /login", "GET /signup", "GET /index.html"],
["OPTIONS sub.domain.com/ajax-cors"],
["GET /login.js"],
["POST /api/login"]
]
}
Doc Editor std UI --
Note each entry in the sequence might contain multiple elements.
We currently run python conf server tests on each new PR, we should add the conf client (non py) tests as well
When creating a new entry under profiling lists (Tag Rules) we have a generic component with category + entry value, we would like to create a component that changes according to the selected category
Annotation should be in a different input field while still working with [value#annotation] in the value field
Annotation should take inner value annotation [value#annotation] over the annotation field [value] [annotation]
Value field should be split to two if category's values are pairs (e.g. category is args, cookies, headers)
Value should be validated using a regex according to category
Version: ba04fdd
To reproduce on a minikube deployment:
export IP=$(minikube ip); pytest --log-level INFO --base-protected-url http://$IP:30081 --base-conf-url http://$IP:30000/api/v1/ --base-ui-url http://$IP:30080 -k 'test_ratelimit_scope_tag' .
Observed error logs for queries associated to this test:
[Envoy (Epoch 0)] [2020-12-21 14:02:04.194][33][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:41: subject has no topointer method
There are many checkboxes throughout the configuration client, some of them use Bulma's wrapper class checkbox
while others do not, we need to locate any checkbox input that doesn't have the wrapper and add it.
The wrapper is not intrusive as it attempts to preserve cross-platform compatibility and the user experience by only adding minor manipulations to the wrapper. e.g. "cursor: pointer"
Describe the bug
Version used: docker tag 7a19c72b7752
Using docker-compose up and going through the steps in https://docs.curiefense.io/installation/getting-started-with-curiefense I stumbled upon the issue that when curieproxy matches a header rule in a profiling list, it throws the error:
[error][lua] [source/extensions/filters/http/lua/lua_filter.cc:683] script log: ./lua/tagprofiler.lua:51: bad argument #2 to 're_match' (string or rex_pcre2_regex expected, got nil)
And fails to further process the request (I do get a response, but the request is not logged as there's no meta data). I found that I can even reproduce it in a fresh setup with:
curl http://curie.demo:30081/with/header2 -H content-type:application/json
When running the curielogger
in debug mode, I see:
curielogger | 2020/11/20 14:44:48 [DEBUG] ====>[&{log_entry:<common_properties:<downstream_remote_address:<socket_address:<address:"172.19.0.1" port_value:47538 > > downstream_local_address:<socket_address:<address:"172.19.0.3" port_value:80 > > start_time:<seconds:1605883488 nanos:103722000 > time_to_last_rx_byte:<nanos:51600 > time_to_first_upstream_tx_byte:<nanos:26714300 > time_to_last_upstream_tx_byte:<nanos:26730100 > time_to_first_upstream_rx_byte:<nanos:27768900 > time_to_last_upstream_rx_byte:<nanos:27981600 > time_to_first_downstream_tx_byte:<nanos:27953200 > time_to_last_downstream_tx_byte:<nanos:28014400 > upstream_remote_address:<socket_address:<address:"172.19.0.2" port_value:8080 > > upstream_local_address:<socket_address:<address:"172.19.0.3" port_value:53392 > > upstream_cluster:"target_site" downstream_direct_remote_address:<socket_address:<address:"172.19.0.1" port_value:47538 > > > protocol_version:HTTP11 request:<request_method:GET scheme:"http" authority:"curie.demo:30081" path:"/with/header2" user_agent:"curl/7.64.1" forwarded_for:"172.19.0.1" request_id:"8f961363-0620-4101-8d72-c34755c7d989" request_headers_bytes:272 > response:<response_code:<value:200 > response_headers_bytes:123 response_body_bytes:333 response_code_details:"via_upstream" > > }]
curielogger | 2020/11/20 14:44:48 [DEBUG] ---> [ 172.19.0.2:8080 172.19.0.3:53392 ] <---
curielogger | 2020/11/20 14:44:48 [DEBUG] No curiefense metadata => drop log entry
API validation has been disabled on release as it was causing issues. We need to reenable it and fix any issues it may have caused in the past
File 'api.py' Line 482
Add a public roadmap
We started with config, then added DB.
Given both are going to expand with new types and uses, perhaps we can unify them as simply config, where types are defined by a JSON schema, and can be added dynamically by simply adding a new schema.
When using the tag autocomplete input with the 'multiple' setting, it does not save unknown tags added when clicking space and only when clicking enter. tags should be submitted with space as well as this is an indicator of a new tag start
When using the tag autocomplete and inserting new tags, they do not show up in the dropdown until the component is reloaded, the new tags should be available immediately
tasker that support two types of tasks.
When there are many tags in autocomplete we currently display all of them, we can add a virutal scrollbar to enhance the user experience
Assuming tasker
is upcoming soon, we will need to set up the convention how to maintain a set of lists that will be visible in the system, yet not editable.
For instance, dynamic banning and releasing of IPs based on violations.
there are two fields which will rule this type of a list:
"id": "..."
and
"source": "..."
similar to the "source": "self-managed"
we should have a value that reflects this status, perhaps automation
or similar.
"restrict" parameter has no effect?
Default waf profile, active, with constraints as shown on the screenshot
WAF Parameter Constraint: header="name-restrict", value="value", restrict is checked
If I sent an invalid value, the request is not blocked. Reproducer:
requests.get('http://192.168.49.2:30081/', headers={"name-restrict": "invalid"})
we will need to add an option which does not include git clone
assuming downloading an arhcive of label x will be good.
I suggest creating an .sh
file that
docker-compose
up from deploy/compose
There is a great demand from whomever I speak to add this DB support.
requests.get("http://192.168.49.2:30081/?TOTO") yields the following error each time
New bug: requests.get("http://192.168.49.2:30081/?TOTO") yields the following error each time:
[Envoy (Epoch 0)] [2020-12-19 15:28:31.341][29][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:81: attempt to index local 'rule' (a nil value)
17:30
I have also seen this error several times, can be reproduced with requests.get("http://192.168.49.2:30081/", headers={"Host": "doesnotmatch"})
[Envoy (Epoch 0)] [2020-12-19 15:27:41.109][30][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:81: attempt to index local 'rule' (a nil value)
right now we only support "OR" relations between enties.
we must add "AND", and perhaps blocks, e.g.
(
{}
OR
{}
)
AND
(
{}
OR
{}
)
I'm sending either an overlong header, or too many headers, but am not getting blocked
(xavier)
There is an account we are running with the market place already.
Will send all informaiton over slack.
When we have no data on a specific entity (for example after the user deletes all flowcontrol through the API) the UI breaks, we should display a user friendly message
Often there is a need to set a subset of attributes or in some cases a single ine (e.g. "active": true
).
Right now, the process is
GET
XPUT
XInstead, I propose adding support to submit a JSON that will be merged into the existing entry.
This can be implemented on the same endpoint we have now, or in a new one e.g.
curl /configs/master/d/urlmaps/e/__default__/ \
--data {"waf_active": true} -H "content-type: application/json"
OR
curl /configs/master/d/urlmaps/e/__default__/attr/ \
--data {"waf_active": true} -H "content-type: application/json"
Version: 1157774
To reproduce on a minikube deployment:
export IP=$(minikube ip); pytest --log-level INFO --base-protected-url http://$IP:30081 --base-conf-url http://$IP:30000/api/v1/ --base-ui-url http://$IP:30080 -k 'test_non_allowlisted_value_norestrict_wafmatch_excludesig[params-regex-no_ignore_alphanum]' .
Observed error logs for queries associated to this test: none
If I follow the diagram on the documentation correctly, this query should follow this path:
regex-norestrict
argument has a defined contrainthtaccess
, does NOT match the Matching Value pattern [v]+[a]{1}l?u*e
sig_id 100140
; it has the wafsig:100140
tag, even though it should not have been evaluated=> This request is rejected, but should not be
curieconfctl sync export
tries to export to the bucket locally, rather than asking the server to do so. This breaks when using docker-compose, where by default the bucket is mounted to /bucket, which does not exist on the host.
-> call the export API instead of calling the cloudstorage API
The Vue.js client should have unit tests for easier maintenance of code
Either schema based UI or fixed for publishinfo, tags and tasks
In the configuration client, under document editor, we have a few different document types
We need to align their implementation to match each other as much as possible:
Some of them have a "name" label above the "name" input and some do not. They should all have it
Some of them have an "ID" label above the "name" input, others have it below, and some to the side. They should all have it in the same line as the "name" label aligned to the right
FROM https://gist.github.com/valyala/ae3cbfa4104f1a022a2af9b8656b1131
Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
Add synchronous_commit = off to postgresql.conf.
Use table inheritance for fast removal of old data:
CREATE TABLE parent ... ;
CREATE TABLE child_1() INHERITS (parent);
CREATE TABLE child_2() INHERITS (parent);
-- always INSERT rows into child_1.
-- SELECT from parent.
-- periodically run the follwing sql for rotating child_1 with child_2:
TRUNCATE TABLE child_2;
BEGIN;
ALTER TABLE child_1 RENAME TO child_tmp;
ALTER TABLE child_2 RENAME TO child_1;
ALTER TABLE child_tmp RENAME TO child_2;
COMMIT;
This is much faster comparing to
DELETE FROM parent WHERE time < now() - interval 'given period'
This also avoids table fragmentation, so SELECT queries work faster on the table.
To Reproduce
curieconfctl conf get master > master.json
curieconfctl conf create -n test master.json
Result:
{
"errors": {
"meta.logs": "None is not of type 'array'",
"delete_documents.limits": "None is not of type 'array'",
"delete_documents.urlmaps": "None is not of type 'array'",
"delete_documents.wafsigs": "None is not of type 'array'",
"delete_documents.wafprofiles": "None is not of type 'array'",
"delete_documents.aclprofiles": "None is not of type 'array'",
"delete_documents.profilinglists": "None is not of type 'array'",
"delete_blobs.geolite2asn": "None is not of type 'boolean'",
"delete_blobs.geolite2country": "None is not of type 'boolean'"
},
"message": "Input payload validation failed"
}
Tested with version cada8ef
AWS user sent separatly.
https://aws.amazon.com/marketplace/management/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.