opencti-platform / client-python Goto Github PK

View Code? Open in Web Editor NEW

86.0 8.0 111.0 6.3 MB

OpenCTI Python Client

Home Page: https://www.opencti.io

License: Apache License 2.0

Python 99.95% JavaScript 0.03% Makefile 0.01%

cti cybersecurity threat-intelligence

client-python's People

Contributors

Stargazers

Watchers

Forkers

jekyc amr-cossi mbellioum suspended totol 2xyo onekill1801 imuledx vvoidranger oooathsign rhaist capitaldata walt1998 anthons-an romainguignard cix-16 lightsecops phucsnguyenv ben12385 transilience-tech-llp mkulasi zweger my-soc hugoob syeedhasan alfon-sec 4n6strider linsicai kaosaudi djds nor3th on1-tech mlodic snipyjulmy unimol moishce-zz pagosasia jekstra ekmixon fscc-samir dinateper minchoi02 cagivajsp mbixler-var perloz kenuosec little-roach pierceoneill cyberpoint northshade yanover cybercercher kohsawa rlynch-ironnet ironnetcybersecurity smarthive-skp commander tonyjabbour pliniocorregio tamirgold srobier djigarjian-deloitte mrong319 bunnyapocalypse telsy-cti cpplam orenzamir5 czechnology nvhoan69 chrisontay zachfleeman 5l1v3r1 stiltz ckane othonam arkan33 mustahsin64 ethicalsecurity-agency tomking2 huseynkhanli fbicyber juliapaluch sqooba cyp-seculyze faizaanalfaisal cgoncalves1 biiigmouth bkeit torettoalfa tobecert obdam cmandich jovanp80 cybervanguardtechnologies sean0427 yeyushilai zi3lak ownsecurity ccu7lsp regturh

client-python's Issues

A new tool is creating whenever trying to add a vulnerability.

Environment:

python 3.7
opencti --version 2.1.2

Problem to Solve

Using PYCTI , whenever trying to add a vulnerability using openct_api_client.vulnerbaility.create ,a new tool is creating instead of vulnerability.

Upgrade the library with methods for new generation connectors (2.0.0)

Problem to Solve

The new connectors need to be able to register to the platform and ping it.

Current Workaround

None.

Proposed Solution

Implement new methods and refactor the library.

Additional Information

None.

Use black formatter

Problem to Solve

Python style is not defined for this project. 😱

It's obviously not such a big deal, but there is a straightforward and easy solution to address this issue. Black.

Black is a project under the PSF umbrella. "Black reformats entire files in place. It is not configurable. It doesn't take previous formatting into account. It doesn't reformat blocks that start with # fmt: off and end with # fmt: on. # fmt: on/off have to be on the same level of indentation. It also recognizes YAPF's block comments to the same effect, as a courtesy for straddling code."

Black is used notable open-source projects: pytest, tox, Pyramid, Django, etc.

Proposed Solution

Add black to setup.py:

extras_require={  # Optional
        'dev': ['black','wheel'],
    },

(wheel for #19)

Apply black to the project:

$ black .

Add $ black --check . to the CI.

--check : Don't write the files back, just return the status. Return code 0 means nothing would change. Return code 1 means some files would be reformatted. Return code 123 means there was an internal error.

Additional Information

Doc: https://black.readthedocs.io/en/stable/

$ git clone https://github.com/OpenCTI-Platform/client-python.git 
$ cd client-python
$ black .
$ git diff --shortstat
 56 files changed, 6126 insertions(+), 4454 deletions(-)

Priority: extra low

Case sensitivity

Description

Firstly, it's possible I'm using the client wrong. I'm creating a connector to import reports, and trying to link entities as knowledge, specifically intrusion-sets, countries, and sectors. I've noticed that these seem to be case sensitive, sometimes resulting in new entities being created. Originally I was just creating a stix bundle and uploading it, so I tried using the client search_stix_domain_entities_by_name method to find the existing id and use that. The feed I'm using has some inconsistencies, particularly in the case of items (plus random pluralisation).

Environment

OS (where OpenCTI server runs): Ubuntu
OpenCTI version: OpenCTI 1.1.2
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

client.search_stix_domain_entities_by_name("Cobalt Group")
client.search_stix_domain_entities_by_name("Cobalt group")

Expected Output

Both should return matches

Actual Output

Only 1 returns a match.

Queue delete leads to messages lost

Description

Queue is deleted when connector reconnects in the connector_helper or at the starting of the connector.

Environment

All
OpenCTI Python client 1.2.12

KillChainPhases are not requested on Indicators in Stix-Domain-Entity

Please replace every line in curly brackets { like this } with appropriate answers, and remove this line.

Problem to Solve

{ Please describe the problem you would like to solve. }

Current Workaround

{ Please describe how you currently solve or work around this problem, given OpenCTI's limitation. }

Proposed Solution

{ Please describe the solution you would like OpenCTI to provide, to solve the problem above. }

Additional Information

{ Any additional information, including logs or screenshots if you have any. }

Possible race condition when creating non-existing objects?

Description

During import of the opencti dataset (https://github.com/OpenCTI-Platform/datasets/blob/master/data/sectors.json), identities can be created more than once when multiple workers are used.

I think this exact issue is related to the method split_stix2_bundle in https://github.com/OpenCTI-Platform/client-python/blob/1.2.15/pycti/opencti_connector_helper.py, as this creates a bundle per SDO or SRO. The pseudo bundles below illustrate (condensed to only include the relevant object IDs) two bundles created by the split_stix2_bundle.

Bundle A: {
    relationship--d4a1c31b-3bdd-458a-ac01-614e0ab871e2,
    identity--7083bf6a-87f7-40a7-b888-a919312f82e5,
}

Bundle B: {
    relationship--4708ce1b-1960-4c67-a408-dc6f26db325d
    identity--7083bf6a-87f7-40a7-b888-a919312f82e5,
}

Note, that because the Identity 7083bf6a-87f7-40a7-b888-a919312f82e5 has multiple relations, it is included in both the bundles, which in and of itself is not a problem, but if the bundles are processed by different workers at the same time, each worker will insert a copy of the Identity with different database/grakn IDs but with the same STIX ID, since the Identiry does not exist yet.

I don't think split_stix2_bundle is directly responsible for this behavior, but it does help highlight a (general?) race condition issue during (parallel?) imports.

I don't know enough about the internal workings of the Elasticsearch and grakn queries used in opencti to have a solution, but Elasticsearch is only Near Real Time, so this might be an issue? If it is, the refresh parameter might be helpful (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html).

Would it help to use the STIX Object ID as the primary/document ID help here, as the second document creation would/should fail since a document with that ID already exists?

Another solution could be to ensure that objects with no relationships are created first (and are querable), before inserting other objects that depend on them. While this would fix the current problem I'm not sure if it would be a viable long-term solution.

Environment

OS (where OpenCTI server runs): Docker containers in swarm mode
OpenCTI version: latest docker image
Elasticsearch and Grakn are on different machines

Reproducible Steps

Steps to create the smallest reproducible scenario:

Setup
1.1. Clean all databases (ES and grakn)
1.2. Ensure that the opencti-connector is included as a service import worker as more than one replica.
Run docker-compose --compatibility up

Expected Output

Only one copy of each sector (such as "Culture and entertainment") should be created.

Actual Output

Multiple copies of the same sector are (sometimes) created.

Additional information

Packaging problem

Description

The ongoing 2.0.0 version is not packaged correctly.

Environment

OpenCTI Python Client 2.0.0

Reproducible Steps

None.

Expected Output

Working.

Actual Output

Import problems.

Additional information

None.

Refactor the client to make it more maintainable/understandable

The client is currently implemented by two primary classes, both of which are rather large (approx. 3600 and 1200 lines of code) and depend on each other. Would it make sense to split the client up? This would make it easier for developers to use and extend the client.

One implementation could be to have individual objects for each of the types used by OpenCTI. These would be responsible for converting corresponding stix2 objects (from the python library stix2) to something the graphql api understands, and adding and validating OpenCTI custom properties.
Futhermore, these objects could also include the graphql queries for the object type (such as 'create', 'create_if_not_exists', etc).

Using this approach, the Identity class would look something like this:

class OpenCTIBaseModel:
    """Base model for OpenCTI objects

    Contains connection info for the graphql db and generic queries + misc helpers.
    """


class Identity(OpenCTIBaseModel):
    STIX_TYPE = stix2.Identity

    def __init__(self, *args, **kwargs):
        self.type = kwargs.get('type', None)
        self.name = kwargs.get('name', None)
        self.description = kwargs.get('description', None)
        self.id = kwargs.get('id', None)
        self.stix_id = kwargs.get('stix_id', None)
        self.created = kwargs.get('created', None)
        self.modified = kwargs.get('modified', None)

    @classmethod
    def get(cls, id):
        query = """
            query Identity($id: String!) {
                identity(id: $id) {
                   id
                    entity_type
                    stix_id
                    stix_label
                    name
                    alias
                    description
                    created
                    modified
                    createdByRef {
                        node {
                            id
                            entity_type
                            stix_id
                            stix_label
                            name
                            alias
                            description
                            created
                            modified
                        }
                    }
                    markingDefinitions {
                        edges {
                            node {
                                id
                                entity_type
                                stix_id
                                definition_type
                                definition
                                level
                                color
                                created
                                modified
                            }
                        }
                    }
                }
            }
        """
	result = cls.query(query, {'id': id})
	return cls.from_query(result)

    @classmethod
    def get_many(cls, limit=10000):
        query = """
            query Identities($first: Int) {
                identities(first: $first) {
                    edges {
                        node {
                            id
                            entity_type
                            stix_id
                            stix_label
                            name
                            alias
                            description
                            created
                            modified
                            createdByRef {
                                node {
                                    id
                                    entity_type
                                    stix_id
                                    stix_label
                                    name
                                    alias
                                    description
                                    created
                                    modified
                                }
                            }
                            markingDefinitions {
                                edges {
                                    node {
                                        id
                                        entity_type
                                        stix_id
                                        definition_type
                                        definition
                                        level
                                        color
                                        created
                                        modified
                                    }
                                }
                            }
                        }
                    }
                }
            }
        """
        result = self.query(query, {'first': limit})
	return [cls(**identity) for identity in result['data']['identities']]

    @classmethod
    def exists(cls, id=None, name=None):
        # implement actual exists query here or use the OpenCTIBaseModel generic queries such as (check_existing_stix_domain_entity)
        pass

    @classmethod
    def from_stix2(cls, obj):
        if isinstance(obj, cls.STIX_TYPE):
            return cls(**obj.serialize())

        return None

    @classmethod
    def from_query(cls, query_result):
        # this most likely needs some transformation of query_result...
        return cls(query_result)

    def save(self, only_if_not_exists=True):
        # query for saving the object
        pass

    def to_stix2(self):
        # this needs some more data mangling
        return cls.STIX_TYPE(**self, allow_custom=True)

With this approach it is easy to add custom properties based on the type of object - as an example the custom properties 'x_opencti_observable_type' and 'x_opencti_observable_value' for indicator/observable could be automatically set by parsing the STIX indicator pattern (assuming it consists of only one value). Another could be to automatically generate external references for Vulnerabilities (as requested in OpenCTI-Platform/opencti#147). These custom changes/additions would be be easy to spot, maintain and document.

The client would then be some convenience methods (such as 'import_bundle'), that can convert the input to these OpenCTI objects and save them using the objects' save methods.

What do you think about this approach?

Introduce methods in API and connectors helper to store connectors states

Problem to Solve

In some cases, connectors developers may need to store a state and use a key/value store.

Current Workaround

None.

Proposed Solution

Implement methods in the connectors helper to store key/value.

Additional Information

None.

Add a method of health check

Problem to Solve

Add a method for health check the API connection.

Current Workaround

None.

Proposed Solution

Develop the method.

Additional Information

None.

Split STIX2 bundle before sending them in ConnectorHelper

Problem to Solve

A STIX2 bundle can be huge (ie. Enterprise Att&ck). So offer a better performance across workers, it has to be transform in multiple chunks.

Current Workaround

None.

Proposed Solution

Introduce STIX2 bundle chunking.

Additional Information

None.

Importing report JSON bundle with tags fails

Description

Importing report JSON bundle with tags fails.

Environment

OS: Ubuntu 18.04.3 LTS
OpenCTI version: 2.1.3
client-python: 2.1.11

Reproducible Steps

Used a connector (helper.send_stix2_bundle) to import following bundle:

{
    "type": "bundle",
    "id": "bundle--d6d095dd-5cb2-4f17-8527-a34b2035dc04",
    "spec_version": "2.0",
    "objects": [
        {
            "type": "identity",
            "id": "identity--b05f0747-adbf-4531-8647-921a56d1372c",
            "created": "2020-01-27T09:05:20.637Z",
            "modified": "2020-01-27T09:05:20.637Z",
            "name": "Test",
            "identity_class": "organization",
            "x_opencti_identity_type": "organization"
        },
        {
            "type": "marking-definition",
            "id": "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9",
            "created": "2017-01-20T00:00:00.000Z",
            "definition_type": "tlp",
            "definition": {
                "tlp": "white"
            }
        },
        {
            "type": "report",
            "id": "report--2e0ce812-0f40-40c8-9eb4-5a16783092fb",
            "created_by_ref": "identity--b05f0747-adbf-4531-8647-921a56d1372c",
            "created": "2020-01-27T09:05:20.640Z",
            "modified": "2020-01-27T09:05:20.640Z",
            "name": "Test Report d11cb227-b761-43f1-9f53-f6a54ed71644",
            "description": "Test Report d11cb227-b761-43f1-9f53-f6a54ed71644 Description.",
            "published": "2020-01-27T09:05:20.640614Z",
            "object_refs": [
                "identity--b05f0747-adbf-4531-8647-921a56d1372c"
            ],
            "labels": [
                "threat-report"
            ],
            "external_references": [
                {
                    "source_name": "Test",
                    "url": "https://example.com/reference/d11cb227-b761-43f1-9f53-f6a54ed71644"
                }
            ],
            "object_marking_refs": [
                "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"
            ],
            "x_opencti_object_status": 0,
            "x_opencti_report_class": "Threat Report",
            "x_opencti_source_confidence_level": 3,
            "x_opencti_tags": [
                {
                    "color": "#6cc221",
                    "tag_type": "Test",
                    "value": "tag_d11cb227-b761-43f1-9f53-f6a54ed71644"
                }
            ]
        }
    ]
}

Expected Output

Report with tags is imported to OpenCTI.

Actual Output

Report is not imported.

Excerpt from Worker log:

INFO:root:Importing a report (id: report--2e0ce812-0f40-40c8-9eb4-5a16783092fb)
ERROR:root:An unexpected error occurred: { 'id' }
Traceback (most recent call last):
  File "worker.py", line 73, in data_handler
    imported_data = self.api.stix2.import_bundle_from_json(content, update, types)
  File "/home/martv/development/OpenCTI-Platform/opencti_1/opencti-worker/venv/lib/python3.6/site-packages/pycti/utils/opencti_stix2.py", line 130, in import_bundle_from_json
    return self.import_bundle(data, update, types)
  File "/home/martv/development/OpenCTI-Platform/opencti_1/opencti-worker/venv/lib/python3.6/site-packages/pycti/utils/opencti_stix2.py", line 1845, in import_bundle
    self.import_object(item, update, types)
  File "/home/martv/development/OpenCTI-Platform/opencti_1/opencti-worker/venv/lib/python3.6/site-packages/pycti/utils/opencti_stix2.py", line 380, in import_object
    embedded_relationships = self.extract_embedded_relationships(stix_object, types)
  File "/home/martv/development/OpenCTI-Platform/opencti_1/opencti-worker/venv/lib/python3.6/site-packages/pycti/utils/opencti_stix2.py", line 173, in extract_embedded_relationships
    if tag["id"] in self.mapping_cache:
KeyError: 'id'                  
INFO:root:Message processed, thread terminated
INFO:root:Message (delivery_tag=1) acknowledged

Add a custom STIX field for file upload from bundle

Please replace every line in curly brackets { like this } with appropriate answers, and remove this line.

Problem to Solve

{ Please describe the problem you would like to solve. }

Current Workaround

{ Please describe how you currently solve or work around this problem, given OpenCTI's limitation. }

Proposed Solution

{ Please describe the solution you would like OpenCTI to provide, to solve the problem above. }

Additional Information

{ Any additional information, including logs or screenshots if you have any. }

OpenCTI Documentation online no Python API doc

Hello,

Regarding : https://opencti-platform.github.io/docs/clients/python/overview

There is nothing in User's guide and API reference

Where can i find this information?

Thank you.

Use "Type Hints" - PEP 484

Problem to Solve

It's hard for developers to understand what exactly is going on in code they haven't written and for type checking tools found in many IDEs that are limited due to the fact that they don't have any indicator.

Proposed Solution

Use type hints (PEP 484).
Add mypy check to CI.

Why Type Hints?

Helps Type Checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.

Examples

From https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#functions

from typing import Callable, Iterator, Union, Optional, List

# This is how you annotate a function definition
def stringify(num: int) -> str:
    return str(num)

# You can of course split a function annotation over multiple lines
def send_email(address: Union[str, List[str]],
               sender: str,
               cc: Optional[List[str]],
               bcc: Optional[List[str]],
               subject='',
               body: Optional[List[str]] = None
               ) -> bool:

Additional Information

https://www.python.org/dev/peps/pep-0484/ - Type Hints
- https://www.youtube.com/watch?v=2wDvzy6Hgxg - Type Hints - Guido van Rossum - PyCon 2015
https://docs.python.org/3/library/typing.html
https://mypy.readthedocs.io/en/stable/index.html
- https://mypy.readthedocs.io/en/stable/existing_code.html#using-mypy-with-an-existing-codebase " how to get started using mypy with an existing, significant codebase that has little or no type annotations"
https://stackoverflow.com/a/32558710 - What are type hints in Python 3.5?

Introduce notes & opinions

Some report dates are in the future on MITRE report creation/ingestion

Description

The improper date is being set on the Publication date field in the report Details component. A screenshot is provided below.

Environment

OS (where OpenCTI server runs): Ubuntu 18.04
OpenCTI version: OpenCTI 1.0.2
OpenCTI client: Frontend
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

Access report
View report Publication Date

Expected Output

Valid report date that coincides with the actual report date

Actual Output

A date in the future that isn't valid based on report Publication Date

Additional information

opencti connector - reports with incoherent dates

Please replace every line in curly brackets { like this } with an appropriate answer, and remove this line.

Description

Several reports imported via the opencti connector have incoherent dates

Examples:

Unit42 BabyShark Feb 2019 --> Date 2/8/2042
ESET Operation Groundbait --> Date 11/8/2020
etc

Environment

OS (where OpenCTI server runs): { e.g. Mac OS 10, Windows 10, Ubuntu 16.4, etc. }
OpenCTI version: { e.g. OpenCTI 1.0.2 }
OpenCTI client: { e.g. frontend or python }
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

{ e.g. Run ... }
{ e.g. Click ... }
{ e.g. Error ... }

Expected Output

{ Please describe what you expected to happen. }

Actual Output

{ Please describe what actually happened. }

Additional information

{ Any additional information, including logs or screenshots if you have any. }

Handle observables export as STIX2 indicators

Problem to Solve

Currently, the observables are not correctly exported in STIX 2 bundles. The content is not STIX 2 compliant.

Current Workaround

None.

Proposed Solution

Implement the observables export as STIX 2 indicators.

Additional Information

None.

Possibility to set the score of an indicator at the creation

Please replace every line in curly brackets { like this } with appropriate answers, and remove this line.

Problem to Solve

The python-client doesn't allow to set the score of an indicator at the creation.

    def create(self, **kwargs):
        name = kwargs.get("name", None)
        description = kwargs.get("description", None)
        indicator_pattern = kwargs.get("indicator_pattern", None)
        main_observable_type = kwargs.get("main_observable_type", None)
        pattern_type = kwargs.get("pattern_type", None)
        valid_from = kwargs.get("valid_from", None)
        valid_until = kwargs.get("valid_until", None)
        id = kwargs.get("id", None)
        stix_id_key = kwargs.get("stix_id_key", None)
        created = kwargs.get("created", None)
        modified = kwargs.get("modified", None)
        created_by_ref = kwargs.get("createdByRef", None)
        marking_definitions = kwargs.get("markingDefinitions", None)
        update = kwargs.get("update", False)
        custom_attributes = """

An update on the entity is necessary to define the score

Current Workaround

Proposed Solution

Give the opportunity to define score at the creation

Additional Information

Fix the STIX2 class for a compliant export

Problem to Solve

According to this issue the STIX2 bundles generated by the OpenCTI application are not valid and do not pass the validation of the STIX2 validator. Issue confirmed by this output:

[-] Results for: report.json
[X] STIX JSON: Invalid
    [!] Warning: marking-definition--455a3fc4-bba5-4ed7-a4af-30e5eb46b472: {101} Custom property 'modified' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: marking-definition--455a3fc4-bba5-4ed7-a4af-30e5eb46b472: {111} Open vocabulary value 'TLP' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: marking-definition--455a3fc4-bba5-4ed7-a4af-30e5eb46b472: {201} Marking definition `definition_type` should be one of: statement, tlp.
    [!] Warning: report--f0ca5274-baca-480d-90ea-959454fe161f: {302} External reference 'marcoramilli' has a URL but no hash.
    [!] Warning: marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168: {101} Custom property 'modified' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {101} Custom property 'sophistication' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {101} Custom property 'last_seen' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {211} primary_motivation contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {211} secondary_motivations contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: {212} resource_level contains a value not in the attack-resource-level-ov vocabulary.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {211} primary_motivation contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {211} secondary_motivations contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {211} personal_motivations contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {212} resource_level contains a value not in the attack-resource-level-ov vocabulary.
    [!] Warning: threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: {221} sophistication contains a value not in the threat-actor-sophistication-ov vocabulary.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {101} Custom property 'sophistication' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {101} Custom property 'last_seen' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {211} primary_motivation contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {211} secondary_motivations contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: {212} resource_level contains a value not in the attack-resource-level-ov vocabulary.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {101} Custom property 'sophistication' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {101} Custom property 'last_seen' should have a type that starts with 'x_' followed by a source unique identifier (like a domain name with dots replaced by hyphen), a hyphen and then the name.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {111} Open vocabulary value '' should be all lowercase and use hyphens instead of spaces or underscores as word separators.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {211} primary_motivation contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {211} secondary_motivations contains a value not in the attack-motivation-ov vocabulary.
    [!] Warning: intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: {212} resource_level contains a value not in the attack-resource-level-ov vocabulary.
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: labels: [] is too short
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: created: '2019-06-27T14:24:43Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: modified: '2019-06-27T14:24:43Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: labels: empty arrays are not allowed
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: x_opencti_aliases: empty arrays are not allowed
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: id: 'user--6e09170b-07bd-4807-9b3f-630e6ce216db' does not start with 'identity--'
    [X] user--6e09170b-07bd-4807-9b3f-630e6ce216db: labels: [] is too short
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: labels: [] is too short
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: created: '2019-06-27T14:25:29Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: modified: '2019-06-27T14:25:29Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: labels: empty arrays are not allowed
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: x_opencti_aliases: empty arrays are not allowed
    [X] report--f0ca5274-baca-480d-90ea-959454fe161f: labels: [] is too short
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: labels: [] is too short
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: created: '2017-06-01T00:00:00Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: modified: '2017-06-01T00:00:00Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: labels: empty arrays are not allowed
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: x_opencti_aliases: empty arrays are not allowed
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: id: 'organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5' does not start with 'identity--'
    [X] organization--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5: labels: [] is too short
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: labels: [] is too short
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: created: '2017-12-14T16:46:06Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: modified: '2019-04-24T15:42:29Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: labels: empty arrays are not allowed
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: goals: '' is not of type 'array'
    [X] intrusion-set--4ca1929c-7d64-4aab-b849-badbfc0c760d: secondary_motivations: '' is not of type 'array'
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: labels: [] is too short
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: created: '2019-06-26T11:44:54Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: modified: '2019-06-26T11:44:54Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: labels: empty arrays are not allowed
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: x_opencti_aliases: empty arrays are not allowed
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: id: 'organization--5d12513d-95ce-4257-adb5-82d4732ed5fe' does not start with 'identity--'
    [X] organization--5d12513d-95ce-4257-adb5-82d4732ed5fe: labels: [] is too short
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: labels: [] is too short
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: created: '2019-06-26T15:40:06Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: modified: '2019-06-26T15:40:06Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: labels: empty arrays are not allowed
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: aliases: empty arrays are not allowed
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: labels: [] is too short
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: aliases: [] is too short
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: goals: '' is not of type 'array'
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: secondary_motivations: '' is not of type 'array'
    [X] threat-actor--aa2cf3bb-6621-447d-b5e5-2628c76c1689: personal_motivations: '' is not of type 'array'
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: labels: [] is too short
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: created: '2018-01-16T16:13:52Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: modified: '2018-10-17T00:17:13Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: labels: empty arrays are not allowed
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: aliases: empty arrays are not allowed
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: aliases: [] is too short
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: goals: '' is not of type 'array'
    [X] intrusion-set--68ba94ab-78b8-43e7-83e2-aed3466882c6: secondary_motivations: '' is not of type 'array'
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: labels: [] is too short
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: created: '2018-04-18T17:59:24Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: modified: '2019-04-25T21:34:56Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: labels: empty arrays are not allowed
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: goals: '' is not of type 'array'
    [X] intrusion-set--269e8108-68c6-4f99-b911-14b2e765dec2: secondary_motivations: '' is not of type 'array'
    [X] relationship--d7ca81de-6960-4af8-b0a3-b80498fb5323: created: '2019-06-27T14:27:52Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--d7ca81de-6960-4af8-b0a3-b80498fb5323: modified: '2019-06-27T14:27:52Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--d7ca81de-6960-4af8-b0a3-b80498fb5323: x_opencti_expiration: null properties are not allowed in STIX.
    [X] relationship--417aba3b-78c3-4c81-b7f0-331a5d697919: created: '2019-06-27T14:27:34Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--417aba3b-78c3-4c81-b7f0-331a5d697919: modified: '2019-06-27T14:27:34Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--417aba3b-78c3-4c81-b7f0-331a5d697919: x_opencti_expiration: null properties are not allowed in STIX.
    [X] relationship--6706a345-ac1c-492c-8373-10355a7d9e8c: created: '2019-06-27T14:27:40Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--6706a345-ac1c-492c-8373-10355a7d9e8c: modified: '2019-06-27T14:27:40Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--6706a345-ac1c-492c-8373-10355a7d9e8c: x_opencti_expiration: null properties are not allowed in STIX.
    [X] relationship--55df051a-afb3-4d2d-a6b9-823248c16bba: created: '2019-06-27T14:27:45Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--55df051a-afb3-4d2d-a6b9-823248c16bba: modified: '2019-06-27T14:27:45Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--55df051a-afb3-4d2d-a6b9-823248c16bba: x_opencti_expiration: null properties are not allowed in STIX.
    [X] relationship--3314b99f-c185-498d-941e-e44d9e948a28: created: '2019-06-27T14:27:49Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--3314b99f-c185-498d-941e-e44d9e948a28: modified: '2019-06-27T14:27:49Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--3314b99f-c185-498d-941e-e44d9e948a28: x_opencti_expiration: null properties are not allowed in STIX.
    [X] relationship--37275e0f-5d63-471d-8d8f-d9d71c56e3be: created: '2019-06-27T14:27:55Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--37275e0f-5d63-471d-8d8f-d9d71c56e3be: modified: '2019-06-27T14:27:55Z' does not match the timestamp format YYYY-MM-DDTHH:mm:ss.sssZ (must be precise to the millisecond)
    [X] relationship--37275e0f-5d63-471d-8d8f-d9d71c56e3be: x_opencti_expiration: null properties are not allowed in STIX.

Current Workaround

None.

Proposed Solution

Fix the export to be full compliant with STIX2.

Additional Information

None.

Handle relations creation with first_seen and last_seen

Problem to Solve

The current relation creation if not exists method only support searching approximate equivalent relations.

Current Workaround

None.

Proposed Solution

Split the method to be able to search relations and handle the creation of multiple relations.

Additional Information

None.

Indicators cannot be parsed

Description

When indicators are parsed, an error occurred to the using of a bad stix2 library method.

Environment

All.

Reproducible Steps

None.

Expected Output

Successful parsing.

Actual Output

Error occurred.

Additional information

None.

Pass tags at the creation of entities

Be able to pass the tags at the creation of entities.

Use proper function to update observables

Description

The observable are updated using the function update_stix_domain_entity_field which lead to a GraphQL error.

Environment

OpenCTI 2.0.0

Reproducible Steps

None.

Expected Output

Update of observable.

Actual Output

Error in the result.

Additional information

None.

Handle update of all entities

Problem to Solve

Handle upgrade of all entities (like attack patterns).

Current Workaround

None.

Proposed Solution

Implement the update.

Additional Information

None.

Wheel package

Problem to Solve

pycti is not distributed as wheels, cf https://www.python.org/dev/peps/pep-0427/

Wheels are the new standard of Python distribution. https://pythonwheels.com/

Proposed Solution

python3 setup.py sdist bdist_wheel

Additional Information

very low priority

removing commercial reports.

i am running an offline instance on my OpenCTI platform.

however, during the installation process there was internet connection and many reports were downloaded in the platform. (1000+).

is there any way to quickly delete all of these reports without deleting their artifacts such as (tools, vulnerabilities, malware etc. )

regards

Update the methods to be able to export all entities in STIX2

Problem to Solve

Currently, it is only possible to export reports and threat actors in STIX2 JSON, related to: opencti/67.

Current Workaround

None.

Proposed Solution

Add all the methods.

Additional Information

None.

Add state management in connector helpers

Problem to Solve

Add the state management in connector helper, related to opencti/296.

Current Workaround

None.

Proposed Solution

Implement the state management.

Additional Information

None.

Fix the requirements versions

Problem to Solve

Fix the dependencies versions.

Current Workaround

None.

Proposed Solution

Fix in requirements.txt and setup.py.

Additional Information

None.

Use **kwargs for all API client methods

Problem to Solve

Currently, all methods of the API client are using ordered **args.

Current Workaround

None.

Proposed Solution

Use **kwargs for more understandable methods.

Additional Information

None.

Introduce integration tests

Problem to Solve

No integration tests are available for testing this library.

Current Workaround

Test manually :)

Proposed Solution

Implement a full test coverage of this Python library.

Additional Information

None.

Create tag when importing entities

Please replace every line in curly brackets { like this } with appropriate answers, and remove this line.

Problem to Solve

{ Please describe the problem you would like to solve. }

Current Workaround

{ Please describe how you currently solve or work around this problem, given OpenCTI's limitation. }

Proposed Solution

{ Please describe the solution you would like OpenCTI to provide, to solve the problem above. }

Additional Information

{ Any additional information, including logs or screenshots if you have any. }

Implement files upload (global or in entity)

Please replace every line in curly brackets { like this } with appropriate answers, and remove this line.

Problem to Solve

{ Please describe the problem you would like to solve. }

Current Workaround

{ Please describe how you currently solve or work around this problem, given OpenCTI's limitation. }

Proposed Solution

{ Please describe the solution you would like OpenCTI to provide, to solve the problem above. }

Additional Information

{ Any additional information, including logs or screenshots if you have any. }

Exporting entities containing observables is broken

Description

When processing the export, if observables are in an exception is thrown processing the notes of "observed-data".

Environment

OpenCTI 3.2.0

Implement methods for export/import upload/download

Problem to Solve

The library do not currently handle files upload/download to the GraphQL API.

Current Workaround

None.

Proposed Solution

Implement the methods.

Additional Information

None.

Relations creation error

Description

Relationships cannot be created when using the custom OpenCTI attributes.

Environment

pycti 2.1.5

Add methods to select observables and their relations

Problem to Solve

Currently, there are not methods for getting observables from OpenCTI.

Current Workaround

None.

Proposed Solution

Implement the methods for getting observables and their relations.

Additional Information

None.

Handle observed data as observables

Problem to Solve

Currently, observed_data are not handled by the Python API.

Current Workaround

None.

Proposed Solution

Handle observed data to import observables.

Additional Information

None.

Create a full documentation

Problem to Solve

Currently, there is no documentation on the Python client. It could be useful :)

Current Workaround

None.

Proposed Solution

Write the documentation, with examples and methods reference.

Additional Information

None.

Unable to read Course of Action from opencti via python api

Description

When trying to read a course of action following here is given:

INFO:root:Listing Course-Of-Actions with filters [{"key": "name", "values": ["bash_profile and .bashrc Mitigation"]}].
INFO:root:{"errors":[{"message":"Unknown type "CourseOfActionsFiltering". Did you mean "CoursesOfActionFiltering", "CoursesOfActionFilter", "CoursesOfActionOrdering", "OrganizationsFiltering", or "ThreatActorsFiltering"?","locations":[{"line":2,"column":46}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}},{"message":"Cannot query field "courseOfActions" on type "Query". Did you mean "courseOfAction" or "coursesOfAction"?","locations":[{"line":3,"column":17}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}]}
Traceback (most recent call last):
File "get_coa.py", line 12, in
coa = opencti_api_client.course_of_action.read(filters=[{'key': 'name', 'values': ['bash_profile and .bashrc Mitigation']}])
File "/usr/local/lib/python3.7/dist-packages/pycti/entities/opencti_course_of_action.py", line 184, in read
result = self.list(filters=filters)
File "/usr/local/lib/python3.7/dist-packages/pycti/entities/opencti_course_of_action.py", line 153, in list
return self.opencti.process_multiple(result["data"]["courseOfActions"])
TypeError: 'NoneType' object is not subscriptable

Environment

Opencti demo enviroment, pycti lib 2.1.14, python3

Reproducible Steps

python snipset

from pycti import OpenCTIApiClient

api_url = 'https://demo.opencti.io'
api_token = '609caced-7610-4c84-80b4-f3a380d1939b'
opencti_api_client = OpenCTIApiClient(api_url, api_token,debug)

coa = opencti_api_client.course_of_action.read(filters=[{'key': 'name', 'values': ['bash_profile and .bashrc Mitigation']}])

print(coa)

Unable to add Stix domain entities to the report.

Description

{ We have inserted all the stix entities data and created a report. While we try to add entities to the report we are getting an error }

Environment

OS (where OpenCTI server runs): { Ubuntu 18.04. }
OpenCTI version: { e.g. OpenCTI 2.1.2 }
Other environment details: Using opencti client i.e PyCTI

Reproducible Steps

Steps to create the smallest reproducible scenario:

{Inserted all the stix entities}
{ Created a report }
{ While adding stix entities to the report using pyCTI getting an error. }

Expected Output

{ }

Actual Output

{ Please describe what actually happened. }

Additional information

{ Any additional information, including logs or screenshots if you have any. }

Handle STIX 2 indicators import

Problem to Solve

The indicators are currently ignored during the import of STIX2 bundles.

Current Workaround

None.

Proposed Solution

Implement the import of indicators as observables.

Additional Information

None.

Segmentation Fault (core dumped)

Description

Segmentation fault (core dumped) appears as an error when running the worker command 'pip3 install -r requirements.txt', from when I do a further breakdown it seems like the pip install works for most of the modules but is having challenges with 4 of them.

Environment

OS (where OpenCTI server runs): Ubuntu 18.04
OpenCTI version: 2.0.1
OpenCTI client: frontend
Other environment details: Appears that the issue is related to python

Reproducible Steps

Steps to create the smallest reproducible scenario:

run pip3 install -r requirements.txt under the file path opencti/worker/

Expected Output

A clean exit, no error messages occurring or at least an error log to show where the issue is occurring.

Actual Output

The tail end of the result shows the following:

Installing collected packages: pytz, pika, six, python-dateutil, regex, datefinder, appdirs, certifi, chardet, urllib3, idna, requests, requests-cache, more-itertools, zipp, importlib-metadata, attrs, pyrsistent, setuptools, jsonschema, simplejson, antlr4-python3-runtime, stix2-patterns, colorama, stix2-validator, PyYAML, stix2, pycti
Segmentation fault (core dumped)

Additional information

I was able to breakdown the actual output and do one-by-one of each installation. Below is what did work and what appeared with an error flag:

What worked
	six  
	python-dateutil  
	pytz  
	pika  
	PyYAML  
	chardet  
	urllib3  
	idna
	certifi  
	requests
	regex
	requests  
	antlr4-python3-runtime  
	stix2-patterns 
	pyrsistent
	attrs
	more-itertools
	zipp
	importlib-metadata
	setuptools
	jsonschema
	colorama
	requests-cache
	appdirs

What didnt work
	simplejson  
		Collecting
		Installing collected packages: simplejson
		Successfully installed simplejson-3.16.0
		Segmentation fault (core dumped)
	stix2  
		(Segmentation Fault core dumped)
	stix2-validator 
		(Segmentation Fault core dumped)
	pycti 
		(Segmentation Fault core dumped)

Hope this helps, let me know what further information is required and I'll try provide when available, otherwise i'll keep searching and see what a solution could be

Handle indicators as observables AND indicators

Problem to Solve

Currently, indicators are handled as observables only.

Current Workaround

None.

Proposed Solution

Handle indicators as observables and indicators, linked to opencti/316.

Additional Information

None.

STIX bundle import - Indicator label is not imported

Description

When a STIX 2.0 json bundle with an indicator is imported (from the web UI), the label of the indicator is lost.

Environment

import_stix2_file.py on https://demo.opencti.io/ instance

Reproducible Steps

Import this bundle (exported from https://exchange.xforce.ibmcloud.com/malware/f246984193c927414e543d936d1fb643a2dff77b )

{
   "spec_version":"2.0",
   "type":"bundle",
   "objects":[
      {
         "id":"indicator--20f5deda-e4a1-5f3f-2426-f509d568369d",
         "type":"indicator",
         "created":"2020-05-06T06:32:26.114Z",
         "modified":"2020-05-06T06:32:26.114Z",
         "labels":[
            "xfe-malware-risk-high"
         ],
         "name":"File hash indicator for sha1 hash f246984193c927414e543d936d1fb643a2dff77b",
         "description":"File hash indicator for sha1 hash f246984193c927414e543d936d1fb643a2dff77b",
         "pattern":"[ file:hashes.'SHA-1' = 'f246984193c927414e543d936d1fb643a2dff77b' ]",
         "valid_from":"2020-05-06T06:32:26.114Z"
      }
   ],
   "id":"bundle--325daaa3-e024-460a-882f-1a6913adb65b"
}

Expected Output

The tag xfe-malware-risk-high is present in the web UI.

Actual Output

The tag is not present. See https://demo.opencti.io/dashboard/signatures/indicators/0e4d8771-6a26-45b3-a74e-3943429bea83

Add a new class for connector helper

Problem to Solve

The new connector model is standalone and will use methods of the OpenCTI Python library to send data to the RabbitMQ.

Current Workaround

None.

Proposed Solution

Implement a ConnectorHelper class.

Additional Information

None.

Importing report JSON bundle with files (x_opencti_files) fails

Description

Importing report JSON bundle with file (x_opencti_files) fails. The file is not uploaded to OpenCTI.

Environment

OS: Ubuntu 18.04.3 LTS
OpenCTI version: 2.1.3
client-python: 2.1.12

Reproducible Steps

Importing report JSON bundle with x_opencti_files custom properties fails. The report is added to the OpenCTI but the PDF file is missing.

Expected Output

The report with the PDF file is imported to OpenCTI

Actual Output

The is an error message in the log:

[opencti_stix_domain_entity] Missing parameters: id or file_name or data

Additional information

I suspect that the add_file method in /pycti/utils/opencti_stix2.py (lines 516-520) is missing id=stix_object_result["id"].