mlsecproject / gglsbl-rest Goto Github PK

I'm seeing this error in my logs... i'm looping through a large list & running it against gglsbl-rest.

mlsecproject/gglsbl-rest:latest-----44d7af4aa467 [2019-10-02 19:30:18,529] ERROR in app: exception handling [http://yoursetforupgradealways.site/] mlsecproject/gglsbl-rest:latest-----44d7af4aa467 Traceback (most recent call last): mlsecproject/gglsbl-rest:latest-----44d7af4aa467 File "/home/gglsbl/app.py", line 29, in _lookup mlsecproject/gglsbl-rest:latest-----44d7af4aa467 return sbl.lookup_url(url) mlsecproject/gglsbl-rest:latest-----44d7af4aa467 File "/usr/lib/python2.7/site-packages/gglsbl/client.py", line 120, in lookup_url mlsecproject/gglsbl-rest:latest-----44d7af4aa467 list_names = self._lookup_hashes(url_hashes) mlsecproject/gglsbl-rest:latest-----44d7af4aa467 File "/usr/lib/python2.7/site-packages/gglsbl/client.py", line 166, in _lookup_hashes mlsecproject/gglsbl-rest:latest-----44d7af4aa467 self._sync_full_hashes(matching_prefixes.keys()) mlsecproject/gglsbl-rest:latest-----44d7af4aa467 File "/usr/lib/python2.7/site-packages/gglsbl/client.py", line 107, in _sync_full_hashes mlsecproject/gglsbl-rest:latest-----44d7af4aa467 self.storage.store_full_hash(threat_list, hash_value, cache_duration, malware_threat_type) mlsecproject/gglsbl-rest:latest-----44d7af4aa467 File "/usr/lib/python2.7/site-packages/gglsbl/storage.py", line 198, in store_full_hash mlsecproject/gglsbl-rest:latest-----44d7af4aa467 dbc.execute(qi, i_parameters) mlsecproject/gglsbl-rest:latest-----44d7af4aa467 OperationalError: database is locked

@asieira would you mind explaining the configuration settings you've used to run gglsbl-rest on ECS? For example:

• Are you using EC2 or Fargate?
• If EC2, what instance types?
• If EC2, how many tasks on each instance?

Any other information you think would be helpful to get started such as autoscaling settings would be great. ECS is pretty new to me.

Thanks in advance!

Hi, love the project. I integrated it into some of our tools and wrote the following for interacting with it programmatically and from the cli.
https://github.com/seanmcfeely/gglsbl-rest-client

Also, I'm using your service with the above python lib as an analysis module here -> https://github.com/ace-ecosystem/ACE/blob/d49fc92dd715b96a9b760b740e3d6c7a74740b97/lib/saq/modules/url.py#L61

As suggested by @neonknight in #1, related to afilipovich/gglsbl#27.

Use https://github.com/arminc/clair-local-scan to scan container for vulnerabilities via CI integration with CircleCI or a similar service.

As described in afilipovich/gglsbl#28.

Increase performance by reusing SafeBrowsingList object across requests when possible.

Since Docker changed its policies, update the repo to use CircleCI or GitHub actions to publish the built images to a public ECR repository.

When running the docker command as given in the README,

docker run -e GSB_API_KEY= mlsecproject/gglsbl-rest -p 127.0.0.1:5000:5000 -i

I got "connection refused" when trying to hit the service. Starting with this command worked properly:

docker run -e GSB_API_KEY= -p 127.0.0.1:5000:5000 -i mlsecproject/gglsbl-rest

Running Ubuntu 16.04 with Docker 17.09.0-ce

Using Image ID 4564f0f5fc40 per mlsecproject/gglsbl-rest:latest

$ docker run -e GSB_API_KEY=$APIKEY -p 127.0.0.1:5000:5000 mlsecproject/gglsbl-rest 
2019-01-25T22:25:22+0000 pid=1 glogging INFO Starting gunicorn 19.9.0
2019-01-25T22:25:22+0000 pid=1 config INFO Initial database load...
2019-01-25T22:25:22+0000 pid=1 config INFO Update started as PID 7
2019-01-25T22:25:22+0000 pid=7 update INFO opening database at /home/gglsbl/db/sqlite.db
Traceback (most recent call last):
  File "update.py", line 32, in <module>
    update_hash_prefix_cache()
  File "/usr/lib/python3.6/site-packages/pid/decorator.py", line 14, in decorator
    return func(*func_args, **func_kwargs)
  File "update.py", line 17, in update_hash_prefix_cache
    sbl = SafeBrowsingList(gsb_api_key, dbfile, True)
  File "/usr/lib/python3.6/site-packages/gglsbl/client.py", line 34, in __init__
    self.api_client = SafeBrowsingApiClient(api_key, discard_fair_use_policy=discard_fair_use_policy)
  File "/usr/lib/python3.6/site-packages/gglsbl/protocol.py", line 81, in __init__
    self.service = build('safebrowsing', 'v4', developerKey=developer_key, cache_discovery=False)
  File "/usr/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/googleapiclient/discovery.py", line 222, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey)
  File "/usr/lib/python3.6/site-packages/googleapiclient/discovery.py", line 272, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1924, in request
    cachekey,
  File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request
    conn, request_uri, method, body, headers
  File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request
    conn.connect()
  File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1356, in connect
    raise socket_err
  File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1289, in connect
    sock.connect((self.host, self.port))
  File "/usr/lib/python3.6/site-packages/httplib2/socks.py", line 501, in connect
    _orgsocket.connect(self, (self.__proxy[1], portnum))
ConnectionRefusedError: [Errno 111] Connection refused
2019-01-25T22:25:22+0000 pid=1 config INFO Update process finished with status code 1