Reporting module error - mongodb binary object has no get about capev2 HOT 25 CLOSED

Working bleeding edge! Something rare nowadays

from capev2.

I've run into this issue as well. A regression was introduced when dev_utils/mongodb.py was split into:

• dev_utils/mongodb.py
• dev_utils/mongo_hooks.py

I reverted these 4 commits from my fork to get it working again:

• f631f67
• 60bcb00
• 9f72942
• 0f42097

My fork is based on: 8be0cc5

The bug is in one of the above 4 commits - 100% sure.

cc: @tbeadle

from capev2.

PS as curiosity I have dropped my mongodb, git clone master branch and did a fresh analysis of 1 tasks to not have any previous data in my cape to simulate fresh install. 0 issues, so for start of digging into your problems I would suggest to start doing diff of configs with repo and see what generate your problem and once again always try master before open issues

from capev2.

@ethhart yes that will be removed

from capev2.

hello, can you share hash to try to reproduce that?

from capev2.

wow, that's a quick reply :D
many thanks!

Which hash do you mean?

from capev2.

file that generate that problem, in your case task n5, otherwise we can't help if we can't reproduce, md5/sha256 hash works, you can attach file zipped with password infected if you want

from capev2.

Sha256: 0490ceace858ff7949b90ab4acf4867878815d2557089c179c9971b2dd0918b9
0490ceace858ff7949b90ab4acf4867878815d2557089c179c9971b2dd0918b9.zip

from capev2.

i can't reproduce your error so i have no idea what is wrong on your side

from capev2.

Thanks for looking into it.
I gonna retry with a fresh setup and report back

from capev2.

do you frequently doing git pull and restart of required services?

from capev2.

no not at all
the sandbox is still undergoing setup and has always been at git-commit

`/opt/CAPEv2$ git log
commit 3987ce5 (HEAD -> master, origin/master, origin/HEAD)
Author: doomedraven
Date: Fri Nov 24 09:05:12 2023 +0100

sync`

The services needed to be restarted a couple of times, but no git pull at all.

from capev2.

well git pull is important always before open any issue as your issue + lying in questionary is bad start [ x] I am running the latest version

from capev2.

sorry about that, didn't thought that the version would get old so fast.

from capev2.

Bleeding edge sandbox 😎

from capev2.

Personally, I think a little bit more quality control on commits generally speaking would be nice ... finding a "working commit" to install off of is not fun. Neither is figuring out what to revert to make it work.

from capev2.

Please take that attitude elsewhere. This is my life's work and reading comments like this after I've spent the last decade working hard on this project for free for others disgusts me quite frankly.

from capev2.

Buddy, I didn't come here to throw shade on your life's work or act unappreciative of all the hard work you've put into this project. Take emotions aside for a second and focus on the objective facts.

1. If the above bug is a bug, it means that anybody who's tried to use your work for since Sept 19th hasn't been able to benefit from any of the subsequent value-add commits on new and updated signatures you've made since - simply because your project won't build correctly OR if it did build correctly, it then crapped out as soon as you ran an analysis on ANY piece of malware. Isn't the fact that you guys didn't know about this until now a concern at all?
2. It's easy to sit on the sidelines and criticize someone else's work, but I'm not doing that ... I spent 2 hours debugging, compiling and testing a fix I then submitted into your cape2.sh logic 2 weeks ago because of change that appears went in entirely untested. I've personally done over 75+ builds of your code in the last 2 months.
3. No matter how good a project is there is ALWAYS room for improvement. You got 2 comments above saying "hey, we're having a hard time using this thing ... can we do something about it?" Not asking for more than a build job that runs once a week or something to make sure this thing can install successfully and run an analysis without crapping out along
   the way. In fact, one might already exist and I don't know about it ... and there may be a bug there.

I think this project is awesome, the work that is going into it is awesome and I'm simply asking for help from you guys to make it a bit more stable. Imagine this open source project was git and from one week to the next you lost the ability to commit code and it didn't get fixed for 3 months ... It's precisely because I find that what you guys are doing here valuable and awesome that I'm saying all this.

If you still find all this disgusting, I'm sorry to hear that ... but again ... no one came here to hurt anyones feelings or diminish/trivialize the work this project represents.

from capev2.

invoking me from PTO...

If the above bug is a bug, it means that anybody who's tried to use your work for since Sept 19th hasn't been able to benefit from any of the subsequent value-add commits on new and updated signatures you've made since - simply because your project won't build correctly OR if it did build correctly, it then crapped out as soon as you ran an analysis on ANY piece of malware. Isn't the fact that you guys didn't know about this until now a concern at all?

• That is so FALSE dude, first go and see my initial answers, we can't reproduce that bug as I started to investigate that directly, we have a private slack with people from the world biggest IT companies, I personally own huge cluster on GCP, and 0 problems, running on latest always. So before coming and said that something is wrong and you need to search working commit is a bit funny, instead of search a working commit I would suggest you to review your setup as it sound mostly like you messed with it. Nobody from advanced users have those problems, we tried to help to newcomer(who opened the issue), but if we can't reproduce it the same that issue doesn't exist aka user messed with his setup.

• About value if you would be one of the advanced user you would see how much value it has. But each user case is different and is impossible to cover all cases.

TL;DR: investigate your setup, the project works just fine if you have proper skills. Found what is wrong? provide as much details as you can to help us reproduce and fix anything or add warning for those who mess due to improper usage.

Anyway detaching for the PTO again.

from capev2.

First off, not my intention to detract from your PTO ... sorry that it has ...

I can appreciate you've tried to reproduce this bug, but just because you haven't been successful doesn't mean it's not there. I reproduced it consistently at least 5 times with installations from vanilla Ubuntu 22.04.3 LTS with latest updates. I've also automated the entire install, config, and launch process as per the instructions listed here: https://capev2.readthedocs.io/en/latest/installation/index.html using a combination of Terraform, Packer and Ansible so the "human error" is mostly out.

I haven't done anything advanced other than run the following on a single VM:

1. install kvm-qemu.sh all cape
2. install kvm-qemu.sh virtmanager cape
3. install cape2.sh all cape
4. poetry install
5. load 3 analysis vms with different versions of windows (w/ cape-agent installed)
6. systemctl restart cape*

I will give the "try latest master approach" another try if you're confident it's my setup and do another pass at verifying that I've followed all the steps in readthedocs.io ... in the process of automating things, I was very thorough ...

P.S. I have not modified my configs much w.r.t. mongodb other than enabling it in reporting.conf and enabling some of the report generations (i.e. litereport, reportsummary w/ screenshots, etc...). Why that would cause cape-processor.service to crap out accessing Mongo during the processing of an analysis run is beyond me.
P.S.S. The exact config of reporting.conf (everything else is default):

    - {section: 'compressresults', key: 'enabled', value: 'yes'}
    - {section: 'litereport', key: 'enabled', value: 'yes'}
    - {section: 'mongodb', key: 'enabled', value: 'yes'}
    - {section: 'reporthtml', key: 'enabled', value: 'yes'}
    - {section: 'reporthtml', key: 'screenshots', value: 'yes'}
    - {section: 'reporthtml', key: 'apicalls', value: 'yes'}
    - {section: 'reporthtmlsummary', key: 'enabled', value: 'yes'}
    - {section: 'reporthtmlsummary', key: 'screenshots', value: 'yes'}

P.S.S.S. Can you confirm that you're running Mongo 7 as per: 6607bff? I doubt this is it .. but this upgrade from 6.0->7.0 also happened in the last 3 months ... just leaving no stone unturned.

from capev2.

i would say your problem is here compressresults disable that and you should not suffer this, i personally want to nuke compressresults and retention modules. Do you know what compressresults does? it make your cape useless :D

its not mongo 6 or 7, as it not reaches there, is compressresults i would say. idk why you enable that, it makes cape useless you can't search later nothing at all.

from capev2.

confirmed

def run(self, results):

        for keyword in ("CAPE", "procdump"):
            if keyword in results:
                try:
                    cape_json = json.dumps(results[keyword], ensure_ascii=False).encode()
                    compressed_data = zlib.compress(cape_json)
                    results[keyword] = Binary(compressed_data) <- your problem here

so about "asking" for better quality commits. as you might see it not our problem, yes CAPE has a lot of modules. We don't use all of them, some are there as some users might have weird user cases or not understand what those modules does

and as you might see in configuration it marked as community module, which means we do not maintain this module

# Community
# Compress results including CAPE output
# to help avoid reaching the hard 16MB MongoDB limit.
[compressresults]
enabled = no

nuked compressresults from cape, git pull + systemctl restart cape-processor and you never will suffer this issue again.

So once again not just for you, but for everyone who comes over this, start with minimal config changes, if you have no clue what module does, DISABLE it as it probably will give you just more problems than benefits

from capev2.

Do we want to remove references to compressresults in compare.py and analysis/views.py?
I don't see any references in community to keep this logic in tact.

from capev2.

Thanks for following up on this @doomedraven - at least we're all on the same page now.

Is it accurate to say that one of the 4 commits I referenced above introduced a side-effect bug in the compressresults feature? As I mentioned earlier, when I backed out the commits my setup resumed working as it had done before.

w.r.t.

so about "asking" for better quality commits. as you might see it not our problem, yes CAPE has a lot of modules. We don't use all of them, some are there as some users might have weird user cases or not understand what those modules does

My asking for better quality commits is not entirely based on this bug - which was effectively a result from a sizeable database change that impacted various parts of the codebase, had an unintended side-effect on an "unsupported feature" (i.e. compressresults), and went in with no evidence of any automated testing (i.e. a passing build job) or anything of the sort. Again, no clue what you guys do in the backend when vetting these changes (manual test?hidden automated tests?) ... the whole process is opaque from where I stand ... a lot of projects use github-actions automation for stuff like this. HOWEVER, my ask also comes on the heels of this fix that I put in a few weeks ago: #1850 ... you guys didn't catch it, I did, and I fixed it ... and I followed "use the latest master" branch which was broken at the time. Still took 2+ hours of trying to figure why my APT packages were all messed up and trace it up to a typo. We are all human, we all make mistakes ... not a big deal .. but things can be better with more testing and automation. That's all I'm going to say on this topic here as it's not related to this issue. I have 15+ years of dev/security experience from Fortune 500 companies and I given you my feedback - what you do with it from this point on is up to you.

from capev2.

well you can call it side effect if you want, but its a community module, as i told, we do not maintain those modules, so for us is not side effect, is that community doesn't update their modules to align with core. That would be a proper sentence.

We don't test and I personally don't care about maintaining community modules as it looks i need to say that lauder and harder.

Dude i spend years doing automation of those scripts and any small change in 3rd part software breaks it, when i push changes to those scripts is when im doing deployment and it works, it can break in any moment and i mention that in README, that IS NOT A SILVER BULLET.

now to the core msg of this all

1. is software it changes = it might break
2. there is no paid support and done by people in their free time, spot bug, help us to fix, we can't cover all user cases
3. want a better core, help us to improve, that is more than welcome.
4. using community modules = it can be broken, it's not our problem is mentioned in readme first line too

but all this just sound like you complain about open source project where nobody gets moneys for doing that and safe time to others, its open source, help to improve or make your fork and have fun, is how open source works right?

from capev2.