What would it take to run a nightly job to update the remote rule sets?
Adding sources to the remote rules makes the unit_test fail.
/opt/binaryalert/rules/clone_rules.py
REMOTE_RULE_SOURCES = {
'https://github.com/Neo23x0/signature-base.git': ['yara'],
'https://github.com/YARA-Rules/rules.git': ['CVE_Rules'],
'https://github.com/SupportIntelligence/Icewater.git': ['']
}
$ ./manage.py unit_test
.........................................................................F
======================================================================
FAIL: test_update_rules (tests.rules.update_rules_test.UpdateRulesTest)
Verify which rules files were saved and deleted.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
return func(*args, **keywargs)
File "/opt/binaryalert/tests/rules/update_rules_test.py", line 52, in test_update_rules
self.assertEqual(expected_files, set(compile_rules._find_yara_files()))
AssertionError: Items in the second set but not the first:
'github.com/SupportIntelligence/Icewater.git/CVE_Rules/cloned.yara'
----------------------------------------------------------------------
Ran 74 tests in 18.957s
FAILED (failures=1)
TEST FAILED: Unit tests failed
Need a way to make sure all of the python libraries are available for the rules.
/opt/binaryalert/rules/clone_rules.py
REMOTE_RULE_SOURCES = {
'https://github.com/Neo23x0/signature-base.git': ['yara'],
'https://github.com/YARA-Rules/rules.git': [''],
'https://github.com/SupportIntelligence/Icewater.git': ['']
}
$ ./manage.py compile_rules
Traceback (most recent call last):
File "./manage.py", line 495, in <module>
main()
File "./manage.py", line 491, in main
manager.run(args.command)
File "./manage.py", line 352, in run
getattr(self, command)() # Command validation already happened in the ArgumentParser.
File "./manage.py", line 421, in compile_rules
compile_rules.compile_rules(COMPILED_RULES_FILENAME)
File "/opt/binaryalert/rules/compile_rules.py", line 36, in compile_rules
externals={'extension': '', 'filename': '', 'filepath': '', 'filetype': ''})
yara.SyntaxError: ./Mobile_Malware/Android_FakeApps.yar(101): invalid field name "app_name"
Would it be better to remove the rules or install the missing python libraries?
/opt/binaryalert/rules/compile_rules.py
for line in yara_filepaths:
try:
test = yara.compile(RULES_DIR+'/'+line)
except:
os.remove(RULES_DIR+'/'+line)
yara_filepaths = {relative_path: os.path.join(RULES_DIR, relative_path)
for relative_path in _find_yara_files()}
Compile requires enough memory to complete. These rules required a t2.small to build.
$ ./manage.py compile_rules
Traceback (most recent call last):
File "./manage.py", line 495, in <module>
main()
File "./manage.py", line 491, in main
manager.run(args.command)
File "./manage.py", line 352, in run
getattr(self, command)() # Command validation already happened in the ArgumentParser.
File "./manage.py", line 421, in compile_rules
compile_rules.compile_rules(COMPILED_RULES_FILENAME)
File "/opt/binaryalert/rules/compile_rules.py", line 45, in compile_rules
externals={'extension': '', 'filename': '', 'filepath': '', 'filetype': ''})
MemoryError
Only a certain number of rules can apply before receiving this error.
$ ./manage.py apply
Traceback (most recent call last):
File "./manage.py", line 495, in <module>
main()
File "./manage.py", line 491, in main
manager.run(args.command)
File "./manage.py", line 352, in run
getattr(self, command)() # Command validation already happened in the ArgumentParser.
File "./manage.py", line 382, in apply
subprocess.check_call(['terraform', 'apply', '-auto-approve=false'])
File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve=false']' returned non-zero exit status 1