I accidentally related a pcap file to itself using some of the new pcap features.

I have a PCAP with a single TCP session in it. I ran metacap on that PCAP file and then clicked the "carve flow" link in the results. The end result is that the original PCAP file now has an extra source instance and also a relationship back to itself.

We should not be doing anything if the parent object is the same as the child object. I know Mike is doing a bunch of work on relationships right now so we can tackle this once he's done with it.

I'm noticing some of the mapreduces not behaving properly. For example, the output of the mapreduce for campaign_date_stats() does not jive with this aggregate query:

db.email.aggregate([
    { $unwind: "$campaign" },
    { $project: { campaign: "$campaign.name" } },
    { $group: { _id: { campaign: "$campaign" }, number: { $sum: 1 } } },
    { $sort: { "_id.campaign": 1 } }
])

We've changed a lot of code and the mapreduces are still very archaic and messy. Might be worth looking them over and cleaning them up.

Sometimes Indicators can be represented as regular expressions (ex: *.foo.com). This shouldn't be taken literally, but more anything matching that regex applies as an Indicator.

Is this something we want to support? If so how should we go about supporting it? How does this affect use with the structured data exchange formats?

Currently MD5 links do a search. Is there a way to provide another link (maybe an icon next to it?) that allows you to do a direct link to the Details page for that MD5 (if the MD5 doesn't exist or you don't have access to it you'll get an error page).

When a service is imported into the system as the web server starts, run some diagnostics on it. This should do some basic checks for common flaws and catch other issues. These should be available in the Services Control Panel so they can be reviewed by an admin/developer for debugging.

There's a couple tables that have still not been converted to jTable.

jTable is suffering from the "try too hard to not duplicate code" issue. The functions are in core and are trying to do everything for everything and it's making the code impossible to manage, understand, and modify.

We should break this code up and put proper functions in each of the handlers that correspond with the objects that are being searched on. This will remove all of the if statements and random hacks that are thrown into the mix to handle special conditions for random data. Then for searches that are truly global in scope, they can leverage all of the handlers across the board in a clean manner.

We may have some code duplication, but to be honest, if it makes things clean, readable, maintainable, and scalable, it is worth it.

Maybe make a jTable class which has a ton of helper functions which can assist in generating the content?

Similar to how we allow Samples to be uploaded as "metadata-only" (we only require an MD5 and a filename) we can do the same with PCAPs.

A common use for this is for people with very large PCAPs stores. They can use the filename field as a way to reference the location/name of the PCAP on disk and the MD5. This will allow people to create relationships within CRITs without having to upload the PCAP.

Future services can then utilize this information so if they need to run against the PCAP they know where to find it.

Write a command-line tool that will allow for dumping/restoring content in CRITs.

It should be able to handle multiple formats including raw mongodumps.

This isn't horrible, but they should be added as datetime objects to be consistent and allow for any potential searching/sorting people wish to do in their custom queries.

Would it make sense to provide a context menu for the UI? Features that come to mind:

• Create Indicator from highlighted value
• Copy ObjectId to clipboard
• Add New ->
  • Relationship
  • Campaign
  • etc
• Delete (admin only)
• Refresh Component
• Run Service

This is geared specifically towards touch devices.

As services become more popular, the amount of data generated for a given document can be overwhelming. In some cases this can lead to maxing out the document at 16MB. This prevents that document from being updated further.

Would it make sense to move analysis results to their own collection with a reference back to the document it is for so they can be retrieved in the future?

This will allow for cleanup without having to manually query the database.

If a session times out and an analyst uses a dialog that's still on the screen, it will look like it's broken when in reality it's because they're not logged in.

We should detect these situations and either redirect the user to the login page, or throw up an error letting them know their session has expired.

Bonus would be if when the session expires, pop up a modal letting the user know and when they acknowledge it, it will send them back to the login screen.

It currently uses its own table but should be moved over to jTable for consistency/usefulness.

Things like Yarahits, the Yara mapreduce, etc. should all get moved to the service. This code doesn't rely on having yara installed as a dependency, but is wasted code if the yara service isn't being used.

This would be a name used in the UI that is nicer than the one currently used which doesn't allow or spaces, etc.

When visiting CRITs, the top bar, bottom bar, nav menu, and search should all stay on the page and their content should be altered dynamically as things in the system change.

The "main content" of the page should be the thing that dynamically changes as you interact with the system. For example:

1. Browse to a Sample
   * Only the main content should dynamically update to show the Details page for the sample.
2. Browse from Sample to Indicator
   * Only the main content should dynamically update to show the Details page for the indicator.

This can be used to also work with dialogs such that they are only loaded once (upon use like they are now) and will stay loaded for the duration of the window/tab/session.

A little helper link that can allow people to get more information about that CVE.

If a service is disabled, the tabs and/or navigation menu items it adds should also not be available.

Also, if they are enabled, they should not be added as buttons to the horizontal Analysis menu if they don't have that type of Analysis service available.

Currently it is a list of StringFields which give the potential for malformed email addresses. Those can have issues when trying to generate Targets off of the email addresses.

This opens up other issues with validating each address and editing them inline in the interface.

It would be useful to allow for screenshots to be uploaded to CRITs. The screenshots would be stored in GridFS and referenced accordingly in their own collection.

Screenshots should have the ability to be tagged with common terms (similar to the bucket list).

Screenshots should be able to be referenced from top-level objects by their ObjectId or tags. For example, if someone has a sample and they use the bucket list to tag "foo", then we might want to recommend any screenshots that have also been tagged with "foo". If the screenshot applies, the analyst can make an official reference to the screenshot.

Screenshots should have their own piece to the interface that is common across all top-level objects (think template widget).

Screenshots shouldn't be top-level specific. For example I can have a screenshot of a popup box and I should be able to reference it in a Sample, Email, PCAP, etc.

Screenshots should also have their own browsing/listing page, be searchable, etc.

If a date is missing in the database, the MongoEngine class will sometimes default it to datetime.datetime.now() (if it is a required field and this is the default).

When rendered in jTable, it gives the false impression that the date for that entry in the database is the current datetime.

Dates should all be converted to UTC before being worked on or stored in the database. They should then be converted for use in the UI. This solves several issues with the fact that MongoDB doesn't understand timezones.

We should have a way to manage how many parallel processes CRITs will spawn when executing services for samples. Currently it will spawn as many as necessary to complete the tasks that are required for all of the uploads. This could get pretty nasty.

This will allow for rapid development and iteration over what this tab can do without tying it to a core release.

Other top-level objects such as Events, IPs, and Domains could benefit from logging Activity.

When executing a queryset, mongoengine will throw an exception if any document in the result does not validate. This could occur because of a bad relationship value, specifically older documents that do not have an objectid in the value field. When the exception is thrown the error message is not super helpful because it tells you the relationship value is an invalid objectid but doesn't tell you which document triggered the exception. This also prevents upgrades from occurring and a number of other situations that can be difficult to troubleshoot.

In talking this over with others it seems like it would be useful to have a command line script that could validate all documents in a collection and log those that are not valid. Optionally this script could move or fix said documents if possible as a remediation.

We should look into MongoEngine signals to see if we can leverage them instead of hand-rolling replacement methods for things like .save() and .delete().

Might make sense for some views of data (like the Recent Activity page) to only show a subset of data and either:

1. Give a link for "More..." to get more data, or
2. Detect scrolling down and fetch new data to render while scrolling is going on.

This adds consistency and prevents a lot of issues that are the results of jamming malformed strings into URLs without proper encoding.

If the service was enabled it should remain enabled after the upgrade unless the configuration requirements have changed.

MongoDB cannot delete content if a value in the index exceeds the max size. This is commonly a problem for people who add exceedingly large value Indicators (greater than 1012 characters in length).

This can be leveraged in many more places throughout the UI to make things more standard and easy for analysts to make quick changes.

Form input should be validated (combination of client and server side).

Examples:

• If a user needs to enter an IP address we should validate the format of the value before accepting it.
• Dates and times should be entered in a format we can convert. If we cannot convert it we should request the date be entered correctly.

We can probably make our own MongoEngine fields, or extend the current ones to do some sort of validation/transformation. If things don't validate we can return the message to the user so they can fix it before submitting again.

Allow for propagating content changes across related items. For example:

1. You have Domain "www.foo.com".
2. You have Indicator "www.foo.com".
3. There is a relationship between the Domain and the Indicator.
4. You add a Campaign to the Domain.
5. The Indicator should reflect attribution to the same Campaign.

This could be as simple as adding a checkbox to the forms to "Propagate to relationships" that does the work. Need to take care to ensure that the propagation doesn't go on forever (only 1st level of relationships).

Instead of only finding top-level objects which also have an Object with the same value, expand the search to also find that value in other places:

• Analysis results
• Top-level object metadata values like hashes, descriptions, etc.

Another developer tested out implementing Django Compressor to reduce the amount of requests. It requires a few more dependencies (six,django-appconf,BeautifulSoup...not sure if BeautifulSoup can be removed or not), but was able to reduce the amount of HTTP requests on the dashboard page by 50% (42 down to 21). Might be something to look into?

Before:

After:

A topic that has been brought up is being able to store associated data markings (like TLP designation) with content within CRITs.

This would involve adding a common interface to CRITs to allow adding data markings to any top-level objects. I'm not sure if the best solution for this would be to add this as a core feature, or as a service tab. There's pros and cons to both.

This feature being a service would allow everyone to enhance the data marking capabilities much faster than waiting for a new release would allow. The problem with being a service is that once standards bindings and APIs support data markings, we would need to write code in core to support importing/exporting that data. That sort of breaks our model of core not relying on a service to function.

The other thing to consider is that markings can be source-specific, throwing a bit of a wrench in how we organize this data within the document structures.

Add the ability to bulk add relationships similar to how we can bulk upload Objects.

Provide some type of management command or script which will accept emails and parse them into emails in CRITs. Would need to figure out if we want the emails to show up as attachments or if simply forwarding the email will do.

Should come with documentation about how to setup/configure/run/etc.

To reproduce:

1. Find (or create) a service with a MULTI_SELECT config option, which has a non-empty default value.
2. Reset the service's config
3. Try to edit the service's config.
4. Notice that "resetting" the config did not properly set the checkboxes to the default values and that there is a form validation error:

Select a valid choice. <XXXXXX> is not one of the available choices.

After fixing this once, there aren't any more validation errors (only the first time after resetting the config).

When adding things like a child sample, we should carry over things like the sources, Campaigns, backdoors, exploits, etc.

This should happen anywhere that makes sense. We only do this in some cases currently.

CRITs currently prevents a user from seeing any content from a source they do not have access to.

Another way to go about this would be allowing users to see all content, but if they do not have access to the source(s) that provided the information, it gets lumped into the "Other" category. This ensures they can work with all of the data in the system (sometimes very necessary in a SOC-type environment) but have no way to attribute the data to a specific source.

Is this something we can expose in the Control Panel? Should it be a global option or specific to certain sources (NDA agreements tend to make the global option a pain)?

Samples can be observed in the wild with different filenames. Currently we only track and store one filename.

It would be nice to track the filename of the sample for every instance of that file being observed. That would include receiving the file from another user/organization and what filename they provided.

A robust wiki page outlining:

• How the code is laid out.
• Where to look for certain things.
• How to compile documentation with Sphinx for web-based code browsing.
• Stylistic requirements for developing code for CRITs.

When looking at the details of an item, if you add to it, eg like objects or relationships, you will dynamically see the table updated.

However if you navigate forward and then back again, you might have the impression that your changes didn't take effect because you will be shown the default html cached from the previous "whole page" load, and no longer see the dynamically updated sections.

If a service is upgraded or downgraded, we need better handling of how the service reacts. This mainly revolves around config comparison and how to handle differences gracefully.

This is an artifact of CybOX having a single relationship in one direction matching to several relationships in the reverse direction.