🐛 Summary

When a snapshot or report fails in create_snapshots_reports_scorecard.py, the script attempts to re-generate the snapshot/report (see #119 where this was implemented). However, the durations for both the failed snapshot/report and the successful snapshot/report are both included in the logging output. Only the successful duration is desired in the output.

To reproduce

Steps to reproduce the behavior:

1. Run extras/create_snapshots_reports_scorecard.py when one or more snapshots or reports fail initially, then succeed during their re-try attempt(s).

Expected behavior

When snapshots or reports fail, only the duration of the successful attempt should appear in the logging output. It should not display the duration of the failed attempt.

Any helpful log output or screenshots

<snip>
2024-04-14 17:16:01,209 INFO - Attempting to regenerate failed third-party reports: [u'CCC', u'BBB']
2024-04-14 17:16:01,209 INFO - [MainThread] Starting third-party report for: CCC
2024-04-14 17:21:47,155 INFO - [MainThread] Successful third-party report generated: CCC (345.95 s)
2024-04-14 17:21:47,155 INFO - [MainThread] Starting third-party report for: BBB
2024-04-14 17:30:20,685 INFO - [MainThread] Successful third-party report generated: BBB (513.53 s)
<snip>
2024-04-14 17:33:11,721 INFO - Longest third-party reports:
2024-04-14 17:33:11,721 INFO -   AAA: 521.3 seconds (8.7 minutes)
2024-04-14 17:33:11,721 INFO -   BBB: 513.5 seconds (8.6 minutes)
2024-04-14 17:33:11,721 INFO -   BBB: 483.4 seconds (8.1 minutes)
2024-04-14 17:33:11,721 INFO -   CCC: 345.9 seconds (5.8 minutes)
2024-04-14 17:33:11,721 INFO -   CCC: 330.2 seconds (5.5 minutes)
2024-04-14 17:33:11,721 INFO -   DDD: 187.7 seconds (3.1 minutes)
<snip>

🐛 Summary

If a CyHy customer report includes a finding with a very long description, the report will fail to generate.

To reproduce

Attempt to generate a CyHy report for an entity with an open ticket for a finding with a very long description (e.g. this one, whose description is currently 51,654 bytes): cyhy-report --cyhy-section cyhy --scan-section scan ENTITY

Note failure output:

Traceback (most recent call last):
  File "/usr/local/bin/cyhy-report", line 11, in <module>
    generate_report.main()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 3643, in main
    was_encrypted, results = generator.generate_report()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 348, in generate_report
    self.__generate_final_pdf()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 3551, in __generate_final_pdf
    assert return_code == 0, "xelatex pass 1 of 3 return code was %s" % return_code
AssertionError: xelatex pass 1 of 3 return code was 1

Expected behavior

The CyHy report PDF should be generated normally.

Any helpful log output or screenshots

XeLaTex output after failure:

Underfull \hbox (badness 1189) in paragraph at lines 2968--2968
[]|\TU/Arial(0)/m/n/10 Splunk Enterprise < 8.1.14,
 []

! TeX capacity exceeded, sorry [input stack size=5000].
\@makeother #1->\catcode `#1
                            12\relax
l.2972 ...ation{'}s self-reported version number.} 
                                                   \\
If you really absolutely need more capacity,
you can ask a wizard to enlarge me.


Here is how much of TeX's memory you used:
 41357 strings out of 494868
 792782 string characters out of 6175714
 1445703 words of memory out of 10000000
 44502 multiletter control sequences out of 15000+600000
 535069 words of font info for 117 fonts, out of 8000000 for 9000
 14 hyphenation exceptions out of 8191
 5000i,19n,8326p,64577b,676s stack positions out of 5000i,500n,10000p,30000000b,80000s
Output written on report.pdf (109 pages).

💡 Summary

Please add a FCEB and Enrolled column to the stakeholders.csv after the cyhy-stakeholder alias is called.

Motivation and context

Internal stakeholders would like to know agencies identified as FCEB and when agencies were enrolled for the service.

Implementation notes

Please provide details for implementation, such as:

• Add a column to the stakeholders.csv denoting FCEB status.
• Add a column to the stakeholders.csv with the enrolled status

Specific stakeholders.py code found here: https://github.com/cisagov/cyhy-reports/blob/dc75e103292137bcfef1ef3bd6464b4e9817e418/cyhy_report/stakeholders/stakeholders.py

Acceptance criteria

• New Column named FCEB with Yes/No for FCEB status
• New Column named Enrolled with date of enrollment

💡 Summary

The parallelization via multiple threads in this repository is not done in the most efficient way. The parallelization should be rewritten to be more efficient.

Motivation

The way parallelization is currently implemented, once a thread completes its assigned work it will sit idle. Better would be to assign threads a single org at a time and give them new work to do (another org) when they are finished. This way no threads would sit idle until there is no more work to be doled out.

Implementation notes

Given that all threads are on the same instance in this case, there probably wouldn't be a lot of savings by rewriting this unless some threads are assigned groups of orgs whose snapshot creation takes considerably less time than others. If we ever move to a Lambda or Kubernetes type of architecture for this processing, where there is heterogeneous hardware, then we would definitely want to rewrite the parallelization to handle a single org at a time.

Acceptance criteria

• The rewritten parallelization gives the same results as the previous parallelization algorithm.
• The rewritten parallelization more efficiently takes advantage of the computing resources available and completes its work more quickly.

💡 Summary

We would like to add additional language to the ad-hoc report to identify the * in the findings.

Motivation and context

Why does this work belong in this project?

This would be useful because it will make the report clearer for the stakeholder.

Implementation notes

Add *** Denotes the possibility of a network management interface.** to the end of the New Potentially Risky Services

Acceptance criteria

• The report will include the verbiage as requested.

Instead of hardcoding a temporary file name like this:

OCSP_FILE = '/tmp/ocsp-crl.csv'

@felddy suggested making use of the Python tmpfile package.

This would probably involve some rewrite of ScorecardGenerator to accept a file-like object instead of a file name, and the main() function in generate_cybex_scorecard.py would also need to be modified somewhat.

🚀 Feature Proposal

For the weekly reporting process, create snapshots in parallel, rather than serially.

Motivation

As of today, creating snapshots is the biggest chunk of our weekly reporting time (about 700 minutes out of 850 minutes total; that's over 80% of the total time and climbing with each new report recipient we add). By creating multiple snapshots at once, we should be able to significantly reduce the total time it takes to create snapshots, just like we have done with report creation.

Since we have to pause the commander while snapshots and reports are generated, shorter reporting time means more weekly uptime for CyHy processing.

🚀 Feature Proposal

Transition from the Pystache package to the Chevron package for rendering Mustache templates in this codebase.

Motivation

We currently use the Pystache package to render Mustache templates for report generation. It has not seen a release in over six years. I think that until we migrate away from using this method for report generation that we should migrate to a maintained package. After looking into this I think that our best bet is the Chevron package.

Pitch

Although it has been over a year since its last release, it has seen more recent developer activity, tries to maintain 100% spec compliance, and claims significantly better performance than what is offered by Pystache. This switch would have us using a newer, and more spec-compliant package. Since we continue to gain new CyHy customers, the speed increase offered might also help report generation under an increasing workload.

A migration appears to be straightforward as the two packages are used in a similar manner. If it is as straightforward as it appears, it would make for a simple transition with testing to verify functionality.

Lastly, as we move forward with a Python 2 to Python 3 transition, it is important to note that the latest version officially supported by Pystache is Python 3.3, In comparison, Chevron offers support through to Python 3.6 (while maintaining support for 2.7 for use pre-transition).

💡 Summary

It has been requested that there be made an addition to the CyHy report of a note regarding the BOD 22-01 requirements. This would just be a static text page, the same for each customer.

Motivation and context

The motivation behind this, other than being a request from Bobby and Rob is to make sure customers have a consistent reminder of the BOD 22-01 requirements and a quick reference on how to get more information on them. Having this information readily available can increase the likelihood for customers to find and report vulnerabilities to add to the CISA catalog.

Implementation notes

Please provide details for implementation, such as:

• Possibly implement as either a new appendix or somewhere in the report description at the beginning.
• Attached is a word doc for exact formatting and wording BOD Language.docx
• Should be a static page, no need for adding any variables or customer names anywhere
• Should be a simple edit to the cyhy-reports/cyhy_report/customer/report.mustache file, with the new wording and formats included in the report template

Acceptance criteria

Work complete when the new addition makes it into the report and is added to the report generation process

💡 Summary

Please add the Region column to the stakeholders.csv after the cyhy-stakeholder alias is called.

Motivation and context

Internal stakeholders would like to know which region agencies belong to based on state location.

Implementation notes

Please provide details for implementation, such as:

• Add a column to the stakeholders.csv denoting the agencies regions based on region allocation found here: https://www.cisa.gov/about/regions

Specific stakeholders.py code found here: https://github.com/cisagov/cyhy-reports/blob/dc75e103292137bcfef1ef3bd6464b4e9817e418/cyhy_report/stakeholders/stakeholders.py

Acceptance criteria

• New column named "Region" exists in CSV containing the correct region number for each stakeholder

@dav3r pointed out that instead of performing several queries to compute counts in cyhy_report/customer/generate_report.py, we can use an aggregation pipeline to perform a single query that returns all the counts. See here for more details.

🐛 Summary

In the CyHy report, findings with long vulnerability descriptions may not have them completely displayed.

To reproduce

Steps to reproduce the behavior:

1. Generate a CyHy report for an entity with an open ticket for a finding with a long description.
2. Note that if the description is longer than there is room for on the page, the remainder of the description is not visible (though it is available in the findings.csv attachment).

Expected behavior

The entire description should be displayed. If there is too much text, it should be truncated and an explanatory message should be displayed (e.g. something like "The full description can be found in the appropriate attachment.")

As far as I can tell, LaTeX does not allow us to create a table cell that spans multiple pages (not authoritative, but there's some discussion about it here).

Screenshot

🐛 Summary

During report generation, a report failed due to OSError: [Errno 12] Cannot allocate memory. This happened on one of the longest-generation-time reports, which may have contributed.

To reproduce

Steps to reproduce the behavior:

1. Run report generation
2. See if one or more reports fail to generate due to an OSError

Expected behavior

We probably want to catch this error and retry that report's generation. This is not a code error but instead is an environment limitation, so we'll want to handle it for future runs.

Any helpful log output or screenshots

Paste the results here:

Traceback (most recent call last):
  File "/usr/local/bin/cyhy-report", line 11, in <module>
    generate_report.main()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 3393, in main
    was_encrypted, results = generator.generate_report()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 341, in generate_report
    self.__generate_final_pdf()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 3299, in __generate_final_pdf
    ["xelatex", "report.tex"], stdout=output, stderr=subprocess.STDOUT
  File "/usr/lib/python2.7/subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 916, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

💡 Summary

Modify extras/create_snapshots_reports_scorecard.py so that third-party snapshots are generated in parallel threads, as was done for "regular" snapshots in #61.

Motivation

Although third-party snapshots don't take much time to create (less than 6 minutes, as of today), it would be cleaner to generate them using threads to be consistent with how "regular" snapshots are created.

Implementation notes

At a minimum, threading should be used for third-party snapshot creation. It may also make sense to move the third-party snapshot code into the generate_snapshots() function. It is important to note that all "regular" snapshots must be created before we begin creating third-party snapshots, since third-party snapshots depend on the "regular" snapshots.

Acceptance criteria

How do we know when this work is done?

• Threading is used to create third-party snapshots in parallel
• Third-party snapshots continue to be created successfully

Can we use the default of shell=False here?

shell=True means that the subprocess Python module spawns the command using the shell process, which makes it much easier for a malicious actor to execute commands.

Originally posted by @dv4harr10 in #101 (comment)

💡 Summary

The number of stakeholders have grown in cyhy-reports and as such we want to optimize the notification process. As discussed recently with @dav3r , #83 is nice but not necessary as this can be solved more simply instead of parallel processing which would be "overkill".

Motivation and context

Currently, the daily notification process takes much longer than necessary. The current process is that we iterate over the list of "CyHy report receiving" organizations (and their descendants) and the notification generator is blindly called for each organization, regardless of whether or not a notification will be generated for that organization.

To speed up the notification process significantly, create_send_notifications.py should be modified to only call the notification generator if we already know it's going to generate a notification.

Implementation notes

The implementation should use mongo queries to build the correct list of organization IDs that require notifications. The rewrite for this process should replace the logic in build_cyhy_org_list as that includes the current implementation for the mongo query.

Acceptance criteria

This will be tested in a Dev environment. The expectation is notifications should be generated significantly faster than previous runs when compared to the amount of time it has taken for processing to complete. The reason being that instead of querying our list of agencies we will focus only those in which notifications have been generated. Thus, decreasing processing time and sending notifications quickly.

This should also be tested against Production data by verifying that the correct list of organization IDs can be generated, though this testing MUST NOT write any data to the Production database.

💡 Summary

Automatically re-attempt to generate failed snapshots and reports

Motivation and context

Occasionally, a snapshot or a report fails to be created successfully. In the past, we have seen this happen with entities that had a large amount of data (e.g. many more tickets than typical) and the snapshot or report generation process required more memory than was available.

It would be great if these failed snapshots and reports were re-attempted once or twice before the script "gave up" on them.

This issue was originally suggested by @jsf9k in #106 (review).

Implementation notes

One simple way to implement this would be to append the failed snapshot or report to the end of the list (either snapshots_to_generate or reports_to_generate). Note that we will probably want to adjust our logic for adding and removing entities from the successful_snapshots, failed_snapshots, successful_reports, and failed_reports lists.

There are other ways to deal with this. More broadly, @mcdonnnj suggested that we could make use of the Python queue module, which might make it easier to accomplish the goal of this issue.

Acceptance criteria

• If a snapshot or report generation fails, the script will automatically re-attempt to generate it a specified number of times before ultimately giving up
• The successful_snapshots, failed_snapshots, successful_reports, and failed_reports lists continue to accurately report status

💡 Summary

Clean up the gen_weekly_reports() function in extras/create_snapshots_reports_scorecard.py so that it is consistent with the changes made in #61.

Motivation

The report generation code should be as consistent as possible with the snapshot generation code.

Implementation notes

• gen_weekly_reports() should be renamed generate_weekly_reports() and include similar logic and logging.
• chunks() should be replaced by the newer make_list_chunks().
• create_reports() should be split into create_reports_from_list() and create_report(), just as we do with create_snapshots_from_list().
• All report-related variable names should be made consistent with their snapshot-related analogs.

Acceptance criteria

• Report generation code is cleaned up and consistent with snapshot creation code
• Reports are successfully created

💡 Summary

The number of stakeholders have grown in cyhy-reports and as such we want to optimize the notification process.

Motivation and context

Currently, notifications take longer than expected so the task is to implement parallelization for create_send_notifications
This will speedup the notification process significantly.

Implementation notes

The implementation will look similar to create_snapshots_reports_scorecard But #62 may also be considered.

Acceptance criteria

This will be tested in a Dev environment. The expectation is notifications should be generated significantly faster than previous runs when compared to the amount of time it has taken for processing to complete.

🐛 Summary

The CyHy reporting process failed to generate a report for the DHS_FEMA org this morning.

To reproduce

Run sudo -u cyhy cyhy-report --cyhy-section cyhy --scan-section scan --final --encrypt DHS_FEMA and watch it fail.

Expected behavior

The report for DHS_FEMA should be produced successfully.

Any helpful log output or screenshots

Paste the results here:

$ sudo -u cyhy cyhy-report --cyhy-section cyhy --scan-section scan --final --encrypt DHS_FEMA
/usr/lib/python2.7/dist-packages/pandas/core/frame.py:6211: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

  sort=sort)
Generating report for DHS_FEMA ...
Traceback (most recent call last):
  File "/usr/local/bin/cyhy-report", line 11, in <module>
    generate_report.main()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 3566, in main
    was_encrypted, results = generator.generate_report()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 344, in generate_report
    self.__generate_attachments()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 2403, in __generate_attachments
    self.__generate_sub_org_summary_attachment()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 2996, in __generate_sub_org_summary_attachment
    for row in self.__results["ss0_descendant_data"]:
KeyError: 'ss0_descendant_data'

🐛 Bug Report

It is possible to set up an organization in CyHy as a "third-party report recipient". Recently, there has been a request to generate a third-party report that includes a CyHy "grouping node". A "grouping node" is simply a RequestDoc that is used as a parent/child node in our database hierarchy, but does not represent an actual organization. For example, there are grouping nodes for each Critical Infrastructure sector containing the CyHy organizations in each sector.

The bug occurs in the create_third_party_snapshots() function in extras/create_snapshots_reports_scorecard.py. If a third-party report recipient contains a CyHy grouping node, we don't currently generate a fresh snapshot for the grouping node.

If that grouping node snapshot doesn't exist, create_third_party_snapshots() will fail to create the snapshot for the third-party report recipient and thus no third-party report can be generated.

Even more fiendishly, if a latest snapshot does exist for that grouping node (even if it's quite outdated), it will be used and the report will contain the outdated data from that snapshot.

To correct this situation, create_third_party_snapshots() should be updated so that a fresh snapshot is always created (using the --use-only-existing-snapshots flag) for all grouping nodes that are children of third-party report recipients prior to generating their reports.

To Reproduce

Steps to reproduce the behavior:

• Execute create_snapshots_reports_scorecard.py

Expected behavior

All third-party snapshots and reports generate successfully.

Any helpful log output

The log output below shows the report generation process failing because there was no snapshot in the database for the grouping node child of the third-party report recipient HEALTHCARE-RR:

2020-08-03 06:48:30,788 ERROR - Stderr failure detail: Generating report for HEALTHCARE-RR ...
Traceback (most recent call last):
  File "/usr/local/bin/cyhy-report", line 11, in <module>
    generate_report.main()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 2301, in main
    was_encrypted, results = generator.generate_report()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 294, in generate_report
    self.__generate_attachments()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 1670, in __generate_attachments
    self.__generate_sub_org_summary_attachment()
  File "/usr/local/lib/python2.7/dist-packages/cyhy_report/customer/generate_report.py", line 1936, in __generate_sub_org_summary_attachment
    for row in self.__results['ss0_descendant_data']:
KeyError: 'ss0_descendant_data'