Regarding “The backlog of user stories will be a maintained in a web-based project management application that will be provided by the Court.”

Can you name the application?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

Description of Work to be Performed, page 3

Question/Comment

How much of the data in the legacy system needs to be migrated in the first year?

What kind of workflows could be triggered within the system (for example: responses missed within particular timeframes, legislation/regulatory requirements that require deadlines, etc.)?

A Tax Court case can be initiated via a paper filing. The RFI states that "many Petitions do not conform with the Tax Court Rules, and some of the documents submitted will not become part of the official docket record. The Court expects a future EF-CMS will allow for the processing of these documents that are not part of the record." How is this information stored currently, and is there any migration of this data that needs to happen?

What kind of processing is needed on these documents?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Kaitlin Devine, Director of Innovation, Ad Hoc LLC

Section of RFP documents

SOO - Description of work to be performed - “The Court expects the development process will consist of 2-week sprint cycles, and that the initial phases of the development process will focus on providing identity and access management, case initiation, payment of fees, search functionality, e-signatures, notes, and content management.”

Question/Comment

Are there any existing user research artifacts that could provide more context on why these particular items were prioritized into the initial phases? For example, these are features but is there more background on the value and outcomes you want these features to achieve?

How will the Tax Court evaluate similar experience beyond the github links that each vendor provides? Where in the response should the vendor describe their technical experience and subject matter expertise?

Where will the new web-based case management system be hosted? cloud.gov?

Please confirm that the selected vendor will have administrative control over any environment they deploy to and will be responsible for managing said environment.

For the purposes of planning, can the Court identify the desired functionality that does not exist in the current system?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Kevin Wehmueller
Business Development Strategist, Taoti Creative

Section of RFP documents

https://github.com/ustaxcourt/case-management-rfq/blob/master/02_SOW.md#deliverables-and-performance-standards

https://github.com/ustaxcourt/case-management-rfq/blob/master/03_Evaluation_Criteria.md#similar-experience

Question/Comment

Are we correct in assuming that source code samples will be evaluated for the extent to which they are well-documented and in line with the performance standards described in the Deliverables section? For reference:

Documentation
All dependencies are listed and the licenses are documented. Major functionality in the software/source code is documented. Individual methods are documented inline using comments that permit the use tools such as JsDoc. System diagram is provided.

Regarding “The Court intends to deploy the new System and convert operations from the current system as soon as the software for the new System has been sufficiently developed to provide the minimal functionality required to support the Court’s basic operations.”

Is there an expected timeline for when this would ideally happen?

Is there a defined minimum functionality that would be expected?

Evaluation criteria states that we need to submit "links to Git repositories​". If these repos are not public, is there a github user that we can add? Is this the only way that the Court will evaluate the relevance of a vendor's technical and subject matter expertise?

Question/Comment on this U.S. Tax Court RFP

On page 7, User Stories, Fees, What is the payment gateway that will be used?

Name and affiliation

Ajie Velasquez, Vice President, Armedia, LLC

Section of RFP documents

Page 7, User Stories, Fees,

Question/Comment

What is the payment gateway that will be used?

What are the existing authentication/authorization mechanisms and do they need to be integrated into the system?

Are electronic submissions allowed only via email to the clerk of the court or directly into the system?

Regarding “The Contractor will not be responsible for hosting of the system or obtaining an Authority to Operate (ATO).”

Who would be responsible for this? How will they gather information to support the ATO?

When registering for electronic submission what requirements exist for remote identity proofing?

How many engineers do you expect would be required to build out this project?

The proposed budget indicates that you may be envisioning a single agile development team (consisting of technical leadership, project management, user research and design, and engineering) over a three year period. However, we advocate for several agile development teams working together over a single year to best satisfy the requested feature set and velocity to replace the existing system. You want to reduce the amount of time you have to maintain multiple systems for the same function. This addresses both the cost of two engineering teams to maintain coexisting systems and the cost to your users/employees of having to switch back and forth between systems.

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Kaitlin Devine, Director of Innovation, Ad Hoc LLC

Section of RFP documents

Appendix B - "The Court expects that the hardware and software comprising the EF-CMS will either be located on premise, or on a FedRAMP-certified cloud storage solution controlled by the Court, or some hybrid thereof."

Question/Comment

Could you talk more about the specific Court needs being met by the self-hosted infrastructure that cannot be met with cloud infrastructure? It would be really helpful to understand the specific needs and the trade-offs the Court is considering in this decision. A hybrid environment would certainly add to the complexity, as well as the portfolio of skillsets needed to manage both environments. But there may be some data or functionality that can't be moved out of the environment that makes the added complexity worth it. Either way, it would be really helpful to explain that here.

Especially since elsewhere in this section, the draft states: "Offerors must state in their quote whether additional hardware, software or ancillary equipment is required for the development, staging, or production environments, and provide details including associated estimated costs."

It's pretty hard to be able to address that section without knowing what the Court specifically wants to use the self-hosted environment for. Is it database management? Some kind of internal user authentication?

Can we use 3rd party services for continuous integration?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Kaitlin Devine, Director of Innovation, Ad Hoc LLC

Section of RFP documents

Appendix B, paragraph 3 - "It will be the responsibility of an EF-CMS Contractor to provide and maintain its own development environment. The Court will provide the internet-connected staging and production environments. "

Question/Comment

What is the motivation for having a dev environment that is located in completely different infrastructure? It seems like you'd want the three environments to mirror each other as closely as possible, to minimize configuration/migration hurtles that may arise only when code is promoted to staging.

The RFI states "The Contractor must provide services that will include all aspects of the software development process, including initial planning, design...and deployment into the Court’s pre-production environment."

Does "the Court's pre-production environment" refer to an environment that the Court has already set up, similar to its current production environment? Or will "the Court's pre-production environment" be one that the Contractor sets up / creates?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Kevin Wehmueller
Business Development Strategist, Taoti Creative

Section of RFP documents

https://github.com/ustaxcourt/case-management-rfq/blob/master/02_SOW.md#description-of-work-to-be-performed

Question/Comment

The RFQ states:

Additional Court representatives will be promptly available as needed to assist in the development process, including for internal user testing. The Court will arrange for any necessary external user testing.

The Court’s current internal management and staffing levels are not adequate to allow for effective coordination with more than one development team, at least during the initial stages of the development process.

How many internal Court Representatives are expected to participate in internal user testing, and how will they interact with the system (especially with regard to initial development goals including identity and access management, case initiation, payment of fees, search functionality, e-signatures, notes, and content management)? Are all representatives located in Washington, DC?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Kevin Wehmueller
Business Development Strategist, Taoti Creative

Section of RFP documents

These questions are referencing the routing for documents, approvals, and other similar functions.
https://github.com/ustaxcourt/case-management-rfq/blob/master/06_Appendix_B.md#docket-management

Question/Comment

The routing defined (in the RFQ) is assumed to be in place, but is there also a need for accommodating the following:

1. Impromptu routing where the user defines a new flow?
2. Exception or temporary routing based on business rules that can be updated and changed at a moment's notice, such as including an external review?
3. Ability to create standard routing templates that can be stored in a "Library" for user access or applied to certain situations based on business rules?

Will the court provide the users needed for research and testing purposes?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Greg Gershman, CEO, Ad Hoc LLC

Section of RFP documents

User Stories

Question/Comment

The draft RFQ clearly outline a set of user stories that detail the features of the application. In reviewing, there does not appear to be any user stories or requirements related to an Application Programming Interface (API). In addition to functionality that will be used by humans, it is often beneficial to build an application with a consideration for how it's data and business logic would be utilized by other systems. For example, there may be other systems with the US Tax Court, or systems outside the Tax Court, that may want or need to integrate information or functionality provided by the new system into their systems. The best way to do this is through an API.

When building a new system, many developers will take an API-first approach. This approach encourages an architecture that separates user-facing functionality from business logic and data. This allows, if needed, business logic and data to be accessed by other systems, either internally or externally, with relatively little effort. Using a RESTful approach to structuring an API also encourages good data modeling practices at the data layer.

If Tax Court sees this as something relevant, it may be worthwhile to define some requirements around an architecture that incorporates an API, and document some users stories for some of the systems that would use that API.

In addition, to the Tax Court's desire to avoid COTS products, an API could help alleviate the risk that the Tax Court takes on by incorporating a COTS product into their architecture. The most effective approach to avoiding vendor lock-in when developing a new system is to ensure that the customer (the government) has control over the data. When a customer gets locked in, it's often not as much the software from a vendor that facilitates the lock-in, it's access to the underlying data and its structure which makes moving to a new solution most difficult. With an API-driven architecture, the Tax Court could define the important data required by the system and its users, and then safely build and incorporate into their solution COTS products that are customized or adapted to utilize that data.

As an example, case management solutions are very common. There are many solutions on the market that provide a wealth of features with a relatively minimal amount of effort. These case management solutions can be integrated with a Tax Court system API so that the case management tool is not the ultimate source of the data; that would be the Tax Court's new system. Some parts of the system, such as the parts that the public interfaces with, could be completely custom-built, while other parts, which can be implemented using commercial products, can be easily integrated with the system's API. This flexibility will enable the Tax Court to leverage both custom and COTS solutions to their ultimate benefit, while at the same time ensuring the Tax Court can transition to another vendor of a COTS product (or decide to build their own version) at any point, since Tax Court owns and controls the data.

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Kaitlin Devine, Director of Innovation, Ad Hoc LLC

Section of RFP documents

SOO - Description of work to be performed

Question/Comment

There doesn't seem to be much mention of product management or product strategy in this section. Do you anticipate this as a need? Is it potentially being filled by government? If so, do you anticipate there will be a need for this in future option years?

"In addition, the System will allow for payment of filing fees by check. If paid by check, a Court employee will scan the check for automatic transmittal to the U.S. Treasury. Checks that cannot be scanned for automatic transmission will be manually delivered to the U.S. Treasury."

Is this done manually right now? Is the EF-CMS system expected to do this automatically?

How many other government systems will this system need to integrate with? Social security administration? A payments system? What are the specific systems with which we would need to integrate?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

General Requirements, page 4

Question/Comment

Is the fee payment system going to be pay.gov?

It appears that the Court will handle product management and product strategy. Could the Court outline their approach/goals for product?

Is there a current system for generating case and filing numbers that needs to be maintained?

Is there a database of practitioners who have been approved to appear before the court (members of tax court bar?) and how is that data going to be made available to the system?

What is the role of the existing court network in the solution other than to provide bandwidth?

Will the Court want to begin this work with an MVP? If so, what basic functionality comprises an EF-CMS MVP?

Are you anticipating the contractor will perform user research?

Note that the Tax Court indicated to vendors we wouldn't attribute their emailed comments when posting on GitHub so that we could cast as wide a net as possible. That's why portions of the following issue are redacted.

This email is in response to the Draft RFQ for the U.S. Tax Courts Enterprise. [Redacted] only has a few questions and comments.

1. We recommend that the term "free open source software" be changed to "open source software". [Redacted]'s open source case management software platform is a paid annual subscription that includes the software license based on Apache 2.0 (https://www.apache.org/licenses/LICENSE-2.0.html), support and maintenance. We would not be able to offer the software free of charge. However, our annual subscription model does provide low cost of entry, predictability and can significantly decrease the amount of time it takes to implement and maintain an application which also reduces labor costs.
   
   While "free" open source software exists, they sometimes come in various forms. A vendor may offer a Community Edition and an Enterprise Edition. A Community version could be free, but would be very limited. For example, community versions do not allow for high availability, they do not offer a dedicated support team and new versions do not go through rigorous Quality Assurance testing. Enterprise Editions, however, are not considered "free" open source. Instead they are considered Open Core and have similar license model as [Redacted].

2. Would enterprise (not free) open source software be considered material under the T&M contract? If not, we would recommend a procurement that allows for the purchase of software and labor.

Will the 27K pending cases need to be imported from the old system into the new? If yes, who will provide that data?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Kaitlin Devine, Director of Innovation, Ad Hoc LLC

Section of RFP documents

SOO - Introduction - "The Court currently uses a legacy case management system that is not web-based. The Court holds a license to use the software that supports that system. Under the terms of that license, the Court is precluded from disclosing any programs, concepts, designs, or source code. This may be an important operating constraint for the Contractor because it will not be able to view or access the current system in any way."

Question/Comment

Does "not web-based" mean this is a desktop client installed on individual workstations that share a common, networked database?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

User Stories, page 6

Question/Comment

Which user stories in the RFQ are part of MVP?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Kevin Wehmueller
Business Development Strategist, Taoti Creative

Section of RFP documents

https://github.com/ustaxcourt/case-management-rfq/blob/master/06_Appendix_B.md#appendix-c-case-processing-workflows

Question/Comment

All actions in the system have to be tracked and recorded per OWASP. This can provide the basis for any statistical and historical analytics. Are other analytics expected that provide more user-related information, such as browsers, time online etc.? Should the system also track user statistics, such as the number of users on the public side of application?

"In addition, the System will allow for payment of filing fees by check. If paid by check, a Court employee will scan thecheck for automatic transmittal to the U.S. Treasury. Checks that cannot be scanned for automatic transmission will be manually delivered to the U.S. Treasury."

Does the EF-CMS system need to scan and verify checks? Or is that done separately and all the EF-CMS system needs to do is manage of payments workflow and documentation?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

Descriptions of Work to be Performed, page 3

Question/Comment

Does the EF-CMS need to support internationalization? If so, which languages?

"A future EF-CMS will allow for calendaring sealed hearings or portions thereof for which Court staff need access to the scheduling information but the public does not, and for permitting an internal search for the name of a petitioner where the public-facing name is “Anonymous”."

Does this mean that most court filings for this system should be easily searchable by the public? What kind of data in the system needs to be made publicly available?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Kevin Wehmueller
Business Development Strategist, Taoti Creative

Section of RFP documents

https://github.com/ustaxcourt/case-management-rfq/blob/master/06_Appendix_B.md#docket-management

Question/Comment

Access/data protection:
As the cases are routed and each person/team conducts their reviews, will notes, comments etc. be accessible to all users with access to the case or will control be administered on each piece of content (e.g., "Note A" requires "Access Level A," while "Note B" requires "Access Level C" or lower)?

User Access and control:
Is there a central database of users (e.g., court practitioners) that can be accessed to verify the identity of users?

If the vendor deploys to pre-production, who would manage the actual production deployments? Who would manage testing and approval for promotion to production? Is the contractor expected to deploy (at any stage) into the existing Windows environment? If so, in what capacity?

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

Description of Work to be Performed, page 3

Question/Comment

How much of the new system needs to be searchable? Do all of the content in the PDFs need to be searchable?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Greg Gershman, CEO, Ad Hoc LLC

Section of RFP documents

Evaluation Criteria - Technical Submissions

Question/Comment

The RFQ requires vendors to provide "links to Git repositories," by which it seems to mean references to version control repositories of other similar work to the work defined in the RFQ.

The first suggestion is to change the phrasing from "Git repositories" to "source code or source code repositories." Git is a specific form of version control, but by no means the only tool used for that purpose. There are many other version control tools, and unless the Tax Court is specifically looking to ascertain a vendor's ability and experience working with Git, we suggest broadening the definition to include other version control systems as well.

While Ad Hoc can basically meet this requirement, we find that these specific kinds of requirements are burdensome on vendors, both in determining if repositories meet the exact criteria, and because much of our work, even though with US federal government agencies, is not made publicly available by our customers.

In regards to the difficulty in meeting the exact criteria, Ad Hoc has worked on a project that we believe should be considered adequate, but is difficult for us to determine if it would be accepted by the government as valid. Ad Hoc is contracted by the US Department of Veterans Affairs to build Vets.gov. Our team employs about 20 software engineers, and our program is over three years. Our team, practically, is broken down into smaller product development teams, which typically consist of 3-5 engineers (along with others, for total team sizes around 7-9 personnel). While we feel this meets the government's requirements, if not exactly in terms, but in spirit, we are concerned because of the requirement that projects have "staffing profiles of approximately five (5) to nine (9) Full Time Equivalent (FTE) personnel." Finding something that matches these exact criteria, is made publicly available by the government, can be very challenging for many vendors. In addition, strict requirements that are met in spirit by vendors may create grounds for protest of the final award.

Further, in our experience, the fact that our work for government is made publicly available is not under a vendor's control. We do work for government agencies that we feel is very relevant to this RFQ's requirements, but is kept private by the decision of the agency. Vendors are not able to easily (or at all) provide access to third parties to view or evaluate this source code, even if the third party is another federal agency. Therefore, we do not believe there is anything distinctive gained by the government in ascertaining that a vendor is able to provide these links. In addition, the practice of making code publicly available is rare to non-existent in the commercial sector, in particular applications such as the one that Tax Court is looking to build. This requirement will eliminate many qualified vendors that have very valid and relevant experience.

Therefore, we suggest revising or removing the criteria to provide links to Git (or version control/source code in general) as a part of this RFQ. We believe doing so will result in better competition from a larger pool of qualified vendors, and ultimately better results for the government.

It would help to better understand what Tax Court is looking to determine with the requirement. If there is a desire to modify the criteria to hone in on specific qualities that Tax Court feels would make a successful vendor, we are happy to provide some feedback on how to do that in ways other than requiring links to publicly-available repositories.

Question/Comment on this U.S. Tax Court RFQ

Name and affiliation

Sarah Ruggless
Flexion

Section of RFQ documents

Question/Comment

Will 18F have a role on the product delivery team for EF-CMS?

Question/Comment on this U.S. Tax Court RFP

Name and affiliation

Greg Gershman, Ad Hoc LLC

Section of RFP documents

Data Rights and Ownership of Deliverables

Question/Comment

In regards to the software and documentation developed as part of this project, the SOO contains references both to "public domain" and "open source." There are differences between the two, we believe the government is interested in ensuring that the code developed by the vendor for this project be freely available to others, and we believe that licensing that code under an open source license (rather than contributing it to the public domain) is a better approach that will ensure the goals of the government are met. Some additional details can be found here: https://opensource.org/node/878