openreferral / api-specification Goto Github PK

I'd like to introduce the concept of API definition filtering to the HSDA design discussion. In an effort to help ensure the API design speak to as wide as possible audience as we can, I'd like to build on discussion around data filtering, and schema filtering, and allow for labeling and packaging up of the API paths using the OpenAPI tagging.

In the OpenAPI, each path can have one or many tags associated with it. We can create a simple set (/organizations/, /locations/, /services/, /search), and a more complete set with all the direct paths to sub elements--reducing cognitive load for new users, while also allowing for access to precise paths for directly updating phone number, or an address without being bothered with entire schema.

I'll work on an example to show how API definition filtering can be used to reduce complexity in API documentation.

Currently ALL POST and PUT endpoints use the body for a transport of the schema.

How we are going to handle pagination in the requests and responses for the APIs. This overlaps with the hypermedia discussion. Currently we are using a page, and per_page parameters for GET requests. Now is the time to change or evolve.

Currently, there is a sample data set here:
https://github.com/openreferral/sample-data
But this only contains a subset of the models in the specification.
We should add the rest of these models so that this data is complete and can be used in prototyping and automated testing.

I'm starting a read for aggregating conversations at the granular level. The looseness or strictness of how the JSON schema interprets HSDS or HSA.

I do not see them in the schema?

I want to open up the conversation around using GraphQL as part the design for the HSDS API. This approach is being considered because HSDS implementations often are data intensive systems, with advanced querying needs. We just want to make sure everything is on the table.

GraphQL - http://graphql.org/

This is the thread to discuss the versioning plan for the API. The starting suggestion is to use the current v1.0 and v1.1, with keeping in alignment with semantic versioning used by major providers like Google - http://apievangelist.com/2017/03/09/guidance-on-versioning-your-api-from-google/.

The current plan is to place the version in the url, right after the baseurl for the API. It's open for discussion for changing strategy with the current release. After that we will probably lock in.

IMO, search URLs are always tough. Right now, the search URL in the Ohana API I've seen is a bit confusing because I don’t know what kind of resource it will return. Will it return locations, or organizations, or any type of resource that fits the keyword? Also, a search isn’t a resource exactly, so having it in the URL doesn’t feel very RESTy (although arguably this doesn’t matter much). My convention would be to just add those search params to the /locations endpoint, unless the search action is meant to return multiple resources, in which case I think a dedicated search URL is appropriate.

On this note, is it necessary to have an endpoint for /locations/nearby when that URL accepts params anyway? Thoughts on just allowing those possible params on the /locations endpoint instead of dedicating a separate endpoint for it?

Aside from Documentation, the links at https://openreferral.github.io are turning up 404...

How will we be sorting information. There is no sorting capability in the current design.

This has been migrated from Ohana (codeforamerica/ohana-api#419 (comment)) - worth revisiting:

JordanLyons commented 23 days ago
Several programs and agencies in my area don't want their physical or mailing addresses possible. I see that this is a required field for several of the CSV's, and I wonder if there's anything that could be done to prevent that kind of sensitive info getting published on the database.
@greggish

greggish commented 22 days ago
This is a known challenge for things like domestic violence shelters, human trafficking services, etc. Before 2016, I would have called it "an important edge case" but now i don't think that 'edge case' does it justice.

The simple scenario seems to be that there's usually a 'front gate' point of contact that is public, without sharing details about where people actually go once they've entered into relationship with the service.

I'm going to cc @kinlane on this (although i suspect that delineating between public and private addresses is not likely to be an issue we can directly address before 2.0). I'm also curious about how other referral providers handle stuff like this, and it might be an issue you could bring to the Google Group for discussion.

How will we be filtering API responses? Right now we depend on the /search/ path to do search, with the following fields:

category:
email:
keyword:
language:
lat_lng:
location:
org_name:
radius:
service_area:
status:

How will this be applied across paths?

I have a working JSON schema API that we can pass any JSON file to it and it will validate against a JSON schema.

I need to finish JSON schema representation for:

• HSDS - A representation to define the rules for the schema
• HSDA - A representation to define the surface area of the API.

I am still polishing my JSON schema chops, but also I'm thinking deeply about a mininum viable definition, as well as potentially more verbose definitions.

I'm thinking that we should be allowing for different levels of compliance.

This thinking overlaps with the schema filtering conversaiton for the API -- whether someone wants to see simple or complex responses, or even POST/PUT to the API - Issue 21.

Introducing a fourth dimension of the scope / filtering conversation -- project level. How do parts of the schema and API definition get grouped, and evolved separately from other projects.

How is this separation described in API definition(s) - OpenAPI + APIs.json, which can then be reflected in documentation, SDKs, and other tooling.

This also needs to be thought about as it impacts the definition for a "HSDA compliant API or schema" - what % compliant is an API. Few will ever be 100% compliant, but providers can pick and choose which project areas they support or don't support.

I will be moving the /search option beyond just returning locations, and give it a collection of organizations, locations, services, and contracts.

I will also give /search an /everything, so that we can have a simple, and expanded results.

THis would be the time to adopt JSON API, HAL, or other evolutionary approach to responses - let's discuss.

Need a plan for status codes, starting with 200, 404, and 500, but then getting more sophisticated in how APIs should be returning errors.

This is a use case as part of the Use Cases meta-issue (#44). It expands on @klambacher 's "Integrators" use case:

Integrators want to combine information into existing systems or datasets. They are committed to the system they use, and may have a substantial existing data set that is outside of the collection policy of the target dataset they want to integrate. They care deeply about ongoing maintenance and updates of the information and often move large quantities of information frequently. Over time, this can sometimes become a bi-directional sharing relationship if the initial integration is successful. Efficient ways to determine changes and keep data in sync between systems is what matters, and search features are generally irrelevant beyond the ability to query changes and do sanity checks. Efficient movement of data in bulk is important and pull-and-display APIs often require too many requests to collect information. "I'd like to integrate your information into our existing database, on this schedule"

Here's what folks are trying to achieve:

1. SUB is an organization that uses Human Services data. They maintain their own database, which they need to use for various operational purposes. PRO is another organization that maintains similar data. Rather than curate an entire database on their own, SUB wants to "subscribe" to the database from PRO as the core of their database.
2. The partnership agreements look something like this:
   • PRO and SUB agree to a scope covered by the subscription (e.g. resources in these taxonomy categories covering this geography)
   • SUB will mirror the relevant part of PRO's database into their own database
   • Whenever PRO updates, adds, or deletes relevant resources, they will notify SUB so that SUB can pull the changes into their copy.
   • Whenever SUB updates or adds a resource, they will submit the change to PRO as a suggested change. PRO will review the changes and either accept them or reject them. Ideally, PRO will notify SUB about the outcome of these updates, especially rejections (accepted edits will come through as the core data set regardless). SUB can submit these changes either as specific changes ("change phone number to X") or as flags ("phone number was disconnected, needs investigation")

I would like to propose a separate HAL representation be developed, allowing for HSDA access to be done via relationships between the data.

This is the second first proposal to use media types for breaking projects into smaller chunks so they do not affect other types of users -- meeting the needs of systems users, without impact say web, spreadsheet, SPA, or conversational interface users.

Implementors can choose to follow core schema, or be HAL compliant, choosing to go a more hypermedia path for all clients.

While I am putting this under v1.2 -- the hypermedia API would branch and take on its own versioning, as separate branch.

This has been migrated from Ohana (codeforamerica/ohana-api#417 (comment)), and worth considering as part of specification:

As part of Code for Miami's involvement with the Knight Cities Challenge, we recently conducted a Civic User Testing Group in Miami with a reskinned version of the Ohana web search. You can find our thoughts here.

We then built a follow-up prototype and performed CUTGroup testing for that prototype as well.

While developing this follow up project, we noticed the API is not configured to return taxonomy data in a keyword search. To get around this, our app ran an initial keyword search to get category data, iterate through each result, and then a second search for that particular organization’s details.

As a result, if a search for the keyword “food” returns 50 organizations, the app must make 51 calls to the API (one for the initial search, and one for each of the 50 organizations in the results). As a result, the app suffered from noticeable performance issues that would not be sustainable at scale.

We can solve this in two ways:

Refactor the API to return taxonomy data in keyword search results; or
Create a second API that allows third parties to copy the vendor provider’s entire data set into their database, which they can then customize for more efficient searches.

I want to begin mapping out the webhooks guidance for future versions of the specification. I'd like to find several webhook implementations we could use to model the external push aspects of an implementation, making things more event oriented.

I didn't quite see any feedback regarding the options on the table for allowing for a more comprehensive approach to schema filtering, so I'm just going to go with the most intuitive option, and add an /everything path to /organizations, /locations, /contacts, and /services.

Ie. when you do /organizations/ you get a simple, flat schema.
when you do /organizations/everything you get a complete schema with all subresources.

We can revisit header options for this, and other approaches further down the road, this should support the concerns on the table.

The table name INTEPRETATION_SERVICES is probably meant to be INTERPRETATION_SERVICES in the .yaml file.

Now that I'm digging though, I'm not sure which git repo file is the correct yaml file. This issue relates to the yaml in the documentation at https://openreferral.github.io/api-specification/definition/yaml/

The major part of this release was ensuring there was POST, PUT, and DELETE methods for the entire surface area of the HSDS schema.

right now we use POST for adding, and PUT for updating. Both use the body and PUT updates the entire resource (ie. organization, location, service, contact). I suggest we add PATCH, allowing for partial updates, as well as OPTIONs further defining the surface area.

The location response is quite big, and on one hand this is awesome, because it ensures the client by default gets the info it needs. However, it’s quite a hefty payload and any mobile clients would prefer something smaller. What about having an option to request this resource with a minimum response (e.g. just the location resource, excluding organization, categories, etc) so that clients concerned about package size can opt for something smaller?

See Irakli's post about one possible approach to this: http://www.freshblurbs.com/blog/2015/06/25/api-representations-prefer.html

Provide guidance on HTTP Status codes to be returned. Add all relevant HTTP Status Codes to individual endpoints using the OpenAPI.

Review and consider this existing AIRS work alongside specification - https://www.airs.org/i4a/pages/index.cfm?pageID=3363

what are the headers, and stragegy for this.

Do we want to provide domain guidance for implementations? Recommend folks do api.example.com, or example.com/api.

Consider offering portal advice too, like developer.example.com.

Need to come up with the standard for returning error and status codes -- there is no plan so far. I'd like to see this span a standard error responses structure, as well as application of HTTP status codes.

HTTP Status Success

• 200 OK - Standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request, the response will contain an entity describing or containing the result of the action.
• 201 Created - The request has been fulfilled, resulting in the creation of a new resource.
• 202 Accepted - The request has been accepted for processing, but the processing has not been completed. The request might or might not be eventually acted upon, and may be disallowed when processing occurs.
• 204 No Content - The server successfully processed the request and is not returning any content.
• 206 Partial Content - The server is delivering only part of the resource (byte serving) due to a range header sent by the client. The range header is used by HTTP clients to enable resuming of interrupted downloads, or split a download into multiple simultaneous streams.

HTTP Status Redirection

• 301 Moved Permanently - This and all future requests should be directed to the given URI.
• 302 Found - Common way of performing URL redirection. An HTTP response with this status code will additionally provide a URL in the location header field. The user agent (e.g. a web browser) is invited by a response with this code to make a second, otherwise identical, request to the new URL specified in the location field.
• 303 See Other - The response to the request can be found under another URI using a GET method. When received in response to a POST (or PUT/DELETE), the client should presume that the server has received the data and should issue a redirect with a separate GET message.
• 304 Not Modified - Indicates that the resource has not been modified since the version specified by the request headers If-Modified-Since or If-None-Match. In such case, there is no need to retransmit the resource since the client still has a previously-downloaded copy.

HTTP Status User Error

• 400 Bad Request - The server cannot or will not process the request due to an apparent client error (e.g., malformed request syntax, too large size, invalid request message framing, or deceptive request routing).
• 401 Unauthorized - Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource.
• 403 Forbidden - The request was a valid request, but the server is refusing to respond to it. The user might be logged in but does not have the necessary permissions for the resource.
• 404 Not Found - The requested resource could not be found but may be available in the future. Subsequent requests by the client are permissible.
• 405 Method Not Allowed - A request method is not supported for the requested resource; for example, a GET request on a form which requires data to be presented via POST, or a PUT request on a read-only resource.
• 406 Not Acceptable - The requested resource is capable of generating only content not acceptable according to the Accept headers sent in the request.
• 408 Request Timeout - The server timed out waiting for the request. According to HTTP specifications: The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time.
• 409 Conflict - Indicates that the request could not be processed because of conflict in the request, such as an edit conflict between multiple simultaneous updates.
• 410 Gone - Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed and the resource should be purged. Upon receiving a 410 status code, the client should not request the resource in the future. Clients such as search engines should remove the resource from their indices. Most use cases do not require clients and search engines to purge the resource, and a 404 Not Found may be used instead.
• 411 Length Required - The request did not specify the length of its content, which is required by the requested resource.
• 412 Precondition Failed - The server does not meet one of the preconditions that the requester put on the request.
• 415 Unsupported Media Type - The request entity has a media type which the server or resource does not support. For example, the client uploads an image as image/svg+xml, but the server requires that images use a different format.
• 422 Unprocessable Entity - The request was well-formed but was unable to be followed due to semantic errors.
• 423 Locked - The resource that is being accessed is locked.
• 428 Precondition Required - The user has sent too many requests in a given amount of time. Intended for use with rate-limiting schemes.
• 429 Too Many Requests - The user has sent too many requests in a given amount of time. Intended for use with rate-limiting schemes.

HTTP Status Server Error

• 500 Internal Server Error - A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.
• 501 Not Implemented - The server either does not recognize the request method, or it lacks the ability to fulfill the request. Usually this implies future availability (e.g., a new feature of a web-service API).
• 503 Service Unavailable - The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

This is a project to begin defining a set of objects to be returned as part of error codes, providing a structure, and common language set to describe common, as well as custom error responses.

I am working on a JSON schema for HSDS, and for HSDA.

I already have an API for validating an API path or JSON snippet against any JSON schema when ready. Will release as extra set of API paths.

Once schema are establish we will need to establish the set of rules being applied, and how strict we intend on being.

Proprosal to introduce a custom HSDA media type for bulk loading, which would be a single data point for GET and POST of data.

Open311 has a similar initiative - http://wiki.open311.org/

While still offering an "everthing/" path on main API, this would allow the core API to reflect the flat nature of schema, keeping usable by spreadsheet users.

This is the first proposal to use media types for breaking projects into smaller chunks so they do not affect other types of users -- meeting the needs of systems users, without impact say web, spreadsheet, SPA, or conversational interface users.

While I am putting this under v1.2 -- the bulk API would branch and take on its own versioning, as separate branch.

Wanted to have a place to have conversations about how we reflect actions in the API design. Right now the API design is very CRUD, and there really aren't any actions beyond just verbs.

I'd like to have an ongoing discussion around how we reflect more advanced actions in the API design, allowing for more actions to be taken after we get more feedback from the community.

Opening up a thread dedicated to the approval, feedback, and auditing flow for the platform, that allows anyone to update a record in the system, but there is an approval flow that can be applied before the change is accepted.

This area would overlap with the meta data conversation which involves the change log and logging - #28

Had a comment come in on a blog post that I want to consider for future road map when it comes to expanding on the regular and holiday scheduling.

Source: http://apievangelist.com/2017/05/23/avoid-moving-too-fast-for-my-api-audience/#comment-3321395153

Abstract: I am very familiar with 211 from the service provider standpoint. The NAACP in Eugene provides a weekly tutoring session at the local community college for students of color. Every year I have to update our listing on 211. Our schedule ends up changing as the school year progresses and we keep all the events on our wordpress site. I have dreamed of a day when a plug in would arrive so the 211 site would "listen" for changes on the events app on word press. This would make it so I could easily update our 211 listing with the most current info. I am probably not alone and a lot of small daycare providers, tutors, camps and counselors are using wordpress and other out of the box (OOB) websites to manage their events. Can we get some API action from the calendar devs on the OOB websites? (please note I am not a developer, just a user)

I'm opening up a 3rd dimension to the whole filtering conversation. It is relevatng data filtering (#22) and schema filtering (#21), but is at a higher level.

It answers other concerns about the overall API design surface area being too large, too many paths. Path filtering in the API documentation would all for a default set of paths (ie. orgs, locations, services, search), then mid, and advanced tiers.

This would reduce cognitive load for developers when first getting going, but allow for knowledgeable users to get at a wealth of more advanced or precise search endpoints.

I am suggesting this also to handle future addition of API paths, such as utility ones like schema validators, or possible universal ID, taxonomy, messaging, etc. Additional projects, relevant to, and sometimes part of core spec, but off in separate namespace.

Adding a parameter for filtering the schema.

Providing comma separated list of fields, and sub-resources.

Suggesting we call it fields=

What are the other API design considerations?

I would also recommend holding off on the API for the metadata portion of the schema - https://openreferral.readthedocs.io/en/latest/reference/#metadata

This area needs some additional considering when it comes to managment, logging, and other aspects of API operations.

This is the thread to discuss use of hypermedia as part of the HSDS v1.x. Now is the time to have this discussion, and discuss the evolution of the project.

I'm suggesting we look at the following formats, based upon their adoption in the API sector:

JSON API - http://jsonapi.org/
Collection+JSON - http://amundsen.com/media-types/collection/
HAL - Hypertext Application Language - http://stateless.co/hal_specification.html
Siren - https://github.com/kevinswiber/siren

I'll be working on a separate entry, outlining the benefits of applying hypermedia design, as well as working examples to help visualize how it will be used.

what is plan for operationId? camelCase?

Migrating from a Slack channel discussion around the number of available APIs I'd like to open a thread specifically about what the core paths GET and POST structures should look like

• /organizations/
• /locations/
• /services/

When you GET, what can be expected? Simple or complete nested (ie. phone, contacts,)?
When you POST or PUT, what can be expected? Simple or complete nested (ie. phone, contacts,)?

This overlaps with other conversation regarding schema filtering, or how you tell the API which schema the user would like returned - #21

This also overlaps with conversation regarding /search/ and it's relationship to /organization/, /locations/, and /services. Where the results should be - #22

My recommendation is that all four core paths have a simple and complete schema representations that the consumer can choose from when GET, or POST and PUT. You choose whether you want to get a full listing or summary listing -- we can also craft other views in the future.

This conversation is probably one of the most pressing regarding the v1.1 design discussion.

Let's say I have an organizational phone id (let's say 12345), and I want to get the whole record from the API. But I don't necessarily know the right organization id.

The format for phones is: /organizations/{orgId}/phones/{phoneId}

Is there a way for me to GET (or PUT changes) without knowing the organization id? Based on my current reading of version 1.1, there isn't. Should we add something like Google that would let me GET /organizations/-/phones/12345 for this situation?

I want to introduce this thread to overlap with discussions around data filtering #22, schema filtering #21, response structure #6, and hypermedia #7.

Basics of content negotiation would be, do we offer application/csv, and text/html, as well as applicaiton/json (which is current default).

Advanced would be offering application/vnd.api+json, application/hal+json, or application/vnd.siren+json.

I'm thinking that offering a simple CSV and JSON response for default make sense for newbies. Content negotiation 101. Then we could start loading up some of the more detailed requests, and solving the "return everything" and "return exactly what I want" with introduction of hypermedia type, or even possibly something like application/vnd.hsds+api.

I'd like to introduce guidance for messaging. Suggesting it is a standalone layer to the system that other areas use. To suppor the approval, and other aspects of the system we will need messaging.

I'm thinking this will include email, SMS, and potentially Github issues and Slack integration when it makes sense. Possibly even a templating system.

We need use cases for people who are going to be using any HSDS/A implementation.

Submit an issue with any format you want about a use case you'd like to see the API serve.

Please provide as much detail as you possibly can.

There are personas in Issue #24 , but I'd like to hear real world scenarios, not just personas.

I wanted to have an issue for continuing discussions around the paths that have been defined so far.

Currently all paths reflect the schema, but are very plain English, and intuitive.

I am using dashes instead of underscores. Except path variables use underscores as defined by the schema.

It's a good start.

Opening up a discussion around a universal unique ID system for records, that all vendors can participate it, allowing single records to have a unique IDs across sytems. Need to conduct more research on best practices, and engage in conversation with vendors.