`500` error encountered on `flows/<flow_id>/runs` requests due to size of payload about metaflow-service HOT 1 OPEN

For others that have run across this issue, one workaround is to use the ui-backend interface (assuming you're using the default outerbounds terraform module), which has much richer server-side filtering features. The original cause of the issue above is that the Metaflow client does a bulk request of all runs from the metadata-service API which lacks filtering features, and then does the filtering client-side.

E.g. to get the latest run for a given flow with given tags:

response = requests.get(f"https://<api_gateway_hostname>/api/runs?_order=-ts_epoch&_limit=30&_group_limit=31&_tags=<tags>&flow_id=<flow_name>&_page=1").json()
if not response['data']:
    print(f"No data")

run_id = response['data'][0]['run_id']

run = Run(f"{flow_name}/{run_id}")

Are there any thoughts on making the metadata API adopt the server-side filtering behavior of the UI backend API?

from metaflow-service.