/iocs/combined/indicator/v1 bugs out on parameters over 10k about gofalcon HOT 12 CLOSED

Most of the CrowdStrike APIs are designed to stop providing results after 10,000. You can see this mentioned in some of the support documentation. I know this happens for /incidents/combined/crowdscores/v1, /devices/queries/host-group-members/v1 and /devices/queries/devices/v1, among others.

Unfortunately, most of the APIs don't explain what's going on. The only unique error I found (rather than a HTTP 500, contact support) is from /incidents/combined/crowdscores/v1. If you attempt limit=2500&offset=10000, it says HTTP 400: total sum of limit (2500) and offset (10000), 12500 is greater than max window value 10000.

Only a select few of the APIs will continue beyond 10,000. /devices/queries/devices-scroll/v1, /spotlight/queries/vulnerabilities/v1 and most of the /intel/ endpoints (plus others I can't remember).

In PSFalcon I wrote some additional error checking at the end of the Invoke-Loop function for when an HTTP error is returned at 10,000 results because this isn't very clear to the end user.

from gofalcon.

Thank you all for the helpful insights. I really appreciate everyone chiming in!

I've created a workaround by querying 1 day worth of IOCs added at a time using a filter, like so:

&filter=(created_on:>='2021-01-01'+created_on:<='2021-01-02')
&filter=(created_on:>='2021-01-02'+created_on:<='2021-01-03')
&filter=(created_on:>='2021-01-03'+created_on:<='2021-01-04')
...
(until present day)

Sample Workaround Code

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/crowdstrike/gofalcon/falcon"

	"github.com/crowdstrike/gofalcon/falcon/client"
	"github.com/crowdstrike/gofalcon/falcon/client/ioc"
)

// getCrowdstrikeIOCs returns a channel of all the Custom IOC values in the system
func getCrowdstrikeIOCs(iocsCh chan string, client *client.CrowdStrikeAPISpecification) error {

	// find the first day an IOC was added and the total number of IOCs in the system
	// we need to do this because there is a 10k limit to CS API queries
	var fLimit int64
	sort := "created_on.asc"
	fLimit = 1
	fParams := ioc.NewIndicatorCombinedV1Params().WithDefaults()
	fParams.SetLimit(&fLimit)
	fParams.SetSort(&sort)
	fRes, err := client.Ioc.IndicatorCombinedV1(fParams)
	if err != nil {
		return err
	}
	fTotal := *fRes.GetPayload().Meta.Pagination.Total
	firstCreatedDate, err := time.Parse(time.RFC3339, fRes.GetPayload().Resources[0].CreatedOn.String())
	if err != nil {
		return err
	}

	// if there, is over 10k IOCs, work forwards in time by 1 day until all IOCs have been returned
	// might not work if you somehow added 10k IOC within a day
	if fTotal > 10000 {
		go func() {
			for firstCreatedDate.Before(time.Now()) {
				// filter :=
				var limit, offset, total int64
				limit = 2000
				offset = 0
				total = 0
				startTime := firstCreatedDate.UTC().Format(time.RFC3339Nano)
				endTime := firstCreatedDate.Add(time.Hour * 24).UTC().Format(time.RFC3339Nano)
				filter := fmt.Sprintf(`(created_on:>='%s'+created_on:<='%s')`, startTime, endTime)
				// fmt.Println(filter)
				for offset <= total {
					params := ioc.NewIndicatorCombinedV1Params().WithDefaults()
					params.SetOffset(&offset)
					params.SetLimit(&limit)
					params.SetSort(&sort)
					params.SetFilter(&filter)
					res, _ := client.Ioc.IndicatorCombinedV1(params)
					for _, ioc := range res.GetPayload().Resources {
						iocsCh <- ioc.Value
					}
					total = *res.GetPayload().Meta.Pagination.Total
					if total > 10000 {
						fmt.Printf("[WARNING]: # IOCs on the day of %s exceeds 10k, not returning some IOCs", startTime)
						break
					}
					offset += limit
				}
				firstCreatedDate = firstCreatedDate.Add(time.Hour * 24)
			}
			close(iocsCh)
		}()
		return nil
	}

	// if there's less than 10k IOCs, just return like normal via pagination, 2000 results at a time
	go func() {
		var limit, offset, total int64
		limit = 2000
		offset = 0
		total = 0
		for offset <= total {
			params := ioc.NewIndicatorCombinedV1Params().WithDefaults()
			params.SetOffset(&offset)
			params.SetLimit(&limit)
			res, _ := client.Ioc.IndicatorCombinedV1(params)
			for _, ioc := range res.GetPayload().Resources {
				iocsCh <- ioc.Value
			}
			total = *res.GetPayload().Meta.Pagination.Total
			offset += limit
		}
		close(iocsCh)
	}()
	return nil
}

func main() {
	falconClientId := os.Getenv("FALCON_CLIENT_ID")
	falconClientSecret := os.Getenv("FALCON_CLIENT_SECRET")
	clientCloud := os.Getenv("FALCON_CLOUD")

	client, err := falcon.NewClient(&falcon.ApiConfig{
		ClientId:     falconClientId,
		ClientSecret: falconClientSecret,
		Cloud:        falcon.Cloud(clientCloud),
		Context:      context.Background(),
		Debug:        false,
	})
	if err != nil {
		return err
	}
	iocsCh := make(chan string, 10000)
	err = getCrowdstrikeIOCs(iocsCh, client)
	if err != nil {
		return err
	}
	c := 0
	for ioc := range iocsCh {
		c++
		fmt.Println(ioc)
	}
	fmt.Printf("Number of IOCs returned: %d\n", c)
	// over 10k
}

from gofalcon.

Reopening. The filtering by day is a work around (eg the filter := fmt.Sprintf(`(created_on:>='%s'+created_on:<='%s')`, startTime, endTime))..... but is there a better way, @jshcodes & @isimluk ?

I'm thinking the right approach here should be using the after parameter from the first example that @kpham-polb gave

The user would need to use the 'after' value in the string to leverage pagination. You would append that after the url '/iocs/combined/indicator/v1?after=longalphanumericstring&limit=XX'.

In this case we'd get the next 2k results (after the first 2k) by using:
https://api.crowdstrilke.com/iocs/combined/indicator/v1?limit=2000&after=WzE2Mjc0Mjk2MDEwODQsImM0MzM5YTk4NDgwNDc3OWE2NjNlOTU5MzNlZWZiZTg1ZTA0ODQ4ODczMjFiYzQzNWRlOTNjYjcxZDc5NDIyMTQiXQ==

Of note, after and offset are mutually exclusive. They should not be used at the same time. When paging after will be removed from the response indicating there are no more resources to return.

from gofalcon.

Reopening. The filtering by day is a work around (eg the filter := fmt.Sprintf(`(created_on:>='%s'+created_on:<='%s')`, startTime, endTime))..... but is there a better way, @jshcodes & @isimluk ?

Perhaps we should publish some pagination examples? IIRC @kylesmartin and/or @crowdstrikedcs did something with pagination recently.

from gofalcon.

I just tested and verified that after works for both /iocs/combined/indicator/v1 and /iocs/combined/queries/v1

Confirmed working and available in FalconPy also (but dropped in Service Classes due to parameter abstraction, they would have to use the Uber class for now.) It's just not listed in the documentation. Will update this as well.

from gofalcon.

/iocs/combined/queries/v1

Endpoint should be /iocs/queries/indicators/v1 for this one.

This parameter has been added to the IOC endpoint module within FalconPy and will be available to Service Classes in versions 0.5.7 and above.

from gofalcon.

+cc @jshcodes for collaboration on this too (who maintains the python SDK, which this also likely affects).

from gofalcon.

Just had a discussion question about this the other day in the FalconPy repo, it does impact how requests are performed via library just like here. (example: QueryDevicesByFilter vs QueryDevicesByFilterScroll)

from gofalcon.

I'm thinking the right approach here should be using the after parameter from the first example that @kpham-polb gave

The user would need to use the 'after' value in the string to leverage pagination. You would append that after the url '/iocs/combined/indicator/v1?after=longalphanumericstring&limit=XX'.

In this case we'd get the next 2k results (after the first 2k) by using:
https://api.crowdstrilke.com/iocs/combined/indicator/v1?limit=2000&after=WzE2Mjc0Mjk2MDEwODQsImM0MzM5YTk4NDgwNDc3OWE2NjNlOTU5MzNlZWZiZTg1ZTA0ODQ4ODczMjFiYzQzNWRlOTNjYjcxZDc5NDIyMTQiXQ==

Of note, after and offset are mutually exclusive. They should not be used at the same time. When paging after will be removed from the response indicating there are no more resources to return.

We should have some of examples this, I'll start on one for the falconpy repo.

from gofalcon.

I just tested and verified that after works for both /iocs/combined/indicator/v1 and /iocs/combined/queries/v1, but it's not specified in swagger.

Thanks for pointing it out... updating PSFalcon to use it instead. after is the better choice whenever it's available because it will not suffer from the 10,000 result limit.

from gofalcon.

How do I use after in gofalcon?

from gofalcon.

How do I use after in gofalcon?

PR #103 brings the ability to set after to the gofalcon.

from gofalcon.