kbrock84 / gatsby-source-fetch Goto Github PK

The current implementation only sends one request and does not check for failure.

Update gatsby-node.js to implement a retry
and better handle errors when the request is not successful.

Users should be able to configure the retry attempts and configure timeout lengths, but there should be acceptable defaults:

module.exports = {
	plugins: [
		// fetch data from some endpoint and load it into graphql
		{
			resolve: `gatsby-source-fetch`,
			options: {
				name: `footer`,
				type: `html`,
				url: `https://www.some-domain.com/common-html/header.html`,

                                retry: {
                                    times: 4,
                                    // possibly some form of configuring timeouts?
                                    timeout: [ 100, 2000, 30000, 30000 ],
                                    // or just
                                    timeout: 2000
                              }
                       }
                }
       ]
}

Thanks for this plugin, it's great.

Is there any way to use the url option of the source plugin as a function? I believe that Gatsby allows for this.

I don't know the url ahead of time because it's on a CDN and it changes on every update.

Right now I have:

{
  resolve: `gatsby-source-fetch`,
  options: {
    name: `brands`,
    type: `brands`,
    url: `https://a-fixed-url.com`,
    method: `get`,
    axiosConfig: {
      headers: { Accept: "text/csv" },
    },
    saveTo: `${__dirname}/src/data/file.csv`,
    createNodes: false,
  },
}

But could it be something like:

{
  resolve: `gatsby-source-fetch`,
  options: {
    name: `brands`,
    type: `brands`,
    url: async () => {
        // Some async request...
        const url = await requestToApi()
        return url,
    },
    method: `get`,
    axiosConfig: {
      headers: { Accept: "text/csv" },
    },
    saveTo: `${__dirname}/src/data/file.csv`,
    createNodes: false,
  },
}

I'm not sure this is possible at all?

Many thanks for your help.

EDIT: I just created a local version for my project and implemented this feature. Would you be open to a PR for this? If so, I'll submit one.