andreasfaust / gatsby-source-custom-api Goto Github PK

View Code? Open in Web Editor NEW

52.0 52.0 25.0 8.87 MB

Source data from any API and transform it to (File-)Nodes.

JavaScript 100.00%

gatsby-source-custom-api's People

Stargazers

Watchers

gatsby-source-custom-api's Issues

Option to add type definition that are not Node type in schema.

Currently in schema whatever fields we define they get converted into node type.
In gatsby you can define type definitions type ABC implements NODE but to define simple objects you can do it this way type XYZ is it possible in the plug in or it should be done manually. That's how nested objects are handled in gatsby link and can be done in this plugin too.

Just like imageKeys if we can differentiate which schema objects would be Nodes and which would be simple nested objects that would be great for type definition

The plugin does not work with gatsby-plugin-image

The plugin works fine with gatsby-image, but as you may know, gatsby-image is now deprecated and replaced with gatsby-plugin-image. Images are being downloaded but they are inaccessible.

image { local { childImageSharp { gatsbyImageData } } }

returns null always.

Username is not defined

There seems to be an issue with the new auth object. It's supposed to be optional, but if I look at the line in the PR, there is no check whether that object is actually provided or not.

This is resulting in the following error and breaking the plugin before it manages to download anything.

TypeError: Cannot read property 'username' of undefined
    at createImageNodes
(/Users/home/Development/test/node_modules/gatsby-source-custom-api/utils/loadImages.js:21:35)
    at /Users/home/Development/test/node_modules/gatsby-source-custom-api/utils/loadImages.js:96:14
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at async Promise.all (index 5)
    at Object.exports.sourceNodes
(/Users/home/Development/test/node_modules/gatsby-source-custom-api/gatsby-node.js:47:14)
    at runAPI (/Users/home/Development/test/n

Is it possible to set up an endpoint with a variable?

For example, the api I'm using pulls a list of job posts from the endpoint: https://myapi.com/jobs/ which will return jobs in the format:
jobs: [{ id: 123, name: job 1 }, { id: 456, name: job 2}]
I then need to query the endpoint https://myapi.com/jobs/job.id to grab the information on that specific job. Any idea how would I set something like that up?

unable to group by nest field

Hi,
I have data in json format with some nested field that I want to use for grouping the result.
however I dont get any result if I group by nested field. e.g
this query doesnt return any result. however if i group by date(which is not a nested field), it works.
{ allVideos { group(field:brand___name) { edges { node { id title } } } } }

videos: [ { title: "xyz", date: "2019-03-24", brand: { name: "documentaries", }, { title: "dddd", date: "2019-03-24", brand: { name: "documentaries", }, { title: "aaa", date: "2019-03-15", brand: { name: "comedy", } }]
Is there anything happening on plugin that preventing it?. thanks

warn Calling "touchNode" with an object containing the nodeId is deprecated.

Image and or missing field issues - You are probably using the wrong version of Gatsby!

I've been having problems with this plugin as other have described.

My issue in particular was that after specifying image types for import the images would not be associated with the parent data objects in GraphQL. I noticed others have commented on potential problems with the version of Gatsby being used.

I have now confirmed this plugin works fine, as described, but as long as you are using Gatsby version 2.3.6 and React version 16.11.0.

The author should update the README.md as such.

Cheers,
Mike

Issue when building pages programmatically

I am experiencing a problem when building pages programmatically. It appears that the data is missing when I make a query.
For example: I feed the plugin with JSON data from an external server. I have an array of deeply nested objects containing data about "Projects". In gatsby-node.js I make a successful query likeso

allProjects {
  edges {
    node {
      slug
      inprogress
      single {
        type
      }
    }
  }
}

And

result.data.allProjects.edges.forEach(({node}) => {
    if (node.slug && !node.inprogress) {
      createPage({
        path: node.slug,
        component: template(node.singleProject.type),
        context: {
          slug: node.slug,
        },
      });
    }
  });

Then when I make query in my template js file for the single project page I get undefined
Here is the query

projects(slug: { eq: $slug }) {
      name
      metaTitle
      metaDescription
      slug
      singleProject {
        ...
      }
    }

In development mode when I try to query in the GraphQL console I get only one project as a result instead of like 10 I have in the data.
Please help!

Getting only first element from array

With the data simillar to the one from manual:

const exampleDataFromApi = [ { url: "post-1", author: "John doe" }, { url: "post-2", author: "Jane Doe" } ];

and following schema:
schemas: { posts: url: String author: String }

with a following query:
query MyQuery { posts { url author } }

I'm getting only the first element not the whole array.

auth is not defined

I thought auth was optional but I appear to be getting an error:

success createSchemaCustomization - 0.006s

 ERROR #11330  PLUGIN

"gatsby-source-custom-api" threw an error while running the sourceNodes lifecycle:

auth is not defined

  45 |
  46 |   // load images or default-dummy-image
> 47 |   entities = await loadImages({
     |              ^
  48 |     entities, imageKeys, createNode, createNodeId, touchNode, store, cache, createContentDigest, auth
  49 |   })
  50 |

Make sure that you don't have a typo somewhere and use valid arguments in sourceNodes lifecycle.
Learn more about sourceNodes here: https://www.gatsbyjs.com/docs/reference/config-files/gatsby-node/#sourceNodes

File: node_modules/gatsby-source-custom-api/gatsby-node.js:47:14

success Checking for changed pages - 0.001s

The api I'm querying is here:

http://sculpture-poetry.du.st/conferences.json

and my config:

{
      resolve: "gatsby-source-custom-api",
      options: {
        url: {
          production: "http://sculpture-poetry-cms.test/conferences.json",
          development: "http://sculpture-poetry-cms.test/conferences.json",
        },
        rootKey: "spConferences",
        schemas: {
          spConferences: `
            data: [data]
          `,
          data: `
            uid: String
            title: String
            description: String
            location: String
            byline: String
            startDate: String
            endDate: String
            button: button
            events: [events]
          `,
          events: `
            uid: String
            slug: String
            url: String
            title: String
            type: String
            description: String
            image: image
            eventLink: eventLink
            speakers: [speakers]
            date: String
            startTime: String
            endTime: String
          `,
          speakers: `
            uid: String
            name: String
            url: String
            slug: String
          `,
          image: `
            url: String
          `,
          button: `
            customText: String
            value: String
          `,
          eventLink: `
            customText: String
            value: String
          `,
        },
      },
    },

Schema hierarchy / image not being downloaded

I've encountered some strange behaviour — I'm building my schemas and it appears that I have to choose between either my images downloading, or to have my api being brought in correctly. The two options I am presented with:

Have my data being brought in with the correct hierarchy (in my case: conferences->events->image), but the images don't get downloaded.
Have my data being brought in with a flat hierarchy (conferences, events->image) and with the images downloading.

My api looks like this:

to have the correct hierarchy I need to use this config:

    {
      resolve: "gatsby-source-custom-api",
      options: {
        url: {
          production: "http://sculpture-poetry-cms.test/conferences.json",
          development: "http://sculpture-poetry-cms.test/conferences.json",
        },
        auth: {
          username: "username",
          password: "password",
        },
        rootKey: "spConferences",
        schemas: {
          spConferences: `
            conference: [conference]
          `,
          conference: `
            uid: String
            title: String
            description: String
            location: String
            byline: String
            startDate: String
            endDate: String
            button: button
            events: [event]
          `,
          event: `
            uid: String
            slug: String
            url: String
            title: String
            type: String
            description: String
            image: image
            eventLink: eventLink
            speakers: [speaker]
            date: String
            startTime: String
            endTime: String
          `,
          image: `
            url: String
            modified: String
          `,
          speaker: `
            uid: String
            name: String
            url: String
            slug: String
          `,
          button: `
            customText: String
            value: String
          `,
          eventLink: `
            customText: String
            value: String
          `,
        },
      },
    },

BUT if I change events: [event] to events: [events] and rename the event schema also to match, the images download — though I can then no longer access the event information through the conferences in GraphQL as they appear as null. The images appear under allImages?

Authentication

It would be swell if we could provide a string that the plugin could pass as an HTTP header, the way [GitHub[(https://developer.github.com/v3/#oauth2-token-sent-in-a-header) does to authenticate API calls.

Missing fields in result

Hey,

for a Schema like this:

        rootKey: 'stations',
        schemas: {
          stations: `
            features: [features]
          `,
          features: `
            properties: properties
            geometry: geometry
          `,
          properties: `
            name: String
          `,
          geometry: `
            coordinates: [Float]
          `,

And a query like this:

query MyQuery {
  allFeatures {
    totalCount
    edges {
      node {
        id
        properties {
          name
        }
      }
    }
  }
}

I get incomplete results (properties is null):

{
  "data": {
    "allFeatures": {
      "totalCount": 5556,
      "edges": [
        {
          "node": {
            "id": "d92c142e-fd57-56e7-8c7e-d5401d119dd2",
            "properties": null
          }
        },
[...]

Querying for allProperties or allGeometry works as expected. Is this a Bug (sounds like #8)?

Build warning message

In my project, i have many warnings type: "Object 'pagesComments': Better provide a schema!" and "The key "id" is restricted in GraphQl!". How can i hide or turn off this messages?

Image downloading but not accessible — also bug depending on url.

Having some difficulty with images and this plugin — there are two main issues I am facing.

Context:

I am using CraftCMS and the ElementAPI to generate a JSON API to use with this plugin.

First bug/issue:

url passed to createRemoteFileNode is either missing or not a proper web uri:

Even though my uri's being passed from craft are like so: /assets/images/L1000642ACR.jpg — it fails to detect/download the image, this appears to match what is in the docs.

To test some things out I appended the site URL to the API so it gave the full URL of the image — this allowed 2 out of 3 test images to download but the third still failed, I can only assume it's something to do with the filename of the image which is quite long and has differing characters within it:

url passed to createRemoteFileNode is either missing or not a proper web uri: http://livvy-cms.test/assets/images/home/Different-Interruptions-of-Dust-41x28cm-photogravure-print-etchin
g-paper-gold-ink-2016-of-the-1979-eruption-of-the-Soufrière-Volcano.jpg
Image downloaded: Post-Card-Image.jpg
Image downloaded: Different-Interruptions.jpg

This isn't ideal for production as the client could upload any image with any filename.
Edit: This is definitely because of the french character è in the filename, not sure how to resolve as of yet.

Second bug/issue

Taking the partial success of the two images that did download — I went into Graphql to see if it had worked. The images downloaded to Gatsby's cache which I assume is correct, but in GraphQL I have no access to:

                    childImageSharp {
                        fluid(maxWidth: 2000) {
                            ...GatsbyImageSharpFluid_withWebp
                        }
                    }

on the local node — any ideas/suggestions would be great.

Edit: I didn't have either sharp plugin installed... hence the images weren't available in GraphQL — worth mentioning in docs? The image name thing is an actual bug though.

How does it discover individual objects?

Hello... Thanks for making this package, it's really useful. I was looking at the code and was wondering how does it discover individual objects given an endpoint?

Suppose there are these two books that the API publishes data on,

https://api.example.com/json/books/1
https://api.example.com/json/books/2

Given a configuration,

{
  resolve: 'gatsby-source-custom-api',
  options: {
    url: 'https://api.example.com/json',
    imageKeys: ['images'],
    rootKey: 'books',
    schemas: {
      book: `
        id: Int
        title: String
        isbn: String
      `,
    },
  },
}

How does it discover the two objects? I wasn't able to make it out from the codebase. If you think this might be also useful for others, maybe it can be added in the documentation. If I'm able to understand it, I'll contribute a PR. Thanks.

"gatsby-source-custom-api" threw an error while running the sourceNodes lifecycle:

All about this issue gatsbyjs/gatsby#17628

I want to get only first 3 elements from an array. Is that possible ?

Let's say we're getting an array of 5 items from the API. Can we get only 3 items, not all the 5 items?

{
  "data": {
    "finance": {
      "data": [
        {
          "id": "590522000022443198",
          "City": "Melaka"
        },
        {
          "id": "590522000025688127",
          "City": "Limassol"
        },
        {
          "id": "590522000037762639",
          "City": "Cyberjaya"
        },
        {
          "id": "590522000037762487",
          "City": "Labuan"
        },
        {
          "id": "590522000022432183",
          "City": "Ipoh"
        }
      ]
    }
  },
}

Unable to use netlify-plugin-gatsby-cache cache

Hi,

it seems that netlify-plugin-gatsby-cache cache is always being deleted when using this plugin.

8:21:45 PM: info One or more of your plugins have changed since the last time you ran Gatsby. As
8:21:45 PM: a precaution, we're deleting your site's cache to ensure there's no stale data.

When I remove this plugin (and the page that is using API data) cache is being read correctly, isn't being deleted and build time on consecutive builds is much shorter.

Add support for source_url for images

Or actually, just total custom childkey to specify what the url key is called. I'm pulling data into a Node backend service from Wordpress, and Wordpress' image urls are labeled "source_url". I don't want to use the gatsby-source-wordpress plugin because I want it all to run through a gateway and backend service first as a proxy. Thus, I am stuck using Wordpress' API and data structures, but they don't fit this plugin's url chilldkey structure.

Unless I am misunderstanding how to use the plugin, this seems way too rigid for a wide range of usage as not everyone is in control of the key names for their images.

build warning messages regarding nodes without an explicit resolver

thanks for this great plugin first of all!

I noticed getting these warnings:
warn Deprecation warning - adding inferred resolver for field .... In Gatsby v3, only fields with an explicit directive/extension will get a resolver.

So I went into getTypeDef.js and edited line 4 to this:
type ${name} implements Node @infer{

(adding @infer)
now my warnings were gone but honestly.. I'm a beginner at this and not sure if this would break anything. Would love to see an update if you are still working on this plugin and help anywhere if I can

Does that work with gatsby v4?

Any reference to migrate this to gatsbyv4?

Add support for image urls that are just strings

My data source api provides images just as a key value of type string (not object with key url).

Therefore, it would be nice if I could specify that the image is just a string.

Add support for env variable urls

When I use an environment variable to store my endpoint, I get a type error saying only absolute urls are supported. Please add support for environment variables

Plugin runs multiple times

Hey, I'm running the plugin v2.3.5 with Gatsby v4.7.2 and I noticed something strange in my logs. I am using 4 instances of gatsby-source-custom-api as per the guidance in the README, all with a valid schema. And the strange thing is, unique file names (downloaded assets) show up 4 times as well. So it appears that if I have X instances of the plugin, it will run all of them for each instance. This is of course increasing build times and may be exacerbating this memory issue on Netlify for me. My config is as follows, is there anything I am doing wrong that would cause this? At first I thought it might be a general issue with my config, but it looks like this is the only plugin doing this.

Update: after more research on my local machine, it looks like it's more arbitrary than the number of instances. On a fast machine, file names are showing up 10x. I've turned off parallel sourcing and removed all page generation from gatsby-node.js to isolate this but it makes no difference.

Update 2: I've now run this same plugin v2.3.5 on the same project in a different branch still running Gatsby v3.14.5 and filenames only show up once. So there appears to be an incompatibility between this plugin and Gatsby 4.

    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "jobCategoriesEu",
        url: "https://boards-api.greenhouse.io/v1/boards/xxxEu/departments.json",
        schemas: {
          jobCategoriesEu: `
              departments: [departments] @link(by: "id", from: "departments___NODE")
            `,
          departments: `
              department: [department] @link(by: "id", from: "department___NODE")
            `,
          department: `
              id: ID!
              name: String!
              parent_id: ID
              child_ids: [Int]
              jobs: [job] @link(by: "id", from: "job___NODE")
            `,
          job: `
              id: ID!
              title: String!
              absolute_url: String!
              internal_job_id: Int
              updated_at: String
              location: location @link(by: "id", from: "location___NODE")
              data_compliance: [compliance] @link(by: "id", from: "compliance___NODE")
            `,
          location: `
              name: String
            `,
          compliance: `
              type: String
              requires_consent: Boolean
              retention_period: String
            `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "jobCategoriesUa",
        url: "https://boards-api.greenhouse.io/v1/boards/xxxUa/departments.json",
        schemas: {
          jobCategoriesUa: `
              departments: [departments] @link(by: "id", from: "departments___NODE")
            `,
          departments: `
              department: [department] @link(by: "id", from: "department___NODE")
            `,
          department: `
              id: ID!
              name: String!
              parent_id: ID
              child_ids: [Int]
              jobs: [job] @link(by: "id", from: "job___NODE")
            `,
          job: `
              id: ID!
              title: String!
              absolute_url: String!
              internal_job_id: Int
              updated_at: String
              location: location @link(by: "id", from: "location___NODE")
              data_compliance: [compliance] @link(by: "id", from: "compliance___NODE")
            `,
          location: `
              name: String
            `,
          compliance: `
              type: String
              requires_consent: Boolean
              retention_period: String
            `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "pressReleases",
        url: "https://api.pr.co/v1/pressrooms/xxxxx/press_releases.json",
        schemas: {
          pressReleases: `
            data: [data] @link(by: "id", from: "data___NODE")
            paging: [paging] @link(by: "id", from: "paging___NODE")
          `,
          data: `
            press: [press] @link(by: "id", from: "press___NODE")
          `,
          press: `
            id: ID!
            pressroom_id: ID!
            title: String!
            subtitle: String
            release_date: String
            release_location: String
            language: String
            social_media_pitch: String
            summary: String
            body_html: String
            permalink: String
            type: String
            state: String
            short_link: String
            pdf: String
            show_in_timeline: Boolean
            content_as_json: String
            draft_content_as_json: String
            freeform_two: Boolean
            reading_time: Int
            updated_at: String
          `,
          paging: `
            page: Int
            limit: Int
            total: Int
          `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "pressKits",
        url: "https://api.pr.co/v1/pressrooms/xxxx/presskits.json",
        imageKeys: ["original"],
        schemas: {
          pressKits: `
            data: [data] @link(by: "id", from: "data___NODE")
          `,
          data: `
            media_counts: [media_counts] @link(by: "id", from: "media_counts___NODE")
            paging: [paging] @link(by: "id", from: "paging___NODE")
            media: [media] @link(by: "id", from: "media___NODE")
          `,
          media_counts: `
            documents: Int
            images: Int
            sounds: Int
            videos: Int
            total: Int
          `,
          paging: `
            page: Int
            limit: Int
            total: Int
          `,
          media: `
            sizes: sizes @link(by: "id", from: "sizes___NODE")
            title: String
            file_size: Int
            content_type: String
          `,
          sizes: `
            original: original @link(by: "id", from: "original___NODE")
            medium: medium @link(by: "id", from: "medium___NODE")
            large: large @link(by: "id", from: "large___NODE")
            square: square @link(by: "id", from: "square___NODE")
          `,
          original: `
            url: String
            height: String
            width: String
          `,
          medium: `
            url: String
          `,
          large: `
            url: String
          `,
          square: `
            url: String
          `,
        },
      },
    },

Trouble fetching REST endpoints

Hi, I have some trouble fetching data from a custom REST API. I can't figure out how to tell the plugin which endpoints to use.

Example:

Endpoint for posts: GET http://localhost:3000/posts
Endpoint for authors: GET http://localhost:3000/authors

{
  resolve: "gatsby-source-custom-api",
  options: {
    url: {
      development: "http://localhost:3000", // on "gatsby develop"
      production: "http://localhost:3000", // on "gatsby build"
    },
    imageKeys: ["image"],
    schemas: {
      posts: `
        id: String
        title: String
        description: String
        image: image
        path: String
      `,
      authors: `
        id: String
        firstName: String
        lastName: String
        path: String
      `,
    },
  },
}

It seems to fetch data only from the base url http://localhost:3000, but the endpoints /authors and /posts are not queried.

Could you help me out here? Thanks in advance!

Can we define default values for a given key?

Hi!

In order to avoid more complexity at the development time and maybe crashes at building time due to the possibility of a server not being running can we add default values on a given schema? For instance I have:

    {
      resolve: 'gatsby-source-custom-api',
      options: {
        url: `${process.env.GATSBY_COMMUNITY_URL}/about.json`,
        rootKey: 'about',
        schemas: {
          about: `
            user_count: Int
          `
        }
      }
    }

Which I would like to say something like:

    {
      resolve: 'gatsby-source-custom-api',
      options: {
        url: `${process.env.GATSBY_COMMUNITY_URL}/about.json`,
        rootKey: 'about',
        schemas: {
          about: `
            user_count: Int = 33000 <-- notice the lame attempt here
          `
        }
      }
    }

It feels a bit overkill that the entire page crashes cause the server is not available. Any thoughts?
Thanks a lot for this solution btw, really appreciate the work have been done here!

andreasfaust / gatsby-source-custom-api Goto Github PK

gatsby-source-custom-api's People

Stargazers

Watchers

Forkers

gatsby-source-custom-api's Issues

Context:

First bug/issue:

Second bug/issue

Edit: I didn't have either sharp plugin installed... hence the images weren't available in GraphQL — worth mentioning in docs? The image name thing is an actual bug though.

Recommend Projects

Recommend Topics

Recommend Org