Goal: Improve file change -> generation latency

https://github.com/parcel-bundler/watcher

Unfortunately in some situations e.g. when changing the document type definitions which causes the TS type definitions to be regenerated, the TS LSP in VSC doesn't automatically pick up those changes. This seems to be a general problem with TS and VSC.

For now users need to manually Restart TS server but we could also investigate other solutions to this problem similar to what the Prisma VSC extension is doing.

Hi. I can't import anything from 'next-contentlayer':

const { withContentlayer } = require('next-contentlayer');

error - Failed to load next.config.js, see more info here https://nextjs.org/docs/messages/next-config-error
ReferenceError: withContentLayer is not defined

The same for:

import { useMDXComponent } from 'next-contentlayer/hooks';

Eslint shows:

Unable to resolve path to module 'next-contentlayer/hooks'

Package.json:

   "contentlayer": "0.0.32",
   "next-contentlayer": "0.0.32",

In some error messages (e.g. for missing required fields) we should make use of the Levenshtein distance to suggest similar field names in case of typos.

I currently have an mdx-bundler wrapper function something like this:

    return await bundleMDXFile(filePath, {
      cwd: path.dirname(filePath),
      esbuildOptions: options => {
        options.platform = 'node'
        // Set the `outdir` to a public location for this bundle.
        options.outdir = `${process.cwd()}/public/img/${getSlug(filePath)}`
        options.loader = {
          ...options.loader,
          // Tell esbuild to use the `file` loader for pngs
          '.png': 'file',
          '.jpg': 'file'
        }
        // Set the public path to /img/about
        options.publicPath = `/img/${getSlug(filePath)}`

        // Set write to true so that esbuild will output the files.
        options.write = true

        return options
      },
      xdmOptions: options => {
        options.remarkPlugins = [
          ...(options.remarkPlugins ?? []),
          ...remarkPlugins
        ]
        options.rehypePlugins = [
          ...(options.rehypePlugins ?? []),
          ...rehypePlugins
        ]
        return options
      }
    })

You'll see that I'm able to use details from the context to configure esbuild for mdx-bundler. I use the filePath I've passed in to set an assets bundle path. I can also set the cwd to the folder of the file being bundled, allowing for easy linking of assets in the same folder as the MDX file (bundleMDXFile does it automatically). These options enable some nice functionality.

In contentlayers config I could imagine something like this:

 mdx: context => {
    return {
      cwd: path.dirname(context.filePath),
      remarkPlugins: [
        remarkGfm
      ],
      rehypePlugins: [
        [rehypePrism, { ignoreMissing: true }]
      ],
      esbuildOptions: options => {
        options.platform = 'node'
        // Set the `outdir` to a public location for this bundle.
        options.outdir = `${process.cwd()}/public/img/${getSlug(context.filepath)}`
        options.loader = {
          ...options.loader,
          // Tell esbuild to use the `file` loader for pngs
          '.png': 'file',
          '.jpg': 'file'
        }
        // Set the public path to /img/about
        options.publicPath = `/img/${getSlug(context.filepath)}`
        // Set write to true so that esbuild will output the files.
        options.write = true
        return options
      }
    }
}

When adding = in the file contents, the dev script fails to compile for contentlayer. I have verified that there are total 3 files so far which have these symbols like (=, <=, >= etc).

Note these characters are directly present and probably parsed into <p></p> tags because there are other files which have these characters but are inside <pre></pre> tag i.e written in MDX inside codeblock section. for eg:

10 >= 9

Attaching screenshot of the error

Error text

 > _mdx_bundler_entry_point-92faee0a-865e-4538-91df-d8bf5b84daff.mdx:48:5:  > _mdx_bundler_entry_point-3c6e9d52-b8b2-4b35-b009-5eb90f57de8e.mdx:41:49: error: error: [plugin: esbuild-xdm] [plugin: esbuild-xdm] Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`
Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`
    41 │ ... line i of the n subsequent lines ( where 0 <    48 │ • 1=  i < n ) contains the i<sup>th</sup> value ge...
       ╵                                                 <= n <= 10^5
       ╵    ^^



 > _mdx_bundler_entry_point-c7153f82-223a-4f26-b087-547faea0ad82.mdx:109:110: error: [plugin: esbuild-xdm] 
Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`
    109 │ ...ray, you have to return the total age of all the persons, given that no person has age <= 20. 
        ╵                                                                                            ^     

Error: Found problems in 3 of 16 documents.

 └── Encountered unexpected errors while processing of 3 documents. This is possibly a bug in Contentlayer. Please open an issue.

     • "blog/hackwithinfy-round-1-experience-with-questions.mdx": UnexpectedMDXError: Error: Build failed with 1 error:
     _mdx_bundler_entry_point-92faee0a-865e-4538-91df-d8bf5b84daff.mdx:48:5: error: [plugin: esbuild-xdm] Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`
     • "blog/hackwithinfy-round-2-experience-with-questions.mdx": UnexpectedMDXError: Error: Build failed with 1 error:
     _mdx_bundler_entry_point-3c6e9d52-b8b2-4b35-b009-5eb90f57de8e.mdx:41:49: error: [plugin: esbuild-xdm] 
Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`
     • "blog/learn-reduce-method-in-javascript-with-code-examples.mdx": UnexpectedMDXError: Error: Build failed with 1 error:
     _mdx_bundler_entry_point-c7153f82-223a-4f26-b087-547faea0ad82.mdx:109:110: error: [plugin: esbuild-xdm] Unexpected character `=` (U+003D) before name, expected a character that can start a name, such as a letter, `$`, or `_`


SourceFetchDataError: {
  "_tag": "HandledFetchDataError"
}

Contentlayer version: 0.0.28-dev.12
OS: Windows 10

Currently the Contentful source plugin doesn't support incremental builds but refetches all the data on every build and implements live-reload via polling all content.

We should add support for incremental builds & live-reloading content for Contentful via the Contentful Sync API

I've tried to download MDX Example for Next.js, but when I tried to run this app in development mode, I get this error. Same thing with implementing it directly to my project.

Error: spawn C:\Users\Desktop\mdx\node_modules\@contentlayer\core\node_modules\esbuild\bin\esbuild ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:480:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (node:internal/child_process:288:12)
    at onErrorNT (node:internal/child_process:480:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn C:\\Users\\Desktop\\mdx\\node_modules\\@contentlayer\\core\\node_modules\\esbuild\\bin\\esbuild',
  path: 'C:\\Users\\Desktop\\mdx\\node_modules\\@contentlayer\\core\\node_modules\\esbuild\\bin\\esbuild',
  spawnargs: [ '--service=0.12.29', '--ping' ]
}

For projects using contentlayer/source-files processing many content files (> 50) it could make sense to process the files in parallel using a Node worker pool in order to speed things up.

This will most likely require some architectural adjustments and further exploration.

One nice feat about mdx-bundler is that it allows for image bundling through import statements.
https://github.com/kentcdodds/mdx-bundler#image-bundling
Implementing this would be especially helpful for the very limited next/image, because there wouldn't be the need for width/size specification and it would provide a blur hash automatically.

ESBuild seems to fail to install

As seen, there is no binary.
Config: https://sourceb.in/BSbso4rW2k

node:events:368
      throw er; // Unhandled 'error' event
      ^

Error: spawn C:\Users\milo5\Desktop\GitHub\website\node_modules\@contentlayer\core\node_modules\esbuild\bin\esbuild ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:477:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (node:internal/child_process:288:12)
    at onErrorNT (node:internal/child_process:477:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn C:\\Users\\milo5\\Desktop\\GitHub\\website\\node_modules\\@contentlayer\\core\\node_modules\\esbuild\\bin\\esbuild',
  path: 'C:\\Users\\milo5\\Desktop\\GitHub\\website\\node_modules\\@contentlayer\\core\\node_modules\\esbuild\\bin\\esbuild',

And that's the error.
More context:

Next.config.js

const { withContentlayer } = require('next-contentlayer');

/**
 * @type {import('next').NextConfig}
 */
module.exports = withContentlayer()({
	reactStrictMode: true,
	async headers() {
		return [
			{
				source: '/(.*)',
				headers: securityHeaders,
			},
		];
	},
	webpack: (config, { dev, isServer }) => {
		// Replace React with Preact only in client production build
		if (!dev && !isServer) {
			Object.assign(config.resolve.alias, {
				react: 'preact/compat',
				'react-dom': 'preact/compat',
			});
		}

		return config;
	},
});

// https://securityheaders.com
const ContentSecurityPolicy = `
  default-src 'self';
  script-src 'self' 'unsafe-eval' 'unsafe-inline' *.youtube.com *.twitter.com cdn.usefathom.com;
  child-src *.youtube.com *.google.com *.twitter.com;
  style-src 'self' 'unsafe-inline' *.googleapis.com;
  img-src * blob: data:;
  media-src 'none';
  connect-src *;
  font-src 'self';
`;

const securityHeaders = [
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
	{
		key: 'Content-Security-Policy',
		value: ContentSecurityPolicy.replace(/\n/g, ''),
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy
	{
		key: 'Referrer-Policy',
		value: 'origin-when-cross-origin',
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
	{
		key: 'X-Frame-Options',
		value: 'DENY',
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
	{
		key: 'X-Content-Type-Options',
		value: 'nosniff',
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-DNS-Prefetch-Control
	{
		key: 'X-DNS-Prefetch-Control',
		value: 'on',
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security
	{
		key: 'Strict-Transport-Security',
		value: 'max-age=31536000; includeSubDomains; preload',
	},
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Feature-Policy
	// Opt-out of Google FLoC: https://amifloced.org/
	{
		key: 'Permissions-Policy',
		value: 'camera=(), microphone=(), geolocation=(), interest-cohort=()',
	},
];

In some cases where you're using type: 'json' you'd like to reference some existing TS type definitions instead of the any type that's currently used for json fields. This applies to both regular fields and computedFields.

Example:

Rough API idea:

{
  // ...  
  typescriptType: { name: 'IReadTimeResults', from: 'reading-time' }
}

Notes

• A potential other direction/idea for this could be to "derive" nested document types based on existing TS types. (Would address #149)

config:

const I18n = defineDocumentType(() => ({
  name: 'I18n',
  filePathPattern: 'i18n/*.yml',
  fileType: 'yaml',
  fields: {
    '*': { type: 'string' }
  }
}));

error msg:

Warning: Found problems in 2 of 4 documents. Skipping those documents.

 └──   2 documents contain field data which isn't defined in the document type definition
     
     • "i18n/en.yml" of type "I18n" has the following extra fields:
       • title: "Willin Wang"
       • subTitle: "Hello World!" 
     • "i18n/zh.yml" of type "I18n" has the following extra fields:
       • title: "王某人"
       • subTitle: "你好，世界！" 

data.yml.json:

{
  "_id": "i18n/en.yml",
  "_raw": {
    "sourceFilePath": "i18n/en.yml",
    "sourceFileName": "en.yml",
    "sourceFileDir": "i18n",
    "bodyType": "none",
    "flattenedPath": "i18n/en"
  },
  "type": "I18n"
}

how could i store localization messages?

Hey 👋

I came across Contentlayer by a Lee Robinson's live about his blog and I instantly fell in love. I'm really excited about this project and I think this is going to be the next standard way of consuming content from multiple sources. I'm really excited to contribute because I feel Contentful is a must for the web dev community, but I'm not used to contributing at all. I'll dedicate my next days studying this repository to really understand it, but if a great soul could provide a overall architechture/understanding of all the packages and how they fit together, that'd be awesome.

Update: I just watched the Effect Time: Intro to Contentlayer and it's a great introduction. It's more complex than I expected 🥲

What I expect about Contentlayer is that it's going to be the single source of any kind of content a project needs. Instead of using multiple libraries to interact with third party services, Contentlayer would be the only one library needed. Is it correct?

One thing I noticed by quickly looking at the examples is that contentlayer.config.ts only exports one source of content. I don't know if such a case exists were a project consumes content from multiple CMS, for example, at the same time. If such a project exists, would it be possible to export multiple content sources like:

import { makeSourcePlugin } from '@contentlayer/source-contentful'
import { makeSource } from '@contentlayer/source-files'

export const mdx = makeSource({});
export const contentful = makeSourcePlugin({});

And using it like:

import { mdx, contentful } from "./contentlayer/data";

mdx.allPost();
contentful.allPost();

// Proposed syntax
//
// mdx.posts.list()
// contentful.posts.list()

I really like the generated Prisma client, and I think the proposed syntax is very elegant.

I've noticed that in some situations Next.js' HMR stops working after some time. I suspect this rather to be a problem in Next.js than in Contentlayer but I'm not certain yet since I can't yet reliably reproduce the problem.

If someone else faces the same problem, please share your situation below in the comments. 🙏

Effect is currently loaded via utils as a dependency while it should be loaded as peerDependency. Linking libs as deps is generally unadvisable for any dependency but poses a critical risk for effect (and its ecosystem packages) in particular given that it doesn't work if multiple copies are loaded.

https://github.com/contentlayerdev/contentlayer/blob/main/packages/%40contentlayer/utils/package.json#L43

Note that multiple copies will be loaded any time a user of the lib installs a different version of effect (even a patch) in a child project, that isn't the case for peerDependencies with a set range of acceptable versions

See https://nextjs.org/blog/next-11-1#builds--data-fetching

Hello! I haven't had any issues with contentlayer until today. Really love the dx so far! Thanks for your contribution to the JS community.

As per the logs ("This error shouldn't have happened. etc...") I'm opening an issue with the stack trace below. Any advice is appreciated.

stack trace:

An unchecked error was produced.

TypeError: Cannot read property 'get' of undefined
    at ApplyFrame.apply (/Users/lucashomer/lucas-homer-dot-com/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Layer/definitions.js:391:60)
    at FiberContext.nextInstr (/Users/lucashomer/lucas-homer-dot-com/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:315:22)
    at FiberContext.evaluateNow (/Users/lucashomer/lucas-homer-dot-com/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:699:56)
    at /Users/lucashomer/lucas-homer-dot-com/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:458:54
    at /Users/lucashomer/lucas-homer-dot-com/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Support/Scheduler/index.js:34:30

Fiber: #2 (started at: 2021-11-12T04:36:33.145Z) was supposed to continue to: <empty trace>

Fiber: #2 (started at: 2021-11-12T04:36:33.145Z) Execution trace: <empty trace>

Fiber: #2 (started at: 2021-11-12T04:36:33.145Z) was spawned by:

Fiber: #0 (started at: 2021-11-12T04:36:33.140Z) was supposed to continue to:

  a future continuation at packages/@contentlayer/core/src/runMain.ts:15:110

Fiber: #0 (started at: 2021-11-12T04:36:33.140Z) Execution trace:

  packages/@contentlayer/core/src/runMain.ts:15:32
  packages/@contentlayer/core/src/runMain.ts:10:12

Fiber: #0 (started at: 2021-11-12T04:36:33.140Z) was spawned by: <empty trace>

Typescript displays error when I import data or type from .contentlayer:

// Module '".contentlayer/data"' has no exported member 'allJavascripts'.ts(2305)
import { allJavascripts } from ".contentlayer/data";
// Module '".contentlayer/types"' has no exported member 'Javascript'.ts(2305)
import { Javascript } from ".contentlayer/types";

While this still works well, I think it's quite annoying and error prone as there are no suggestions.

Repo:

• https://github.com/renomureza/test-contentlayer-next
• https://github.com/renomureza/test-contentlayer-next/blob/main/pages/javascript/%5Bslug%5D.tsx#L1

Screenshot:

https://twitter.com/threepointone/status/1460634204472029184?s=21

The author is some static value I retrieve from a different file. (field is not visible in output)
The coverImage I want to get if no value is provided (field is not visible)
The categories and tags is a list of strings (field is also not visible)

See below for the code, frontmatter and the result in contentlayer folder.

contentlayer.config.js

const computedFields = {
  readingTime: { type: "json", resolve: (doc) => readingTime(doc.body.raw) },
  wordCount: {
    type: "number",
    resolve: (doc) => doc.body.raw.split(/\s+/gu).length,
  },
  slug: {
    type: "string",
    resolve: (doc) => doc._raw.sourceFileName.replace(/\.md$/, ""),
  },
  author: { type: "json", resolve: (doc) => publisher },
  coverImage: {
    type: "json",
    resolve: (doc) => getCoverImage(doc, defaultData, "coverImage"),
  },
};

const Post = defineDocumentType(() => ({
  name: "Post",
  filePathPattern: `blog/*.md`,
  fields: {
    title: { type: "string", required: true },
    date: { type: "date", required: true },
    tags: {
      type: "list",
      of: { type: "string" },
      typeField: "string",
      required: false,
    },
    categories: {
      type: "list",
      of: { type: "string" },
      typeField: "string",
      required: true,
    },
    coverImage: { type: "string", required: false },
  },
  computedFields,
}));

My blogs frontmatter:

---
title: Concurrency in Go
tags:
  - concurrency
  - parallelism
categories:
  - Software development
  - Golang
date: 2019-01-17T17:21:56.000Z
coverImage: /images/be41df8a96b934821f14f5844949b52604c8b074.png
---

Result:

{
  "title": "Concurrency in Go",
  "date": "2019-01-17T17:21:56.000Z",
  "coverImage": "/images/be41df8a96b934821f14f5844949b52604c8b074.png",
  "body": {
    "raw": "\nA reason to choose Go over other programming languages could be to have the need for building software which requires concurrency. Go is built with concurrency…"
  },
  "_id": "blog/concurrency-in-go.md",
  "_raw": {
    "sourceFilePath": "blog/concurrency-in-go.md",
    "sourceFileName": "concurrency-in-go.md",
    "sourceFileDir": "blog",
    "bodyType": "markdown",
    "flattenedPath": "blog/concurrency-in-go"
  },
  "type": "Post",
  "readingTime": {
    "text": "9 min read",
    "minutes": 8.22,
    "time": 493200,
    "words": 1644
  },
  "wordCount": 1646,
  "slug": "concurrency-in-go"
}

Currently, if I set a field to the date type and enter a partial date, e.g., 2021-10-21 I'm given a date object initialized with a value in UTC, like this 2021-10-21T00:00:00.000Z. Ideally I'd be able to have that partial date interpreted correctly for my timezone — I'm in America/New_York so I'd like to see 2021-10-21T04:00:00.000Z.

A possible, straightforward solution would be the date-fns-tz libraries zonedTimeToUtc function and an optional IANA timezone in the contentlayer config. E.g.

zonedTimeToUtc(doc.createdAt, opts.timezone)

If you're interested in a contribution I could work on a PR.

After having trouble installing contentlayer on my nextjs site, I tried it on a bare new project and still run into the same error. I hope I don't miss anything, but I always end with the following error

ready - started server on 0.0.0.0:3000, url: http://localhost:3000
error - Failed to load next.config.js, see more info here https://nextjs.org/docs/messages/next-config-error
/Users/sk/Desktop/nextjs-blog/node_modules/next-contentlayer/dist/index-cjs.cjs:23
                return nextConfig.redirects?.() ?? [];
                                            ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/Users/sk/Desktop/nextjs-blog/next.config.js:1:30)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ dev: `next dev`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Stepts to reproduce (example repo):

• created a brand new project via
  npx create-next-app nextjs-blog --use-npm --example "https://github.com/vercel/next-learn/tree/master/basics/learn-starter"
• installed contentlayer via npm install contentlayer next-contentlayer
• npm run dev sucessfully starts the server until this step
• added a next.config.js file with the following content:

  const { withContentlayer } = require("next-contentlayer");
  module.exports = withContentlayer()({});

• npm run dev crashes with the error above

Thanks for creating this repo, can't wait to try it out!

Running yarn build locally works fine, however on vercel the build fails with error

I feel this might be similar to the issue mentioned here #41 as it seems the build fails when type checking is done on vercel during compilation step.

Error on windows while running leerob portfolio

Full error:

Warning: Contentlayer might not work as expected on Windows
This error shouldn't have happened. Please consider opening a GitHub issue with the stack trace below here:
https://github.com/contentlayerdev/contentlayer/issues

An unchecked error was produced.

TypeError: Cannot read properties of undefined (reading 'tracer')
    at C:\Users\Ricky\Desktop\leerob.io-main\node_modules\src\Effect\derive.ts:88:76
    at IRead.f (C:\Users\Ricky\Desktop\leerob.io-main\node_modules\src\Effect\has.ts:113:33)
    at FiberContext.evaluateNow (C:\Users\Ricky\Desktop\leerob.io-main\node_modules\src\Fiber\context.ts:1130:51)
    at C:\Users\Ricky\Desktop\leerob.io-main\node_modules\src\Fiber\context.ts:571:33
    at C:\Users\Ricky\Desktop\leerob.io-main\node_modules\src\Support\Scheduler\index.ts:13:23

Fiber: #27 (started at: 2021-11-10T05:05:40.765Z) was supposed to continue to:

  a future continuation at packages/@contentlayer/utils/src/node/fs.ts:59:12
  a future continuation at packages/@contentlayer/utils/src/node/version.ts:19:10
  a future continuation at packages/@contentlayer/utils/src/node/version.ts:20:15
  a future continuation at packages/@contentlayer/core/src/ArtifactsDir.ts:29:10
  a future continuation at packages/@contentlayer/source-files/src/fetchData/index.ts:48:116

Fiber: #27 (started at: 2021-11-10T05:05:40.765Z) Execution trace:

  packages/@contentlayer/core/src/ArtifactsDir.ts:25:13
  packages/@contentlayer/core/src/DataCache.ts:44:36
  packages/@contentlayer/core/src/DataCache.ts:43:10

Fiber: #27 (started at: 2021-11-10T05:05:40.765Z) was spawned by:

Fiber: #22 (started at: 2021-11-10T05:05:40.668Z) was supposed to continue to: <empty trace>

Fiber: #22 (started at: 2021-11-10T05:05:40.668Z) Execution trace:

  packages/@contentlayer/core/src/generation/generate-dotpkg.ts:76:13
  packages/@contentlayer/core/src/generation/generate-dotpkg.ts:72:16

Fiber: #22 (started at: 2021-11-10T05:05:40.668Z) was spawned by:

Fiber: #21 (started at: 2021-11-10T05:05:40.668Z) was supposed to continue to: <empty trace>

Fiber: #21 (started at: 2021-11-10T05:05:40.668Z) Execution trace: <empty trace>

Fiber: #21 (started at: 2021-11-10T05:05:40.668Z) was spawned by:

Fiber: #15 (started at: 2021-11-10T05:04:55.847Z) was supposed to continue to: <empty trace>

Fiber: #15 (started at: 2021-11-10T05:04:55.847Z) Execution trace:

  packages/@contentlayer/utils/src/effect/Stream.ts:32:83
  packages/@contentlayer/core/src/getConfig/index.ts:168:13
  packages/@contentlayer/core/src/getConfig/index.ts:159:26
  packages/@contentlayer/core/src/getConfig/index.ts:158:30
  packages/@contentlayer/core/src/getConfig/index.ts:148:26
  packages/@contentlayer/core/src/getConfig/index.ts:147:31
  packages/@contentlayer/core/src/getConfig/index.ts:136:26
  packages/@contentlayer/core/src/getConfig/index.ts:135:15
  packages/@contentlayer/core/src/getConfig/index.ts:121:10
  packages/@contentlayer/core/src/getConfig/esbuild.ts:114:12
  packages/@contentlayer/core/src/getConfig/esbuild.ts:117:14
  packages/@contentlayer/core/src/getConfig/esbuild.ts:66:15
  packages/@contentlayer/core/src/getConfig/esbuild.ts:65:10
  packages/@contentlayer/core/src/getConfig/esbuild.ts:64:10
  packages/@contentlayer/core/src/getConfig/esbuild.ts:47:22
  packages/@contentlayer/core/src/getConfig/esbuild.ts:72:10
  packages/@contentlayer/core/src/getConfig/esbuild.ts:71:12
  packages/@contentlayer/core/src/getConfig/esbuild.ts:71:46
  packages/@contentlayer/core/src/getConfig/esbuild.ts:85:20
  packages/@contentlayer/core/src/getConfig/esbuild.ts:84:12
  packages/@contentlayer/core/src/getConfig/esbuild.ts:83:10
  packages/@contentlayer/core/src/getConfig/esbuild.ts:113:12
  packages/@contentlayer/core/src/getConfig/index.ts:52:13
  packages/@contentlayer/core/src/getConfig/index.ts:107:10
  packages/@contentlayer/utils/src/node/fs.ts:101:22

Fiber: #15 (started at: 2021-11-10T05:04:55.847Z) was spawned by:

Fiber: #0 (started at: 2021-11-10T05:04:55.792Z) was supposed to continue to:

  a future continuation at packages/@contentlayer/core/src/runMain.ts:15:110

Fiber: #0 (started at: 2021-11-10T05:04:55.792Z) Execution trace:

  packages/@contentlayer/core/src/runMain.ts:15:32
  packages/@contentlayer/core/src/runMain.ts:12:19
  packages/@contentlayer/core/src/runMain.ts:10:12

Fiber: #0 (started at: 2021-11-10T05:04:55.792Z) was spawned by: <empty trace>
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I've used this repo as a starting point for an ongoing very WIP effort to create a blog for myself.

When starting the dev server with next dev, though, I end up in a compiling loop, here is a snippet from the terminal:

event - compiled successfully in 330 ms (454 modules)
wait  - compiling...
event - compiled successfully in 327 ms (454 modules)
wait  - compiling...
event - compiled successfully in 336 ms (454 modules)
wait  - compiling...
event - compiled successfully in 342 ms (454 modules)
wait  - compiling...
event - compiled successfully in 332 ms (454 modules)
wait  - compiling...
event - compiled successfully in 554 ms (454 modules)
wait  - compiling...
event - compiled successfully in 339 ms (454 modules)
wait  - compiling...
event - compiled successfully in 361 ms (454 modules)
wait  - compiling...
event - compiled successfully in 347 ms (454 modules)
wait  - compiling...
event - compiled successfully in 345 ms (454 modules)
wait  - compiling...
event - compiled successfully in 334 ms (454 modules)
wait  - compiling...
event - compiled successfully in 362 ms (454 modules)
wait  - compiling...
event - compiled successfully in 344 ms (454 modules)
wait  - compiling...
event - compiled successfully in 336 ms (454 modules)

I remember seeing something similar a few days ago at work, the problem was this: getsentry/sentry-javascript#4123

I wonder if contentlayer has a similar issue, or if I configured something badly. here is the code causing the problem: https://github.com/balazsorban44/balazsorban.com/tree/feat/blog

next build seems to work fine.

Any ideas?

I want to use shiki for syntax highlighting, but its getHighlighter function is async. Since top-level await is not implemented in esbuild, it is not possible for me to export an object for the config. If an async function was accepted, I could get the highlighter and return the config.

There is an open issue about contentlayer not working on windows #7
Based on the comment here, I have installed the 0.0.28-dev.12 version. The package is able to compile in the first run but any modifications in MDX file while in dev mode crashes the script.
This happens for both dev script for Nextjs as well as yarn contentlayer dev script.

Attaching a video for demo

Seems like MDX v2 is just around the corner so we should investigate upgrading to it soon.

We should also verify whether mdx-bundler is still needed with MDX v2.

I installed next js typescript with next-content layer. when running yarn dev command, it returns error.

Repo: https://github.com/kicut/learn-nextjs-contentlayer

Error in Terminal:

➜  learn-nextjs-contentlayer git:(main) yarn dev 
yarn run v1.22.17
$ next dev
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
warn  - SWC minify beta enabled. https://nextjs.org/docs/messages/swc-minify-enabled
Error: Cannot find module '@opentelemetry/sdk-trace-base'
Require stack:
- /home/kicut/code/project/next-contentlayer/node_modules/@effect-ts/otel/Processor/Simple/index.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.mod._resolveFilename (/home/kicut/code/project/next-contentlayer/node_modules/next/dist/build/webpack/require-hook.js:171:28)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/kicut/code/project/next-contentlayer/node_modules/@effect-ts/otel/Processor/Simple/index.js:52:26)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/kicut/code/project/next-contentlayer/node_modules/@effect-ts/otel/Processor/Simple/index.js'
  ]
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It would be great if this tool optionally included all the MDX packages you would need, so you could remove dependencies from your repository like mdx-bundler (and esbuild which is needed for it).

apparently it's not possible to use array type for field. It's normal ? thanks 👍🏼

Seems to be related to Yarn 1.

Hello

I am having issues when replacing React with Preact in production build.

My project can be found at jahirfiquitiva/jahir.dev

And this is the build error I am getting:

info  - Creating an optimized production build  
Failed to compile.

./node_modules/next-contentlayer/dist/hooks/useMDXComponent.js
Module not found: Package path ./compat/jsx-runtime.js is not exported from package /Users/jahir/dev/jahir/website/node_modules/preact (see exports field in /Users/jahir/dev/jahir/website/node_modules/preact/package.json)

Import trace for requested module:
./node_modules/next-contentlayer/dist/hooks/index.js
./src/pages/blog/[slug].tsx


> Build failed because of webpack errors
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I'm not sure why it started failing if it was working just fine before 😕

I've setup Contentlayer with Fuse.js to have simple yet effective fuzzy. For this I'm creating lists for Fuse.js with allDocuments from Contentlayer. My content types have however different "titles", eg. for posts title and for people name. These document titles are what I present in the search results. In order to do this with just allDocuments and not having to import all content types and doing mutations on the different arrays (which does not seem ideal for maintainability), I've simply set up a second frontmatter field on some content types, so that all of them have for example title as a field, but that's a bit a hassle when working with content written by others. That's why it would be great if there was an option in Contentlayer to create an alias, perhaps similar to how defaults are declared.

Perhaps something like this:

  fields: {
    name: { type: 'string', required: true },
    title: { type: 'string', alias: 'name' }
  }

Currently many packages (e.g. @contentlayer/core) export much more than a typical user would need. We should move the non-common exports from the main module ./ to a sub-module ./internal.

There's a number of common utility functions which are useful in Contentlayer projects (e.g. a type-safe pick function) like below or here:

We should provide these utility functions out of the box with Contentlayer. Candidates:

• pick: Type-safe function similar to TS Pick type
• A function that helps with getting the "next" and "previous" document (common in docs pages)
• A function that helps with sorting documents
• findById

If you'd like to see an other utility function added, please share your requests/ideas in the comments below. 🙏

Hey schickling, looking to use contentlayer on my docs site at urlbox.io after seeing it in leerob's repo, looks great.

Quick question I have before getting too stuck in, how would I be able to structure the content, I'm currently using a manifest.json like next.js docs (https://github.com/vercel/next.js/blob/master/docs/manifest.json) and formik-docs (https://github.com/formium/formik/blob/master/docs/manifest.json) in order to structure the individual mdx posts, so that they can be structured correctly in the sidebar.

This also allows adding next and previous links at the bottom of each article to navigate to sibling docs. Just wondering if this is possible currently with contentlayer or how to go about it?

A lot of the examples seem to just order a list of content by date, whereas for docs sites, we'd need more structure / a pre-defined order.

Context

I've been trying out a combination of Next.js and Contentlayer, which works great. However, with Next 12 and the React Server Components alpha, there's a hurdle of Next being unable to build the server pages. It'll throw something like the following:

./pages/notes/[slug].server.tsx
Module not found: Package path ./data is not exported from package /x/node_modules/.contentlayer (see exports field in /x/node_modules/.contentlayer/package.json)

Looking at the exports field in the generated package.json, it is well-formed, e.g.

{
  "exports": {
    "./data": {
      "import": "./data/index.mjs"
    },
    "./types": {
      "import": "./types/index.mjs"
    }
  }
}

That would suggest that, when building the pages, Next uses CommonJS, right? As a quick sanity-check, I tried copying over the generated .contentlayer module, renaming the .mjs files to plain .js files, and pointing the exports field to those, e.g.

{
  "exports": {
    "./data": "./data/index.js",
    "./types": "./types/index.js"
  }
}

Then, I pointed the imports from ./pages/notes/[slug].server.tsx to the copy, and the build passes once more.

Request

It'd be awesome if the generated .contentlayer module would support importing via CommonJS as well as ES modules. While I'm sure there's a whole bunch of things to sort out with both React and Next before the server components eventually come out of alpha and beta, it seems to me that this would be enough to have a rough initial support. 😅

'Link' implicitly has type 'any' because it does not have a type annotation and is referenced directly or indirectly in its own initializer.

Workaround: Use any

cc @seancdavis

Related to #55 I tried removing the .contentlayer folder. Once removed it will not be recreated until you do a

yarn add contentlayer next-contentlayer

It would be better if it gets recreated when running yarn dev.

We should improve the CLI logging messages e.g. in the following cases:

• timing: we should add how long a step has taken (e.g. 50ms to generate 20 documents)
  • This should also provide a summary of how time is spent in different remark plugins etc
• Adjust log messages for incremental builds based on file watching. (Currently the log messages indicate that all documents have been re-generated even though just a single file has changed)
• The log messages should also indicate when using cache from file system

Ideally we can even easily test the wanted behaviour above in unit tests.

Hey there... I started getting this error with v 0.0.30, it was working fine with v 0.0.27

./contentlayer.config.ts:160:19
Type error: Type 'DocumentType<"Blog">' is not assignable to type 'DocumentType<string>'.
  Types of property 'def' are incompatible.
    Type 'Thunk<DocumentTypeDef<"Blog">>' is not assignable to type 'Thunk<DocumentTypeDef<string>>'.
      Type 'DocumentTypeDef<"Blog">' is not assignable to type 'DocumentTypeDef<string>'.
        Types of property 'computedFields' are incompatible.
          Type 'ComputedFields<"Blog"> | undefined' is not assignable to type 'ComputedFields<string> | undefined'.
            Type 'ComputedFields<"Blog">' is not assignable to type 'ComputedFields<string>'.
              Type 'string' is not assignable to type '"Blog"'.

  158 | const contentLayerConfig = makeSource({
  159 |   contentDirPath: 'data',
> 160 |   documentTypes: [Blog, InspirationItem],
      |                   ^
  161 |   mdx: {
  162 |     remarkPlugins: [remarkGfm],
  163 |     rehypePlugins: [
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

This is the content of contentlayer.config.ts

import { ComputedFields, defineDocumentType, makeSource } from 'contentlayer/source-files';
import readingTime from 'reading-time';
import rehypeAutolinkHeadings from 'rehype-autolink-headings';
import rehypeSlug from 'rehype-slug';
import remarkGfm from 'remark-gfm';

const computedFields: ComputedFields = {
  readingTime: { type: 'json', resolve: (doc) => readingTime(doc.body.raw) },
  slug: {
    type: 'string',
    // eslint-disable-next-line no-underscore-dangle
    resolve: (doc) => doc._raw.sourceFileName.replace(/\.mdx$/, ''),
  },
};

const Blog = defineDocumentType(() => ({
  name: 'Blog',
  filePathPattern: 'posts/*.mdx',
  bodyType: 'mdx',
  fields: {
    title: { type: 'string', required: true },
    date: { type: 'string', required: true },
    hero: { type: 'string', required: true },
    excerpt: { type: 'string' },
    description: { type: 'string' },
    author: { type: 'json' },
    ogImage: { type: 'json' },
  },
  computedFields,
}));

const computedInspirationFields: ComputedFields = {
  favicon: {
    type: 'string',
    resolve: async (doc) => {
      return `${doc.link}/static/images/favicon.png`;
    },
  },
};

export const InspirationItem = defineDocumentType(() => ({
  name: 'InspirationItem',
  filePathPattern: 'inspiration/*.json',
  fields: {
    title: { type: 'string', required: true },
    link: { type: 'string', required: true },
    favicon: { type: 'string' },
  },
  computedFields: computedInspirationFields,
}));

export default makeSource({
  contentDirPath: 'data',
  documentTypes: [Blog, InspirationItem],
  mdx: {
    remarkPlugins: [remarkGfm],
    rehypePlugins: [rehypeSlug, rehypeAutolinkHeadings],
  },
});

I Can't upgrade to 0.0.28 successfully, And I'm getting this weird error(locally & vercel):

22:32:38.406 | This error shouldn't have happened. Please consider opening a GitHub issue with the stack trace below here:
-- | --
22:32:38.406 | https://github.com/contentlayerdev/contentlayer/issues
22:32:38.407 |  
22:32:38.407 | An unchecked error was produced.
22:32:38.407 |  
22:32:38.407 | TypeError: Cannot read property 'get' of undefined
22:32:38.408 | at ApplyFrame.apply  (/vercel/path0/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Layer/definitions.js:391:60)
22:32:38.408 | at FiberContext.nextInstr  (/vercel/path0/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:315:22)
22:32:38.408 | at FiberContext.evaluateNow  (/vercel/path0/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:699:56)
22:32:38.408 | at /vercel/path0/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Fiber/context.js:458:54
22:32:38.408 | at /vercel/path0/node_modules/@contentlayer/utils/node_modules/@effect-ts/system/Support/Scheduler/index.js:34:30
22:32:38.408 |  
22:32:38.408 | Fiber: #2 (started at: 2021-11-08T21:32:38.391Z) was supposed to continue to: <empty trace>
22:32:38.408 |  
22:32:38.410 | Fiber: #2 (started at: 2021-11-08T21:32:38.391Z) Execution trace: <empty trace>
22:32:38.410 |  
22:32:38.410 | Fiber: #2 (started at: 2021-11-08T21:32:38.391Z) was spawned by:
22:32:38.410 |  
22:32:38.410 | Fiber: #0 (started at: 2021-11-08T21:32:38.383Z) was supposed to continue to:
22:32:38.410 |  
22:32:38.411 | a future continuation at packages/@contentlayer/core/src/runMain.ts:15:110
22:32:38.411 |  
22:32:38.411 | Fiber: #0 (started at: 2021-11-08T21:32:38.383Z) Execution trace:
22:32:38.411 |  
22:32:38.411 | packages/@contentlayer/core/src/runMain.ts:15:32
22:32:38.411 | packages/@contentlayer/core/src/runMain.ts:10:12
22:32:38.411 |  
22:32:38.411 | Fiber: #0 (started at: 2021-11-08T21:32:38.383Z) was spawned by: <empty trace>

Additionally to defining document properties in frontmatter, MDX also supports defining properties as export statements (see MDX docs). We should consider to support this feature in Contentlayer.

Hey, I'm not sure if this is possible already, but I would like to be able to build data from a single file

Example ... data/projects.json

There I would have something like:

[
  {
    "title": "Blueprint",
    "description": "Dashboard for creating Android icon packs",
    "icon": "/static/images/projects/android/blueprint.png",
    "preview": "/static/images/projects/android/blueprint-preview.png",
    "link": "https://github.com/jahirfiquitiva/Blueprint/",
    "color": "#4d8af0",
    "tag": "android",
    "stack": [
      "android",
      "kotlin",
      "material design"
    ]
  },
  {
    "title": "Frames",
    "description": "Dashboard for creating Android wallpapers apps",
    "icon": "/static/images/projects/android/frames.png",
    "preview": "/static/images/projects/android/frames-preview.png",
    "link": "https://github.com/jahirfiquitiva/Frames/",
    "color": "#35b4ef",
    "tag": "android",
    "stack": [
      "android",
      "kotlin",
      "material design"
    ]
  }
]

And contentlayer would be able to return each item as a document of type project, or something like that

While I guess that it is currently possible to localize MDX based on frontmatter tags, doing this would mean that the filenames would need to be different, resulting in different slugs which would certainly not be ideal. I guess there are other ways to achieve it, but none of them seem ideal, that's why it would be great to support l10n (and also just because Contentlayers goal is to be the abstraction layer between data and code).

I assume that there could be several ways for structuring the MDX files, but what comes to my mind first is either via file name eg. /pages/about.mdx /pages/about.de.mdx or via subfolders eg. /pages/en/about.mdx /pages/de/about.mdx.

Edit: the following structure seems pretty nice, but would depend on #11

├── data
│   ├── pages
│   │   ├── about
│   │   │   ├── index.mdx
│   │   │   ├── index.de.mdx
│   │   │   ├── picture.jpeg
│   │   ├── privacy
│   │   │   ├── index.mdx
│   │   │   ├── index.de.mdx
│   │   │   ├── cover.jpeg

For Contentful one just needs to add &locale= https://www.contentful.com/developers/docs/references/content-delivery-api/#/reference/localization