Metalsmith and Filenames, What can we do to improve them? about metalsmith HOT 26 OPEN

Just found myself in need of filename metadata and found this thread.

From my standpoint (without knowing anything about the internals):

• metalsmith core could offer filename.source and filename.destination as default file metadata (as these are part of the core API keywords/entry points), and
• plugins which mutate filenames should add their own key to that such as filename.my_plugin.

This way downstream plugins could decide which filename mutation to use for complex cases. Not sure that's sane. It seems it could even be a plugin actually to keep with the minimalist core philosophy...

from metalsmith.

What makes metalsmith so simple and powerful is the simplicity of the files object. The files object is the current state of the files being processed by metalsmith. At any time the files object can be output to the build folder using only the object key for the file path and the file object's contents property for the file contents. You can make these changes without use and understanding of a process that would require function calls.

Also there is a very good reason to store the files in an object indexed by file path: preventing duplicate files.

I agree that there are cases where a key for the current file's path on each file object is nessisary. This has become nessisary for the metalsmith-navigation plugin I am still working on (not released yet). When the navigation plugin is done execution, the path data it set and used on each file object is no longer needed and is removed.

IMO the lifecycle of a metalsmith plugin should be roughly like this:

(There should really be a plugin guide like this somewhere with intended conventions, expectations etc.)

• When inspecting file objects:
  • Do so as if this was their initial and only state (how the current state was achieved should be irrelevant).
  • The files object keys (the file path that will be output for that file) should be the only place this file's path is maintained (metalsmith-collections breaks this rule).
• When executing
  • Modify metalsmith.metadata() and files objects how ever you like.
• When done execution:
  • Properties assigned to file objects or metalsmith.metadata() by the plugin:
    • Not Intended to be used by other plugins:
      • Should be removed.
      • Modified files object keys (file paths) and contents property of file objects should be returned to a usable state.
    • Intended to be used by other plugins:
      • Should be documented and maintained consistently for backward compatibility.
      • Should have namespaced or configurable keys to avoid collision with other plugins.
  • Data placed on the metalsmith.metadata() object with references to file object should not depend on file path keys for reference
    • Doing this durring execution is fine, other plugins should not be expected to rely on data structured this way (metalsmith-collections breaks this rule)
  • The state of the files object should be such that it is ready to be output immediately without futher action or modification.
• Ideally if a plugin has behavior that breaks these rules it should be designed such that the behavior is opt in via configuration.

Enough ranting

Solving the problem

The file path problem could easily be solved with a very simple middleware function. I think this should be added to metalsmith with the documented convention that the file object path property (or whatever key) is the place to look for a file object's path.

var updatePaths = function(files, metalsmith, done){
    for(var key in files){
        files[key].path = key;
    }
    done();
};

var metalsmith = Metalsmith(__dirname)
    .use(collections)
    .use(changePaths)
    .use(updatePaths)
    .use(templates)
    .build(function(err) {
        if (err){
            throw err;  
        } 
    });

from metalsmith.

“These aren’t the droids files you’re looking for…“

Now that I’ve had some time to get to know Metalsmith better, I don’t think the core needs to be modified.

I think this is actually a documentation problem—not a Metalsmith shortcoming.

Let’s Educate!

Something like the following needs to be displayed prominently on the website or README:

Each plugin accepts the following arguments (files, metalsmith, done).

The files object is where Metalsmith plugins do their smithing.
It looks something like this:

{
    'current/filename.html': {
        'filename': 'src/original/filename.md',
        // various other properties
        ..., 
        ...       
}

So each plugin gets both the current and the original filename. That’s all the information necessary to do any filename manipulation imaginable.

EDIT: I made a mistake in assuming that the filename property above was generated by Metalsmith itself. It appears that a plugin I was using was inserting this property for me. Thanks to @Ajedi32 for setting me straight. :)

EDIT: Also, although the filename property may not be inserted by Metalsmith itself, the principles of simplicity and easily adding this property remain. Just add a plugin at the start of your chain that inserts the original (i.e. source) filename, and you’re all set.

A quick example

I just wrote a custom plugin to modify the current filename myself. Here it is:

/**
 *  This Metalsmith plugin removes a prefix from each file's current state.
 *  
 *  Its original purpose was to keep all site content grouped together
 *  in a directory, but “shift up” those files by one directory.
 *  
 *  This is performed by simply removing the prefix from the file’s
 *  current name.  
 */
const elevateContents = (PREFIX = '_content/', DEBUG = false) => {

    if (DEBUG === true) {
        console.log('PREFIX = ' + PREFIX);
        console.log('DEBUG = ' + DEBUG); 
    }

    const pluginFunction = (files, metalsmith, done) => {
        for (let key in files) {
            if (key.startsWith(PREFIX)) {
                let newKey = key.replace(PREFIX, '');
                files[newKey] = files[key]; // set new “current filename” to the same file value
                delete files[key]; // delete original key, now that it lives under `newKey`
            }
        }
        done();
    }

    return pluginFunction;
}

from metalsmith.

@hollowdoor I'm confused. What part of @unstoppablecarl's list of conventions above do you think metalsmith-markdown is breaking? There's nothing in that list of conventions that says plugins shouldn't move files around or rename them.

Not as written by @unstoppablecarl, no.

If I am reading people’s concerns correctly, I think part of the concern is the very simplicity and flexibility that makes Metalsmith so awesome.

When people first try out Metalsmith, they could be coming from many different previous experiences. They might not have used a static site generator before at all. Or perhaps they have used a popular one, but ran up against some limitations that caused them to look elsewhere. One popular example is Jekyll. Another is DocPad. (Which is what I used for a time, before becoming frustrated with it and searching around till I found Metalsmith!)

Both Jekyll and DocPad allow plugins.

In Jekyll, plugins are described as falling into one of five categories: Generators, Converters, Commands, Tags, and Hooks.

In DocPad, plugins are described as one of the following: Renderers, Helpers, Deployers, and Admin Interfaces.

Finally, both Jekyll and DocPad have documentation dedicated to teaching users about understanding, using, and authoring plugins. This documentation, as well as giving the reader categories of plugins, give the reader something to latch onto as they read. These things ground the user.

But if you go to http://metalsmith.io and read down the page, plugins are introduced in the middle of explaining how Metalsmith works. There’s nothing technically wrong with that, but it seems somewhat roundabout compared with the documentation mentioned above.

By offering no categories—not even with a big disclaimer like, “These are not strict categories! They’re just for convenience, and a plugins can do just about anything…”—and no page dedicated to the topic of plugins (there aren’t even any navigation links on http://metalsmith.io ! 😡), whenever someone new to Metalsmith is excited about the tool, but wants some direction, they are forced to scroll through a long page with no clearly defined section on plugins.

This adds a lot of friction to the process.

This is a documentation problem. Metalsmith is doing all the right things in terms of its design—now its documentation needs the same loving attention!

from metalsmith.

Looking back on the idea of adding this to metalsmith core months later I am not sure it should be added. I think by having metalsmith core add any properties you are loosing some of the simplicity that makes it so great.

Pros:

• When developing a plugin it will be very convenient to be able to assume that .path is the file object's path without having to create a mechanism yourself. (although as you can see in the code I posted above it isn't that complicated to do)

Cons:

• Probably should not add functionality that runs after every plugin to assign data plugins may or may not need.
• There is not an implicit way communicate that the path property is intended to be read only.
• Users would likely think that changing the value of the path property would change the file name.
• The problem of file objects not knowing their key should probably not be magically abstracted away.
• Plugin developers would need to know and remember to update the path property when changing file names (if their plugin used that property, I could see myself making this mistake and not be able to easily figure out why things were not going as expected).

As I said in my rant I think the file object keys should be the only place that a file object's path is maintained.

I think a better solution to this problem would be more guides and documentation about how to handle this problem when developing plugins.

from metalsmith.

I think something simple that might helps is to add the original filename. Some plugins might want it)
I am doing a better watcher (metalsmith-watch is very limited) and I need to udpate collections by hand since this plugins works with simple array (in order to avoid duplicate). Having something like file._originalFilename or similar will helps me a lot to prevent duplicate.

from metalsmith.

@KrishnaPG One possible solution would be to write another plugin to set the path property on all the files in that directory before passing it to handlebars.

from metalsmith.

Oh and by the way, of course an easy workaround this for now is to add a slug key and value to all files that need that :)

from metalsmith.

I have come up with a bit of a workaround for my metalsmith-slug plugin that does as much as it can without touching Metalsmith internals. It allows you to set a renameFiles property, which just duplicates a given file object with the new slug name as the key, deleting the original key/value. This is working ok for my case but is certainly not ideal.

from metalsmith.

@unstoppablecarl That sounds like an interesting solution, if we just spliced that quick filler function between every middleware function it would solve the issue and make a sort of read-only file.path property.

I'd be down to try that out if anyone wanted to submit a PR with tests and docs.

from metalsmith.

File names can change though. Look at the permalinks plugin.

from metalsmith.

I think a filename property would be a good solution. Several templating engines require it when using includes and extends, so it's already a standard of sorts.

But maybe it doesn't have to be added to core since there's @MoOx's metalsmith-filenames. Would allow things to stay modular.

from metalsmith.

I have indeed a problem about it.

This is my code: https://github.com/kud/diary/blob/feat.multilanguage-2/tasks/markup-en.js and this is my structure: https://github.com/kud/diary/tree/feat.multilanguage-2/src

The result is everything which uses collections plugin are located to dist/en but everything which does not use collections plugins are in dist/en/posts/en.

I've tried to change the path of the files via something like:

var updatePaths = function(files, metalsmith, done){
    for(var key in files){
        files[key].path = key;
    }
    done();
};

but it didn't work.

Any clue?

from metalsmith.

Here's a couple of plugin definitions that might help:

/**
 * Adds source filename to file like front matter
 */
var original_filename = function (files, metalsmith, done) {
    Object.keys(files).forEach(function (file) {
        files[file].original_filename = file;
    });
    done();
};

or if you wanted to track the changes, this would initialise a metadata object with the source filenames.

/**
 * Adds filename tracking object to metadata
 */
var filename_tracking = function (files, metalsmith, done) {
    var metadata = metalsmith.metadata();
    var file_list = [];
    Object.keys(files).forEach(function (file) {

        file_list.push({source_file: file});
    });

    metadata.filetracking = file_list;

    done();
};

Like @ismay and @unstoppablecarl have said, it doesn't seem like core needs to be changed for people to work in this way if they want to.

I use collections to group my files and either use the original filename (not path) or a slug with permalinks to put things in the right place for the build.

from metalsmith.

I'd like to suggest the following idea:

Before Metalsmith starts processing the chain, it creates a deep copy of all original data, use Object.defineProperty to set writable to false and puts them into something like an "originalState" property (including all files, metadata etc...).

e.g. for plugins accessible on this way:

    .use(function(files, metalsmith, done) {
        console.log(metalsmith.originalState);
        done();
    })

from metalsmith.

Going back to the solutions in the original post, being able to use native Array.prototype methods would be really powerful for creating more testable and declarative plugins with less for in noise

I realise that this would break most of the current plugins, but it would be easy to mitigate, providing a mutation plugin in order to continue using plugins that are not yet converted. We could even provide an additional .use method (.useLegacy perhaps) that would mutate into and out of the object style into a new array setup.

from metalsmith.

@MrDeclare I do not think that should be built in. It would be easy enough for any plugins that needed it to implement it themselves. Making your own middleware that implemented this would be simple enough.

@mildfuzz I do not think an array is appropriate for a list of objects indexed by unique file path.

see my previous commment #58 (comment)

from metalsmith.

What about a metalsmith: … metadata key?

Underneath that, you could add the filename or anything else necessary for Metalsmith’s processing pipeline. I can’t imagine any accidental name collisions if everything like this is grouped under a key with the same name as the tool used to build the site.

(Aside: I’m ambivalent about both sides of this issue. I just thought I’d offer the above suggestion as a compromise.)

from metalsmith.

@Zearin Yep, that's correct.

One minor correction though: that 'filename' key containing the "original" filename actually doesn't get created by Metalsmith. You can, however, easily add it with a plugin if needed.

from metalsmith.

One minor correction though: that 'filename' key containing the "original" filename actually doesn't get created by Metalsmith. You can, however, easily add it with a plugin if needed.

…Oh! Oops. :-/

I’m definitely seeing that property, but I guess it’s being created by one of the plugins that I’m already using. I should have thought to check if that was the case!

Thanks for the correction.

from metalsmith.

(Edited my earlier post with a note reflecting @Ajedi32’s correction. Hopefully that will prevent my earlier post from misleading future readers who might be skimming the thread without looking closely.)

from metalsmith.

Right now metalsmith-markdown plugin changes the file name key from .md to destination .html. Is this what is expected of plugins? I read @unstoppablecarl's conventions post above, and I didn't quite understand all of it with respect to what metalsmith-markdown is doing by modifying the metadata file name keys.

from metalsmith.

@hollowdoor It's valid for plugins to change the filename or extension if it's part of what the plugin does and is apparent to the consumer of that plugin. The documentation for metalsmith-markdown should be more explicit that the file extension will change. It would be good to do a PR for that.

from metalsmith.

@hollowdoor I'm confused. What part of @unstoppablecarl's list of conventions above do you think metalsmith-markdown is breaking? There's nothing in that list of conventions that says plugins shouldn't move files around or rename them.

from metalsmith.

@Zearin Exactly this is my first time ever using a static site builder. Also my first time making a plugin (yet to be published) for one. For the most part my experience with Metalsmith has been wonderful. That I can read javascript probably helps. ;)

@woodyrew That's true about the doc updates. I saw the behavior, and wondered what is the suggested usage. As far as I'm concerned -- in my ignorance -- anything can be a convention because I'm not the one deciding all this. :)

@Ajedi32 I'm not really concerned with what the conventions allow by omission. I was wondering what the going opinion is on the behavior of changing names. I guess I got my answer. Sorry. I think I didn't word my first comment very well. Also notice I said "I didn't quite understand". This could just be me not being very smart.

from metalsmith.

Came across this thread while trying to solve a filename problem I am facing.

I have a source file (named: my-file.hbs) that, through some handlebar plugins, is getting transformed to my-file.html in the output folder. But I am facing challenge on how to get this transformed name my-file.html from the metadata? The path property in the collections is showing the original name my-file.hbs, not the final name. The metadata debug shows as below:

articles:
  metalsmith:metadata    [ { title: 'Unfinished article',
  metalsmith:metadata        draft: false,
  metalsmith:metadata        collection: [Array],
  metalsmith:metadata        comments: true,
  metalsmith:metadata        contents: <Buffer 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 3e 0d 0a 3c 68 74 6d 6c 20 78 6d 6c 6e 73 3d 22 68 74 74 70 3a 2f 2f 77 77 77 2e 77 33 2e 6f 72 67 2f 31 39 ... >,
  metalsmith:metadata        mode: '0666',
  metalsmith:metadata        stats: [Object],
  metalsmith:metadata        path: 'my-file.hbs',
  metalsmith:metadata        next: [Object],
  metalsmith:metadata        rootPath: '',
  metalsmith:metadata        paths: [Object] },

  metalsmith:files   'my-file.html':
  metalsmith:files    { title: 'Unfinished article',
  metalsmith:files      draft: false,
  metalsmith:files      collection: [ 'articles' ],
  metalsmith:files      comments: true,
                    ...
  metalsmith:files      path: 'my-file.hbs',
  metalsmith:files      next:

I am creating a blog index file that points to all the other htmls files, something like below, but have no clue on to get the final path of the file.

	{{#each articles }}
		<li>
			<div class="title"><a href="{{ path }}">{{ title }}</a></div>
		</li>
	{{/each}}

Right now, it is generating <a href="my-file.hbs">...</a> which is incorrect, since it should be my-file.html.

Is there anyway we can access the final filename from the metadata, so that we can use it as vairable for href above?

from metalsmith.