The core API evolutionated over time, it has some weird param names, etc

Let's redesign it thinking in ease of use

Based on:

• #26
• #25

Add support for Circle CI, it is basically only adding documentation explaining how to set it up and propagating secrets. Container Images are already built

Using ajv-cli: https://github.com/ajv-validator/ajv-cli

{
  lintSchema = {
    schema = ./schema.json;
    targets = [ ./data.yaml ./data.json ];
  };
}

depending on how you name folders on the makes/ folder, the outputs are going to look like-this, or likeThis, or whatever you name the directories.

for consistency with makes built-ins, we should translate directory names to camelCase: https://levelup.gitconnected.com/converting-a-string-to-camelcase-in-javascript-f88a646a22b4

so people can have directories like /makes/my-derivation/main.nix, but execute m .myDerivation

nix has no built-in, and nixpkgs does not have a function in lib.strings, so we should create our own in this function:

there are some builtins people can invoke in makes.nix

for workflows not covered there people can create any combination of directories and files under the makes/ folder in their project. For each main.nix, at any location, an output is created and can be listed when running m. The main.nix format is a function that receives some arguments:

we should document all of this in a 'Extending makes' section

Currently makes can be used locally, in order to support Gitlab we need to:

• add the required args and modules to build and deploy container images
• deploy a container image on Gitlab
• add documentation about setting makes in the gitlab CI

Similar as asBashArray, but for associative-arrays instead of for lists

local map=(
  [key]=val
)

WIth: https://github.com/kamadorueda/mailmap-linter

Add support for bandit: https://github.com/PyCQA/bandit

{
  securePythonWithBandit = {
    cli = {
      python = "3.8";
      target = "/src/cli";
    };
  };
}

If you consider it valuable, would be great that the documentation mention why we create the tool in relation to the existing build tools? Why this is better than X or Y current technologies?

Thanks in advance for your great contribution to the community

Add support for https://cachix.org/ so people can push stuff there, and fetch them transparently, this way they only build things once

formatting, testing, applying, common modules?

This can help us with keeping commands output up-to-date in the documentation

• https://github.com/zimbatm/mdsh

So people can import for instance different versions of nixpkgs and pinning commits, passing arguments, accepting licenses, etc

For https://github.com/fluidattacks/makes/blob/main/src/modules/outputs/builtins/default.nix:

Is it possible to append the import of the default.nix from each builtin(subfolder) so we don't actually have to touch this file every time we add a new builtin?

So users can propagate their parser.js and config.js

https://github.com/terryyin/lizard

Initially I'm adding support for:

--lenght Threshold for maximum function length warning.
-- CCN Threshold for cyclomatic complexity number warning

Create formatPython and lintPython, based on:

https://gitlab.com/fluidattacks/product/-/blob/master/makes/utils/lint-python

The mentioned URL is a single job, here we want to have 2 for maximum granularity

Add the required docs

Based on:

• #26
• #25

Add support for Bitbucket pipelines & Bamboo, it is basically only adding documentation explaining how to set it up and propagating secrets. Container Images are already built

Currently, actions like updating a table of contents or pushing containers to protected registries fail when running in forks.

Let's look for a comfortable way of marking jobs so they only run in the main repo fluidattacks/makes.

Since in Autonomic we want to have a granular cache at the solution level, we find it necessary to build each solution in a single derivation.

https://github.com/splintchecker/splint

Based on:

• #26
• #25

Add support for Travis CI, it is basically only adding documentation explaining how to set it up and propagating secrets. Container Images are already built

Docs: http://cppcheck.sourceforge.net/

Create formatNix (similar as formatBash),
and lintNix based on:

https://gitlab.com/fluidattacks/product/-/tree/master/makes/applications/makes/lint

the mentioned URL is a single job that formats and lints at the same time, here in makes we want to have 2 modules: one for linting, one for formatting

add the required documentation

Let's make asBashArray to use files so we can source them. This will help us simplify function usage.

Makes currently only works on files that are in the HEAD commit of the git repository

In order to make developing easy, let's propagate files that are not in the gitignore, too, which is a perfect balance between reproducibility and ease of use

Based on this: https://github.com/shellspec/shellmetrics ?

I'm looking for a way to build all related test at the same time
for example, something like this

lintWithLizard = {
  targets = {
    c = [ "pathfiles" ];
    py = [ "pathfiles" ];
    lobster  = [ "pathfiles" ];
  };
};

this going to produce outputs related by the name of the test

/lintWithLizard/c
/lintWithLizard/py
/lintWithLizard/lobster

The idea is to use makeScript to rerun m . from a bash file, get the names of the related tests and run them from bash.

Now respect to this, how could I add makes to my makeScript env?

if this a good approach? or should I try something different?

Currently you can do: M_FROM=url-of-git-clone-of-a-project m and list outputs from that remote git project

For user experience we shold shift that to the CLI, like m url-of-git-clone-of-a-project

Since writing the full url is long, we can create builtin aliases in makes, like: m skims. Internally makes would resolve skims to the product url, etc

The registry can be a JSON in the repo

@kamadorueda Let's consider adding pipeline support to makes so people can run all jobs associated with a pipeline.

• Consider parallelism
• Consider needs:
• Consider artifacts:

based on: https://gitlab.com/fluidattacks/product/-/tree/master/makes/utils/lint-markdown

Document it.

Since in Autonomic we want granuralize the cache by solution, I trying to build in my extended makes a main.nix file that allow me from a main derivation collect multiple lazy derivations

1. Get all .lobster files from my repo (pathsMatching)
2. Create the lazy makeDerivations
3. Collect all created lazy makeDerivations

So I made this based on some builtins

{ makeDerivation
, path
, inputs
, makeSearchPaths
, pathsMatching
, ...
}:
let

  ## Get lobster solutions
  solutionNames = pathsMatching {
    regex = ".*\\.lobster";
    target = "/code";
  };

  lib = inputs.nixpkgs.lib;

  ## Build a single solution file
  makeSolution = name: { src }: {
    name = "/solution/lobster/${name}";
    value = makeDerivation {
      env = {
        envTargets = path src;
      };
      name = "build-solutions-lobster-${name}";
      searchPaths = {
        bin = [
          inputs.nixpkgs.lobster
        ];
      };
      builder = ./entrypoint.sh;
    };
  };
  
  ## Lazy functions
  solutions = builtins.map
    (solutionName: {
      name = "/solution/dirOfSolution/lobster/${solutionName}";
      value = (makeSolution solutionName {
        src = "${solutionName}";
      }).value;
    })
    solutionNames;

in
## Collect lazy functions
makeDerivation {
  env = {
    envSolutions = lib.attrsets.catAttrs "value" solutions;
  };
  builder = "echo $envSolutions > $out";
  name = "solution-dir-of-solutions-for-lobster";
}

Initially, I thought that since a single derivation was created for each file these derivations will be cached, however, seems it wasn't like that, I made the following:

I run this test once to build all derivations, after that, I modified only the file code-codeabbey-002-alejotru3012.lobster and I run again my test, but, I notice that all derivations were re-built and not only the derivation for the file that I modified

I got this output

these derivations will be built:
  /nix/store/0llv41qzl5qm5l1c3hax7hj1431w25nv-build-solutions-lobster--code-codeabbey-002-alejotru3012.lobster.drv
  /nix/store/0ym4c8hk55c8c9add8hlsq0as8jrjfxv-build-solutions-lobster--code-codeabbey-100-alejotru3012.lobster.drv
  /nix/store/572hjp4iiysfn4h1rk7bnhhphsv6nxah-build-solutions-lobster--code-codeabbey-120-alejotru3012.lobster.drv
  /nix/store/7ygd265h0xsvpp8rdhbqq7kqphrbrrik-build-solutions-lobster--code-codeabbey-023-alejotru3012.lobster.drv
  /nix/store/9nyr7y4zg44nqhwwn0d6f31qvqi0zvyg-build-solutions-lobster--code-codeabbey-102-alejotru3012.lobster.drv
  /nix/store/b9sin6417g6636y50ig1fnx93cd97797-build-solutions-lobster--code-codeabbey-015-alejotru3012.lobster.drv
  /nix/store/bldsbih8kzq9ajnwpbm2jz2nrv6gsjca-build-solutions-lobster--code-codeabbey-101-alejotru3012.lobster.drv
  /nix/store/czvgd8awqgqjblb1100cm1rw00aq1kq7-build-solutions-lobster--code-codeabbey-044-alejotru3012.lobster.drv
  /nix/store/dhm16pcffyy6czi6bd6ldvzh0kxavm95-build-solutions-lobster--code-codeabbey-038-alejotru3012.lobster.drv
  /nix/store/djyhj53xdpz0lfclwdbk2isz3p42iwnq-build-solutions-lobster--code-codeabbey-006-alejotru3012.lobster.drv
  /nix/store/gj1pabpzykpr6lh58cmhwj5ppv8vzw5p-build-solutions-lobster--code-codeabbey-043-alejotru3012.lobster.drv
  /nix/store/ikfjj6y6g0437sw4wak0qzz0a317yw9a-build-solutions-lobster--code-codeabbey-011-alejotru3012.lobster.drv
  /nix/store/kqpcl2bci0wam75fhf2g92qjz0gig7kz-build-solutions-lobster--code-codeabbey-003-alejotru3012.lobster.drv
  /nix/store/n4aznc7h9alz4h33k302i7j6xxiibyzz-build-solutions-lobster--code-codeabbey-085-alejotru3012.lobster.drv
  /nix/store/rjh21wsa8dqy5aw9pbrq97nnxcj6lf7k-build-solutions-lobster--code-codeabbey-016-alejotru3012.lobster.drv
  /nix/store/s52lmcb28akfh6i82vq1h0nfrhxnxfad-build-solutions-lobster--code-codeabbey-031-alejotru3012.lobster.drv
  /nix/store/v2f98dc5m3n33nkyvfkjf9q3njvqqidj-build-solutions-lobster--code-codeabbey-080-alejotru3012.lobster.drv
  /nix/store/ynpww3cy6kzppmlwyx2hwq1kxq0cbr2x-build-solutions-lobster--code-codeabbey-053-alejotru3012.lobster.drv
  /nix/store/p60pkp1g1wzvhs6c8xm7h2vdzvzpq9mh-solution-dir-of-solutions-for-lobster.drv
building '/nix/store/0llv41qzl5qm5l1c3hax7hj1431w25nv-build-solutions-lobster--code-codeabbey-002-alejotru3012.lobster.drv'...
building '/nix/store/kqpcl2bci0wam75fhf2g92qjz0gig7kz-build-solutions-lobster--code-codeabbey-003-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/002/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/003/alejotru3012.lobster
building '/nix/store/djyhj53xdpz0lfclwdbk2isz3p42iwnq-build-solutions-lobster--code-codeabbey-006-alejotru3012.lobster.drv'...
building '/nix/store/ikfjj6y6g0437sw4wak0qzz0a317yw9a-build-solutions-lobster--code-codeabbey-011-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/006/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/011/alejotru3012.lobster
building '/nix/store/b9sin6417g6636y50ig1fnx93cd97797-build-solutions-lobster--code-codeabbey-015-alejotru3012.lobster.drv'...
building '/nix/store/rjh21wsa8dqy5aw9pbrq97nnxcj6lf7k-build-solutions-lobster--code-codeabbey-016-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/015/alejotru3012.lobster
building '/nix/store/7ygd265h0xsvpp8rdhbqq7kqphrbrrik-build-solutions-lobster--code-codeabbey-023-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/016/alejotru3012.lobster
building '/nix/store/s52lmcb28akfh6i82vq1h0nfrhxnxfad-build-solutions-lobster--code-codeabbey-031-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/023/alejotru3012.lobster
building '/nix/store/dhm16pcffyy6czi6bd6ldvzh0kxavm95-build-solutions-lobster--code-codeabbey-038-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/031/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/038/alejotru3012.lobster
building '/nix/store/gj1pabpzykpr6lh58cmhwj5ppv8vzw5p-build-solutions-lobster--code-codeabbey-043-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/043/alejotru3012.lobster
building '/nix/store/czvgd8awqgqjblb1100cm1rw00aq1kq7-build-solutions-lobster--code-codeabbey-044-alejotru3012.lobster.drv'...
building '/nix/store/ynpww3cy6kzppmlwyx2hwq1kxq0cbr2x-build-solutions-lobster--code-codeabbey-053-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/044/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/053/alejotru3012.lobster
building '/nix/store/v2f98dc5m3n33nkyvfkjf9q3njvqqidj-build-solutions-lobster--code-codeabbey-080-alejotru3012.lobster.drv'...
building '/nix/store/n4aznc7h9alz4h33k302i7j6xxiibyzz-build-solutions-lobster--code-codeabbey-085-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/080/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/085/alejotru3012.lobster
building '/nix/store/0ym4c8hk55c8c9add8hlsq0as8jrjfxv-build-solutions-lobster--code-codeabbey-100-alejotru3012.lobster.drv'...
building '/nix/store/bldsbih8kzq9ajnwpbm2jz2nrv6gsjca-build-solutions-lobster--code-codeabbey-101-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/101/alejotru3012.lobster
building '/nix/store/9nyr7y4zg44nqhwwn0d6f31qvqi0zvyg-build-solutions-lobster--code-codeabbey-102-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/100/alejotru3012.lobster
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/102/alejotru3012.lobster
building '/nix/store/572hjp4iiysfn4h1rk7bnhhphsv6nxah-build-solutions-lobster--code-codeabbey-120-alejotru3012.lobster.drv'...
I'm building for this file /nix/store/y4bhzlcjidzd0id5gg3lcsrv0sy784ks-head/code/codeabbey/120/alejotru3012.lobster
building '/nix/store/p60pkp1g1wzvhs6c8xm7h2vdzvzpq9mh-solution-dir-of-solutions-for-lobster.drv'...
/nix/store/bxsjkisbd3bf70j9z6vpwmnyl9caxwvn-solution-dir-of-solutions-for-lobster

so I have these questions

Is this the expected behavior? maybe I'm misunderstanding the cache? or am I forgetting something to get a solution for my problem?.
Is it possible to have a granular cache using the approach that I propose?

Finally, I know that I can use an approach like /lintWithLizard to get a single derivation by solution, so at this moment, I'm looking for a method to call all that individuals outputs /lintWithLizard/ at the same time.

Create lintBash (similar as formatBash) based on:

https://gitlab.com/fluidattacks/product/-/tree/master/makes/applications/makes/lint

the mentioned URL is a single job that formats and lints at the same time, here in makes we want to have 1 modules that lints only

add the required documentation

For some outputs is important to know if they are being executed locally in the dev machine, or in the remote CI/CD system

let's abtract this

Based on:

• #26
• #25

Add support for Jenkins, it is basically only adding documentation explaining how to set it up and propagating secrets. Container Images are already built

In order to authenticate into workflows we will need an authentication backend,

we currently solve this problem at Fluid Attacks with environment variables and with Mozilla's sops

environment variables usually need to be remapped, for instance:

• AWS_ACCESS_KEY_ID is set from XXXX_AWS_ACCESS_KEY_ID

sops files must be decrypted and exported into console, for instance:

• (some magic big script goes here)

What I imagine is having in the makes.nix a way to declare such authentication backends,
which outputs makes templates that can the be passed as arguments to other args/modules and inject the required steps for authenticating,

this allows us to allow extending makes into more authentication backends, reuse secrets as-code (no bash magic), and being really secure as the magic happens behind curtains and is reviewed by us

/usr/bin/env /bin/env, /bin/xxxx do not exist in Nix, so we need to replace them with the /nix/store version

for maximum reproducibility

Based on: https://shellspec.info/

https://shellspec.info/comparison.html

Create lintCommitMsg, based on:

https://gitlab.com/fluidattacks/product/-/tree/master/makes/applications/makes/lint-commit-msg

Add the required docs.

https://gitlab.com/fluidattacks/product/-/issues/4679

Currently makes can be used locally, in order to support GitHub we need to:

• add the required args and modules to build and deploy container images, very likely will be implemented in #25
• deploy a container image on GitHub
• Setting up ourselves in the GitHub Marketplace
• add documentation about setting makes in GitHub Actions

Use https://github.com/thlorenz/doctoc for autogenerating the table of contents for our README.

Make sure we use thlorenz/doctoc#128 in order to set indentation to 4 spaces.

• lintBash
• lintNix