Giter Club home page Giter Club logo

user-flow's Introduction

@push-based/user-flow

Runtime performance measurements done right, with Lighthouse user flows!

npm


This is a library & CLI to organize and run Lighthouse user flows in an organized and scalable way ๐Ÿ›ธ with CI automation in place


What is it?

A CLI tool to measure performance continuously and also integrate it into your CI. It provides lot's of DX features, nice commands with rich arguments and integration with the latest dev tooling.

Why to use it?

It will enable you to measure bootstrap as well as runtime performance with minimum effort, speed up your performance test development and reduced the needed code and configuration to a minimum.

user-flow-code-reduction

In addition, it is always up-to-date with the latest Chrome DevTools features.

Benefits

Install

Run npm i @push-based/user-flow --save-dev or yarn add @push-based/user-flow --dev to install the library.

After that you can run:
user-flow --helpor user-flow --help

Run without install

You can also use npx to run it in e.g. the CI setup: npx @push-based/user-flow --help

Quick Start

As the CLI needs a npm project to run in we explain 2 common things, using the package in an existing project and using it in a fresh project. Both ways require a node and npm project setup to install user-flow and folders to store the reports and test files.

  1. have node v14.X.X installed
    run node -v and npm -v to check it.

To start from scratch read setup an empty project

Set up and run user flows in an existing npm project

In this chapter we will learn how to install and configure user flows, as well as create a first example test and see the resulting performance report.

  1. Install:
npm i @push-based/user-flow --save-dev
  1. Set up the .user-flowrc.json config file

Run

npx @push-based/user-flow init

or if you already installed it,

npx user-flow init

in the console and accept the default value for every question.

Set up user-flow in existing project gif

This results in the following file:

./.user-flowrc.json

{
  "collect": {
    "url": "https://coffee-cart.netlify.app/",
    "ufPath": "./user-flows"
  },
  "persist": { "outPath": "./measures", "format": ["html"] }
}
  1. The CLI automatically creates an example user-flow. (./user-flows/basic-navigation.uf.ts)

It is a simple navigation measurement to start from.

./basic-navigation.uf.ts

import {
  UserFlowInteractionsFn,
  UserFlowContext,
  UserFlowProvider
} from '@push-based/user-flow';

// Your custom interactions with the page 
const interactions: UserFlowInteractionsFn = async (ctx: UserFlowContext): Promise<any> => {
  const { page, flow, browser, collectOptions } = ctx;
  const { url } = collectOptions;

  // Navigate to URL
  await flow.navigate(url, {
    stepName: `Navigate to ${url}`,
  });

};

const userFlowProvider: UserFlowProvider = {
  flowOptions: {name: 'Order Coffee'},
  interactions
};

module.exports = userFlowProvider;
  1. Run CLI You can directly run the cli command. The typescript files will get resolved and compiled live.

npx user-flow collect or just npx user-flow as collect is the default.

This will execute the user flow and opens the HTML report in the browser:

getting-started-resulting-navigation-report

For more information on how to write user-flows read in the Writing user flows for the CLI section.

Optionally you can pass params to overwrite the values form .user-flowrc.ts in the file directly or over the CLI:

npx user-flow --ufPath=./user-flows-new --outPath=./user-flows-reports --url=https://localhost:4200

๐Ÿค“ DX Tip:
For a faster development process you can use the --dryRun option to skip measurement and perform the interactions only
This is a multitude faster e.g. 3s vs 53s for a simple 2 step flow with navigation

CLI

You can read more about tricks and DX the general CLI features in our docs.

Global Options

Option Type Default Description
--help, -h boolean undefined Show help
--version boolean undefined Show version number of cli
--rcPath, -p string ./user-flowrc.json Path to user-flow.config.json. e.g. ./user-flowrc.json
--verbose, -v boolean undefined Run with verbose logging
--interactive -i boolean true (false in CI mode) When false questions are skipped with the values from the suggestions. This is useful for CI integrations.

Commands

* command

Run the default command over:
@npx @push-based/user-flow [options]

Description:
The default command forwards all options to the collect command.

Run command over:
@npx @push-based/user-flow init [options]

Description:
This command helps you to set up a .user-flowrc.json and asks for input over CLI prompts.

Option Type Default Description
-h, --generateFlow boolean n/a Generate basic user-flow file under ufPath
-g, --generateGhWorkflow boolean n/a Generate user-flow.yml file under .github/workflows
-x, --generateBudgets boolean n/a Generate budget.json file under the current working directury
--lhr string n/a Used together with --generateBudgets. Path to lighthouse report for initial budget

getting-started-resulting-navigation-report

As a result we get a .user-flowrc.json and an example flow if answered with yes.

๐Ÿค“ DX Tip: Set up user flows in a sub directory:
npx @push-based/user-flow init --rcPath ./path/to/project/.user-flowrc.json

Run command over:
@npx @push-based/user-flow collect [options] or @npx @push-based/user-flow [options] as it is the default command.

Description:
This command executes a set of user-flow definitions against the target URL and saves the output.

Option Type Default Description
-t, --url string n/a URL to analyze
-u, --ufPath string ./user-flows Path to user-flow file or folder containing user-flow files to run. (*.uf.ts or*.uf.js)
-c, --configPath string n/a Path to the lighthouse config.json file
-b, --budgetPath string n/a Path to the lighthouse budget.json file
-s, --serveCommand string n/a Runs a npm script to serve the target app. This has to be used in combination with --awaitServeStdout
-a, --awaitServeStdout string .user-flowrc setting Waits for stdout from the serve command to start collecting user-flows
-f, --format string html, json setting Format of the creates reports ( html, json, md, stdout)
-o, --outPath string ./measures output folder for the user-flow reports
-e, --openReport boolean true Opens file automatically after the user-flow is captured
-d, --dryRun boolean false When true the user-flow test will get executed without measures (for fast development)

๐Ÿ’ก Pro Tip: CLI arguments that accept multiple values can be set by using the param multiple times in a row:

As an example we could apply two different formats as output for the collect command: npx user-flow collect --format=json --format=md

Configuration

The CLI supports the official user-flow/lighthouse configuration.

Details on how to work with configurations can be found in the configuratin section.

Writing user flows for the CLI

You can think of user flows as front end e2e tests which measures performance related information during the test.

Write basic user flows leveraging all 3 measurement modes of lighthouse.

User flow measurement modes

Icon Mode Measure Performance Accessibility Best Practices SEO PWA
user-flow_navigation-icon Navigation Page load 100% / 30 100% / 30 100% / 30 100% / 30 โœ” / 7
user-flow_timespan-icon Timespan User Interaction 10 / 10 โŒ 7 / 7 โŒ โŒ
user-flow_snapshot-icon Snapshot Current page state 4 / 4 16 / 16 5 / 5 9 / 9 โŒ

When you execute and open the user-flow report you will see the measurement modes also visualized there.

user-flow--example

Organizing testing logic is an art. If you don't own that knowledge, the amount of low-level code get's a night mare to maintain in bigger projects...

This is the reason we introduced UFO's! Organize clutter code ๐Ÿ‘ฝ in developer friendly shells ๐Ÿ›ธ

user-flow--advanced-architecture

See ufo-architecture for more details.

Chrome DevTools provides a feature to help with record and export user interactions. This can replace any handwritten code and organizes interactions in a JSON structure. user-flow--replay-scripts

This library provides a way to replay and enrich those interactions over the CLI.

See recorder-exports for more details.

Implementing performance improvements without breaking something is hard.
Even harder is it, to keep it that way. ๐Ÿ”’

img-budgets-mode-support

Automatically create budgets with:
npx user-flow init --generateBudgets
Automatically create budgets from an existing lhr with:
npx user-flow init --generateBudgets --lhr path/to/lhr.json

See performance-budgets for more details.

With just a few steps you can run your user flows in as a GitHub workflow to enrich your PR's with report summaries as comments.

Automatically create a workflow with:
npx user-flow init --generateGhWorkflow

user-flow-gh-action-cover

See github-workflow-integration for more details.

With just a few steps you can run your user flows in as a Nx workspace to enrich your DX with a nx-plugin.

Automatically generate/execute/migrate with user-flow-nx-plugin:

  • nx g @push-based/user-flow-nx-plugin:install
  • nx g @push-based/user-flow-nx-plugin:target e2e

See user-flow-nx-plugin for more details.

Examples

Resources


made with โค by push-based.io

user-flow's People

Contributors

biophoton avatar christopherpholder avatar edbzn avatar hoebbelsb avatar karnaukhov-kh avatar push-based-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

user-flow's Issues

Implement DX for first usage

The setup command targets scaffolding of tests, budgets, .rc files etc.

It should focus on a good user experience from the CLI perspective. This means the user should be guided by questions and options to setup a testing repository for lighthouse user flows.

As we want to do a good job for our users I will map some challenges a potential user can have.

As there are different users let's define a persona first

In general this issue targets direct users of @push-based/user-flows, developer, performance engineers etc that install and run the CLI.

Persona

Personal data
Chang, 38, located in Brazil in a IT city, good developer in front-end development, tooling and some experience with node scripts with a high level of experience

Background
Heard about it on twitter and is curios what it really is.

Knowledge
He knows lighthouse already but is not a power user.
In general he sensed or know already that there is no good tool out there to measure runtime performance.

There is no puppeteer knowledge present in his bent, but he already wrote cypress tests and back then a small protractor tests suit.
The concept of Fixtures/Page-Objects etc. in testing architectures is nothing he uses on a daily basis but he knows the general idea and practical benefits.

Environment and Tooling
There is no CI setup for run-time performance in place in their infrastructure/company.
Not even for bootstrap performance with LHCI as the setup and maintenance was too time consuming.

Motivation
He is actively looking for new tools and tests them occasionally if the docs seem promising, in particular if he would already see how the user interaction and the resulting report is viewed.

He would even request some time to do a PoC on their product if the results are convincing and easy to explain to the product manager.

First contact and usage
In the afternoon he saw a tweet about the tool on twitter:

๐Ÿ“ฃ๐Ÿ“ฃ๐Ÿ“ฃ
I released push-based/user-flow 1.1.x

a CLI that helps with @____lighthouse
Runtime #WebPerformance performance tests

  • ๐Ÿ—ƒ CI integration
  • ๐Ÿ”’ GH Actions
  • ๐Ÿ‘โ€๐Ÿ—จPR comments
  • ๐Ÿฆบ scaffolding and nice prompts
  • ๐Ÿ“˜ updated docs

snippet-prompt
Uploading SpZcfNnbmKnbub3l8FWe.avifโ€ฆ

โญor sponsorship โ™ฅ on GH
https://github.com/push-based/user-flow/tree/main/packages/cli#readme

Gripped by the cool animation of the CLI he clicked the tweet to see details and swipes the images. "Pretty cool.... hmm"
He clicks the link and lands on the main readme page. He scrolls through the page and sees the npm badge, some logo and a toc.
As the ToC is too much to read he scrolls further down over the CLI option tables and discovers the image about the user interactions and the different scores... then the Architecture headline with the ufo and some links... Then he spots the letters "budgets" and the images which shows the highlighted area "Over budget".
Goosebumps... damn, I have to go...

As he had to drive home by car quickly to meet someone there was no time to read more about the budgets, so he did not know that they only work for the initial navigation. But the misconception intensified his interest enough to have the lib in mind for the next moment there was enough time to at least read more of the docs.

...

It's Thursday, so there most probably will be at least a little time in the coming the weekend to look it up.

The meeting with his friend was pretty cool so he stayed longer and had some nice conversations about headspace, a "observation technique" on thoughts that is described there and the movie inception. It was quite late when he came home so there was no energy left for anything else then getting ready to sleep.

After he arrived on Friday afternoon after quite a hard day (never do important stuff on a Friday) he sits on the sofa and scrolls the phone when the new tool popped up again in his mind triggered by the twitter notification badge on the app icon.

He quickly opens the notifications, scrolls some centimeters in the thread, closes it and opens the @push-based/user-flow docs on GitHub.

He skims the Quick start section for some information on how to connect the tool to his applications URL, it's little enough code to not scroll further and really read the lines.

DX to implement

  • Setup the CLI
  • Run it and see a result
    • [ ] add script to package.json to run the measure e.g. "user-flow":"npx user-flow collect"
    • open viewer and display the report

Change Markdown table Gather mode for Emoji's

Change the markdown table to use emojis instead of using the name of gather modes
Example

<span title="navigation">๐Ÿงญ<span>
Icon Mode Emoji Measure Performance Accessibility Best Practices SEO PWA
user-flow_navigation-icon Navigation ๐Ÿงญ Page load 100% / 30 100% / 30 100% / 30 100% / 30 โœ” / 7
user-flow_timespan-icon Timespan โฑ User Interaction 10 / 10 โŒ 7 / 7 โŒ โŒ
user-flow_snapshot-icon Snapshot ๐Ÿ“ท Current page state 4 / 4 16 / 16 5 / 5 9 / 9 โŒ

Exclude none typescripts from execution as user-flows

Files in the collect user-flow path/directory should only be executed if:

  • they have a ts or js extension
  • they export a module with the right required keys and rough types
    • for now lets only validate the shallow of the exported object

Add trace to our errors

It is nice to have a user friendly error message but we also need to have a way to understand the real cause. therefore I suggest to add the traces from the original error to the new one.

Example of missing traces:
Error while running user flows. TypeError: Cannot set properties of undefined (setting 'config')

It would be really helpful to see the actual file and the full trace here.

Solution:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause

try {
  connectToDatabase();
} catch (err) {
  throw new Error('Connecting to database failed.', { cause: err });
}

TODO:

  • Improve Error for serve command - Error while executing npm run start:ssr:prod
  • adopt tsconfig to latest es version so the 'cause' feature works
  • add cause to all catched or thrown errors

Include puppeteer/replay to execute steps from a `.json` export

We should create a custom RunnerExtension

This extension will detect user-flow modes over the type property.
We can later either execute the user-flow modes programmatically or add a custom replay action in the json export.

We can start with some experiments and document them in a readme.

Later we can work on a custom runner-extension and document everything.

We also have to save raw exports of original and enriched recordings.

Add log for location of persistent report on verbose collect

run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
// -- Add out path in verbose long 
Output path: /example/path ... 
// -- 
Duration: Order Coffee in CI: 46.425
25h25h25h

Reduce verbose log to only necessary logs

Current verbose output:

run "collect" as a yargs command with args:
potentialExistingCfg:  {
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows',
    serveCommand: undefined,
    awaitServeStdout: undefined
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: 'md'
  }
}
25l25l25lUpdate config under ./packages/user-flow-ci-integration/.user-flowrc.json
readRcConfig: ./packages/user-flow-ci-integration/.user-flowrc.json {
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows'
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: [ 'md' ]
  }
}
No updates for ./packages/user-flow-ci-integration/.user-flowrc.json to save.
{
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows',
    serveCommand: undefined,
    awaitServeStdout: undefined
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: [ 'md' ]
  }
}
run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
Duration: Order Coffee in CI: 46.605
25h25h25hrun "collect" as a yargs command with args:
potentialExistingCfg:  {
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows',
    serveCommand: undefined,
    awaitServeStdout: undefined
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: 'md'
  }
}
25l25l25lUpdate config under ./packages/user-flow-ci-integration/.user-flowrc.json
readRcConfig: ./packages/user-flow-ci-integration/.user-flowrc.json {
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows'
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: [ 'md' ]
  }
}
No updates for ./packages/user-flow-ci-integration/.user-flowrc.json to save.
{
  collect: {
    url: 'https://coffee-cart.netlify.app/',
    ufPath: './packages/user-flow-ci-integration/user-flows',
    serveCommand: undefined,
    awaitServeStdout: undefined
  },
  persist: {
    outPath: './packages/user-flow-ci-integration/measures',
    format: [ 'md' ]
  }
}
run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
Duration: Order Coffee in CI: 46.605
25h25h25h

Implement custom argv handling until we update to the new yargs versoin

We have a couple of functions handling the passed CLI parameters.
This is normally handled by yargs's option normalization features , but as as we have to derive a couple of options before yargs starts we have to rely on a custom implementation.

As a side note I want to mention that yargs's new version will ship a couple of features making this possible more easily. After the upgrade we most probably can delete this code.

Allow Saving flow results in multiple formats

Expected behavior

When setting multiple persist formats it should save the results in those formats
"persist": { "outPath": "./measures", "format": ["html", "json"] }

Actual behavior

The first option is saved but an error occurs when attempting to generate the second report.

Relevante code

type PersistFn = (cfg: Pick<PersistOptions, 'outPath'> & { flow: UserFlow, name: string }) => Promise<string>;

const _persistMethod = new Map<string, PersistFn>();

_persistMethod.set('html', async ({ outPath, flow, name }) => {
  const report = await flow.generateReport();
  const fileName = join(outPath, `${toFileName(name)}.uf.html`);
  writeFile(fileName, report);
  return fileName;
});

_persistMethod.set('json', async ({ outPath, flow, name }) => {
    const report = await flow.createFlowResult(); // <== Error occurs here 
    const fileName = join(outPath, `${toFileName(name)}.uf.json`);
    writeFile(fileName, JSON.stringify(report));
    return fileName;
});

export async function persistFlow(flow: UserFlow, name: string, { outPath, format }: PersistOptions): Promise<string[]> {
  // @Notice: there might be a bug in user flow and Promise's
  return Promise.all(format.map((f: string) => (_persistMethod.get(f) as any)({ flow, name, outPath })));
}

Error message

Error: no known mark: lh:runner:audit
    at Object.exports.stop (/Users/username/Applications/user-flow/node_modules/marky/lib/marky.cjs.js:104:13)
    at Function.timeEnd (/Users//usernamer/Applications/user-flow/node_modules/lighthouse-logger/index.js:129:11)
    at Function.audit (/Users//username/Applications/user-flow/node_modules/lighthouse/lighthouse-core/runner.js:82:11)
    at async auditGatherSteps (/Users//username/Applications/user-flow/node_modules/lighthouse/lighthouse-core/fraggle-rock/user-flow.js:215:20)

Possible Solution

Instead of using a new Map for the _persistMethod we could use an object method that returns the list of fileNames.
The format options would then be handled with a condition switch statement.

The potential solution was already tested by persisting both formats on the same Map.

Init command produces a example user flow with incorrect import path

When running the init command the file produce imports the use flow from '../../../' instead of '@push-based/user-flow'

Current output:

// Your custom interactions with the page
import {
  UserFlowContext,
  UserFlowInteractionsFn,
  UserFlowProvider,
} from "../../..";

Expected output:

// Your custom interactions with the page
import {
  UserFlowContext,
  UserFlowInteractionsFn,
  UserFlowProvider,
} from  '@push-based/user-flow';

Command to reproduce:
npx user-flow init

Versioning workflow

Hey, I noticed that the versioning job in the CI is failing. Seems like your last tag is not attached to main so semver fails (130839c).

I wonder how you end up with that state as it's not possible with the actual configuration. Are you doing manual steps that I ignore when you release?

DX: Implement DX for second contact

The "second contact" focuses on more advanced tests and CI integration.

It should focus on a good user experience from the CLI perspective, as well as a smooth setup of the GitHub action to integrate user-flow into the CI. This means the user should be guided by good documentation and maybe a command with questions and options to set it up.

In general this issue targets direct users of @push-based/user-flows, developer, performance engineers etc. that have user-flow already installed and running, but not implemented a real live integration.
This story is a follow up on #32

Furthermore it will help to separate the CLI and the GH action repositories.

Related repositories:

  • ๐Ÿ’ป the CLI - @push-based/user-flow
  • โš™ the CI Action - CI related code e.g. GitHub Action @push-based/user-flow-gh-action

Persona

Personal Data

Chris, 26, full-stack developer, self educated

Background

Needs an easy way to stop regression on an ongoing JS application. He tried user flow already #32 and remembered it after another uncaught regression in his live system.

Knowledge

He knows lighthouse and puppeteer, also knows user-flow.

Environment and Tooling

There are no performance tests in the application
There are very few tests in the CI

Second contact

Chris already tested out user-flow and was impressed by this new method of testing the performance of user actions. However, he only tested the basics and left it at that point as there was no more time.

Some 1,5 month later, while testing the performance of the application he was working on, he noticed the performance of the application got worse and he had to figure out why.

The app is pretty busy maintained from many developers and there is no good release management in place to track changes, nor any continuous performance measurement, so it was quite a hassle to spot the issue.

After all the issue was found, and additionally, not only the performance that had gotten worse, but had also a couple of SEO critical bugs introduced.
What a nice ending of that Friday... and hello to a weekend of cleaning up others mess.

In the middle of the next week, not quite recovered from the work intense weekend he started to reason about possible solutions to this unfortunately recurring problem and user-flow jumped in his mind again.

He suggested to set up using user-flow to avoid regressions, but the conversation with his CTO was unpleasant.
Just by looking at some GitHub repository and no clear time estimation on how much it takes to get it running it just got zero attention next to all the high priority tickets in the queue.

But Chris's curiosity got the best of him and he keep playing around with it, knowing that he could convince his CTO to implement user-flow if a PoC could catch some regression in the CI.

Problems from test run

To fix for second contact

  • Using puppeteer to manipulate the application was more time consuming that expected - fixed by #66
  • Every time I wanted to add a test I had to run the whole test white which took way to long. - fixed by #141
  • I still had no idea how to integrate it into the CI - fixed by #66
  • New results overwrote the previous ones and didnโ€™t allow me to check if there was a difference in results - fixed by #109
  • I wasnโ€™t really sure where to put files and what should be where and realized it would eventually become a massive code base or a bowl of spaghetti code - fixed by #64
  • If I wanted to import recording from chrome new recorder I had to introduce a bunch of new functions and didnโ€™t know where they should go, also since they are written in js and not TS I ended up just adding //@ts-ignore all over the place since it would require effort I wasnโ€™t willing to invest at the moment to fix the type errors - fixed by #66
  • A couple of times I had to change puppeteer configurations and just kind of dumped the new configs in the file. - fixed
  • ~generate multiple formats - fixed by #181 ~

To fix for third contact

  • Some action required the user to log in and it added unnecessary steps, plus I could figure out how to maintain the user logged in between different user flow test runs - fixed by #20
  • To set up a new test I ended up duplicating a user flow file and modifying it.

DX to Implement

Next steps

  • #161
  • โš™ run it against the main branch to check
  • ๐Ÿ’ป get a warning / error when a PR causes performance regression that below the budget
  • ๐Ÿ’ปโš™docs for authentication #20
  • ๐Ÿ’ป setup scalable testing architecture
    • add a test
    • add a page
    • add a UI element
    • add fixtures
    • add budget

flow file name should get derived from url

Reports persisted with collect should have:

  • the url in the flow name
  • the url and ISO date of execution in the persisted file name (e.g. LH has this name github.com-20221216T184910.json)

Typing changes:

  • name should be an optional parameter of flowOptions

flow name should be:
<url>-<name(optional)>-<date>

init Command:

  • initial flow should get renamed neutral.

Documentation:

  • Examples should be updated with new names.

docs: multiselect handling

Select choices

These key combinations may be used on prompts that support multiple choices, such as the multiselect prompt, or the select prompt when the multiple options is true.

command description
space Toggle the currently selected choice when options.multiple is true.
number Move the pointer to the choice at the given index. Also toggles the selected choice when options.multiple is true.
a Toggle all choices to be enabled or disabled.
i Invert the current selection of choices.
g Toggle the current choice group.

environment variables and CI mode

Motivation

Several things in the way how the CLI works are specifically designed for humans that use and interact with it.
A couple of examples are prompts, file generation, visualizing data, reporting progress etc

There are a couple of scenarios where we have no humans involved and or just no need for a specific feature to be active.
One way of doing this is to configure the CLI over comments. E.g. don't open a report automatically --open false.

To avoid a large set of CLI parameters or many different config files I suggest to do some of those configurations automatically.

Examples

CI Configurations Draft:

  • --open false - No one will look at it in the CI
  • --interactive false - no one can answer
  • --format md --format <all others> - Because we need it for the GitHub action as md text
  • headless - we need to configure the lighthouse for a headless run

Before:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json --interactive false -e false -f html -f md

After:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json

Test Configurations Draft:

  • --open false - We only need it to test that the --open parameter is working
  • --interactive false - Only the prompts need it, the rest is defined by config files
  • --dryRun false - We want to have as many tests in dry run to save time
  • headless - we need to test a headful run only once

Before:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json --interactive false -e false --dryRun true

After:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json

Details

To group the different pre-configurations I suggest to introduce the concept of environments to the CLI.

Environments:

  • LOCAL - on the developers machine
  • CI - in the a CI setup like GitHub pages
  • TEST - in the test environment

State of the art - How to detect CI in all the services like GitHub?

In general the name CI is used and set to true or 1.

The following platforms have a CI key:

Unknown:

Todos

  • introduce checks for CI and Test environment
  • introduce env variable and check
  • setup pre-configuration sets
  • test configuration overwrite

Add conversion aid for Chrome Dev Tools Recorder

To reduce the friction for new users to create user flow we should implement some aid to convert Chrome DevTools Recorder outputs into user-flow scripts.

Recorder user flows can be exported and shared in 3 ways/formats:

  1. JSON file
  2. @puppeteer/replay script
  3. Puppeteer script

Potentially we could create a Chrome extension to export replay scripts in user-flow format directly.
As there are already multiple extensions for costume exports (examples: Cypress extension and WebPage Test extension) as well as a Recorder extention API.

Handle error for rc path

Error

> npx @push-based/user-flow -p packages/ngx-fast-icon-demo/perf collect

internal/fs/utils.js:312
    throw err;
    ^

Error: EISDIR: illegal operation on a directory, read
    at Object.readSync (fs.js:614:3)
    at tryReadSync (fs.js:383:20)
    at readFileSync (fs.js:420:19)
    at readFile (C:\Users\micha\git\ngx-fast-icon\node_modules\@push-based\user-flow\src\lib\core\utils\file.js:13:38)
    at readRcConfig (C:\Users\micha\git\ngx-fast-icon\node_modules\@push-based\user-flow\src\lib\core\rc-json\index.js:9:48)

Catch error and log that the rc path needs to point to file and not a directory.

Compare and visualise two reports in MD format

Motivation

Every change in the code base could potentially impact the application's performance.
We need a way to inform developers if the changes were performance slowdowns or improvements.

To accomplish this, we need a way to measures from a current audit with measures from a previous audit.

Implementation

There are different building blocks necessary to implement this feature:

Building blocks

  • Report comparison - a way to compare results with previous results
    • Selecting comparison - a way to specify that the comparison should be created
    • Specifying baseline - a way to specify a report to compare the new measures with
  • Report comparison visualization - a human-readable version of the comparison eg. MD table
    • Persistance option - logic to name and save the md report

Suggested solution

Report Comparison

The report would expand the reduced report by adding baseline measures.
This reduced report comparison would then be used to generate the Md table.

{
  "name": "Sandbox Setup StaticDist",
  "steps": [
    {
      "name": "Navigation report",
      "gatherMode": "navigation",
      "results": {
        "performance": 0.95,
        "accessibility": 1,
        "best-practices": 0.92,
        "seo": 1,
        "pwa": 0.3
      },
      "baseline":  {
        "performance": 0.89,
        "accessibility": 1,
        "best-practices": 0.95,
        "seo": 0.9,
        "pwa": 0.2
      }
    },
  ]
}

Selecting Comparison

TODO
Related question:

  • Should this be the default?
  • Should this be a optional RC & CLI param?

Specifying Baseline

TODO
Related question:

  • Should specifying the baseline be the selection method ?
  • Should specifying the baseline mean that Md and Stdout are automatically comparisions?

Report Comparison Visualization

The report would add the comparison number next to the current measurement

Step Name Gather Mode Performance Accessibility Best Practices Seo Pwa
Navigation report navigation 95 (+10) 100 82 (-11) 100 30 (-5)
Timespan report timespan 10/11 (-2) - 5/7 (-1) - -
Snapshot report snapshot ร˜ 3/4 10/10 4/5 9/9 -

The report would be styled to make it easier to spot which measures when up or down.
Some options for this styling are:

  • emojis ( ๐ŸŸข | โšช | ๐Ÿ”ด ), (๐Ÿ’š| ๐Ÿ”ป), (๐Ÿ”บ| ๐Ÿ”ป), ( โซ | โฌ )
  • line breaks <br />
  • parenthesis and plus-minus (+ #), (- #)
Step Name Gather Mode Performance Accessibility Best Practices
Option 1 navigation 95 (+10 ๐Ÿ’š) 100 82 (-11 ๐Ÿ”ป)
Option 2 navigation 95 (+10 ๐ŸŸข) 100 (โšช) 82 (-11 ๐Ÿ”ด)
Option 3 navigation 95 (+10 โซ) 100 (โšช) 82 (-11 โฌ )
Option 4 navigation 95
๐Ÿ’š +10
100
82
๐Ÿ”ป -11
Option 5 navigation 95
๐ŸŸข +10
100
โšช
82
๐Ÿ”ด -11
Option 6 navigation 95
+10 โซ
100
82
-11 โฌ

Persistance Option

TODO
Related question:

  • Should this be simplfy an extention of MD and Stdout or a seperate persist format?
  • Should the persisted report include a naming convention so specify its not the same as the MD and avoid name colitions?

References

DX: Improving output of user-flow --help command

Currently the docs start with:

Install
Run npm i @push-based/user-flow --save-dev or yarn add @push-based/user-flow --dev to install the library.

After that you can run:
user-flow --helpor user-flow --help

Run without install

You can also use npx to run it in e.g. the CI setup: npx @push-based/user-flow --help

but the output of the command is:

% npx user-flow --help

Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]

feat: add config-path options to CLI

Add --config-path options to CLI

Add a new parameter the the collect command named config-path. This path should specify the location of a lighthouse config file.

TODOS:

  • implement types from Configuration
  • create docs for CLI options: --config-path - path to configuration . e.g. --config-path=confiuration.json
  • Add resources to docs: Lighthouse configuration

Passing parameters to the cli overwrites the rc

When running the collect file and passing in an additional parameter like format it will overwrite the rc json file.
npx user-flow --format md will overwrite the json file and remove the previous configuration.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.