>  ____   ___   ____  _   _        _    _   ___    ___  _  _  ____
> |  _ \ / _ \ /  __)| |_| | ____ | \  / | / _ \  / __)| |/ )/  __)
> |  __/( (_) )\__  \|  _  |(____)|  \/  |( (_) )( (__ |   ( \__  \
> |_|    \___/ (____/|_| |_|      |_|  |_| \___/  \___)|_|\_)(____/

A mock-up set of PowerShell 7 functions that filter text-object input from the pipeline(stdin) and return text-object (this concept is closer to Bash than PowerShell).Inspired by Parsing Text with PowerShell (3/3), Steve Lee, January 28th, 2019.

• A set of functions for interactive text processing with connecting pipes.
• For my personal work and hobby use.
• For use in UTF-8 Japanese environments on windows.
• Required PowerShell7+ on Windows
  • Some script may work with PowerShell5
  • Some script may work with PowerShell7+ on Linux
• Note:
  • The code is spaghetti (due to my technical inexperience).
  • Insufficient tests and error handlings.
  • Processing speed is not fast.
• Workflow-Examples:
  • See: examples.md

Function list:

# one-liner to create function list
# (add internal link to each function's section)
cat README.md `
    | grep '^#### ' `
    | grep -o '\[[^[]+\]' `
    | sed 's;\[([^\]]+)\];[$1][];g' `
    | sort {
        -join ( [int[]] ("$_".ToCharArray()) | ForEach-Object { [System.Convert]::ToString($_, 16)})
    } -Unique `
    | flat -ofs ", " `
    | Set-Clipboard

• Add-LineBreakEndOfFile, Add-LineBreak, Add-Quartile, Add-Stats, Apply-Function, Cast-Date, Cast-Decimal, Cast-Double, Cast-Integer, ConvImage, Decrease-Indent, Delete-Field, Detect-XrsAnomaly, Drop-NA, Edit-Function, Edit-Property, Execute-RMarkdown, Execute-TinyTeX, Get-AppShortcut, Get-ClipboardAlternative, Get-DateAlternative, Get-Histogram, Get-OGP, Get-Ticket, GetValueFrom-Key, Grep-Block, GroupBy-Object, Inkscape-Converter, Invoke-Link, Join2-Object, Measure-Quartile, Measure-Stats, Measure-Summary, Override-Yaml, Plot-BarChart, Rename-Normalize, Replace-ForEach, Replace-NA, Select-Field, Set-NowTime2Clipboard, Shorten-PropertyName, Shutdown-ComputerAFM, Sleep-ComputerAFM, Sort-Block, Test-isAsciiLine, Transpose-Property, Trim-EmptyLine, Unique-Object, Unzip-Archive, addb, addl, addr, addt, cat2, catcsv, chead, clip2file, clip2hyperlink, clip2img, clip2normalize, clip2push, clip2shortcut, conv, count, csv2sqlite, csv2txt, ctail, decil, delf, dot2gviz, filehame, fillretu, flat, flow2pu, fpath, fval, fwatch, gantt2pu, getfirst, getlast, grep, gyo, han, head, image2md, jl, json2txt, juni, keta, kinsoku, lastyear, lcalc2, lcalc, linkcheck, linkextract, list2table, logi2dot, logi2pu, man2, map2, math2tex, mdfocus, mdgrep, mind2dot, mind2pu, movw, nextyear, pawk, percentile, pu2java, push2loc, pwmake, pwsync, retu, rev2, rev, say, sed-i, sed, self, seq2pu, sleepy, sm2, summary, table2md, tac, tail-f, tail, tarr, tateyoko, teatimer, tenki, tex2pdf, thisyear, toml2psobject, uniq, vbStrConv, watercss, wrap, yarr, ycalc, ysort, zen

General: This repository contains a set of filter commands for pattern matching on amorphous strings. The filter commands expect UTF-8 + space-delimited + line-oriented string data (text object) through the pipeline as input. The output is also a string, not a typical PowerShell object. Some functions accept Object-input through pipeline, and some functions are "like PowerShell cmdlets" that output object. The design of the filters is similar to Unix/Linux shells such as Bash.

File Structure: The file structure under the src directory is one file and one function. The function name is a string excluding _function.ps1 from the file name. Basically, filters are not dependent on other filters, so if you move the function file and read it with dot source, you can use it anywhere. (Some filters depend on other filters.) Also, note that some filters only work with PowerShell7+ on Windows.

Note: The filters in this repository are command sets intended for personal use and are mockups without proper error handling.

1. Comment out unnecessary lines (functions) in operator.ps1
2. Dot sourcing operator.ps1 (and optional operator-extra.ps1)
   • (However, this only works for the current process. When you close the window (end the process), the dot-sourced functions are forgotten)

# install all functions
. path/to/posh-mocks/operator.ps1
# option
. path/to/posh-mocks/operator-extra.ps1

The functions are designed to be as independent from other functions as possible. You can also load single function using dot-sourcing. In that case, it is recommended to first set the encoding of the current process to UTF-8 as shown below, because this functions basically assumes UTF-8 for input and output through the pipeline.

# install favorite functions for japanese environment
# set encode
if ($IsWindows){
    chcp 65001
    [System.Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding("utf-8")
    [System.Console]::InputEncoding  = [System.Text.Encoding]::GetEncoding("utf-8")
    # compartible with multi byte code
    $env:LESSCHARSET = "utf-8"
}
# sourcing dot files
. path/to/posh-mocks/src/hoge_function.ps1
. path/to/posh-mocks/src/piyo_function.ps1

The motivation for creating each function and usage.

List the function names (file names) under src directory. With this function, I don't have to remember the name of the function I created.

• Usage
  • man2 man2
  • man2 <function-name> [-p|-Paging]
• Params
  • man2
    • [[-f|-FunctionName] <String>]
    • [-c|-Column <Int32>]
    • [-e|-Exclude <String>]
    • [-p|-Paging]
    • [-i|-Include <String>]
    • [-Examples]
    • [-Independent]
    • [-l|-Line]
• Behavior
  1. Remove _function.ps1 strings from the file names of the src/*_function.ps1 files in the same hierarchy as this function file
  2. Enumerate function name
• Dependencies
  • flat, tateyoko, keta
    • Use the -Independent switch to create a table without depending on other functions in this repository (use only built-in Format-Wide function to formar the output).
• Examples
  • Get command list as table
    • man2
    • man2 -c 5
  • Get help man2 <command>
    • man2 man2
    • man2 man2 -p
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: man2
• Inspired by
  • Format-Wide (Microsoft.PowerShell.Utility) - PowerShell
  • Select-Object (Microsoft.PowerShell.Utility) - PowerShell

Examples:

Function names are sorted lexicographically from top to bottom. This behavior is similar to linux ls command. The reason for depending on flat, tateyoko and keta is to obtain this output.

man2

Add-LineBreakEndOfFile clip2file      head        say
Add-LineBreak          clip2hyperlink image2md    sed-i
Add-Stats              clip2img       jl          sed
Apply-Function         clip2normalize json2txt    self
ConvImage              clip2push      juni        seq2pu
Delete-Field           clip2shortcut  keta        sleepy
Detect-XrsAnomaly      conv           kinsoku     sm2
Drop-NA                count          lcalc       summary
Get-AppShortcut        csv2sqlite     linkcheck   table2md
Get-First              csv2txt        linkextract tac
Get-Histogram          ctail2         list2table  tail-f
Get-Last               ctail          logi2dot    tail
Get-OGP                decil          logi2pu     tarr
GroupBy-Object         delf           man2        tateyoko
Invoke-Link            dot2gviz       map2        teatimer
Measure-Stats          filehame       mdfocus     tenki
Override-Yaml          fillretu       mdgrep      tex2pdf
Plot-BarChart          flat           mind2dot    toml2psobject
Rename-Normalize       flow2pu        mind2pu     uniq
Replace-NA             fpath          movw        vbStrConv
Select-Field           fval           pawk        watercss
Shorten-PropertyName   fwatch         percentile  wrap
addb                   gantt2pu       pu2java     yarr
addl                   gdate          push2loc    ycalc
addr                   getfirst       pwmake      ysort
addt                   getlast        pwsync      zen
cat2                   grep           retu
catcsv                 gyo            rev2
chead                  han            rev

Edit-Function [func|alias] [editor]

• Usage
  • man Edit-Function
  • man e
• Params
  • Edit-Function
    • [[-f|-Function] <String>] ...function name or alias
    • [[-e|-Editor] <String>] ...specify editor
    • [-o|-AsObject] ...output only object
    • [-q|-DryRun] ...what-if

exapmles

Edit man2 function with default text editor

Edit-Function -Function man2

# or
Edit-Function -f man2

# or
Edit-Function man2

# or
e man2

output:

Directory: path/to/the/posh-mocks/src

Mode   LastWriteTime    Length Name
----   -------------    ------ ----
-a---  2023/11/25 8:44  12133 man2_function.ps1

Edit man2 function with specified text editor

Edit-Function -Function man2 -Editor notepad

# or
Edit-Function -f man2 -e notepad

# or
Edit-Function man2 notepad

# or
e man2 notepad

Output only Object (do not run editor)

e i -o

Directory: path/to/the/posh-mocks/src

Mode  LastWriteTime 　 Length Name
----  ------------- 　 ------ ----
-a--- 2023/10/27 12:00 13459  Invoke-Link_function.ps1

If there are no arguments, return the function directory

e

Directory: path/to/the/posh-mocks

Mode  LastWriteTime   Length Name
----  -------------   ------ ----
d-r-- 2023/11/25 8:17        src

If there are no arguments, return the function list

e

Open text with editor if an existing file is specified

e ../bin/posh-mocks/operator.ps1

Replace input line by line. A degraded copy of the sed command used in Unix/Linux environments. Roughly equivalent to "string" | ForEach-Object{ $_ -replace 'reg','str' }.

sed is a command that I use every day, so I made it available with similar syntax on Bash and PowerShell.

• Usage
  • man2 sed
  • sed 's;<regex-pattern>;<replace-strings>;g'
    • The second characters from the left is recognized as a delimiter. This behavior is used when replacing delimiter ;. For example, you can write sed 's@;;;@aaa@g' to replace ;.
  • sed 's;<regex-pattern>;<replace-strings>;'
    • If there is no g at the end, only the first match pattern from the left of the line is replaced.
    • Note that this use case is case-sensitive
  • sed "s;`t;<replace-strings>;g"
    • Control characters such as `t and ``` rn `` can be replaced by surrounding them with double quotes.
    • Note that it is not \t or \n line sed on Bash.
• Inspired by Unix/Linux Commands
  • Command: sed

Examples:

# g flag - replace all strings matching a pattern
'a1b1c1' | sed 's;1;2;g'

a2b2c2

# replace only first match 
# (Note that this mode is case sensitive)
'a1b1c1' | sed 's;1;2;'

a2b1c1

# delete tab (use double quote)
cat a.txt | sed "s;`t;;g"

# replace CrLf to space
(cat a.txt) -join " "

# print mode.

# input data
$dat = "aaa", "bbb", "ccc", "ddd", "eee"
$dat

aaa
bbb
ccc
ddd
eee

# Output between "bbb" and "ddd"
$dat | sed 'p;^bbb;^ddd;'

bbb
ccc
ddd

# delete mode.

# input data
$dat = "aaa", "bbb", "ccc", "ddd", "eee"
$dat

aaa
bbb
ccc
ddd
eee

# Delete between "bbb" and "ddd"
$dat | sed 'd;^bbb;^ddd;'

aaa
eee

Edit files in-place. A degraded copy of sed -i on Unix/Linux. To prevent files from being accidentally overwritten due to incorrect substitution, this command does not actually overwrite files until you specify the -Execute swicth.

• Usage
  • man2 sed-i
  • sed-i 's;pattern;replace;g' file [-Execute] [-DoNotCreateBackup|-OverwriteBackup]
  • The default behavior is dry run and creates a backup file (.bak) in the current directory
  • Overwrite files by specifying -Execute switch
• Examples
  • sed-i 's;abc;def;g' file -Execute
    • Equivalent to sed -i.bak 's;abc;def;g' file on Unix/Linux (backup original file to .bak file.)
  • sed-i 's;abc;def;g' file -Execute -DoNotCreateBackup
    • Equivalent to sed -i 's;abc;def;g' file on Unix/Linux (do not create backup file.)
  • sed-i 's;pattern1;replace1;g','s;pattern2;replace2;g',... file
    • Multiple replacement strings can be specified by separating them with commas
• Inspired by Unix/Linux Commands
  • Command: sed

Examples:

"abcde" > a.txt; sed-i 's;abc;def;g' a.txt

ifile: ./a.txt
ofile: ./a.txt.bak
defde

ls *.txt

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          2022/09/29    21:41              7 a.txt
-a---          2022/09/29    21:41              7 b.txt

ls *.txt | %{ sed-i 's;abc;def;g' $_.FullName }

ifile: a.txt
ofile: a.txt.bak
defde

# Replace and overwrite original file and create backup
ls *.txt | %{ sed-i 's;abc;hoge;g' $_.FullName -Execute }

./a.txt > ./a.txt.bak
./b.txt > ./b.txt.bak

# Replace and overwrite original file and *do not* create backup
ls *.txt | %{ sed-i 's;abc;hoge;g' $_.FullName -Execute -DoNotCreateBackup }

./a.txt > ./a.txt
./b.txt > ./b.txt

Single-line oriented pattern matching using regular expressions for windows. A degraded copy of the grep command on Linux environment. Roughly equivalent to Select-String -Pattern <reg>. Since grep is one of the commands that I often use in my daily tasks, I aimed for an equivalent usability on both Bash and PowerShell environments.

By default, it is case insensitive, but -CaseSensitive switch makes case sensitive.

By default, the specified pattern is interpreted as a regular expression, but the [-s|SimpleMatch] option allows it to be treated as literal strings.

• Usage
  • man2 grep
  • grep '<regex>' -H file1,file2,...
  • cat file1,file2,... | grep '<regex>' [-v][-f][-s][-C <int>[,<int>]][-l]
  • cat file1,file2,... | grep '<regex>' [-o][-l]
• Inspired by Unix/Linux Commands
  • Command: grep
• Learn
  • about Splatting - PowerShell (ja-jp)

The search speed of this command is not fast. (In my environment, it is faster to simply use Select-String cmdlet.) Therefore, if you are searching with simple conditions, it is faster to use Set-Alias -Name grep -Value Select-String.

# Select-String (fast)
1..10 | %{ Measure-Command{ 1..100000 | sls 99999 }} | ft

Days Hours Minutes Seconds Milliseconds
---- ----- ------- ------- ------------
0    0     0       0       437
0    0     0       0       386
0    0     0       0       394
0    0     0       0       385
0    0     0       0       407
0    0     0       0       715
0    0     0       0       424
0    0     0       0       424
0    0     0       0       443
0    0     0       0       423

# grep (slow)
1..10 | %{ Measure-Command{ 1..100000 | grep 99999 }} | ft

Days Hours Minutes Seconds Milliseconds
---- ----- ------- ------- ------------
0    0     0       1       84
0    0     0       1       74
0    0     0       1       287
0    0     0       1       81
0    0     0       1       186
0    0     0       1       186
0    0     0       1       79
0    0     0       1       382
0    0     0       1       178
0    0     0       1       183

Both grep and Select-String are faster to search by specifying a file as an argument than reading input via the pipeline. Therefore, when processing big size data, it is a good practice to use grep instead of cat at the beginning of the pipeline and specify a file as an argument to reduce the amount of data.

# slow
cat big.data | grep "hoge" | grep "piyo"
cat big.data | sls  "hoge" | sls  "piyo"

# fast!
grep "hoge" big.data | grep "piyo"
sls  "hoge" big.data | sls  "piyo"

Examples

• ref: Select-String (Microsoft.PowerShell.Utility) - PowerShell

# Find a case-sensitive match (grep 'regex' -CaseSensitive)
'Hello', 'HELLO' | grep 'HELLO' -CaseSensitive -SimpleMatch

HELLO

# Find a pattern match (grep 'regex')
grep '\?' -H "$PSHOME\en-US\*.txt"

https://go.microsoft.com/fwlink/?LinkID=108518.
or go to: https://go.microsoft.com/fwlink/?LinkID=210614
or go to: https://go.microsoft.com/fwlink/?LinkID=113316
    Get-Process -?         : Displays help about the Get-Process cmdlet.

# Find matches in text files (grep 'regex' -H file,file,...)
Get-Alias   | Out-File -FilePath .\Alias.txt   -Encoding UTF8
Get-Command | Out-File -FilePath .\Command.txt -Encoding UTF8
grep 'Get\-' -H .\*.txt | Select-Object -First 5

Alias.txt:7:Alias       cal2 -> Get-OLCalendar
Alias.txt:8:Alias       cat  -> Get-Content
Alias.txt:28:Alias      dir  -> Get-ChildItem
Alias.txt:44:Alias      gal  -> Get-Alias
Alias.txt:46:Alias      gbp  -> Get-PSBreakpoint

# Skip blank lines (grep ".")
"aaa","","bbb","ccc"

aaa

bbb
ccc

"aaa","","bbb","ccc" | grep .

aaa
bbb
ccc

# Find a string in subdirectories (grep 'regex' -H file,file,... [-r|Recurse])
grep 'tab' -H '*.md' -r [-FileNameOnly|-FileNameAndLineNumber]

Table: caption
{.table2col}
| table |

# The following commands are also approximately equivalent
ls *.md -Recurse | grep "table"

table2col.md:10:Table: caption
table2col.md:12:{.table2col}
table2col.md:66:| table |

# Find strings that do not match a pattern (grep 'regex' [-v|-NotMatch])
Get-Command | Out-File -FilePath .\Command.txt -Encoding utf8
cat .\Command.txt | grep "Get\-", "Set\-" -NotMatch | Select-Object -Last 5

Cmdlet Write-Output   7.0.0.0  Microsoft.PowerShell.Utility
Cmdlet Write-Progress 7.0.0.0  Microsoft.PowerShell.Utility
Cmdlet Write-Verbose  7.0.0.0  Microsoft.PowerShell.Utility
Cmdlet Write-Warning  7.0.0.0  Microsoft.PowerShell.Utility

# Use double quotes when searching for tab characters (grep "`t")
"1,2,3", "4,5,6", "7,8,9", "" | %{ $_ -replace ',', "`t" } | grep "`t[28]"

1       2       3
7       8       9

# Find lines before and after a match (grep "regex" -C <int>,<int> )
Get-Command | Out-File -FilePath .\Command.txt -Encoding utf8
cat .\Command.txt | grep 'Get\-Computer' -C 2, 3

  Cmdlet   Get-Command        7.3.1.500  Microsoft.PowerShell.Core
  Cmdlet   Get-ComputeProcess 1.0.0.0    HostComputeService
> Cmdlet   Get-ComputerInfo   7.0.0.0    Microsoft.PowerShell.Management
  Cmdlet   Get-Content        7.0.0.0    Microsoft.PowerShell.Management
  Cmdlet   Get-Counter        7.0.0.0    Microsoft.PowerShell.Diagnostics
  Cmdlet   Get-Credential     7.0.0.0    Microsoft.PowerShell.Security

# Tips: use Out-String -Stream (alias:oss) to greppable
cat .\Command.txt | grep 'Get\-Computer' -C 2, 3 | oss | grep '>'

> Cmdlet   Get-ComputerInfo   7.0.0.0    Microsoft.PowerShell.Management

# Find all pattern matches (grep 'regex' -o)
cat "$PSHOME\en-US\*.txt" | grep "PowerShell"

PowerShell Help System
Displays help about PowerShell cmdlets and concepts.
PowerShell Help describes PowerShell cmdlets, functions, scripts, and
modules, and explains concepts, including the elements of the PowerShell
PowerShell does not include help files, but you can read the help topics
You can find help for PowerShell online at
    1. Start PowerShell with the "Run as administrator" option.
    Get-Help About_Modules : Displays help about PowerShell modules.

cat "$PSHOME\en-US\*.txt" | grep "PowerShell" -o

PowerShell
PowerShell
PowerShell
PowerShell
PowerShell
PowerShell
PowerShell
PowerShell
PowerShell

The above example can also be written as follows using Select-String (alias:sls)

# Adding parentheses "( )" when connecting pipelines requires rework and is troublesome
(cat "$PSHOME\en-US\*.txt" | sls "PowerShell" -AllMatches).Matches.Value

PowerShell
PowerShell
...

If you simply want to grep 'regex', it is faster to use sls 'regex'. However, slightly complicated (but frequently used in my tasks) options, such as grep 'regex' -o and grep 'regex' -H <files>, it can be written shorter than using sls.

# An example of using "sls" without parentheses.
# In this case as well, compared to using "grep -o",
# you will have to connect two more pipes, which is cumbersome.�
cat "$PSHOME\en-US\*.txt" `
    | sls "PowerShell" -AllMatches `
    | select -ExpandProperty Matches `
    | select -ExpandProperty Value

PowerShell
PowerShell
...

# Convert pipeline objects to strings using Out-String -Stream
$hash = @{
    Name     = 'foo'
    Category = 'bar'
}

# !! NO output, due to .ToString() conversion
$hash | grep 'foo'

# Out-String converts the output to a single multi-line string object
$hash | Out-String | grep 'foo'

Name           Value
----           -----
Name           foo
Category       bar

# Out-String -Stream converts the output to a multiple single-line string objects
$hash | Out-String -Stream | grep 'foo'

Name           foo

# Leave header line
grep 'virginica' iris.csv -LeaveHeader `
    | head -n 5

sepal_length,sepal_width,petal_length,petal_width,species
6.3,3.3,6.0,2.5,virginica
5.8,2.7,5.1,1.9,virginica
7.1,3.0,5.9,2.1,virginica
6.3,2.9,5.6,1.8,virginica

cat iris.csv `
    | grep 'virginica' -LeaveHeader `
    | head -n 5

sepal_length,sepal_width,petal_length,petal_width,species
6.3,3.3,6.0,2.5,virginica
5.8,2.7,5.1,1.9,virginica
7.1,3.0,5.9,2.1,virginica
6.3,2.9,5.6,1.8,virginica

入力文字列の最初の数行または最後の数行を出力する。 Linux環境で使うhead、tailのような使用感で文字列を置換する。 "string" | Select-Object -First <int> / -Last <int>と同じ効果を得る。

• Usage
  • man2 head
  • man2 tail
• Examples
  • 1..20 | head [-n <int>]
  • 1..20 | tail [-n <int>]
  • head *.*
  • tail *.*
• Inspired by Unix/Linux Commands
  • Command: head, tail

A PowerShell implementation of the tail -f command using Get-Content a.txt -Wait -Tail 1 -Encoding utf8.

• Usage
  • man2 tail-f
• Examples
  • tail-f a.txt
  • 1..100 | %{ (Get-Date).ToString('yyyy-MM-dd HH:mm:ss') >> a.txt; sleep 1 }
• Inspired by Unix/Linux Commands
  • Command: tail

Example:

1..100 | %{ (Get-Date).ToString('yyyy-MM-dd HH:mm:ss') >> a.txt; sleep 1 }
("another process")
tail-f a.txt

2023-04-07 05:33:29
2023-04-07 05:33:30
2023-04-07 05:33:31
...

入力文字列の最初の数行または最後の数行を削除（カット）して出力する。 "string" | Select-Object -Skip <int> / -SkipLast <int>と同じ効果を得る。

• Usage
  • man2 chead
    • chead [-Num <Int32>] [[-Files] <String[]>]
  • man2 ctail
    • ctail [-Num <Int32>] [[-Files] <String[]>] [-InputText <String[]>]
• Examples
  • 1..20 | chead [-n <int>]
  • 1..20 | ctail [-n <int>]
• Inspired by Unix/Linux Commands
  • Command: chead

Examples:

chead - Cut the first part of files

# read from stdin
1..5 | chead

2
3
4
5

1..5 | chead -n 2

3
4
5

# read from file
1..5 > a.txt; chead a.txt

2
3
4
5

1..5 > a.txt; chead -n 2 a.txt

3
4
5

ctail - Cut the last part of files

# read from stdin
1..5 | ctail

1
2
3
4

1..5 | ctail -n 2

1
2
3

# read from file
1..5 > a.txt; ctail a.txt

1
2
3
4

1..5 > a.txt; ctail -n 2 a.txt

1
2
3

入力から隣接する（連続する）重複行をなくし一意とする。大文字小文字は区別しない。事前ソート必要。 Group-Object -NoElement、Get-Uniqueと似ている。

• Usage
  • man2 uniq
  • uniq [-c|-d]
• Examples
  • 1,2,3,4,5,3 | sort | uniq
  • 1,2,3,4,5,3 | sort | uniq -c
• Inspired by Unix/Linux Commands
  • Command: uniq

テキストファイルのコンテンツを取得。 複数ファイルを指定する方法は、Get-Contentの際の「カンマ区切り」ではなく「半角スペース区切り」。 引数にハイフン「-」指定で標準入力から読み込み。

• Usage
  • man2 cat2
  • cat2 file1 file2 file3...
• Inspired by Unix/Linux Commands
  • Command: cat

入力を行単位逆順に出力する。

• Usage
  • man2 tac
• Examples
  • 1..5 | tac
  • tac a.txt,b.txt
• Inspired by Unix/Linux Commands
  • Command: tac

Examples:

1..5 | tac

5
4
3
2
1

文字列を行内で反転する。

• Usage
  • man2 rev
• Examples
  • "aiueo" | rev
• Inspired by Unix/Linux Commands
  • Command: rev

Examples:

"aiueo" | rev

oeuia

半角スペースで区切られた列をリバースする。 列内の文字列はリバースしない。 入力はパイプのみ受け付け。

• Usage
  • man2 rev2
• Inspired by Unix/Linux Commands
  • Command: rev
• Inspired by greymd/egzact: Generate flexible patterns on the shell - GitHub
  • License: The MIT License (MIT): Copyright (c) 2016 Yasuhiro, Yamada
  • Command: mirror

Examples:

Write-Output "01 02 03" | rev2

03 02 01

Write-Output "01 02 03" | rev2 -e

01 02 03
03 02 01

半角スペース区切り行列の転置（半角スペース区切り文字列の縦横変換）。 列数は不揃いでもよい。

• Usage
  • man2 tateyoko
• Examples
  • "1 2 3","4 5 6","7 8 9" | tateyoko
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: tateyoko

Examples:

"1 2 3","4 5 6","7 8 9"

1 2 3
4 5 6
7 8 9

"1 2 3","4 5 6","7 8 9" | tateyoko

1 4 7
2 5 8
3 6 9

半角スペース区切りレコードの列数を最大列数にそろえる。 不足列を埋める、で、fill（埋める）＋retu（列）。 列数がそろっていると何かと都合よい。

• Usage
  • man2 fillretu
  • cat dat.txt | fillretu

Input:

cat dat.txt

2018 3
2018 3
2018 3
2017 1
2017 1
2017 1
2017 1
2017 1
2022 5
2022 5

cat dat.txt | grep . | yarr

2018 3 3 3
2017 1 1 1 1 1
2022 5 5

cat dat.txt | grep . | yarr | fillretu

2018 3 3 3 _ _
2017 1 1 1 1 1
2022 5 5 _ _ _

cat dat.txt | grep . | yarr | fillretu -NaN 0

2018 3 3 3 0 0
2017 1 1 1 1 1
2022 5 5 0 0 0

tateyokoとのコンビネーション。

cat dat.txt | yarr | fillretu | tateyoko | keta

2018 2017 2022
   3    1    5
   3    1    5
   3    1    _
   _    1    _
   _    1    _

各行の行数を列挙

• Usage
  • man2 juni
  • juni [-z]
• Examples
  • cat a.txt | juni
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: juni

Output:

"a".."d" | juni

1 a
2 b
3 c
4 d

半角スペース区切りの標準入力から任意の列のみ抽出する。 すべての列は0で、最終列はNFで指定することもできる

1.2.3と指定すると、1列目の2文字目から3文字を切り出し 切り出し文字数が対象文字数よりも多い場合は切り取れる範囲のみ切り出し。

• Usage
  • man2 self
  • self <num> <num>...
• Examples
  • "1 2 3","4 5 6","7 8 9" | self 1 3
    • How to read : Select fields 1st and 3rd columns.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: self

Examples:

# select field 1 and 3
"1 2 3","4 5 6","7 8 9"

1 2 3
4 5 6
7 8 9

"1 2 3","4 5 6","7 8 9" | self 1 3

1 3
4 6
7 9

# select 2nd field and and
# cut out 2 characters from the 2nd character
"123 456 789","223 523 823"

123 456 789
223 523 823

"123 456 789","223 523 823" | self 2.2.2

56
23

# select entire line and add 2nd field,
# and cut out 2 characters from the 2nd character in the 2nd field
"123 456 789","223 523 823" | self 0 2.2.2

123 456 789 56
223 523 823 23

# select the 1st field from the leftmost field and
# select the 2nd field from the rightmost field(=NF)
"1 2 3 4 5","6 7 8 9 10"

1 2 3 4 5
6 7 8 9 10

"1 2 3 4 5","6 7 8 9 10" | self 1 NF-1

1 4
6 9

半角スペース区切りの標準入力から指定列のみ削除する 最終列をNFで指定することもできる

• Usage
  • man2 delf
  • delf <num> <num>...
• Examples
  • "1 2 3","4 5 6","7 8 9" | delf 1 3
    • How to read : Delete fields 1st and 3rd column.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: delf

Examples:

# delete field 1 and 2
"1 2 3", "4 5 6", "7 8 9"

1 2 3
4 5 6
7 8 9

"1 2 3", "4 5 6", "7 8 9" | delf 1 2

3
6
9

# delete field 1 and 2nd field from right
"1 2 3 4 5", "6 7 8 9 10"

1 2 3 4 5
6 7 8 9 10

"1 2 3 4 5", "6 7 8 9 10" | delf 1 NF-1

2 3 5
7 8 10

半角スペース区切りの標準入力から指定列の合計を算出（サムアップ）する。

sm2 <k1> <k2> <s1> <s2>と指定することで、 <k1>列から<k2>列をキーとして<s1>列から<s2>列までを合計する。

sm2 0 0 <s1> <s2>と指定すると全行サムアップ。 ファイルのキーの事前ソートが必要。 大文字小文字を区別しない。

• Usage
  • man2 sm2
• Params
  • sm2 [+count] <k1> <k2> <s1> <s2>
• Options
  • +count: Output the total number of rows in the leftmost column. 合計した行数を最左列に出力
• Examples
  • "A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 1 2 3 3
    • How to read : Using the value obtained by concatenating from 1st to 2nd columns as a key, sum-up for each column values from 3rd to 3rd columns.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: sm2

Examples:

# input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

# Sort by key column before connecting pipeline to sm2 command
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 1 2 3 3

A 1 20
B 1 10
C 1 10

# Result if you forget to sort
"A 1 10","B 1 10","A 1 10","C 1 10" | sm2 1 2 3 3

A 1 10
B 1 10
A 1 10
C 1 10

# input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

# +count option
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 +count 1 2 3 3

2 A 1 20
1 B 1 10
1 C 1 10

# input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

# calculator mode
"A 1 10","B 1 10","A 1 10","C 1 10" | sm2 0 0 2 2

4

# calc average with sm2 and lcalc command
## input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

## sum up
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 +count 1 2 3 3

2 A 1 20
1 B 1 10
1 C 1 10

## calc average
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 +count 1 2 3 3 | lcalc '$0;$NF/$1'

2 A 1 20 10
1 B 1 10 10
1 C 1 10 10

半角スペース区切りのロング型データをクロス集計する。入力データはヘッダなし、かつ、キーが事前に一意に集計されていること。

Input data must be preprocessed to ensure unique keys and no header line.

• Usage
  • man2 map2
  • map2 -n <n>[,<m>]
    • e.g. map2 -n 2,1 means: Cross tablate the 2 columns from the left as vertical-key, 1 column as horizontal-key, and the rest as value fields
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: map

Examples:

値列が1列のみの場合。

# Input data example1:
# (Case: vkey1, vkey2, hkey, value)
cat data.txt

location-A store-A target-A 1
location-A store-B target-B 2
location-A store-C target-C 3
location-B store-A target-A 4
location-B store-B target-B 5
location-B store-C target-C 6
location-C store-A target-A 7
location-C store-B target-B 8
location-C store-C target-C 9

cat data.txt | map2 -n 2,1 | keta

         *       * target-A target-B target-C
location-A store-A        1        0        0
location-A store-B        0        2        0
location-A store-C        0        0        3
location-B store-A        4        0        0
location-B store-B        0        5        0
location-B store-C        0        0        6
location-C store-A        7        0        0
location-C store-B        0        8        0
location-C store-C        0        0        9

cat data.txt | map2 -n 1,2 | keta

         *  store-A  store-B  store-C
         * target-A target-B target-C
location-A        1        2        3
location-B        4        5        6
location-C        7        8        9

値列が複数列の場合。

# Input data example2:
# (Case: vkey, hkey, value1, value2)
cat data.txt

loc-1 tar-1 1 10
loc-1 tar-2 2 20
loc-1 tar-3 3 30
loc-2 tar-1 4 40
loc-2 tar-2 5 50
loc-2 tar-3 6 60
loc-3 tar-1 7 70
loc-3 tar-2 8 80
loc-3 tar-3 9 90

# ("A".."Z" is given according to the number of value-columns)
cat data.txt | map2 -n 1,1 -ifs " " -ofs "`t"

*       *       tar-1   tar-2   tar-3
loc-1   A       1       2       3
loc-1   B       10      20      30
loc-2   A       4       5       6
loc-2   B       40      50      60
loc-3   A       7       8       9
loc-3   B       70      80      90

# -yarr switch
cat data.txt | map2 -n 1,1 -ifs " " -ofs "`t" -yarr

*       tar-1   tar-1   tar-2   tar-2   tar-3   tar-3
*       a       b       a       b       a       b
loc-1   1       10      2       20      3       30
loc-2   4       40      5       50      6       60
loc-3   7       70      8       80      9       90

半角スペース区切りの標準入力における列同士の計算。

• Usage
  • man2 lcalc
  • lcalc [-d] 'expr; expr;...'
• Options
  • exprはクオート内において;区切りで複数の計算式を指定可能
  • -d: calculator mode. 電卓(dentaku)モード
• Note
  • 計算列の指定
    • $1,$2,... : 列指定は$記号＋列数
    • $0 : 全列指定
    • $NF : 最終列のみこのように書くことができる。ただし$NF-1とは書けない点に注意。
  • 短縮形で使用できる関数
    • 丸め : round($1,num)
    • 平方根 : sqrt($1)
    • べき乗 : pow($1,2)
    • 絶対値 : abs($1)
    • 対数 : log($1)
    • 対数base=2 : log2($1)
    • 常用対数 : log10($1)
    • パイ : PI
• Examples
  • "8.3 70","8.6 65","8.8 63" | lcalc '$1+1;$2/10'
  • lcalc -d '1+1'
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: lcalc

Examples:

# input
"8.3 70","8.6 65","8.8 63"

8.3 70
8.6 65
8.8 63

# lcalc
"8.3 70","8.6 65","8.8 63" | lcalc '$1+1;$2/10'

9.3 7
9.6 6.5
9.8 6.3

# calculator mode does not require
# standard input (from pipline)
lcalc -d '1+1'

2

lcalc -d '1+sqrt(4)'

3

lcalc -d 'pi'

3.14159265358979

# 短縮形で使用できる関数以外の関数も使用できる
lcalc -d '[math]::Ceiling(1.1)'

2

# calc average with sm2 and lcalc command
## input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

## sum up
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 +count 1 2 3 3

2 A 1 20
1 B 1 10
1 C 1 10

## calc average
"A 1 10","B 1 10","A 1 10","C 1 10" | sort | sm2 +count 1 2 3 3 | lcalc '$0;$NF/$1'

2 A 1 20 10
1 B 1 10 10
1 C 1 10 10

Column-to-column calcurations with script block on space delimited stdin.

Auto-Skip empty row. Multiple expr with ";" in scriptblock. Built-in variables:

$1,$2,... : Column indexes starting with 1
$NF       : Rightmost column
$NR       : Row number of each records

• Usage
  • man2 lcalc2
  • lcalc2 {expr; expr;...} [-d "delim"] [-c|-Calculator]
• Options
  • -c|-Calculator: calculator mode
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: lcalc

Examples:

# Multiple expr using ";" in scriptblock
# data
"8.3 70","8.6 65","8.8 63"

8.3 70
8.6 65
8.8 63

# calc
"8.3 70","8.6 65","8.8 63" `
    | lcalc2 {$1+1;$2+10}

8.3 70 9.3 80
8.6 65 9.6 75
8.8 63 9.8 73

# Output only result
# input
"8.3 70","8.6 65","8.8 63"

8.3 70
8.6 65
8.8 63

# output 1
"8.3 70","8.6 65","8.8 63" `
    | lcalc2 {$1+1; $2+10} -OnlyOutputResult

9.3 80
9.6 75
9.8 73

# output 2
#   Put result on the left,
#   put original field on the right,
#   with -OnlyOutputResult and $0
"8.3 70","8.6 65","8.8 63" `
    | lcalc2 {$1+1; $2+10; $0} -OnlyOutputResult

9.3 80 8.3 70
9.6 75 8.6 65
9.8 73 8.8 63

# Get row number of record
1..5 | lcalc2 {$NR}

1 1
2 2
3 3
4 4
5 5

# Calculator mode
lcalc2 -Calculator {1+1}

2

# calculator mode does not require
# standard input (from pipline)
lcalc2 -c {1+[math]::sqrt(4)}

3

lcalc2 -c {[math]::pi}

3.14159265358979

lcalc2 -c {[math]::Ceiling(1.1)}

2

# Calculate average with sm2 and lcalc2 command
## input
"A 1 10","B 1 10","A 1 10","C 1 10"

A 1 10
B 1 10
A 1 10
C 1 10

## sum up
"A 1 10","B 1 10","A 1 10","C 1 10" `
    | sort `
    | sm2 +count 1 2 3 3

2 A 1 20
1 B 1 10
1 C 1 10

## calc average
"A 1 10","B 1 10","A 1 10","C 1 10" `
    | sort `
    | sm2 +count 1 2 3 3 `
    | lcalc2 {$NF/$1}

2 A 1 20 10
1 B 1 10 10
1 C 1 10 10

半角スペース区切りの標準入力に対する行指向のパターンマッチングプロセッサ。

pawk [-fs "delim"] [-Pattern { condition }] -Action { action }

pawk reads the input a line at a time, scans for pattern, and executes the associated action if pattern matched.

As a feature, pipeline processing can be applied only to specific columns for multiple column inputs.

# input line (csv: comma separated values)
$dat = "abc,def,ghi","jkl,mno,pqr","stu,vwz,012"

abc,def,ghi
jkl,mno,pqr
stu,vwz,012

# apply rev commnand only 2nd columns
PS> $dat | pawk -fs "," -Pattern {$1 -match "^j"} -Action { $2=$2|rev;$0 }

jkl,onm,pqr

# -Begin, -Process and -End block like AWK
PS> 1..10 | pawk -Begin { $sum=0 } -Action { $sum+=$1 } -End { $sum }

55

• Usage
  • man2 pawk
  • pawk [-fs "delim"] [-Pattern { condition }] -Action { action }
• Params
  • [[-a|-Action] <ScriptBlock>] ...action script
  • [-p|-Pattern <ScriptBlock>] ...pattern criteria
  • [-b|-Begin <ScriptBlock>] ...run before reading input
  • [-e|-End] <ScriptBlock> ...run after reading input
  • [-fs|-Delimiter <String>] ...input/output field separator
  • [-ifs|-InputDelimiter <String>] ...input field separator
  • [-ofs|-OutputDelimiter <String>] ...output field separator
    • If -ifs or -ofs are not specified,-fs delimiter will be used as both the input and output delimiter.
    • If -ifs and/or -ofs are specified together, -fs value will be overridden.
  • [-AllLine] ...output all input even if not match pattern (but action is only apply to matching rows)
  • [-SkipBlank] ...continue processing when empty row is detected
• Note
  • -Action { action }, -Pattern { criteria }, -Begin { action }, -End { action } options should be specified in the script block.
  • The column specification symbols are $1,$2,...,$NF. (The left most column number is 1 and counts up to the right.)
    • Specifying $0 means the entire row.
    • Note that it is not allowed to assign value to $0 ($0=$val is not allowed in script block).
  • Each column value is interpreted as System.Double if it looks like a number, and otherwise, as System.String.
    • Note that zero starting numbers are treated as system.string exceptionally.
    • Underscore _ can also be used as a numeric delimiter. (e.g. 123_456)
  • Built-in variables and options:
    • $NF : the last element of the current line. (This is one of the behaviors that is different from awk)
    • $NR : current row number
• Inspired by Unix/Linux Commands
  • Command: awk, gawk

Examples:

# sum from 1 to 10 and output the result
1..10 | pawk -Begin {$sum=0} -Action {$sum+=$1} -End {$sum}

55

# output all line using $0 in -Action script block
1..10 | pawk -Begin {$sum=0} -Action {$sum+=$1;$0} -End {"=====","sum: $sum"}

1
2
3
4
5
6
7
8
9
10
=====
sum: 55

# If both -Action {$0} and -AllLine switch are
# used at the same time, the outputs are duplicated.
1..3 | pawk -Begin {$sum=0} -Action {$sum+=$1;$0} -End {"=====","sum: $sum"} -AllLine

1
1
2
2
3
3
=====
sum: 6

# All of the following get all line output
1..3 | pawk -Begin {$sum=0} -Action {$sum+=$1} -End {"=====","sum: $sum"} -AllLine
1..3 | pawk -Begin {$sum=0} -Action {$sum+=$1;$0} -End {"=====","sum: $sum"}

1
2
3
=====
sum: 6

# Although -Action {$0} and -AllLine switch have different outputs, 
# there is no difference in that the action
# is executed only pattern-mathed rows.

## Case1: -Action {$0} 
1..3 | pawk -Begin {$sum=0} -Action {$sum+=$1 ; $0} -End {"=====","sum: $sum"} -Pattern {$1 % 2 -eq 1}

1
3
=====
sum: 4

## Case2: -AllLine switch. Total value is the same as above. (sum=4)
## (Action skipped not mathed rows)
1..3 | pawk -Begin {$sum=0} -Action {$sum+=$1} -End {"=====","sum: $sum"} -Pattern {$1 % 2 -eq 1} -AllLine

1
2
3
=====
sum: 4

# Note that if -AllLine switch is used,
# it duplicates the output if there is
# an output with -Action {action}

# notes on interpreting numbers and strings

# input data (zero padding numbers)
$dat = 1..5 | %{ "{0:d3}" -f $_ }
$dat

001
002
003
004
005

$dat | pawk -Pattern {$1 -eq 1}

match nothing.
because numbers starting with zero are considered strings.

# Cast [string]"001" to [int]"001"       
$dat | pawk -Pattern {[int]$1 -eq 1}
$dat

001   # matched!

# Match if you compare a zero-filled number as a string
$dat | pawk -Pattern {$1 -eq "001"}

001

# Inversion of the above criteria ( -eq to -ne )
$dat | pawk -Pattern {$1 -ne "001"}

002
003
004
005

# Zero-filled numbers are strings,
# so their sum with a number is a
# concatenation of strings.
$dat | pawk -Pattern {$1 -eq "001"} -Action {$1+1}

0011

# -AllLine switch outputs all lines that
#  do not match the pattern. However, 
# the action is executed only on lines that
# match the pattern
$dat | pawk -Pattern {$1 -eq "001"} -Action {$1=$1+1} -AllLine

0011
002
003
004
005

# Cast 1st column of zero-filled numbers to an integer
# and then takeing the numeric sum gives the expected behaviour.
$dat | pawk -Pattern {$1 -eq "001"} -Action {[int]$1+1}

2

# -AllLine switch
$dat | pawk -Pattern {$1 -eq "001"} -Action {$1=[int]$1+1} -AllLine

2
002
003
004
005

# Column specification using $0

$dat = "a b c 1","d e f 2","g h i 3"
$dat

a b c 1
d e f 2
g h i 3

$dat | pawk -Action {$0 + " zzz"}

a b c 1 zzz
d e f 2 zzz
g h i 3 zzz

# Replace 2nd column
$dat | pawk -Action {$2="zzz" ; $0}
$dat | pawk -Action {$2="zzz"} -AllLine
$dat

a zzz c 1
d zzz f 2
g zzz i 3

# Read csv data

$dat = "a b c 1","d e f 2","g h i 3" | %{ $_ -replace " ",","}
$dat

a,b,c,1
d,e,f,2
g,h,i,3

$dat | pawk -fs "," -Action {$2=$2*3 ; $0}

a,bbb,c,1
d,eee,f,2
g,hhh,i,3

# Convert csv to tsv
$dat | pawk -fs "," -Action {$0} -ofs "`t"

a       b       c       1
d       e       f       2
g       h       i       3

# Pattern match and execute Action

$dat = "a b c 1","d e f 2","g h i 3" | %{ $_ -replace " ",","}
$dat

a,b,c,1
d,e,f,2
g,h,i,3

# Pattern match
$dat | pawk -fs "," -Pattern {$NF -gt 1}
$dat

d,e,f,2
g,h,i,3

$dat | pawk -fs "," -Pattern {$NF -gt 1 -and $2 -match 'e'}

d,e,f,2

$dat | pawk -fs "," -Pattern {$NF -le 1}

a,b,c,1

# Pattern match and replace 1st field
$dat | pawk -fs "," -Pattern {$NF -gt 1} -Action {$1="aaa";$0}

aaa,e,f,2
aaa,h,i,3

# Pattern match and replace 1st field and output all rows,
# but -Action script is applied only pattern matched rows.
$dat | pawk -fs "," -Pattern {$NF -gt 1} -Action {$1="aaa"} -AllLine

a,b,c,1
aaa,e,f,2
aaa,h,i,3

# Handling zero padding numbers

$dat = "001,aaa,1","002,bbb,2","003,ccc,4","005,ddd,5"
$dat

001,aaa,1
002,bbb,2
003,ccc,4
005,ddd,5

# Zero padding numbers are not double but string
$dat | pawk -fs "," -Pattern {$1 -eq 2}

# not match

$dat | pawk -fs "," -Pattern {$1 -eq "002"}

002,bbb,2

# Cast as double
$dat | pawk -fs "," -Pattern {[int]$1 -eq 2}

002,bbb,2

# Use -begin -end example

$dat = "001,aaa,1","002,bbb,2","003,ccc,4","005,ddd,5"
$dat

001,aaa,1
002,bbb,2
003,ccc,4
005,ddd,5

# Add 3rd field values and output result
$dat | pawk -fs "," -Begin {$sum=0} -Action {$sum+=$3} -End {$sum}

12

$dat `
    | pawk `
        -fs "," `
        -Begin {$sum=0} `
        -Action {$sum+=[math]::Pow($3,2);$0+","+[math]::Pow($3,2)} `
        -End {$sum}

001,aaa,1,1
002,bbb,2,4
003,ccc,4,16
005,ddd,5,25
46

# As a feature, pipeline processing can be applied only to
# specific columns for multiple column inputs, like below.

# Input
$dat = "abc,def,ghi","jkl,mno,pqr","stu,vwz,012"
$dat

abc,def,ghi
jkl,mno,pqr
stu,vwz,012

# Apply rev commnand only 2nd columns
$dat | pawk -fs "," -Action {$2=$2|rev;$0}
$dat | pawk -fs "," -Action {$2=$2|rev} -AllLine

abc,fed,ghi # reverse 2nd column
jkl,onm,pqr # reverse 2nd column
stu,zwv,012 # reverse 2nd column

# Apply rev commnand only 2nd columns and only pattern matched rows
$dat | pawk -fs "," -Action {$2=$2|rev} -Pattern {$1 -match '^j'} -AllLine

abc,def,ghi  # not match
jkl,onm,pqr  # reverse 2nd column
stu,vwz,012  # not match

# Select column

# Input data
$dat = "abc,def,ghi","jkl,mno,pqr","stu,vwz,012"
$dat

abc,def,ghi
jkl,mno,pqr
stu,vwz,012

# The following is probably not expected behavior
$dat | pawk -fs "," -Action {$1,$2}

abc
def
jkl
mno
stu
vwz

# Use -join operator
$dat | pawk -fs "," -Action {$1,$2 -join ","}

abc,def
jkl,mno
stu,vwz

# Use @() to specify an array
$dat | pawk -fs "," -Action {@($1,$2) -join ","}

abc,def
jkl,mno
stu,vwz

# Equivalent alternate solution.Using the fact that input rows　are
# separated by delimiters and stored in a variable of array named "$self".
# note that the index is zero start in this case.
$dat | pawk -fs "," -Action {$self[0..1] -join ","}

abc,def
jkl,mno
stu,vwz

# Various column selections

# Input data
$dat = "abc,def,ghi","jkl,mno,pqr","stu,vwz,012"
$dat

abc,def,ghi
jkl,mno,pqr
stu,vwz,012

# Duplicate columns
$dat | pawk -fs "," -Action {$1,$1,$1,$1 -join ","}

abc,abc,abc,abc
jkl,jkl,jkl,jkl
stu,stu,stu,stu

# Select max Number of field(column)
$dat | pawk -fs "," -Action {$NF}

ghi
pqr
012

# Select max Number -1 of field(column)
$dat | pawk -fs "," -Action {$self[-2]}

def
mno
vwz

$dat | pawk -fs "," -Action {$self[$self.count-2]}

def
mno
vwz

# Select n to last columns
$dat | pawk -fs "," -Action {$self[1..($self.count-1)] -join ","}

def,ghi
mno,pqr
vwz,012

# Manipulation of specific columns

# Input
$dat = "001,aaa,2022-01-01","002,bbb,2022-01-02","003,ccc,2022-01-03","005,ddd,2022-01-04"
$dat

001,aaa,2022-01-01
002,bbb,2022-01-02
003,ccc,2022-01-03
005,ddd,2022-01-04

# Add days -10 to 3rd column (cast [datetime])
$dat | pawk -fs "," -Action {$3=(Get-Date $3).AddDays(-10).ToString('yyyy-MM-dd')} -AllLine

001,aaa,2021-12-22
002,bbb,2021-12-23
003,ccc,2021-12-24
005,ddd,2021-12-25

# Manipulation of specific columns using pipe

# Input
$dat = "001,aaa,20220101","002,bbb,20220102","003,ccc,20220103","005,ddd,20220104"
$dat

001,aaa,20220101
002,bbb,20220102
003,ccc,20220103
005,ddd,20220104

# Format date for 3rd column.
# (Column symbols ($1,$2,...) in single quotes are escaped.
# so that $1,$2,... symbols in the ForEach-Object command has the expected behavior.)
$dat | pawk -fs "," -Action {$3=$3|ForEach-Object{$_ -replace '([0-9]{4})([0-9]{2})([0-9]{2})','$1-$2-$3'}; $0}

001,aaa,2022-01-01
002,bbb,2022-01-02
003,ccc,2022-01-03
005,ddd,2022-01-04

# Equivalent alternative solution using [datetime]::ParseExact
$dat | pawk -fs "," -Action {$3=([datetime]::ParseExact($3,"yyyyMMdd",$null)).ToString('yyyy-MM-dd'); $0}

001,aaa,2022-01-01
002,bbb,2022-01-02
003,ccc,2022-01-03
005,ddd,2022-01-04

# Usage of build-in variables ($NF, $NR)

# Input
$dat = "1,aaa,111","2,bbb,222","3,ccc,333"
$dat

1,aaa,111
2,bbb,222
3,ccc,333

$dat | pawk -fs "," -Pattern {$NF -ge 222}

2,bbb,222
3,ccc,333

$dat | pawk -fs "," -Pattern {$NR -ge 1}

1,aaa,111
2,bbb,222
3,ccc,333

# when using the column specification operator "$1" in double uotes,
# add the subexpression operator like "$($1)"

"a b c 1","d e f 2","g h i 3" | pwk -Action {"id: $($4), tag: ""$($1)"""}

id: 1, tag: "a"
id: 2, tag: "d"
id: 3, tag: "g"

# Re-arrange 2-4 characters of an undelimited strings.

# Input
"aiueo","12345","abcde"

aiueo
12345
abcde

# Re-arrange 2-4 chars of each row.
"aiueo","12345","abcde" | pawk -fs '' -Action {$self[0,3,2,1,4] -join ''}

aeuio
14325
adcbe

# Equivalent to the above
"aiueo","12345","abcde" | pawk -fs '' -Action {@($1,$4,$3,$2,$5) -join ''}

aeuio
14325
adcbe

# If an empty string is specified as the delimiter,
# the first and last elements are dropped from the array.

半角スペース区切り入力の列数を出力。

• 同じ列数の場合は、重複を削除して列数を出力
• 列数が変化するごとに列数を出力する
• 空行はゼロを出力

すべての行の列数が同じか否かを検知するタスクなどで使う。

• Usage
  • man2 retu
  • retu [-c]
• Examples
  • cat a.txt | retu
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: retu

Output:

# If all column numbers are equal,
# duplicates are removed and
# only one column numbers is output.
"a".."z" | retu

1

# Output for each change in the
# number of columns.
"a a","b b b","c c c","d d"

a a
b b b
c c c
d d

"a a","b b b","c c c","d d" | retu

2
3
2

# With the "-c" switch, all rows
# are output with column numbers.
"a a","b b b","c c c","d d" | retu -c

2 a a
3 b b b
3 c c c
2 d d

半角スペース区切り入力のキー数のカウント。 <k1>列から<k2>列をキーフィールドとみなし、 重複するキーフィールド数をカウントする。 -cスイッチで、カウントアップしながら全行出力。

• Usage
  • man2 count
  • count [-c] <k1> <k2>
• Examples
  • cat a.txt | count [-c] <k1> <k2>
    • How to read : Count the number of duplicate records using the value obtained by concatenating from 1st to 2nd columns as a key.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: count

Input:

cat a.txt

01 埼玉県 01 さいたま市 100
01 埼玉県 02 川越市 100
01 埼玉県 03 熊谷市 100
02 東京都 04 新宿区 100
02 東京都 05 **区 100
02 東京都 06 港区 100
02 東京都 07 千代田区 100
02 東京都 08 八王子市 100
02 東京都 09 立川市 100
03 千葉県 10 千葉市 100
03 千葉県 11 市川市 100
03 千葉県 12 柏市 100
04 神奈川県 13 横浜市 100
04 神奈川県 14 川崎市 100
04 神奈川県 15 厚木市 100
04 神奈川県 16 小田原市 100

Output:

cat a.txt | grep . | sort | count 1 2

3 01 埼玉県
6 02 東京都
3 03 千葉県
4 04 神奈川県

cat a.txt | grep . | sort | count -c 1 2

1 01 埼玉県 01 さいたま市 100
2 01 埼玉県 02 川越市 100
3 01 埼玉県 03 熊谷市 100
1 02 東京都 04 新宿区 100
2 02 東京都 05 **区 100
3 02 東京都 06 港区 100
4 02 東京都 07 千代田区 100
5 02 東京都 08 八王子市 100
6 02 東京都 09 立川市 100
1 03 千葉県 10 千葉市 100
2 03 千葉県 11 市川市 100
3 03 千葉県 12 柏市 100
1 04 神奈川県 13 横浜市 100
2 04 神奈川県 14 川崎市 100
3 04 神奈川県 15 厚木市 100
4 04 神奈川県 16 小田原市 100

半角スペース区切り入力から、 同一キーの最初行のデータを出力。 大文字小文字を区別しない。

• Usage
  • man2 getfirst
  • getfirst <k1> <k2>
    • <k1>列から<k2>列をキーとみなす
• Examples
  • cat a.txt | getfirst 1 2
    • How to read : Get only the first row with the same key with the value concatenated from 1st to 2nd column as the key.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: getfirst

Output:

cat a.txt

01 埼玉県 01 さいたま市 100
01 埼玉県 02 川越市 100
01 埼玉県 03 熊谷市 100
02 東京都 04 新宿区 100
02 東京都 05 **区 100
02 東京都 06 港区 100
02 東京都 07 千代田区 100
02 東京都 08 八王子市 100
02 東京都 09 立川市 100
03 千葉県 10 千葉市 100
03 千葉県 11 市川市 100
03 千葉県 12 柏市 100
04 神奈川県 13 横浜市 100
04 神奈川県 14 川崎市 100
04 神奈川県 15 厚木市 100
04 神奈川県 16 小田原市 100

cat a.txt | getfirst 1 2

01 埼玉県 01 さいたま市 100
02 東京都 04 新宿区 100
03 千葉県 10 千葉市 100
04 神奈川県 13 横浜市 100

半角スペース区切り入力から、 同一キーの最下行のデータを出力。 大文字小文字を区別しない。

• Usage
  • man2 getlast
  • getlast <k1> <k2>
    • <k1>列から<k2>列をキーとみなす
• Examples
  • cat a.txt | getlast 1 2
    • How to read : Get only the last row of the same key with the value concatenated from 1st to 2nd column as the key.
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: getlast

Output:

cat a.txt

01 埼玉県 01 さいたま市 100
01 埼玉県 02 川越市 100
01 埼玉県 03 熊谷市 100
02 東京都 04 新宿区 100
02 東京都 05 **区 100
02 東京都 06 港区 100
02 東京都 07 千代田区 100
02 東京都 08 八王子市 100
02 東京都 09 立川市 100
03 千葉県 10 千葉市 100
03 千葉県 11 市川市 100
03 千葉県 12 柏市 100
04 神奈川県 13 横浜市 100
04 神奈川県 14 川崎市 100
04 神奈川県 15 厚木市 100
04 神奈川県 16 小田原市 100

cat a.txt | getlast 1 2

01 埼玉県 03 熊谷市 100
02 東京都 09 立川市 100
03 千葉県 12 柏市 100
04 神奈川県 16 小田原市 100

縦型（ロング型）の半角スペース区切りレコードを、 指定列をキーに横型（ワイド型）に変換する。

• Usage
  • man2 yarr
  • cat a.txt | yarr [-n|-num <int>]
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: yarr

Input(long type data):

cat a.txt

2018 1
2018 2 9
2018 3
2017 1
2017 2
2017 3
2017 4
2017 5 6
2022 1
2022 2

Output(wide type data):

# num=1で左から1列目をkeyとしてロング型をワイド型に変換。
cat a.txt | grep . | yarr -n 1

2018 1 2 9 3
2017 1 2 3 4 5 6
2022 1 2

※ grep .で空行をスキップ（＝1文字以上の行のみヒット）

Equivalent in PowerShell

cat a.txt | grep . | ForEach-Object -begin{$h=@{}} -process{$a=$_ -split " ", 2; $h[$($a[0])]+=" $($a[1])"} -end {foreach($k in $h.keys){$k+$h[$k]}}

2022 1 2
2018 1 2 9 3
2017 1 2 3 4 5 6

横長（ワイド型）の半角スペース区切りレコードを、 指定列をキーに縦長（ロング型）に変換する。

• Usage
  • man2 tarr
  • cat a.txt | tarr [-n|-num <int>]
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: tarr

Input(wide type data):

cat a.txt

2018 1 2 3
2017 1 2 3 4
2022 1 2

Output(long type data):

# num=1で左から1列目をkeyとしてワイド型をロング型に変換。
cat a.txt | grep . | tarr -n 1

2018 1
2018 2
2018 3
2017 1
2017 2
2017 3
2017 4
2022 1
2022 2

※ grep .で空行をスキップ（＝1文字以上の行のみヒット）

半角スペース区切り文字列を任意列数となるように整える。 -ifsで入力テキストの区切り文字を、
-ofsで出力テキストの区切り文字をそれぞれ指定できる。
それぞれ空文字も指定可能。（-ifs "", -ofs ""）

• Usage
  • man2 flat
• Examples
  • 1..9 | flat
  • 1..9 | flat 4
  • "aiueo" | flat 3 -ifs "" -ofs ""
• Inspired by greymd/egzact: Generate flexible patterns on the shell - GitHub
  • License: The MIT License (MIT): Copyright (c) 2016 Yasuhiro, Yamada
  • Command: flat

Examples

1..9 | flat

1 2 3 4 5 6 7 8 9

1..9 | flat 4

1 2 3 4
5 6 7 8
9

"aiueo" | flat 3 -ifs "" -ofs ""

aiu
eo

半角スペース区切り文字列を任意の書式でくるむ。 -ifsで入力テキストの区切り文字を、 -ofsで出力テキストの区切り文字をそれぞれ指定できる。 それぞれ空文字も指定可能。（-ifs "", -ofs ""）

• Usage
  • man2 wrap
  • wrap [[--f|Format] <String>] [-p|-Placeholder <String>] [-fs|-Delimiter <String>]
• Examples
  • "A B C D" | wrap '[*]'
  • "A B C D" | wrap '[?]' -Placeholder '?'
  • "A B C D" | wrap '[*]' -fs "_"
  • "ABCD" | wrap '[*]' -fs ''
• Inspired by greymd/egzact: Generate flexible patterns on the shell - GitHub
  • License: The MIT License (MIT): Copyright (c) 2016 Yasuhiro, Yamada
  • Command: wrap

Examples

"A B C D" | wrap '[*]'

[A] [B] [C] [D]

"A B C D" | wrap '[?]' -Placeholder '?'

[A] [B] [C] [D]

"A B C D" | wrap '[*]' -fs "_"

[A B C D]

"ABCD" | wrap '[*]' -fs ""

[A][B][C][D]

"A B C D","E F G H" | wrap '<td>*</td>' | addt '<table>','<tr>' | addb '</tr>','</table>'

<table>
<tr>
<td>A</td> <td>B</td> <td>C</td> <td>D</td>
<td>E</td> <td>F</td> <td>G</td> <td>H</td>
</tr>
</table>

改行を挿入する。 Add-LineBreakは、文字列中に`r`nを見つけるとそこに改行を挿入する。 Add-LineBreakEndOfFileは、入力の最後に改行を1行挿入する。 Get-Content -Delimiterと似た挙動である。

• Usage
  • man2 Add-LineBreak
  • man2 Add-LineBreakEndOfFile

Remove empty lines from the beginning and end of line.

• Usage
  • man Trim-EmptyLine
  • man etrim

EXAMPLE

# Remove empty lines from the beginning and end of line
## input
[string[]] $dat = @("",""),@("a".."c"),@("",""),@("d".."f"),@("","") | % { $_ }
$dat

  <- Empty line
  <- Empty line
a
b
c
  <- Empty line
  <- Empty line
d
e
f
  <- Empty line
  <- Empty line

## output
$dat | Trim-EmptyLine

a
b
c
  <- Empty line
  <- Empty line
d
e
f

-Uniq switch

# The -Uniq switch merges consecutive blank lines in the body into a single line
## input
[string[]] $dat = @("a".."c"),@("",""),@("d".."f") | % { $_ }
$dat

a
b
c
  <- Consecutive blank line
  <- Consecutive blank line
d
e
f

## output
$dat | Trim-EmptyLine -u

a
b
c
  <- A single blank line
d
e
f

入力の上下左右に文字列を挿入する。

• addt: add-top: 入力行の先頭行の上に文字列を追加
• addb: add-bottom: 入力行の末尾行の下文字列を追加
• addr: add-right: 各入力行の最右列に文字列を追加
• addl: add-left: 各入力行の最左列に文字列を追加

addtとaddbは、ヘッダやフッタを追加する。
addtとaddbはカンマ区切りで複数行指定も可能。
addrとaddlは、左や右に列を追加するのに便利。

• Usage
  • man2 addb
  • man2 addl
  • man2 addr
  • man2 addt
• Examples
  • "A B C D" | addt '<table>','' | addb '','</table>'
  • "B C D" | addr " E" | addl "A "
• Inspired by greymd/egzact: Generate flexible patterns on the shell - GitHub
  • License: The MIT License (MIT): Copyright (c) 2016 Yasuhiro, Yamada
  • Command: addt, addb, addr, addl

入力行を任意列数ごとに畳み込み演算（convolution）のように折り返す。 文章のN-gramを求める。

Convolution operation with the specified number of columns for each input record. Instead of just wrapping lines, each lines shifts the previous line to the left by one element.

Default field separator is space.

• Usage
  • man2 conv
  • conv <num> [-fs DELIMITER] [-r] [-f]
• Option
  • -r : output number of records(rows) in the left column
  • -f : output number of fields(columns) in the left column
  • -fs : input/output field separator
  • -ifs : input field separator
  • -ofs : output field separator
  • -nfs : field separator for only "-r" and "-f" option
• Examples
  • @(1..10) -join " " | conv 3
  • "にわにはにわにわとりがいる" | conv -fs '' 2
    • N-gram of text
• Inspired by greymd/egzact: Generate flexible patterns on the shell - GitHub
  • License: The MIT License (MIT): Copyright (c) 2016 Yasuhiro, Yamada
  • Command: conv

Examples:

convolution operation

@(1..10) -join " " | conv 3
# or 1..10 | flat | conv 3

1 2 3
2 3 4
3 4 5
4 5 6
5 6 7
6 7 8
7 8 9
8 9 10

N-gram of text

Write-Output "にわにはにわにわとりがいる" | conv -fs '' 2

にわ
わに
には
はに
にわ
わに
にわ
わと
とり
りが
がい
いる

# output -n (NumOfRecord), and -f (NumOfField)
Write-Output "にわにはにわにわとりがいる" | conv -fs '' 2 -r -f

1 1 にわ
1 2 わに
1 3 には
1 4 はに
1 5 にわ
1 6 わに
1 7 にわ
1 8 わと
1 9 とり
1 10 りが
1 11 がい
1 12 いる

半角スペース区切り入力の桁そろえ。 端末上で半角スペース区切り入力を確認するときに見やすい。 マルチバイト文字対応。

• Usage
  • man2 keta
• Examples
  • "aaa bbb ccc","dddddd eeee ffff" | keta
    • デフォルトで右揃え
  • "aaa bbb ccc","dddddd eeee ffff" | keta -l
    • -lスイッチで左揃え
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: keta

Examples:

# input
"aaa bbb ccc","dddddd eeee ffff"

aaa bbb ccc
dddddd eeee ffff

# right padding (default)
"aaa bbb ccc","dddddd eeee ffff" | keta

   aaa  bbb  ccc
dddddd eeee ffff

# left padding (-l switch)
"aaa bbb ccc","dddddd eeee ffff" | keta -l

aaa    bbb  ccc 
dddddd eeee ffff

入力文字列の行数をカウント。 (1..20 | Measure-Object).Countと同じ効果。

• Usage
  • man2 gyo
• Examples
  • 1..20 | gyo
  • gyo *.*
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: gyo

Examples:

# simple usage
1..20 | gyo

20

# output row numbers for each text files
# in current directory
gyo *.*

11 .textlintrc
16 a.md
76 b.md
72 c.md
29 a.json
29 t.md
118 table2col.md
2 uri-list.txt

全角文字を半角に変換。For japanese locale on windows。 [-k|-Kana]スイッチで全角カタカナは変換しない （半角カナを全角に変換するわけではない）

• Usage
  • man2 han
• Params
  • [-k|-Kana] - Do not convert full-width(zenkaku) KANA
• Examples
  • "input" | han | zen -k
    • → 英数字記号を半角に、カナのみ全角に変換
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: han
• Thanks
  • 【PowerShell】全角英数字記号を半角に変換する方法 - buralog
• Learn
  • Regex.Replace Method (.NET 7) - Replace(String, String, MatchEvaluator)
    • Regex.Replace Method (System.Text.RegularExpressions)
    • Regex.Replace メソッド (System.Text.RegularExpressions)

Examples:

"パピプペポ１２３４５６７８９＝Ａ" | han | zen -k

パピプペポ0123456789=A

説明
==============
入力から英数字記号を半角に、カナのみ全角に変換

"パピプペポ０１２３４５６７８９＝Ａ" | han

ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ0123456789=A

"パピプペポ０１２３４５６７８９＝Ａ" | han -k

パピプペポ0123456789=A

"パピプペポ０１２３４５６７８９＝Ａ" | han -k | zen

パピプペポ０１２３４５６７８９＝Ａ

"パピプペポ０１２３４５６７８９＝Ａ" | han | zen

パピプペポ０１２３４５６７８９＝Ａ

"パピプペポ０１２３４５６７８９＝Ａ" | han | zen -k

パピプペポ0123456789=A

全角文字を半角に変換する。For japanese locale on windows。 [-k|-Kana]スイッチで半角カタカナのみ全角カタカナに変換

• Usage
  • man2 zen
• Options
  • [-k|-Kana] - Convert only half-width kana to full-width kana
• Examples
  • "input" | han | zen -k
    • → 英数字記号を半角に、カナのみ全角に変換
• Inspired by Open-usp-Tukubai - GitHub
  • License: The MIT License (MIT): Copyright (C) 2011-2022 Universal Shell Programming Laboratory
  • Command: zen
• Thanks
  • 【PowerShell】全角英数字記号を半角に変換する方法 - buralog
• Learn
  • Regex.Replace Method (.NET 7) - Replace(String, String, MatchEvaluator)
    • Regex.Replace Method (System.Text.RegularExpressions)
    • Regex.Replace メソッド (System.Text.RegularExpressions)

Examples:

"パピプペポ０１２３４５６７８９＝Ａ" | han | zen -k

パピプペポ0123456789=A

説明
==============
入力から英数字記号を半角に、カナのみ全角に変換

"パピプペポ０１２３４５６７８９＝Ａ" | han

ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ0123456789=A

"パピプペポ０１２３４５６７８９＝Ａ" | han -k

パピプペポ0123456789=A

"パピプペポ０１２３４５６７８９＝Ａ" | han -k | zen

パピプペポ０１２３４５６７８９＝Ａ

"パピプペポ０１２３４５６７８９＝Ａ" | han | zen

パピプペポ０１２３４５６７８９＝Ａ

"パピプペポ０１２３４５６７８９＝Ａ" | han | zen -k

パピプペポ0123456789=A

文字列の変換。

• Usage
  • man2 vbStrConv
• Params
  • vbStrConv
    • [-Lowercase]
    • [-Uppercase]
    • [-ProperCase]
    • [-Wide]
    • [-Narrow]
    • [-Hiragana]
    • [-Katakana]
• Examples
  • "input" | vbStrConv -ProperCase

Examples:

"input" | vbStrConv -ProperCase

Input

シンプルな日付の入力（e.g. 1/23）に年を付与して出力。 筆者は「年数」をしばしばミスタイプしてしまうためその予防として作成。

Add this/next/last year to month/day input. To prevent mistyping the number of year.

Output to both clipboard and stdout by default. with -stdout switch, output only to stdout.

# Result of execution on a certain day in **2023**.

thisyear 2/28

2023-02-28

nextyear 2/29

2024-02-29

lastyear 2/28

2022-02-28

• Usage
  • man2 thisyear
  • man2 nextyear
  • man2 lastyear
  • man2 *year
• Syntax
  • thisyear / nextyear / lastyear
    • [-Date] <String[]>
    • [-RedirectTo <String>]
    • [-Prefix <String>]
    • [-Suffix <String>]
    • [-Format <String>]
    • [-FormatJP]
    • [-FormatJPZeroPadding]
    • [-Slash]
    • [-GetDateTimeFormat]
    • [-Year2]
    • [-DayOfWeek]
    • [-DayOfWeekWithRound]
    • [-AddDays <Int32>]
    • [-AddMonths <Int32>]
    • [-AddYears <Int32>]
    • [-Duplicate <Int32>]
    • [-NoSeparator]
    • [-EndOfMonth]
• Examples
  • thisyear 1/23
  • nextyear 1/23
  • lastyear 1/23

Examples:

# Result of execution on a certain day in **2023**.

thisyear 2/28 -s "_this_year"

2023-02-28_this_year

nextyear 2/29 -s "_next_year"

2024-02-29_next_year

lastyear 2/28 -s "_last_year"

2022-02-28_last_year

任意列の値を整形する。

"1111 2222 3333" | fval '#,0.0' 2, 3
1111 2,222.0 3,333.0

• Usage
  • man2 fval
• Params
  • fval
    • [-f|-Format] <String>
    • [[-n|-Num] <Int32[]>]
    • [-SkipHeader]

Example:

"1111 2222 3333" | fval '#,0.0' 2, 3

1111 2,222.0 3,333.0

Returns only the values that match the specified key from text data stored in space-delimited (space or tab) key-value format.

Whitespaces before and after keys and values are trimmed and ignored.

• Usage
  • man2 getvalkey
    • cat file | getvalkey [[-k|-Key] <Regex[]>]
    • getvalkey [[-k|-Key] <Regex[]>] [[-p|-Path] <String[]>]
• Params
  • cat file | getvalkey [[-k|-Key] <Regex[]>]
    • [-d|-Delimiter <String>]
    • [-f|-File]
    • [-v|-NotMatch]
    • [-s|-SimpleMatch]
    • [-CaseSensitive]
    • [-r|-Recurse]
    • [-Split <n>]
    • [-Get <n>]

Example:

cat keyval-dict.txt

key value
hog val hoge
fug val fuga

output values that match hog

cat keyval-dict.txt | getvalkey hog

val hoge

output all values

cat keyval-dict.txt | getvalkey .

value
val hoge
val fuga

output with key (using -Get option)

cat keyval-dict.txt | getvalkey hog -Get 0,1

hog val hoge

半角スペース区切りデータにpercentileまたは四分位数（quartile）を適用してランク付け。

percentileとsummaryはどちらも「ヘッダありスペース区切りデータの縦一列ぶんの要約統計量を算出する」点は同じ。 percentileはCategory列を指定するとCategoryごとに要約統計量を計算できる。

Calculate and ranking with percentile and quartiles on space-delimited data with/without headers.

• Usage
  • man2 percentile
• Params
  • percentile
    • [[-Val|-v] <Int32>]
    • [[-Key|-k] <Int32[]>]
    • [-Delimiter|-fs <delim>]
    • [-Rank|-Level5]
    • [-NoHeader]
    • [-Cast <String>]

Examples:

cat iris.csv `
    | percentile -v 1 -k 5 -fs "," `
    | Format-Table

field        key        count    sum mean stdev  min Qt25 Qt50 Qt75
-----        ---        -----    --- ---- -----  --- ---- ---- ----
sepal_length setosa        50 250.30 5.01  0.35 4.30 4.80 5.00 5.20
sepal_length versicolor    50 296.80 5.94  0.52 4.90 5.60 5.90 6.30
sepal_length virginica     50 329.40 6.59  0.64 4.90 6.20 6.50 7.00

Input

## Input
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } }

a 1
a 2
a 3
a 4
a 5
b 1
b 2
b 3
b 4
b 5
c 1
c 2
c 3
c 4
c 5
d 1
d 2
d 3
d 4
d 5

Output

## calc summary of 2nd field
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | percentile 2 -NoHeader

field : F2
count : 20
sum   : 60
mean  : 3
stdev : 1.45095250022002
min   : 1
Qt25  : 2
Qt50  : 3
Qt75  : 4
max   : 5
IQR   : 2
HiIQR : 7
LoIQR : -1

## same as below (calc rightmost field by default)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | percentile -NoHeader

## percentile 2 -k 1 :
##  means summary 2nd field using 1st field as key
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | percentile 2 -k 1 -NoHeader `
    | Format-Table

field key count   sum mean stdev  min Qt25 Qt50 Qt75
----- --- -----   --- ---- -----  --- ---- ---- ----
F2    a       5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F2    b       5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F2    c       5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F2    d       5 15.00 3.00  1.58 1.00 1.50 3.00 4.50

## -k 1,2 means fields from 1st to 2nd are considered keys
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $s $_" } } `
    | percentile 3 -k 1,2 -NoHeader `
    | Format-Table

field key count   sum mean stdev  min Qt25 Qt50 Qt75
----- --- -----   --- ---- -----  --- ---- ---- ----
F3    a a     5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F3    b b     5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F3    c c     5 15.00 3.00  1.58 1.00 1.50 3.00 4.50
F3    d d     5 15.00 3.00  1.58 1.00 1.50 3.00 4.50

## -Rank means ranking with quartile
##   add cumulative-ratio and quartile-labels
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | percentile 2 -Rank -NoHeader `
    | keta

F1 F2 percentile label
 a  1     0.0167   Qt1
 b  1     0.0333   Qt1
 c  1     0.0500   Qt1
 d  1     0.0667   Qt1
 a  2     0.1000   Qt1
 b  2     0.1333   Qt1
 c  2     0.1667   Qt1
 d  2     0.2000   Qt1
 a  3     0.2500   Qt2
 d  3     0.3000   Qt2
 b  3     0.3500   Qt2
 c  3     0.4000   Qt2
 a  4     0.4667   Qt3
 b  4     0.5333   Qt3
 d  4     0.6000   Qt3
 c  4     0.6667   Qt3
 b  5     0.7500   Qt4
 a  5     0.8333   Qt4
 c  5     0.9167   Qt4
 d  5     1.0000   Qt4

## -Level5 means ranking by 20% cumurative ratio
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | percentile 2 -Level5 -Rank -NoHeader `
    | keta

F1 F2 percentile label
 a  1     0.0167     E
 b  1     0.0333     E
 c  1     0.0500     E
 d  1     0.0667     E
 a  2     0.1000     E
 b  2     0.1333     E
 c  2     0.1667     E
 d  2     0.2000     D
 a  3     0.2500     D
 d  3     0.3000     D
 b  3     0.3500     D
 c  3     0.4000     C
 a  4     0.4667     C
 b  4     0.5333     C
 d  4     0.6000     B
 c  4     0.6667     B
 b  5     0.7500     B
 a  5     0.8333     A
 c  5     0.9167     A
 d  5     1.0000     A

Another example:

cat iris.csv `
    | percentile -fs "," 1 `
    | Format-Table

field        count    sum mean stdev  min Qt25 Qt50 Qt75  max
-----        -----    --- ---- -----  --- ---- ---- ----  ---
sepal_length   150 876.50 5.84  0.83 4.30 5.10 5.80 6.40 7.90

cat iris.csv `
    | percentile -fs "," 1,2,3,4 `
    | Format-Table

field        count    sum mean stdev  min Qt25 Qt50 Qt75  max
-----        -----    --- ---- -----  --- ---- ---- ----  ---
sepal_length   150 876.50 5.84  0.83 4.30 5.10 5.80 6.40 7.90
sepal_width    150 458.60 3.06  0.44 2.00 2.80 3.00 3.35 4.40
petal_length   150 563.70 3.76  1.77 1.00 1.55 4.35 5.10 6.90
petal_width    150 179.90 1.20  0.76 0.10 0.30 1.30 1.80 2.50

cat iris.csv `
    | percentile -fs "," 1,2,3,4 -k 5 `
    | Format-Table

field        key        count    sum mean stdev  min Qt25 Qt50 Qt75
-----        ---        -----    --- ---- -----  --- ---- ---- ----
sepal_length setosa        50 250.30 5.01  0.35 4.30 4.80 5.00 5.20
sepal_length versicolor    50 296.80 5.94  0.52 4.90 5.60 5.90 6.30
sepal_length virginica     50 329.40 6.59  0.64 4.90 6.20 6.50 7.00
sepal_width  setosa        50 171.40 3.43  0.38 2.30 3.15 3.40 3.70
sepal_width  versicolor    50 138.50 2.77  0.31 2.00 2.50 2.80 3.00
sepal_width  virginica     50 148.70 2.97  0.32 2.20 2.80 3.00 3.20
petal_length setosa        50  73.10 1.46  0.17 1.00 1.40 1.50 1.60
petal_length versicolor    50 213.00 4.26  0.47 3.00 4.00 4.35 4.60
petal_length virginica     50 277.60 5.55  0.55 4.50 5.10 5.55 5.90
petal_width  setosa        50  12.30 0.25  0.11 0.10 0.20 0.20 0.30
petal_width  versicolor    50  66.30 1.33  0.20 1.00 1.20 1.30 1.50
petal_width  virginica     50 101.30 2.03  0.27 1.40 1.80 2.00 2.30

Missing/Empty value detection/removal/replacement examples

# Empty value control (empty record detection/removal/replacement)

# input (include empty value field)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s,$_" } } `
    | %{ $_ -replace ',5$',',' }

a,1
a,2
a,3
a,4
a,
b,1
b,2
b,3
b,4
b,
c,1
c,2
c,3
c,4
c,
d,1
d,2
d,3
d,4
d,

# detect empty line and stop processing (use -isEmpty option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s,$_" } } `
    | %{ $_ -replace ',5$',',' } `
    | percentile -fs "," -NoHeader -isEmpty

percentile: Detect "Empty" : a,

# fill empty values with "NaN" (use -FillNaN option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s,$_" } } `
    | %{ $_ -replace ',5$',',' } `
    | percentile -fs "," -NoHeader -FillNaN

field : F2
count : 20
sum   : NaN
mean  : NaN
stdev : NaN
min   : NaN
Qt25  : 1
Qt50  : 2
Qt75  : 3
max   : 4

# NaN control (Missing value detection/removal/replacement)

"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' }

a 1
a 2
a 3
a 4
a NaN
b 1
b 2
b 3
b 4
b NaN
c 1
c 2
c 3
c 4
c NaN
d 1
d 2
d 3
d 4
d NaN

"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' } `
    | percentile 2 -NoHeader

field : F2
count : 20
sum   : NaN
mean  : NaN
stdev : NaN
min   : NaN
Qt25  : 1
Qt50  : 2
Qt75  : 3
max   : 4

# Output "NaN" information (use -Detail option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' } `
    | percentile 2 -NoHeader -Detail

field      : F2
count      : 20
sum        : NaN
mean       : NaN
stdev      : NaN
min        : NaN
Qt25       : 1
Qt50       : 2
Qt75       : 3
max        : 4
IQR        : 2
HiIQR      : 6
LoIQR      : -2
NaN        : 4
DropNaN    : 0
FillNaN    : 0
ReplaceNaN : 0

# Detect "NaN" and stop processing (use -isNaN option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' } `
    | percentile 2 -NoHeader -isNaN

percentile: Detect "NaN" : a NaN

# Drop "NaN" (use -DropNaN option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' } `
    | percentile 2 -NoHeader -DropNaN -Detail

field      : F2
count      : 16
sum        : 40
mean       : 2.5
stdev      : 1.15470053837925
min        : 1
Qt25       : 1
Qt50       : 2.5
Qt75       : 4
max        : 4
IQR        : 3
HiIQR      : 8.5
LoIQR      : -3.5
NaN        : 4
DropNaN    : 4
FillNaN    : 0
ReplaceNaN : 0

# Replace "NaN" (use -ReplaceNaN option)
"a".."d" `
    | %{ $s=$_; 1..5 | %{ "$s $_" } } `
    | %{ $_ -replace '5$','NaN' } `
    | percentile 2 -NoHeader -ReplaceNaN 0 -Detail

field      : F2
count      : 20
sum        : 40
mean       : 2
stdev      : 1.45095250022002
min        : 0
Qt25       : 1
Qt50       : 2
Qt75       : 3
max        : 4
IQR        : 2
HiIQR      : 6
LoIQR      : -2
NaN        : 4
DropNaN    : 0
FillNaN    : 0
ReplaceNaN : 4

デシル分析表の生成。入力データを10ブロックに分けてラベル付け。percentileとの違いは、値をみずに単純に「行数」だけをみてそれを10等分する点。

percentileでのpercentileやquartileは外れ値を除外したり**値をみるが、このdecilはどちらかというと指定列の「上位の少数レコード」が総計の多く（たとえば8割）を占めている、ということを確認するのによく使う。

• Usage
  • man2 decil
• Params
  • decil
    • [[-Key] <Int32>]
    • [[-Val] <Int32>]
    • [-Delimiter <String>]
    • [-Rank]
    • [-NaN <String>]
    • [-NoHeader]
    • [-Quartile]
    • [-Outlier]
    • [-StandardDeviation]
• thanks:
  • デシル分析表の作成 with Excel

Examples:

Input:

cat data.txt

Customers Sales
AC01 6340834
AC02 6340834
AC03 6340834
・・・
U036 6158245
U040 6500047
U041 6751113

Output:

cat data.txt `
    | decil `
    | Format-Table

Name  Seg Count          Sum       Mean Ratio Cumulative-Ratio
----  --- -----          ---       ---- ----- ----------------
Sales D01    57 431538439.00 7570849.81  0.14             0.14
Sales D02    57 384099747.00 6738592.05  0.12             0.26
Sales D03    57 382266775.00 6706434.65  0.12             0.38
Sales D04    56 374027341.00 6679059.66  0.12             0.49
Sales D05    56 353450955.00 6311624.20  0.11             0.60
Sales D06    56 309655714.00 5529566.32  0.10             0.70
Sales D07    56 303489528.00 5419455.86  0.10             0.80
Sales D08    56 302052324.00 5393791.50  0.09             0.89
Sales D09    56 266710113.00 4762680.59  0.08             0.98
Sales D10    56  76811269.00 1371629.80  0.02             1.00

Another examples:

cat data.txt `
    | decil -NoHeader `
    | Format-Table

Name Seg Count          Sum       Mean Ratio Cumulative-Ratio
---- --- -----          ---       ---- ----- ----------------
H2   D01    57 431538439.00 7570849.81  0.14             0.14
H2   D02    57 384099747.00 6738592.05  0.12             0.26
H2   D03    57 382266775.00 6706434.65  0.12             0.38
H2   D04    56 374027341.00 6679059.66  0.12             0.49
H2   D05    56 353450955.00 6311624.20  0.11             0.60
H2   D06    56 309655714.00 5529566.32  0.10             0.70
H2   D07    56 303489528.00 5419455.86  0.10             0.80
H2   D08    56 302052324.00 5393791.50  0.09             0.89
H2   D09    56 266710113.00 4762680.59  0.08             0.98
H2   D10    56  76811269.00 1371629.80  0.02             1.00

description
=========================
NoHeader input

cat data.txt

Customers Salse
AC01 6340834
AC02 6340834
AC03 6340834
・・・
U036 6158245
U040 6500047
U041 6751113

cat data.txt `
    | decil -Rank `
    | head

Seg Customers Sales
D01 BZ30 9830001
D01 CZ31 9600101
D01 GZ96 9500965
D01 TZ11 8998608
D01 CZ35 8920822
D01 EZ64 8691211
D01 GZ87 8615511
D01 FZ09 8614123
D01 U022 8594501

description
=========================
-Rank option

cat data.txt

AC01 6340834
AC02 6340834
AC03 6340834
・・・
U036 6158245
U040 6500047
U041 6751113

cat data.txt `
    | decil -Rank `
    | chead `
    | ratio 3 `
    | head

D01 BZ30 9830001 0.0030872127736867039417159664
D01 CZ31 9600101 0.0030150103174844539891268974
D01 GZ96 9500965 0.0029838756385019996555041486
D01 TZ11 8998608 0.0028261052631631841729778897
D01 CZ35 8920822 0.0028016757709572328253828774
D01 EZ64 8691211 0.0027295640781731753488107647
D01 GZ87 8615511 0.0027057897156916167519817411
D01 FZ09 8614123 0.0027053538000360764173397506
D01 U022 8594501 0.0026991913094071049142092472
D01 CZ84 8470022 0.0026600974009877927269611624

半角スペース区切りデータのうち数値列1列分の基礎統計量を算出する。

percentileとsummaryはどちらも「ヘッダありスペース区切りデータの縦一列ぶんの要約統計量を算出する」点は同じ。 percentileはCategory列を指定するとCategoryごとに基礎統計量を計算できる。

• デフォルトで四分位数を計算
  • 四分位数の**値はExclusive
  • 四分位数の外れ値は (Qt1st - IQR*1.5)以下、(Qt3rd + IQR*1.5)以上
  • 標準偏差はn-1で割った値（標本標準偏差）
• オプションで外れ値を加味した四分位数、標準偏差を算出

cat iris.csv | summary 1 -d ","

Name        : sepal_length
Count       : 150
Mean        : 5.84333333333333
Min         : 4.3
Qt25%       : 5.1
Qt50%       : 5.8
Qt75%       : 6.4
Max         : 7.9
IQR         : 1.3
IQD         : 0.65
Sum         : 876.5
Count-NaN   : 0
Count-Total : 150

• Usage
  • man2 summary
• Params
  • summary
    • [[-Num] <Int32>]
    • [-Delimiter <String>]
    • [-NaN <String>]
    • [-noHeader]
    • [-Quartile]
    • [-Outlier]
    • [-StandardDeviation]

Examples:

一列分しか演算してくれない。

cat iris.csv | summary 1 -d ","

Name        : sepal_length
Count       : 150
Mean        : 5.84333333333333
Min         : 4.3
Qt25%       : 5.1
Qt50%       : 5.8
Qt75%       : 6.4
Max         : 7.9
IQR         : 1.3
IQD         : 0.65
Sum         : 876.5
Count-NaN   : 0
Count-Total : 150

以下のように工夫すると任意の複数列の基礎統計量が計算できる。

1..4 | %{ cat iris.csv | summary $_ -d "," } | ft

Name         Count Mean  Min Qt25% Qt50% Qt75%  Max  IQR  IQD
----         ----- ----  --- ----- ----- -----  ---  ---  ---
sepal_length   150 5.84 4.30  5.10  5.80  6.40 7.90 1.30 0.65
sepal_width    150 3.06 2.00  2.80  3.00  3.30 4.40 0.50 0.25
petal_length   150 3.76 1.00  1.60  4.35  5.10 6.90 3.50 1.75
petal_width    150 1.20 0.10  0.30  1.30  1.80 2.50 1.50 0.75

ムービング・ウィンドウ・アプローチ（異常検知の考え方・傾向分析（TREND ANALYSIS）の一種）のPowerShell実装。

Implementation of moving window approach.

通常のサンプリング計画では同一ロットから決められたサンプル数(n)を採取して検査し，その中で基準値(m)を超えるものが(c)個以内であれば合格と判定するが（2階級法），Moving windowでは，比較的大きな数のサンプル数n個を一定の期間，決められた頻度で採取して検査し，最新の結果が加わるたびに最古の検査結果をn個の枠から削除し，そのn個のなかで，基準値(m)を超えるものが(c)個以内であればその工程または食品安全管理システムは適切に管理されていると判断する手法であり，サンプル日ごとの検査結果を表に表した場合，n個の枠が検査結果が加わるたびに日々，枠（窓）を移動するように見えるので，ムービング・ウインドウと呼ばれている．

from グローバル化と食品微生物規格の考え方 - J-Stage

• references:

  • Guidelines | CODEXALIMENTARIUS FAO-WHO
    • CXG 21-1997 Principles and Guidelines for the Establishment and Application of Microbiological Criteria Related to Foods
  • Application of the moving window approach in the verification of the performance of food safety management systems - ScienceDirect

• Usage

  • man2 movw

• Params

  • movw
    • [[-Num] <Int32>]
    • [-WindowSize <Int32>]
    • [-AcceptableLevel <Double>]
    • [-MaxLimit <Double>]
    • [-MaxFrequency <Double>]
    • [-ReverseLimit]
    • [-OnlyResult]
    • [-NoHeader]
    • [-Delimiter <String>]
    • [-NaN <String>]

• Exmples:

  • cat iris.txt | movw -Num 1 -WindowSize 5 -AcceptableLevel 4.7 -MaxLimit 5.5 -MaxFrequency 3

• Notes:

  • Corrective action is triggered by exceeding either (c) or (M):
    • Limit that will start corrective measures when this value deviates at least once in the window
      • (M) Max limit
    • Levels that will start corrective measures when -AcceptableLevel deviations occur more than -MaxFrequency times in "-WindowSize" range
      • (m) AcceptableLevel: a marginally acceptable level
      • (n) WindowSize: moving window size
      • (c) Maximum frequency (c) of all samples taken during the specified period (n)
  • The evaluation result is output in the "Res" field.
    The meaning of the numerical value is as follows:
    • 0: Clear (no need to start corrective action)
    • 1: Over MaxFrequency (need to start corrective action)
    • 2: Over MaxLimit (need to start corrective action)
    • 3: Over MaxFrequency and over MaxLimit (same as 1, 2)
  • The moving window approach for implementing microbiological criteria is described.
  • The approach can be a cost-effective means to demonstrate acceptable performance of a food safety management system.
  • The approach is appropriate where the between-lot variability of samples is less than the within-lot variability.

Examples:

cat iris.txt `
    | movw `
        -Num 1 `
        -WindowSize 5 `
        -AcceptableLevel 4.7 `
        -MaxLimit 5.5 `
        -MaxFrequency 3

s_l AcceptableLevel MaxLimit MaxFrequency WindowSize Res
--- --------------- -------- ------------ ---------- ---
5.1 NaN             NaN      NaN          NaN        NaN
4.9 NaN             NaN      NaN          NaN        NaN
4.7 NaN             NaN      NaN          NaN        NaN
4.6 NaN             NaN      NaN          NaN        NaN
5.0 4.7             5.5      3            5          1
5.4 4.7             5.5      3            5          1
4.6 4.7             5.5      3            5          0
5.0 4.7             5.5      3            5          1
4.4 4.7             5.5      3            5          1
4.9 4.7             5.5      3            5          1
5.4 4.7             5.5      3            5          1
4.8 4.7             5.5      3            5          1

水平方向の値ソート。任意のキーフィールドを無視して並び替えることもできる。

"key1 key2 2 4 7 1 3" | ysort -n 2

key1 key2 1 2 3 4 7

左列のキーを無視できる以外は、おおよそ以下のスクリプトと等価。 Roughly equivalent to the script below:

"3 1 2 11" | %{ ($_ -split " " | sort) -join " " }
"3 1 2 11" | %{ ($_ -split " " | sort { [double]$_ }) -join " " }

• Usage
  • man2 ysort
• Params
  • ysort
    • [[-n|-Num] <Int32>]
    • [-Cast <String>]
    • [-Descending]
    • [-CaseSensitive]
    • [-Unique]
    • [-SkipHeader]
• Options
  • -SkipHeader: skip first record
  • -Cast <datatype>: any data type can be specified.
  • -Ordinal: ordinal sort
• Notes
  • Auto detect types:
    • double [System.Double]
    • datetime [System.DateTime]
    • version [System.Version]
    • string [System.String]
  • Empty records are skipped.

Examples

simple usage

"C B A" | ysort

A B C

basic usage (sort value fields as System.Double)

# input
$dat = @("3 2 2 1 12 22", "123 75 12 12 22 01 87 26", "98 21 21 67 59 1")
$dat

3 2 2 1 12 22
123 75 12 12 22 01 87 26
98 21 21 67 59 1

$dat | ysort

1 2 2 3 12 22
01 12 12 22 26 75 87 123
1 21 21 59 67 98

$dat | ysort -Unique

1 2 3 12 22
01 12 22 26 75 87 123
1 21 59 67 98

$dat | ysort -Descending

22 12 3 2 2 1
123 87 75 26 22 12 12 01
98 67 59 21 21 1

$dat | ysort -Cast double

1 2 2 3 12 22
01 12 12 22 26 75 87 123
1 21 21 59 67 98

$dat | ysort -Cast string

1 12 2 2 22 3
01 12 12 123 22 26 75 87
1 21 21 59 67 98

sort key-value fields (-n <n> option)

# input (leftmost value is key string)
$dat = @("ABC 2 2 3 12 22", "DEF 123 75 12 22 01", "XYZ 98 21 67 59 1")
$dat

ABC 2 2 3 12 22
DEF 123 75 12 22 01
XYZ 98 21 67 59 1

# "-n 1" means key fields from 1 to 1, others are value fields
# or "-n <n>" means sort by ignoring 1 to <n> fields
$dat | ysort -n 1

ABC 2 2 3 12 22
DEF 01 12 22 75 123
XYZ 1 21 59 67 98

# If the key string is in the leftmost column,
# it will be sorted as a string if no key field is specified.
$dat | ysort  ## Oops! forgot to specify a key field (-n 1)!

12 2 2 22 3 ABC
01 12 123 22 75 DEF
1 21 59 67 98 XYZ

If detect different data type, raise error and stop processing

# error example

# sort data type detected from first record
# input (second record type (string) is different fron first (double))
$dat = @("3 2 2 1 12 22", "ABC 75 12 12 22 01 87 26", "98 21 21 67 59 1")
$dat

3 2 2 1 12 22
ABC 75 12 12 22 01 87 26 ## leftmost field is different data type to 1st record
98 21 21 67 59 1

# so the following raises an error and stop processing
$dat | ysort

1 2 2 3 12 22
Sort-Object: ysort_function.ps1:152
Line |
    152 |                  | Sort-Object { [double] $_ } `
        |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        | Cannot convert value "ABC" to type "System.Double". Error: "The input string 'ABC' was not in a correct format."
ysort: Different record type detected: 2 : ABC 75 12 12 22 01 87 26

Version-sort example

# version sort example
"192.1.3.2 192.11.3.1 192.2.3.5" | ysort
"192.1.3.2 192.11.3.1 192.2.3.5" | ysort -Cast version

192.1.3.2 192.2.3.5 192.11.3.1

Datetime-sort example

# datetime sort example
"2021-4-10 2021-4-1 2021-4-5" | ysort

2021-4-1 2021-4-5 2021-4-10

"2021/4/10 2021/4/1 2021/4/5" | ysort -Cast datetime

2021-4-1 2021-4-5 2021-4-10

Miscellaneous

# specified delimiter
"ABC321" | ysort -n 3 -fs ""

ABC123

Ordinal sort

# ordinal sort

"abc1 Abc2 abc3 Abc4" | ysort

abc1 Abc2 abc3 Abc4

"abc1 Abc2 abc3 Abc4" | ysort -Ordinal

Abc2 Abc4 abc1 abc3

水平方向の集計。任意のキーフィールドを無視しての集計も可能。 デフォルトでヘッダありスペース区切りデータを期待する。

# non-key data
"11 12 33" | ycalc -NoHeader -Sum

F1 F2 F3 sum
11 12 33 56

# key-value data
"k1 k2 12 24 37 11 23" | ycalc -n 2 -NoHeader -Sum -Average -Minimum -Maximum

F1 F2 F3 F4 F5 F6 F7 sum ave max min
k1 k2 12 24 37 11 23 107 21.4 37 11

# "ycalc -n 2 -Sum":
#     means ignore fields 1-2 as keys,
#     sum remaining fields horizontally.

• Usage
  • man2 ycalc
• Params
  • ycalc
    • [[-n|-Num] <Int32>]
    • [-Sum]
    • [-Average]
    • [-Mean]
    • [-Maximum]
    • [-Minimum]
    • [-StandardDeviation]
    • [-AllStats]
    • [-NoHeader]
• Example
  • cat data | ycalc -n 2 -Sum
    • Means ignore fields 1-2 as keys, sum remaining fields horizontally
• Options
  • -NoHeader: No header data

Examples

# non-key data
"11 12 33" | ycalc -NoHeader -Sum

F1 F2 F3 sum
11 12 33 56

# key-value data
"k1 k2 12 24 37 11 23" `
    | ycalc -n 2 -NoHeader -Sum -Average -Minimum -Maximum

F1 F2 F3 F4 F5 F6 F7 sum ave max min
k1 k2 12 24 37 11 23 107 21.4 37 11

# "ycalc -n 2 -Sum":
#     means ignore fields 1-2 as keys,
#     sum remaining fields horizontally.

A toolset for categorical analysis and anomaly detection for time series data.

The core concept of this section, the idea of performing data analysis on the console, was heavily inspired by the following sites:

• thanks to
  • Nicholas Dille, Data Analysis using #PowerShell, Published on 21 Mar 2017
  • GitHub - nicholasdille/PowerShell-Statistics: Statistical analysis of data on the command line
    • License: Apache License 2.0
    • See CREDIT section

As a basis for the anomaly detection mechanism, I adopted the X-control chart method, in which process variation is determined from the average of the difference between successive observations.

• JISZ9020-2:2016 管理図 - 第2部：シューハート管理図 - Part 2: Shewhart control charts

The functions of the anomaly detection mechanism used in this section to be considered in its use are completely similar to those of an X-control chart.

Most of the functions presented in this section are also compartible with other type of data exploration.

X管理図については，次によって幾つかの注意を払うことが望ましい。

• a) X管理図は，群に基づく管理図ほど，工程変化に敏感ではない
• b) 工程分布が正規分布でない場合は，X管理図の解釈に注意しなければならない
• c) X管理図は，連続する観測値の間の差の平均値から工程変動を求める。したがって，データは時系列に沿ったものであり，また連続する二つの個々の測定値の収集の間に，いかなる大きな変化も工程内では発生していないことが条件となる。例えば，生産ラインを途中で停止した場合，二つの不連続のバッチ化学製品の生産活動からデータを収集し，最初の活動の最終バッチと次の活動の最初のバッチとの間の移動範囲を計算することは賢明なことではない

from: JISZ9020-2:2016 管理図－第2部：シューハート管理図　Control charts-Part 2: Shewhart control charts

Example script to calculate basic statistics by category:

# Code example 1
# Categorical analysis with this section's toolset
Import-Csv -Path iris.csv `
    | Sort-Object -Property "species" -Stable `
    | Apply-Function -Key "species" {
        Measure-Stats -Value "sepal_length" -Key "species" -Sum -Average } `
    | Format-Table