librarycarpentry / lc-shell Goto Github PK

Lesson #1 and #3 have the same questions at the top (and on the overall schedule), but different objectives. This can cause confusion for students new to Carpentry lessons and new to the shell. Since the questions listed are directly addressed in Lesson #1, I suggest more targeted questions for Lesson #3:

• How can I make a new file?

• How do I edit and rename files?

• Can I cancel a command in the shell?

• How can I reuse commands or view shell history?

Additionally, I have some thoughts on the resources used for teaching the UNIX Shell lesson.

Both times I took this lesson, I found it very confusing because neither instructor who taught the lesson shared the Reference. In the Guide, there is only a line under “Preparing to Teach” that says “reference.md can be printed out and given to students as a reference, your choice.” I think that this should be edited that the guide should be shared with students (not your choice), because it gives an overview of many of the arguments that will be used throughout the lesson. This might be me, but I’d suggest that always providing students with this document reduces cognitive overload by giving them the space to focus on learning how the shell works without having to remember arguments.

Reference.md could be updated with a few more key arguments. It is lacking some that may be important to students being able to apply the lesson quickly to their work:

• man -- important for students to be able to navigate the shell and complete tasks on their own

• history -- key for remembering functions and reproducing the same exercise when learning

• touch -- this argument may not make logical sense to new learners

per @aadoek

The unix date command takes different options on Mac OS (because Mac uses BSD command line utilities rather than GNU). For this reason some of the examples that use date -I (to return the system date in ISO-8601 format) don't work.

A suggested fix is to specify the full date format for Mac:
date +%Y-%m-%d instead.

https://librarycarpentry.org/lc-shell/06-free-text/index.html
Option 2, Optical character recognised text, uses a file called 000003160_01_text.json; however, the command below uses a different file name (which is then also used subsequently):

$ tr -d [:punct:] < 201403160_01_text.json > 201403160_01_text-nopunct.txt

It seems that macOS 10.15 (Catalina), due for general release Fall 2019, uses zsh instead of bash by default. I don't think the lesson strays into areas where the syntax differs between bash and zsh, but the Setup page will need to be updated for accuracy at least.

I don't have a Mac so not I'm not ideally placed to comment on whether it would be better for Catalina users to do the lesson under zsh or to provide instructions on how to switch to bash. (Will people need to install it? If so, can they install the latest bash instead of the current default version?)

The exercise "Searching with regular expressions" in "Counting and Mining with the Shell" and its proposed solution could probably use some re-wording: If the file created is to only contain the found ISSNs, the command already requires the -o flag (which is only pointed out in the exercise below). Also, the -E flag version fails rather silently (i.e. simply doesn't produce any result, rather than giving an error message) on my system (Ubuntu). When redirecting output to a file immediately, this can easily be missed, so maybe learners should be encouraged to check the results in some way. It could be helpful to also include the (clumsy, but -E/P-less) version of

grep -o '[0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]' 2014-01_JA.tsv

 The output will be a path to your home directory. Let’s check if we recognise it by listing the   
 contents of the directory. To do that, we use the ls command:

I would introduce here the abbreviation and say 'ls stands for list.'

Two additional tidbits that I think students might find valuable:

• Give an example (probably in "Working with files and folders") of using cd - as kind of like a command line ALT + Tab to cycle back and forth between the current and previous directory
• Mention (in the "History" section) that Bash defaults to saving a user's last 500 commands. So the command history will be truncated in relative short order if you're a heavy shell user.

I'd recommend making an addition to the "Using history" section of ep. 3 to include instructions on pressing CTRL + r repeatedly to cycle through all past commands matching the search criteria.

A sentence similar to the following:

If multiple past commands contain the text you input, you can CTRL + r repeatedly to cycle through them.

Could be placed after "The past command will autocomplete."

Could not tell if this is an already reported issue.
Environment: macOS Catalina - iMac and MacAir

In episode Working with files and directories / Using the echo command
echo "Library Carpentry is awesome!"

Returns a request for final quote:
dquote>

however the "!" (exclamation point) is truncated when final quote is added.
Library Carpentry is awesome

Leaving a space between the exclamation point and the quote
$ echo "Library Carpentry is awesome! "

produced correct result:
Library Carpentry is awesome!

Is this an issue for anyone else?

Thanks for contributing! If this contribution is for instructor training, please send an email to [email protected] with a link to this contribution so we can record your progress. You’ve completed your contribution step for instructor checkout just by submitting this contribution.

If this issue is about a specific episode within a lesson, please provide its link or filename.

Please keep in mind that lesson maintainers are volunteers and it may be some time before they can respond to your contribution. Although not all contributions can be incorporated into the lesson materials, we appreciate your time and effort to improve the curriculum. If you have any questions about the lesson maintenance process or would like to volunteer your time as a contribution reviewer, please contact The Carpentries Team at [email protected].

The free text lesson includes an option (option 1) to work with a file ("gulliver.txt") that is not currently in the shell-lesson.zip archive. The file should be added to the archive or this option should be removed from the lesson.

See: https://github.com/LibraryCarpentry/lc-shell/blob/gh-pages/_episodes/06-free-text.md

In lesson 3 (https://librarycarpentry.org/lc-shell/03-working-with-files-and-folders/index.html), the first objective is to "Create files and directories from the command line". But "creating files" is never really introduced or mentioned in the lesson. This will be confusing for a novice learner.

I would recommend either to remove "Create files" from the objectives or have at least one explicit mention on how files can be created (e.g.: by introducing the redirect ">" concept here instead of in lesson 5).

One of the files in the shell-lesson folder is a JSON file called "000003160_01_text." The long string of zeros is difficult to type while live coding. Perhaps a few (or all of) the zeroes could be removed from the name of this file. Thanks!

This lesson has been migrated here to the Library Carpentry organisation.

All issues that need work are still at the old location:

Find issues to resolve for this lesson here:
https://github.com/data-lessons/library-shell-DEPRECATED/issues

Find issues to resolve there and fix them here.

If your Maintainer team has decided not to participate in the June 2019 lesson release, please close this issue.

To have this lesson included in the 18 June 2019 release, please confirm that the following items are true:

• Example code chunks run as expected
• Challenges / exercises run as expected
• Challenge / exercise solutions are correct
• Call out boxes (exercises, discussions, tips, etc) render correctly
• A schedule appears on the lesson homepage (e.g. not “00:00”)
• Each episode includes learning objectives
• Each episode includes questions
• Each episode includes key points
• Setup instructions are up-to-date, correct, clear, and complete
• File structure is clean (e.g. delete deprecated files, insure filenames are consistent)
• Some Instructor notes are provided
• Lesson links work as expected

When all checkboxes above are completed, this lesson will be added to the 18 June lesson release. Please leave a comment on carpentries/lesson-infrastructure#26 or contact Erin Becker with questions ([email protected]).

Dear Maintainers - I received the following .md file as a submission through email to [email protected].

title: "Automating the tedious with loops"
teaching: 20
exercises: 10
questions:

• "What is a loop?"
• "How can a loop be used to repeat a task?"
  objectives:
• "Build a concept for how loops can be used to repeat tasks"
• "Implement a loop to rename several files"
  keypoints:
• "Looping is the foundation for working smarter with the command line"
• "Loops help us to do the same (or similar) things to a bunch of items"

Writing a Loop

Loops are key to productivity improvements through automation as they allow us to execute
commands repetitively. Similar to wildcards and tab completion, using loops also reduces the
amount of typing (and typing mistakes).
Suppose we have several hundred document files named project_1825.doc, project_1863.doc, XML_project.docand so on.
We would like to change these files, but also save a version of the original files, naming the copies
backup_project_1825.doc and so on.

We can use a loop to do that.
Here's a simple example that creates a backup copy of four text files in turn.

Let's first create those files:

$ touch a.doc b.doc c.doc d.doc

This will create four empty files with those names. It is easy to use the shell to create a batch of files in one go.

Now we will use a loop to create a backup version of those files. First let’s look at the general form of a loop:

for thing in list_of_things
do
    operation_using $thing    # Indentation within the loop is not required, but aids legibility
done

{: .language-bash}

We can apply this to our example like this:

$ for filename in *.doc
> do
> echo "$filename"
>    cp "$filename" backup_"$filename"
> echo "backup complete for"   
> done

{: .bash}

backup complete for a.doc
backup complete for b.doc
backup complete for c.doc
backup complete for d.doc

{: .output}

When the shell sees the keyword for,
it knows to repeat a command (or group of commands) once for each thing in a list.
For each iteration,
the name of each thing is sequentially assigned to
the loop variable and the commands inside the loop are executed before moving on to
the next thing in the list.
Inside the loop,
we call for the variable's value by putting $ in front of it.
The $ tells the shell interpreter to treat
the variable as a variable name and substitute its value in its place,
rather than treat it as text or an external command.

Double-quoting variable substitutions

Because real-world filenames often contain white-spaces,
we wrap $filename in double quotes ("). If we didn't, the
shell would treat the white-space within a filename as a separator
between two different filenames, which usually results in errors.
Therefore, it's best and generally safer to use "$..." unless
you are absolutely sure that no elements with white-space can ever
enter your loop variable (such as in [episode 5]({{ page.root }}/05-counting-mining/index.html#using-a-loop-to-count-words)).
{: .callout}

In this example, the list is four filenames: 'a.doc', 'b.doc', 'c.doc', and 'd.doc'
Each time the loop iterates, it will assign a file name to the variable filename
and run the cp command.
The first time through the loop,
$filename is a.doc.
The interpreter runs the command cp on a.doc,
and then prints the filename to the screen (because we asked it to echo each filename as it works its way through the loop).
For the second iteration, $filename becomes
b.doc. This time, the shell runs cp on b.doc
and then prints the filename to the screen. The loop performs the same operations for c.doc and then for d.doc and then, since
the list only included these four items, the shell exits the for loop at that point.

Follow the Prompt

The shell prompt changes from $ to > and back again as we were
typing in our loop. The second prompt, >, is different to remind
us that we haven't finished typing a complete command yet. A semicolon, ;,
can be used to separate two commands written on a single line.
{: .callout}

Same Symbols, Different Meanings

Here we see > being used as a shell prompt, but > can also be
used to redirect output from a command (i.e. send it somewhere else, such as to a file, instead of displaying the output in the terminal) ---
we'll use redirection in [episode 5]({{ page.root }}{% link _episodes/05-counting-mining.md %}).
Similarly, $ is used as a shell prompt, but, as we saw earlier,
it is also used to ask the shell to get the value of a variable.

If the shell prints > or $ then it expects you to type something,
and the symbol is a prompt.

If you type > or $ yourself, it is an instruction from you that
the shell to redirect output or get the value of a variable.
{: .callout}

We have called the variable in this loop filename
in order to make its purpose clearer to human readers.
The shell itself doesn't care what the variable is called.

For loop exercise

Complete the blanks in the for loop below to print the name, first line, and last line
of each text file in the current directory.

___ file in *.txt
__
	echo "_file"
	head -n 1 _______
	____ __ _ _______
____

{: .bash}

Solution

for file in *.txt
do
	echo "$file"
	head -n 1 "$file"
	tail -n 1 "$file"
done

{: .bash}
{: .solution}
{: .challenge}

This is our first look at loops. We will run another loop in the
[Counting and Mining with the Shell]({{ page.root }}{% link _episodes/05-counting-mining.md %}) episode.

Running the loop from a Bash script

Alternatively, rather than running the loop above on the command line, you can
save it in a script file and run it from the command line without having to rewrite
the loop again. This is what is called a Bash script which is a plain text file that
contains a series of commands like the loop you created above. In the example script below,
the first line of the file contains what is called a Shebang (#!) followed by the path to the interpreter
(or program) that will run the rest of the lines in the file (/bin/bash). The second line demonstrates how
comments are made in scripts. This provides you with more information about what the script does.
The remaining lines contain the loop you created above. You can create this file in the same directory
you've been using for the lesson and by using the text editor of your choice (e.g. nano) but when you save the
file, make sure it has the extension .sh (e.g. my_first_bash_script.sh). When you've done this, you can run the
Bash script by typing the command bash and the file name via the command line (e.g. bash my_first_bash_script.sh).

#!/bin/bash
# This script loops through .txt files, returns the file name, first line, and last line of the file
for file in *.txt
do
	echo $file
	head -n 1 $file
	tail -n 1 $file
done

{: .bash}
Download/copy my_first_bash_script.sh. For more on Bash scripts, see Bash Scripting Tutorial - Ryans Tutorials.
{: .callout}

Under the searching and mining portion of the lesson, the code:' *grep -iwEo 'Cont[ae][cn]t' .csv ' (under Mining and Searching) has learners use the grep -o flag, however, some users have an older version of grep that does not recognize the flag and this can cause confusion among learners despite it being a legitimate flag. It would be appropriate to add a tip blurb explaining why this may happen (and how to fix it by installing a newer grep version) and I think it would reassure instructors and learners who are not aware of the issue. We made this amendment in our library carpentry workshop and also split Counting and Mining into two separate lessons since it was so long (but thats a different circus).

Counting and Mining Lesson Link: https://librarycarpentry.org/lc-shell/05-counting-mining/index.html

Thanks for contributing! If this contribution is for instructor training, please send an email to [email protected] with a link to this contribution so we can record your progress. You’ve completed your contribution step for instructor checkout just by submitting this contribution.

If this issue is about a specific episode within a lesson, please provide its link or filename.

Please keep in mind that lesson maintainers are volunteers and it may be some time before they can respond to your contribution. Although not all contributions can be incorporated into the lesson materials, we appreciate your time and effort to improve the curriculum. If you have any questions about the lesson maintenance process or would like to volunteer your time as a contribution reviewer, please contact The Carpentries Team at [email protected].

'Solution' suggests using date --help. This only works for Windows users. man date should be added for OS/Linux users.

This is a recommended exercise that can be added to Episode 5 - counting and mining from @weaverbel and @jduckles. Builds on the cut Selecting columns from our article dataset exercise.

It uses grep, cut, sort, uniq and shows them how to whittle things down and then filter those results to a file for later work. Filtering and subsetting data from big files was something a lot of people would use.

grep 2009 2014-01_JA.tsv | grep INTERNATIONAL | cut -f 12 | sort | uniq -c | sort -nr > results/titles.tsv

This refers to Lesson 2 in the Unix Shell - https://librarycarpentry.org/lc-shell/02-navigating-the-filesystem.html

When arguments/flags are first mentioned in Lesson 2, their function could be made clearer.
For example, an explanation that with command ls:
-l is for outputting as a long list;
-h is for outputting in human-readable form; and
-S is for sorting (largest first).

These are covered in more detail later, but when they first appear, it is quite cryptic!

This relates to "Mining or Searching" here: https://librarycarpentry.org/lc-shell/05-counting-mining/index.html

This command throws two errors on macOS Mojave:

$ grep -iw revolution *.tsv > results/$(date -I)_JAiw-revolution.tsv

First error is documented in this issue. After solving for this by removing the -I it still throws this error:

-bash: results/$(date)_JAi-revolution.tsv: ambiguous redirect

One fix is to put quotes around $(date):

$ grep -iw revolution *.tsv > results/"$(date)"_JAiw-revolution.tsv

The explanation in the paragraph "Double-quoting variable substitutions" and the example above (writing a loop) differ in the order. The explanation says "The interpreter runs the command cp on a.doc, and then prints the filename to the screen". In the example the filename is printed first.

Episode 6 currently states: "We’re going to work with the gulliver.txt file we made in the previous lesson." Has it been considered to make this more precise by saying something like "We're going to work with the gulliver.txt file, which we made from the 829-0.txt file in Episode 3, Working with files and folders." (Current formulation is slightly confusing since the previous episode actually renames a chapter from Little Women. I see there were some earlier issues raised about this file so thought this may be a confusing bit for others as well.

Submitted by email as part of Instructor Training checkout:

Here is my feedback/contribution on the Library Carpentry: The Shell Lesson, specifically in episode three and five.

The tip on Canceling Commands is confusing in its current section.

It references cancelling the previous command to print the contents of a book, but the output is on the screen almost instantly (or in a few instances takes 1-2 seconds).

This causes confusion for learners as they read through the lesson/follow along because nothing happens, when they "cancel".

This tip would be more helpful in episode five in the section on grep as it takes more time for the shell to process and search. The lines of text "cascade" by in the window like is mentioned in episode three. The tip would be helpful after this script:

$ grep -iwE 'fr[ae]nc[eh]' *.tsv

This gives learners time to input ctrl + c and see the print stop. They also know what to expect as they saw the full printout of grep earlier.

The Counting number of files part I and II exercises in the "Counting and Mining in the Shell" episode can be easily done by the command ls | wc -l since the total is not included in an ls command. When we use the ls -l | wc -l command a line total in the ls -l output throws the count off by one. We then come back to this exercise later in the episode to fix this "one more" problem with grep, though all this is really unnecessary. This was pointed out to me by a student the last time I taught this lesson. Maybe we don't need this grep exercise, or another one can be used that might be better. Below are the exercises I am talking about.

Counting and mining in the shell episode challenge:
Counting number of files, part I
Let’s make a different pipeline. You want to find out how many files and directories there are in the current directory. Try to see if you can pipe the output from ls into wc to find the answer, or something close to the answer.

Solution
You get close with

$ ls -l | wc -l
but the count will be one too high, since the “total” line from ls is included in the count. We’ll get back to a way to fix that later when we’ve learned about the grep command.

Counting number of files, part II
In the earlier counting exercise in this episode, you tried counting the number of files and directories in the current directory.

Recall that the command ls -l | wc -l took us quite far, but the result was one too high because it included the “total” line in the line count.
With the knowledge of grep, can you figure out how to exclude the “total” line from the ls -l output?
Hint: You want to exclude any line starting with the text “total”. The hat character (^) is used in regular expressions to indicate the start of a line.
Solution
To find any lines starting with “total”, we would use:

$ ls -l | grep -E '^total'

The 01-intro-shell.md lesson covers quite a bit of territory:

• navigating shell
• cp, mv and mkdir
• creating a loop
• inspecting files

I counted at least 10 commands in this one lesson. We break it up into multiple topical lessons like SWC unix shell.

In option 1 (working with gulliver.txt), there is this line:

Note: there are a few bits of punctuation in here - I’ve left these in deliberately as you should always bug fix! The internet is a always a good place to start searching for why this might have happened (something about the punct command we used…)

The remaining punctuation marks in the file appear to be curly (or "smart") quotes: “ and ‘ (and corresponding counterparts). Additional information should probably be provided.

• An easy way to identify that there are still punctuation marks in the file. Note grep [“‘”’] gulliver-final.txt works but might be a little heavy-handed.
• A little more guidance on investigating the issue (like what to search the Internet for other than punct).

Migrated from data-lessons/library-shell-DEPRECATED#67

In Lesson 2: Navigating the filesystem, it says:

Let’s start by opening the shell. This likely results in seeing a black window with a cursor flashing next to a dollar sign.

I use MacOS Catalina, and with the upgrade the default shell is now zsh rather than bash. I would note that some Mac users might see a white window and a dollar sign, or a white window, tilde, and percentage sign.

I just used https://github.com/LibraryCarpentry/lc-shell/raw/gh-pages/data/shell-lesson.zip for a recent workshop and noticed that the Little Women file is missing. @jt14den or anyone else, do you have the file and can you add it to the zip file?

On https://librarycarpentry.org/lc-shell/02-navigating-the-filesystem/index.html - in the box "Getting help", second paragraph, Windows is not capitalized and shell is capitalized.

@drjwbaker I wasn't able to find a solution to the sed challenges in the working with free text lesson ( removing smart quotes using sed). What I found didn't work, and I am not sure if it is because I am on a Windows machine or not. I think it is important for learners to understand how to find answers online, but I also think we should be providing answers to solutions too.

In Episode 5, the callout "Invalid option – o?" says

If you get an error message “invalid option – o” when running the above command, it means you use a version of grep that doesn’t support the -o flag. This is for instance the case with the version of grep that comes with Git Bash on Windows.

This was true when the callout was added (2 June 2017 in 1969f75) but someone may have been listening because it's not the case any longer. If I'm reading the right bit of source code, the option was added to Git Bash on Windows on 9 July 2018 in commit git-for-windows/git@9d8db06e, and first released in version v2.19.0-rc0.windows.1 on 21 Aug 2018.

The -o option is supported on macOS and in GNU grep (hence on Linux and in WSL shells on Windows 10), so does anyone know of any remaining greps that don't support -o?

Probably for now the best thing would be to put this warning in the past tense:

This was the case with the version of grep that came with Git Bash on Windows before September 2018. Since the flag is not crucial to this lesson, please just relax and ignore the problem. If you want the flag in future, however, you could install another version of grep, perhaps by upgrading your version of Git Bash on Windows, or installing a Bash shell for Windows 10 using the Windows Subsystem for Linux.

But once we're comfortable that everyone has upgraded past Git Bash on Windows v2.18, I think this callout and the references to it could be removed.

Submitted via email Aug 4, 2020:

"I checked the training material of all carpentries and I was wondering
why there is no "Hello World" example in all fields available. In
general, when I started programming in different languages and with the
command line, there was always in the beginning an example including
something with "Hello World", e.g. for Unix

echo Hello World

was my first command in the shell. Learners that have started in the
field will appreciate "Hello World" examples, because they can
understand and compare such common examples if they are interest in more
programming languages. Therefore I suggest to add "echo Hello World" for
example as first command for the Unix part in the Library Carpentry."

Hi if you start from, https://librarycarpentry.github.io/lc-shell/
and then click on the setup (http://librarycarpentry.github.io/lcshell/setup/) link you get....

404
File not found
The site configured at this address does not contain the requested file.
If this is your site, make sure that the filename case matches the URL.
For root URLs (like http://example.com/) you must provide an index.html file.
Read the full documentation for more information about using GitHub Pages.
GitHub Status — @githubstatus

On this page: https://librarycarpentry.github.io/lc-shell/setup.html

The link to shell-lession.zip is incorrect. It links to the Github page (https://github.com/LibraryCarpentry/lc-shell/blob/gh-pages/data/shell-lesson.zip) but should be the 'raw' download (https://raw.githubusercontent.com/librarycarpentry/lc-shell/gh-pages/data/shell-lesson.zip).

Can we change the title of this to line up with the new title for the Introduction, so it doesn't expressly say librarians?

Thanks!

this may be too advanced, but I'm not so sure. What about file permissions & ownership? I noticed there is no mention of this anywhere, yet this is one of the largest issues I face when working in the shell. This is especially important on a server, where multiple users are accessing the machine.

And then for automating tasks, there was no mention of the cron. Creating a script is great and is a huge step in the right direction, but if you really want to automate a task, you need to be able to use the cron and schedule a script to run. Maybe this could be added as an addendum?

Originally posted by @jmcgranahan in #50 (comment)

We need to create some scenarios and use cases for the shell lesson, and this could be a good sprint task.

We have counting and mining data, we have grep to find info within files, and basic shell navigation, including FOR loops. At my workshop today, people wanted specific tasks the shell can do for librarians. It would be good to create a list.

More info here: data-lessons/librarycarpentry#31

Suggestions from Jamie @jamieviva that we add something on the general structure of bash. So, something like:

COMMAND -FLAG FILE.USED

Then explain structure of what happens thereafter, so:

if you add nothing = print to shell; if you add > FILE.OUTPUT = save to file; if you add '|' = save to memory for next command.

It probably should do somewhere in https://github.com/data-lessons/library-shell/blob/gh-pages/_episodes/01-intro-shell.md not before doing work in the shell, but as a "look at what you've also learned!" outcome.

Thoughts?

More: data-lessons/library-shell-DEPRECATED#28

Per https://www.bleepingcomputer.com/news/microsoft/microsoft-launches-multi-shell-windows-terminal-console-app/
"One of the most anticipated features is the ability to open tabs for multiple console windows in one Windows Terminal window. When opening tabs, you will be able to select the shell you wish to use, which includes the normal CMD shell, PowerShell, and any associated shells from Linux distributions installed via the Windows Subsystem for Linux."

Note on Setup https://librarycarpentry.org/lc-shell/setup.html the associated shell from the Linux distributions on Windows 10.

See lesson 5 'Counting and mining with the shell', section 'Automatically adding a date prefix'.

The example in the text uses date -I. This does not work on my system (MacOS 10.14.5, GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)). I was unable to check for other systems. I find a reference to the -I flag here, but not here.

Something similar can be achieved by date "+%Y %m %d". But then it might be necessary to explain the formatting commands.

I just submitted a PR ( #98 ) for a typo in this area, but I am curious about it's meaning. In the introduction under "Where is my shell?" there is the following:

"For Windows users, popular shells such as Cygwin or Git Bash provide a Unix-like interface, but may need to be installed separately. In Windows 10, the Power Shell natively provides that functionality."

It is unclear to me what is meant by PowerShell natively supports that functionality? Given the previous sentence I read it as saying that PowerShell provides a Unix-like interface. While it provides some aliases for popular Unix commands (ls for Get-ChildItem/dir ), it does not implement a full bash shell, including commands used later in the lessons (i.e. there's no sed or grep).

I worry it could confuse learners, but I may just be misinterpreting it.

The output of ls from within shell-lesson does not match the current contents of the directory (See also #8). The current output for the archive (https://github.com/LibraryCarpentry/lc-shell/blob/gh-pages/data/shell-lesson.zip) looks like:

total 139M
-rw-rw-r-- 1 jcoliver jcoliver 384K Feb 22  2017 000003160_01_text.json
-rw-r--r-- 1 jcoliver jcoliver 3.6M Jan 31  2017 2014-01-31_JA-africa.tsv
-rw-r--r-- 1 jcoliver jcoliver 7.4M Jan 31  2017 2014-01-31_JA-america.tsv
-rw-rw-r-- 1 jcoliver jcoliver 126M Jun 10  2015 2014-01_JA.tsv
-rw-r--r-- 1 jcoliver jcoliver 1.4M Jan 31  2017 2014-02-02_JA-britain.tsv
-rw-r--r-- 1 jcoliver jcoliver 583K Feb  2  2017 33504-0.txt
-rw-r--r-- 1 jcoliver jcoliver 598K Jan 31  2017 829-0.txt
-rw-rw-r-- 1 jcoliver jcoliver  19K Feb 22  2017 diary.html

Hi Library Carpentry maintainers,

I was wondering if we could mention/briefly explain what does this output in episode 2 means.

total 0
drwx------+ 6 riley staff 204 Jul 16 11:50 Desktop
drwx------+ 3 riley staff 102 Jul 16 11:30 Documents
drwx------+ 3 riley staff 102 Jul 16 11:30 Downloads
drwx------@ 46 riley staff 1564 Jul 16 11:38 Library
drwx------+ 3 riley staff 102 Jul 16 11:30 Movies
drwx------+ 3 riley staff 102 Jul 16 11:30 Music
drwx------+ 3 riley staff 102 Jul 16 11:30 Pictures
drwxr-xr-x+ 5 riley staff 170 Jul 16 11:30 Public

Here we are focused on "ls -l" command, but we do not explain what does "drwx------" or "riley" or "staff", etc. If I was attending this workshop I will ask what all of those things mean, and I believe it would be good to have a short explanation.

Proposed addition:
In this output, we can see file permission listed "drwx------+", number of links "3", owner name "riley", group owner name "staff", number of bytes "102", last modified time " Jul 16 11:30" and directory name "Music".

Please let me know what do you think.
Thanks.

https://librarycarpentry.org/lc-shell/06-free-text/index.html
$ tr ' ' '\n' < gulliver-clean.txt | sort | uniq -c | sort -r > gulliver-final.txt
should be
$ tr ' ' '\n' < gulliver-clean.txt | sort | uniq -c | sort -rn > gulliver-final.txt
for numerical counting; maybe we should also explain the -r flag?

In episode 6 "Working with free text", the files 829-0.txt (that becomes gulliver.txt) and diary.txt are provided with Windows line endings (CRLF or \r\n if you prefer).

On Windows, sed automatically strips out the \r characters, leaving the UNIX line ending (LF, \n), so when we get to the grand finale, e.g.

tr ' ' '\n' < gulliver-clean.txt | sort | uniq -c | sort -rn > gulliver-final.txt

the translation works as expected to put all words on their own line. But on other platforms, the \r characters are preserved, so the translation effectively turns the text into a UNIX file where some words end in CR (\r) and some do not. The variants are seen as different by uniq, so the operation gives the wrong results.

This issue does not affect option 2 as 000003160_01_text.json (or 201403160_01_text.json, see #100) does not have any line breaks.

I would like to see instructor guides developed for all Library Carpentry lessons. This could be a good sprint project. Librarians will probably like to have some guidance on how to teach what for many will be new material. Not sure what instructor guides should consist of? Then see some of the guides for Software Carpentry lessons, e.g. shell - http://swcarpentry.github.io/shell-novice/guide/, python - http://swcarpentry.github.io/python-novice-inflammation/guide/.

Guides like this help with context setting, things to remember to explain, and other useful info.

History of this issue: data-lessons/librarycarpentry#27

Recommend pwd instead of cd to test/demo that everything is working and provide a brief description: The output will be a path to your home directory.

In https://librarycarpentry.org/lc-shell/setup.html

Hi starting at https://github.com/search?utf8=%E2%9C%93&q=Shell+Lessons+for+Librarians&type=

data-lessons/library-shell-DEPRECATED
Unix shell lesson for librarians NOW MOVED > https://github.com/LibraryCarpentry/lc-shell

The "NOW MOVED", link takes you to, https://github.com/LibraryCarpentry/lc-
where it should take you to, https://github.com/LibraryCarpentry/lc-shell

Walter Griffatong

The shell lesson needs 1 or more breaks. I'd suggest after episodes 3, 4, and 5.

it looks like the instructor guide hasn't kept up with the modification of the lesson and needs to be updated in preparation for the June 2019 release.