Record some ueful Linux Tricks that I have been using
find . -maxdepth 1 -mtime -1
find . -mtime -1
set -gx PATH /home/myt007/miniconda3/bin $PATH
alias ta='tmux attach'
alias yg='you-get '
alias ytdl="youtube-dl --rm-cache-dir -i --no-check-certificate --merge-output-format mp4 --format 'bestvideo[ext=mp4]+bestaudio[ext=m4a]' --add-metadata --embed-thumbnail "
alias ytdlmax="youtube-dl --rm-cache-dir -i --no-check-certificate --merge-output-format mp4 --format 'bestvideo+bestaudio[ext=m4a]' --add-metadata --embed-thumbnail "
alias subl='sublime'
alias nvishow='watch -n1 nvidia-smi'
alias tree='tree -N'
alias blogserver='JEKYLL_ENV=production bundle exec jekyll serve -w'
alias atomblog='atom ~/blog'
alias filecount='find . | wc -l'
alias fcount='find . | wc -l'
alias ta='tmux attach'
alias brewup='brew update; brew upgrade; brew cleanup; brew doctor'
alias gits='git status'
alias aria2c='aria2c -x 16 -s 16 '
alias ca='source activate dl'
alias cadl='conda activate dl'
alias ipyserver='jupyter notebook --no-browser --ip=127.0.0.1 --port=8888'
alias ipyconvert='ipython nbconvert --to script '
alias kalicontainer='sudo docker run -it --net="host" --privileged kali:1 /bin/bash'
alias untar='sudo tar -xzvf'
alias update='sudo apt update; sudo apt upgrade -y; wget -O - https://raw.githubusercontent.com/laurent22/joplin/master/Joplin_install_and_update.sh | bash'
alias install='sudo apt-get install'
alias h265izehere='h265ize -v -m fast -q 25 -x --no-sao --aq-mode 3 --stats --delete'
alias rmv='rsync --remove-source-files -avhP'
alias rcp='rsync -avhP'
alias ncduroot='sudo ncdu -x /'
alias minuimushere='find . -type f -print0 | xargs -0 -P 8 -n 8 minuimus.pl'
alias dc='docker-compose '
alias enhance='function ne() { docker run --rm -v "$(pwd)/`dirname ${@:$#}`":/ne/input -it alexjc/neural-enhance ${@:1:$#-1} "input/`basename ${@:$#}`"; }; ne'
enhance --zoom=1 --model=repair images/broken.jpg
conda create -n gpu python pip ipykernel tensorflow-gpu
conda activate gpu
python -m ipykernel install --user --name gpu --display-name gpu
find . -type f -name 'download.sh' -exec sed -i 's/youtube-dl /youtube-dl --download-archive ytdl-archive-record.txt /g' {} \;
rsync -r --ignore-existing --progress source dest
- Install youtube-dl package
- can install it with brew
- Use
youtube-dl -F url
to check what format is available on youtube - Useful options include --ignore-errors \ --no-playlist \ --yes-playlit \ --embed-thumbnail \ -x (output audio)
I use:
youtube-dl -f 140 --yes-playlist --embed-thumbnail --ignore-errors url
to download the playlist with specifying file format code as 140 (.m4a file).
If the file code 140 is not available, one might consider use -x option to force ffmpeg to convert an audio fril. So iCloud Music Library doesn't support the aac and opus from youtube-dl. Maybe the mp3 format will work. So it does work. Here is the recommanded commands for mp3 transformation:
youtube-dl -x --audio-format mp3 --yes-playlist --embed-thumbnail --ignore-errors url
If you want to fix it locally, you may have to modify your .profile to execute the shell you want:
if [ "$SHELL" != "/bin/bash" ]
then
export SHELL="/bin/bash"
exec /bin/bash -l # -l: login shell again
fi
ls | xargs -I@ convert @ -trim @
watch -n 1 cat $(ls -1t | head -1)/logfile
tail -f $(ls -1t | head -1)/logfile
watch -n 1 tail $(ls -1t | head -1)
ref: https://superuser.com/questions/117596/how-to-tail-the-latest-file-in-a-directory
du --exclude={relative/path,path2} -cBM --max-depth=1 2> >(grep -v 'Permission denied') | sort -n
or
ncdu
Do
%env CUDA_DEVICE_ORDER=PCI_BUS_ID
%env CUDA_VISIBLE_DEVICES=0
or
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
Use the following to test
from tensorflow.python.client import device_lib
print device_lib.list_local_devices()
THEANO_FLAGS=device=cuda0
THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=cuda0
THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu0 python train.py
THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu2 python
Noted that the order of GPUs is reversed on dd0 and dd1.
CUDA_VISIBLE_DEVICES=0,1,2 python
CUDA_VISIBLE_DEVICES=2 python
CUDA_VISIBLE_DEVICES=“” python
KERAS_BACKEND=theano THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu6 python resnet_custom_dataset_theano.py
python -m cProfile -s cumtime lwn2pocket.py
Use this if want to write to file
python -m cProfile -s cumtime lwn2pocket.py > profile.txt
Use this if want to duplicate the output to file while keep the one on screen.
python -m cProfile -s cumtime lwn2pocket.py | tee profile.txt
Use this to generate the binary output file.
python -m cProfile -s cumtime -o profile.txt lwn2pocket.py
Read the profile binary file (generated by -o option)
python -m cprofilev -f profile.txt -p 7777
nvidiashow
自定义的显示nvidia-smi 的命令
nvidia-smi
“the status of nvidia gpu card."
ljubljana.iems.northwestern.edu: 129.105.36.99 开始tensorboard
tensorboard --logdir=./log
ssh -L 16006:127.0.0.1:6006 -i ~/.ssh/ljubljana [email protected]
refer to http://127.0.0.1:16006/ in a local browser
Source:https://coderwall.com/p/ohk6cg/remote-access-to-ipython-notebooks-via-ssh
jupyter notebook --no-browser --port=7777
tmux
tux new -s svrg
Dettach: [ctrl+b][d]
tux attach: tux a -t svrg
source activate svrg
rsync -avz -e "ssh -i /home/yma/.ssh/celje" [email protected]:/home/yma/imagenet /home/yma/imagenet --progress```
rsync with LAN acceleration:
```rsync -aHAXxv --numeric-ids -e "ssh -i /home/yma/.ssh/celje -T -c arcfour -o Compression=no -x" [email protected]:/home/yma/imagenet /home/yma/imagenet --progress
As you have discovered you cannot use rsync with a remote source and a remote destination. Assuming the two servers can't talk directly to each other, it is possible to use ssh to tunnel via your local machine.
Instead of
rsync -vuar host1:/var/www host2:/var/www
you can use this
ssh -R localhost:50000:host2:22 host1 'rsync -e "ssh -p 50000" -vuar /var/www localhost:/var/www'
In case you're wondering, the -R option sets up a reverse channel from port 50000 on host1 that maps (via your local machine) to port 22 on host2. There is no direct connection from host1 to host2.
- “gits” stands for “git status”
- “gitacp “XXX” ” stands for “git add *” + “git commit -m “XXX”” + “git push origin”
- alias sa='source activate dl'
- alias sshcontainer='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -L 7777:127.0.0.1:7777 [email protected]'
function ta() {
tmux attach
}
alias subl='sublime'
alias nvishow='watch -n1 nvidia-smi'
alias tree='tree -N'
alias blogserver='JEKYLL_ENV=production bundle exec jekyll serve -w'
alias atomblog='atom ~/blog'
alias filecount='find . | wc -l'
alias fcount='find . | wc -l'
alias ta='tmux attach'
alias brewup='brew update; brew upgrade; brew cleanup; brew doctor'
alias gits='git status'
# git push
alias gpd="git push origin develop"
alias gpm="git push origin master"
# Remove git from a project
alias ungit="find . -name '.git' -exec rm -rf {} \;"
alias aria2c='aria2c -x 16 -s 16 '
alias ytdl="youtube-dl --rm-cache-dir -i --no-check-certificate --merge-output-format mp4 --format 'bestvideo[ext=mp4]+bestaudio[ext=m4a]' --add-metadata --embed-thumbnail "
alias ygl='you-get -l '
alias yg='you-get '
alias ca='source activate dl'
alias cadl='conda activate dl'
alias ipyserver='jupyter notebook --no-browser --ip=127.0.0.1 --port=8888'
alias tbhere='tensorboard --logdir=$(pwd)'
alias ipyconvert='ipython nbconvert --to script '
alias kalicontainer='sudo docker run -it --net="host" --privileged kali:1 /bin/bash'
alias dockertag='function _dockertag(){ curl -s -S "https://registry.hub.docker.com/v2/repositories/library/$@/tags/" | jq ".\"results\"[][\"name\"]" | sort};_dockertag'
# look and kill for local program that uses certain port
alias findport='function _findport(){ lsof -n -i4TCP:$@ | grep LISTEN};_findport'
alias killport='function _killport(){ lsof -n -i4TCP:$@ | grep LISTEN | awk "{print \$2}" | xargs kill};_killport'
alias untar='sudo tar -xzvf'
alias update='sudo apt-get update; sudo apt-get upgrade'
alias mkcd='foo(){ mkdir -p "$1"; cd "$1" }; foo '
alias install='sudo apt-get install'
# ls better
## Colorize the ls output ##
alias ls='ls --color=auto'
## Use a long listing format ##
alias ll='ls -la'
## Show hidden files ##
alias l.='ls -d .* --color=auto'
alias la="ls -aF"
alias ld="ls -ld"
alias lt='ls -At1 && echo "------Oldest--"'
alias ltr='ls -Art1 && echo "------Newest--"'
alias ..="cd .."
alias ...="cd ..; cd .."
alias cd..='cd ..'
alias ...='cd ../../../'
alias ....='cd ../../../../'
alias .....='cd ../../../../'
alias .4='cd ../../../../'
alias .5='cd ../../../../..'
alias grep='grep --color=auto'
# handy short cuts #
alias h='history'
alias j='jobs -l'
alias path='echo -e ${PATH//:/\\n}'
alias vi=vim
# Stop after sending count ECHO_REQUEST packets #
alias ping='ping -c 5'
# Use netstat command to quickly list all TCP/UDP port on the server:
alias ports='netstat -tulanp'
#16: Add safety nets
# do not delete / or prompt if deleting more than 3 files at a time #
alias rm='rm -Iv --preserve-root'
# confirmation #
alias mv='mv -iv'
alias cp='cp -iv'
alias ln='ln -i'
alias cl="clear;ls;pwd"
alias editsshconfig="subl ~/.ssh/config"
alias editbashrc="subl ~/.bashrc"
alias editrc="subl ~/.zshrc"
alias sourcerc="source ~/.zshrc && source ~/.bashrc"
alias src="source "
## pass options to free ##
alias meminfo='free -m -l -t'
## get top process eating memory
alias psmem='ps auxf | sort -nr -k 4'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
## get top process eating cpu ##
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
## Get server cpu info ##
alias cpuinfo='lscpu'
## older system use /proc/cpuinfo ##
##alias cpuinfo='less /proc/cpuinfo' ##
## get GPU ram on desktop / laptop##
alias gpumeminfo='grep -i --color memory /var/log/Xorg.0.log'
# top
alias cpu='top -o cpu'
alias mem='top -o rsize' # memory
#copy output of last command to clipboard
alias cl="fc -e -|pbcopy"
# copy and paste current pwd
alias cppath='pwd|pbcopy'
alias pbpath='cd $(pbpaste)'
#27 Resume wget by default
alias wget='wget -c'
## set some other defaults ##
alias df='df -H'
alias du='du -ch'
alias nvishow='watch -n1 nvidia-smi'
alias ca='conda activate dl'
alias fcount='find . | wc -l'
function gitacp() {
git add *
git add -u
git commit -m "$1"
git push origin
}
function gitacm() {
git add *
git commit -m "$1"
git push origin master
}
function gitacd() {
git add *
git commit -m "$1"
git push origin dev
}
alias sshcontainer='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -L 7777:127.0.0.1:7777 [email protected]'
alias gc='gcalcli'
alias gca='gcalcli agenda '
alias gcm='gcalcli calm '
alias gcw='gcalcli calw '
export HOME=/data/ymaab
alias ipyserver='jupyter notebook --no-browser --ip=127.0.0.1 --port=7777'
alias nvishow='watch -n1 nvidia-smi'
alias sa='source activate tf'
alias rsynccopy="rsync --partial --progress --append --rsh=ssh -r -h "
alias rsyncmove="rsync --partial --progress --append --rsh=ssh -r -h --remove-sent-files"
- https://stackoverflow.com/questions/36394101/pip-install-locale-error-unsupported-locale-setting
- unset LC_ALL
- export LC_ALL=C
https://stackoverflow.com/questions/39817645/cuda-cudnn-installed-but-tensorflow-cant-use-the-gpu
https://gist.github.com/jmsaavedra/62bbcd20d40bcddf27ac
It turns out that imageio has a function to download this for you. Read the help information!
ssh lxe1647 -p 2207
ssh lxe1646 -p 2201
ssh lxe1646 -p 2200
The PyTorch Container:
ssh lxe1646 -p 2203
python -m CycleGAN_TensorFlow.create_cyclegan_dataset --image_path_a=./CycleGAN_TensorFlow/input/syncars/trainA --image_path_b=./CycleGAN_TensorFlow/input/syncars/trainB --dataset_name="syncars_train" --do_shuffle=0
python -m CycleGAN_TensorFlow.create_cyclegan_dataset --image_path_a=./CycleGAN_TensorFlow/input/syncars/testA --image_path_b=./CycleGAN_TensorFlow/input/syncars/testB --dataset_name="syncars_test" --do_shuffle=0
CUDA_VISIBLE_DEVICES=3 python -m CycleGAN_TensorFlow.main --to_train=1 --log_dir=CycleGAN_TensorFlow/output/cyclegan/syncars --config_filename=CycleGAN_TensorFlow/configs/syncars_train.json
CUDA_VISIBLE_DEVICES=2 python -m CycleGAN_TensorFlow.main --to_train=0 --log_dir=CycleGAN_TensorFlow/output/cyclegan/syncars --config_filename=CycleGAN_TensorFlow/configs/syncars_test.json --checkpoint_dir=CycleGAN_TensorFlow/output/cyclegan/syncars/20171108-105517
for n in $(seq 9431 9999); do mv "IMG_$n.jpg" some_other_dir; done
find / -name car_damage* 2>&1 | grep -v "Permission denied"
https://askubuntu.com/questions/339/how-can-i-install-a-package-without-root-access
- ncdu - visualize the data usage
for file in $(ls -p | grep -v / | tail -100)
do
mv $file /other/location
done
For conda kernel support: https://github.com/Anaconda-Platform/nb_conda_kernels
For solyly extensions: https://github.com/ipython-contrib/jupyter_contrib_nbextensions
MultiIndex CrossSelection: The xs method of DataFrame additionally takes a level argument to make selecting data at a particular level of a MultiIndex easier.
df.xs('one', level='second')