Skip to main content

Shell

conda/miniconda

# get miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-py38_23.9.0-0-Linux-x86_64.sh

# create env
conda env remove --name env_name
conda create --name qlib_env python=3.8

# see the env
conda env list
conda info -e

# remove env
conda remove -n env_name --all

# asign the source
pip install -i pypi.douban.com/simple tensorflow-gpu==1.14

configure ssh

  1. Upload the public key to the server
# generate key
ssh-keygen 

ssh-copy-id -i ~/.ssh/id_rsa.pub -p PORT_NUMBER root@ipaddress

Crontab

# edit
crontab -e

# list
crontab -l
# 20 16 * * * /path/to/deploy.sh
# start at 16:20 everyday

deploy.sh demo is as below:

#!/bin/bash
/home/zhonghao/miniconda3/envs/data/bin/python /home/zhonghao/path/to/scrip.py

Referenceopen in new window

Cuda

When install the cuda, you need to set environment variables.

export PATH=/usr/local/cuda-11.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:$LD_LIBRARY_PAT

Where PATH is the path to the cuda bin directory and LD_LIBRARY_PATH is the path to the cuda lib64 directory and LD is the dynamic library.

Then you can install pytorch from by qurey the cuda version. For example, you are using cuda 11.2, you can install pytorch by

pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html

11.2 is the version of the cuda and cu111 is the version of the pytorch. cu111 means the pytorch is built with cuda 11.1.

More details can be found in the official websiteopen in new window.

find

find /path/to/source_folder -type f -exec cp {} /path/to/destination_folder \;
# where {} is the file found by find command
# -type f means find files
# -exec is the command to execute
# cp is the command to copy
# \; is the end of the command

frp

docsopen in new window

  • config for server

    • bindPort is the port that the frps listens on the server
    bindPort = 7000

auth.method = "token"
auth.token = "{mytoken}"

webServer.addr = "0.0.0.0"
webServer.port = 7100
webServer.user = "{username}"
webServer.password = "{password}"

  • config for client

    serverAddr = "{Public IP}"
serverPort = 7000
auth.method = "token"
auth.token = "{mytoken}"

[[proxies]]
name = "XXXX"
type = "tcp"
localIP = "127.0.0.1"
localPort = 22
remotePort = {port}

history

To display the time of the command, set the HISTTIMEFORMAT environment variable.

export HIST_STAMPS="yyyy-mm-dd"

ipv6

The format of the IPv6 address is xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx, where x is a hexadecimal digit. The address can be shortened by removing leading zeros and consecutive sections of zeros. For example, 2001:0db8:0000:0042:0000:8a2e:0370:7334 can be shortened to 2001:db8:0:42:0:8a2e:370:7334.

The localhost address is ::1 while ipv4 is 127.0.0.1.

In some services, you need to specify the bind address as :: or * to allow both ipv4 and ipv6 connections.

jupter2script

pip install nbconvert

jupyter nbconvert --to script detailed_workflow.ipynb

Log

Standard output and error are redirected to the file nohup.out if neither STDOUT nor STDERR are redirected to another destination (e.g. by using > or 2> in the command line).

# STDOUT
nohup python -u main.py > log.txt 2>&1 &

# STDERR
nohup python -u main.py 2> log.txt &

# Both
nohup python -u main.py > log.txt 2> log.txt &

mlflow


pkill -f "mlflow server"

mlflow server --backend-store-uri file:////root/mlflow_sqlite

MongoDB Installationopen in new window

mount windows file

mkdir ~/mnt/directory_name

# mount
sudo mount -t cifs -o username=USERNAME,password=PASSWORD //WINDOWS_HOST_IP/SHARE_NAME ~/mnt/directory_name

# unmount
sudo umount ~/mnt/directory_name

Referenceopen in new window

mysql

# install
sudo apt update
sudo apt install mysql-server

# start service
sudo service mysql start
sudo mysql_secure_installation

mysql -u root -p

# show port
mysql -u root -p -e "SHOW VARIABLES LIKE 'port';"

# connect from other machine
# Create a user account for localhost:
CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' WITH GRANT OPTION;

# Create a user account for any host (%):
CREATE USER 'username'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'username'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

  • When using Aliyun, you need to pay attention to the security rules in the security group(安全组中设置安全规则).

  • If your MySQL password contains a @ symbol, you can handle this by URL encoding the password when constructing the connection URI. In URL encoding, the @ symbol is represented as %40. Therefore, if your password is, for example, my@password, you would include it in the connection URI like this:

    mysql+pymysql://username:my%40password@localhost:3306/mydatabase

Reference

netstat

netstat -tulpn

pkill -f "mlflow server"

ngrok

oh-my-zshopen in new window

sudo apt update
sudo apt install git
sudo apt install zsh
sh -c "$(wget -O- https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

# install zsh-autosuggestions
git clone https://github.com/zsh-users/zsh-autosuggestions ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions

# .zshrc
ZSH_THEME="candy"
ZSH_THEME="obraun"
ZSH_THEME="jonathan"

plugins=(git zsh-autosuggestions)

qlibopen in new window

# install with pip
pip install pyqlib

# install from source
pip install numpy
pip install --upgrade  cython
git clone https://github.com/microsoft/qlib.git && cd qlib
pip install .

quick initialize

bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohmybash/oh-my-bash/master/tools/install.sh)"
wget https://repo.anaconda.com/miniconda/Miniconda3-py38_23.9.0-0-Linux-x86_64.sh
sh Miniconda3-py38_23.9.0-0-Linux-x86_64.sh

# qlib
pip install numpy
pip install --upgrade  cython
git clone https://github.com/microsoft/qlib.git && cd qlib
pip install .

systemd

# 使用 apt 安装 systemd（Debian/Ubuntu）
apt install systemd

# start service
sudo systemctl start frpc.service

# stop service
sudo systemctl stop frpc.service

# restart service
sudo systemctl restart frpc.service

# enable service
sudo systemctl enable frpc.service

Below is an example.

  • Service file name: frpc.service
  • Service file path: /etc/systemd/system/frpc.service
  • Contents:
    [Unit]
Description = frp server
After = network.target syslog.target
Wants = network.target

[Service]
Type = simple
ExecStart = /path/to/frp/frpc -c /path/to/frp/frpc.toml

[Install]
WantedBy = multi-user.target

Referenceopen in new window

Ubuntu

# create user
sudo adduser username
sudo passwd username

# sudo assess
sudo usermod -aG sudo username

# RAM
free -h

# GPU
lspci | grep -i nvidia

vuepress(hope-theme)

# install nvm, npm, node
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | zsh
source ~/.zshrc
nvm install node
npm install -g pnpm
export NPM_CONFIG_USERCONFIG="$HOME/.config/pnpm/.npmrc"
source ~/.zshrc

# install vuepress
pnpm install -D vuepress

# clone github repo
git clone git@github.com:ZhongHaoAustin/zhonghao-pages.git
cd zhonghao-pages

# run dev
pnpm docs:dev
pnpm run docs:dev

wexterm

configopen in new window

Create a file named .wezterm.lua in your home directory, with the following contents:

-- Pull in the wezterm API
local wezterm = require 'wezterm'

-- This table will hold the configuration.
local config = {}

-- In newer versions of wezterm, use the config_builder which will
-- help provide clearer error messages
if wezterm.config_builder then
  config = wezterm.config_builder()
end

-- This is where you actually apply your config choices

-- For example, changing the color scheme:
config.color_scheme = 'Cloud (terminal.sexy)'
-- and finally, return the configuration to wezterm
return config