BLACK CAT PROGRAMMER

bash script notes

Get input arguments

test.sh -n=Tom –phone=1234

#!/bin/bash

for i in "$@"
do
case $i in
    -n=*|--name=*)
    NAME="${i#*=}"
    ;;
    -p=*|--phone=*)
    PHONE="${i#*=}"
    ;;
esac
done

echo NAME = $NAME
echo POST = $PHONE

Get positional arguments

Save below script to test.sh and run it as

#!/bin/bash

echo $0 $1 $2

>> test.sh a bb cc

result:
test.sh a bb

Get process id of last executed command

python3 HelloForever.py &

pid=$!
echo $pid

Assign variables

GREETING_WORD="Hello world!"
echo $GREETING_WORD

>> Hello world!

Loop

for (( i=0; i<5; i++)) do
	echo $i
done

echo date

TODAY=`date +%Y%m%d`
echo $TODAY

Changing all files in working directory to lowercase

for i in *; do mv $i `echo $i | tr [:upper:] [:lower:]`; done

Adding prefix to all files in working directory

for a in *.png; do mv "$a" "$1$a"; done

Posted in notesTagged ,

vim notes

vim 真是太好用了

The basic

In view mode:

CommandUsage
:w
:q
:wq
:wq!
Save
quit
Save and quit
confirm save and quit
:number
eg: :129
Go to line number
Go to line 129
20|
25h
25l
Go to column 20
Move cursor 25 characters ahead
move cursor 25 character after
/keyword
?keyword
n
N
Search keyword from beginning of file
Search keyword from the end of file
Go to next find result
Go to previous find result
$
9
:1
G
Go to beginning of current line
Go to the end of current line
Go to line 1
Go to the end of file
ctrl + d
ctrl + u
Move half of the view downwards
Move half of the view upwards
i
a
r
Insert: start editing at the current cursor position
Append: start editing after the current cursor position
Replace: replace the character at the current cursor position
u
Ctrl + r
Undo
Redo
h
j
k
l
Move the cursor to left
Move the cursor to down
Move the cursor to up
Move the cursor to right
e
E

w
W
Go to the end of a word separated by special character
Go to the end of a word separated by space

Go to the start of a word separated by special character
Go to the start of a word separated by space

Display and not display line number:

:set number
:set nonumber

marco

// record a marco and named it "a"
// in view mode, enter "q" then the name of the marco, a-z

ie: qa

// then vim will enter recording mode. You may see "recording @a" in bottom left corner

// To play the marco once
// in view mode enter "@" followed by the name of the marco, in my case, a

ie: @a

// To play the marco several times
// in view mode, enter the number of times followed by "@" and the name of the marco

ie: 10@a

Posted in notesTagged ,

IP Table

iptables 就可以當作是 linux 裡的防火牆。

Each table has many chains which each chain has many rules

Common tables are nat and filter

filter is for controlling the incoming and outgoing packets

 CHAINS
  1. INPUT
  2. FORWARD
  3. OUTPUT

while

nat is for forwarding packets

 CHAINS
  1. PREROUTING
  2. INPUT
  3. OUTPUT
  4. POSTROUTING

Save the current iptables rules

iptables-save > file_path.rules

Restore iptables rules

iptables-restore < /etc/iptables.rules

Examples

1. List current iptable rules of NAT
iptables -t nat -L -v

2. List current iptables rules of INPUT line number
iptables -t nat -L -v --line-numbers

3.1. Adding a new rule accepting port 1194 traffic
3.1) Append mode
 iptables -t nat -A INPUT -i eth0 -p tcp -m tcp --dport 1194 -j ACCEPT
3.2) Insert mode
iptables -t nat -I INPUT [line number ]-i eth0 -p tcp -m tcp --dport 1194 -j ACCEPT
3.3) Insert mode v2
iptables -I INPUT 7 -p tcp --dport 8443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

4.Change the source IP from 10.8.0.0/24 to 192.168.13.11
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j SNAT --to-source 192.168.13.11

5. delete a rule
 1) first, show the current rules by
 iptables -t nat -L -v --line-numbers 
 
 2) delete the rule
 iptables -t nat -D POSTROUTING 1

REF: https://opensource.com/article/18/9/linux-iptables-firewalld

Posted in notesTagged ,

Installing Raspbian to respberry using MAC

Installing the OS on SD card

  • donwload the image
  • insert the SD card in to MAC
  • find out the address of the SD card diskutil list
Blackcat-MacBook-Pro:Volumes blackcat$ diskutil list
/dev/disk0 (internal):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         251.0 GB   disk0
   1:                        EFI EFI                     314.6 MB   disk0s1
   2:                 Apple_APFS Container disk1         250.7 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +250.7 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD            161.3 GB   disk1s1
   2:                APFS Volume Preboot                 21.9 MB    disk1s2
   3:                APFS Volume Recovery                519.0 MB   disk1s3
   4:                APFS Volume VM                      7.1 GB     disk1s4

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *63.3 GB    disk2
   1:             Windows_FAT_32 boot                    46.0 MB    disk2s1
   2:                      Linux                         63.2 GB    disk2s2

the SD card is /dev/disk2 in my case

  • unmount the SD card sudo diskutil unmount /dev/disk2
  • write the image to SD card
    sudo dd bs=1m if=2017-04-10-raspbian-jessie.img of=/dev/disk2
  • eject the SD card sudo diskutil eject /dev/rdiskX

Connecting to raspberry

Reference

  • Create an empty file ssh on root directory of the SD card touch /ssh
  • Create a file vi /wpa_supplicant.conf on root directory
country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
    ssid="<wifi ssid>"
    psk="<wifi password>"
}
  • ssh pi@<IP of the raspberry>
  • default password is raspberry
Posted in notesTagged

Hong Kong ID

There are two types of Hong Kong AB123456(A) and A123456(3)

Algorithm of generating the check digit
A=10,B=11,...,Z=35, space=36

Example 1

HKID: AB123456(check digit)
0 = [A*9 + B*8 + 1*7 + 2*6 + 3*5 + 4*4 + 5*3 + 6*2 + (check digit)] % 11
check digit = A*9 + B*8 + 1*7 + 2*6 + 3*5 + 4*4 + 5*3 + 6*2 
= 11 - 255 % 11 
= 11 - 2
= 9

ps: if check digit == 10 then return A

Example 2

HKID: A123456
check digit = 11 - (36*9 + A*8 + 1*7 + 2*6 + 3*5 + 4*4 + 5*3 + 6*2)
= 11 - 481 % 11 
= 11 - 8
= 3

ps: if check digit == 10 then return A
Posted in notesTagged

python request

金田一出了新的故事,上網睇固然開心,不用擔心存放問題,但網頁的計設真是很爛,要scroll up and down 去看一整頁,又多廣告,索性把它下載下來好一點


import urllib.request

# http://cache.someou.com/Uploads/files/2020/66134/001.jpg

PAGES = [25, 23, 22, 22, 22, 24, 21, 23, 26, 23, 23, 21, 21, 22, 25, 22, 23, 21, 23, 24, 23, 23, 22, 23, 26, 23, 22, 24, 24, 22]
BASE_URL = 'http://cache.someou.com/Uploads/files/2020/{:d}/{:03d}.jpg'


for chp in range(30):
    for page in range(PAGES[chp]):
        link = BASE_URL.format(66134+chp, page+1)
        print("donwloading {} chapter {} page {}".format(link, chp, page))

        req = urllib.request.Request(link)
        req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36')
        req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3')
        req.add_header('Cookie', '__cfduid=d28d5f69ff39f960a0571508905d2e3b21556458631')
        response = urllib.request.urlopen(req)

        if response.status == 200:
            #result = f.read().decode('utf-8')
            f = open('{:03d}_{:03d}.jpg'.format(chp+1,page+1), 'wb')
            f.write(response.read())
            f.close()
        else:
            print("Failed in download chp {} page {}".format(chp, page))
            print("status {} reason {}".format(response.status, response.reason))

print("done")

小小玩具,自己做一個記錄

Posted in notesTagged

Linux command cheatsheet

Finding files

# finding a file
# 1. find hello.txt in /src
find /src -name hello.txt

# Find large files in linux
$ sudo du -a /dir/ | sort -n -r | head -n 20

Add a new user

adduser <username> --shell /bin/bash --home <home directory>

Add a user to an existing to group

usermod -aG <group name> <username>

# eg1: add user to sudo group
usermod -aG sudo <username>

Remove user from a group

gpasswd -d <user name> <group name>

# show the group of the user
groups <user name>

Creating symbolic link

ln -s <source folder> <new shortcut>

Show current memory usage

> free -h

              total        used        free      shared  buff/cache   available
Mem:           875M        141M        155M         80M        579M        592M
Swap:           99M          0B         99M

or

> free -m

              total        used        free      shared  buff/cache   available
Mem:            875         141         155          80         579         592
Swap:            99           0          99

Remove password requirement

sudo vim /etc/pam.d/common-password

remove obscure to disable complexity check

add minlen=3 to add min password length

Service control

# Start a service
sudo service nginx start

# check the status of a service
sudo service nginx statua

# stop a service
sudo service nginx stop

# list all service
service --status-all
service --status-all | more
service --status-all | grep ntpd
service --status-all | less

Posted in notesTagged

Adding routing rule on macOS

After setting up StrongSwan, everything should be done, ie you can connect to the vpn server and the subnet behind. However, in some circumstance, the local subnet mask cover the remote subnet, ie the remote subnet is 192.168.55.0/24, while the local subnet is 192.168.0.0/32.

In this case, the packet which is intended go to remote 192.168.55.1 will go to local gateway. Therefore, we have to add a custom routing rule on the local 

mac os

  • First list the current routing tables by netstat -nr
  • Remove the existing one if any route delete 192.168.55.0/24
  • Add a new rule to route the traffic to vpn server route add 192.168.55.0/24 10.10.10.1 where 10.10.10.1 is the vpn gateway
Posted in notesTagged

VPN server IP changed

平常家的ip 十年都唔轉一次,上次被大陸ban 了ip ,打電話去CS,都改不了,最後要改了router 的 mac address。但今次就無啦啦轉了一個新的IP ,令到早前set 好的vpn server 用不了。

如果IP 轉了,需要改一下conf, secret 和重新 issue 條cert

  1. 改了 /etc/ipsec.conf 的 leftid
  2. 改了 /etc/ipsec.secrets 的 ip
  3. 重新issue server cert
$sudo ipsec pki --pub --in vpn-server-key.pem \
--type rsa | ipsec pki --issue --lifetime 1825 \
--cacert server-root-ca.pem \
--cakey server-root-key.pem \
--dn "C=US, O=VPN Server, CN=server_ip" \
--san server_ip \
--flag serverAuth --flag ikeIntermediate \
--outform pem > vpn-server-cert.pem

不過要用回之前的 server-root-ca.pemserver-root-key.pemvpn-server-key.pem,不然就要重新弄過了

$ sudo cp ./vpn-server-cert.pem /etc/ipsec.d/certs/vpn-server-cert.pem
Posted in notesTagged

Useful mysql command

Installation

Install mysql on rapberry pi

sudo apt-get install mysql-server mysql-client

Check version of mysql

mysql –version

Configure db

sudo vim /etc/mysql/my.cnf

Change password

SET PASSWORD FOR 'user name'@'localhost'='new password';

For later version, use authentication_string

Create user

GRANT ALL PRIVILEGES ON *.* TO 'your_username_here'@'localhost' IDENTIFIED BY 'your_chosen_password_here';
FLUSH PRIVILEGES;

Create database

CREATE DATABASE db_name DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Querying

List all tables in the database

SELECT * FROM information_schema.tables

or

show tables

Logging

Enable SQLs logging

An alternative way to log SQLs – logging to table. It is approriate when you don’t want to restart the sql server

  1. Create a table for the log
  CREATE TABLE `slow_log` (
   `start_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP 
                          ON UPDATE CURRENT_TIMESTAMP,
   `user_host` mediumtext NOT NULL,
   `query_time` time NOT NULL,
   `lock_time` time NOT NULL,
   `rows_sent` int(11) NOT NULL,
   `rows_examined` int(11) NOT NULL,
   `db` varchar(512) NOT NULL,
   `last_insert_id` int(11) NOT NULL,
   `insert_id` int(11) NOT NULL,
   `server_id` int(10) unsigned NOT NULL,
   `sql_text` mediumtext NOT NULL,
   `thread_id` bigint(21) unsigned NOT NULL
  ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log'
  
  CREATE TABLE `general_log` (
   `event_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
                          ON UPDATE CURRENT_TIMESTAMP,
   `user_host` mediumtext NOT NULL,
   `thread_id` bigint(21) unsigned NOT NULL,
   `server_id` int(10) unsigned NOT NULL,
   `command_type` varchar(64) NOT NULL,
   `argument` mediumtext NOT NULL
  ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='General log'
  1. Enable it by
SET global general_log = 1;
SET global log_output = 'table';

The log will be in the table general_log

  1. view the log by select * from mysql.general_log
  2. disable it when finished by SET global general_log = 0;

Export / import

Export database

mysqldump -u [username] -p [database name] > [database name].sql

Example:

15 2 * * * root mysqldump -u root -pPASSWORD –all-databases | gzip > /mnt/disk2/database_`date +”%Y%m%d”`.sql.gz

Import database

CREATE DATABASE newdatabase;

mysql -u [username] -p newdatabase < [database name].sql

Clone a database

mysqldump db_name | mysql new_db_name

Posted in notesTagged ,