WenJunjie Blog

sysadmin

Digitalocean Ssd Vps

| Comments

DigitalOcean

  • Simple Cloud Hosting
  • SSD cloud server

DigitalOcean

Price Comparison 价格对比

provider DigitalOcean Amazon Linode rockspace
Price $20/month $60/month $79.95/month $87.60/month
RAM 2GB 1.7GB 2GB 2GB
Bandwidth 2¢ per GB if over 3,000GB 12¢ per GB if over 1GB/mo. 10¢ per GB if over 800GB 18¢ per GB Bandwith Out
Spin Up Time 55 Seconds Up to 10 Minutes 2-3 Minutes per GB 2-3 Minutes per GB
Disk Space 40GB SSD 160GB 80GB 80GB

结论:性价比相当高

registrations acount 注册流程

  • Sign UP
  • Add a Payment Method

    Credit Cards or PayPal (Pay $5 USD for test)

  • Create Droplets(Virtual Server)

    Select Droplet Type & Size 512MB / 1 CPU 20GB SSD Disk Select Droplet Region New York 1 Select Droplet Image CentOS 6.3 x64

  • Droplet History

Event Initiated Execution Time
Create 1 minute ago 42.0 Seconds
  • Check Mail(IP root password)

CentOS Init

vi /etc/ssh/sshd_config Port 52038 PermitRootLogin no PermitEmptyPasswords no PasswordAuthentication no UseDNS no AllowUsers user /etc/init.d/sshd reload ssh -p 52038 user@xxx.xxx.xxx.xxx

ssh keys ssh-keygen -t rsa ssh-copy-id user@xxx.xxx.xxx.xxx cat .ssh/id_rsa.pub | ssh user@xxx.xxx.xxx.xxx “cat >> ~/.ssh/authorized_keys”

ssh user@xxx.xxx.xxx.xxx mkdir .ssh chmod 700 .ssh touch .ssh/authorized_keys chmod 600 .ssh/authorized_keys

vi /etc/ssh/sshd_config PermitRootLogin without-password PermitRootLogin no reload ssh

LEMP Install

  • CentOS 6
Install

#rpm -Uvh http://repo.webtatic.com/yum/el6/latest.rpm #yum –enablerepo=webtatic php54w

#rpm -ivh nginx-release-centos-6-0.el6.ngx.noarch.rpm

  • Command Reff 命令参考 yum –enablerepo=remi,remi-test list mysql mysql-server nginx php yum –enablerepo=remi list mysql mysql-server nginx php yum list mysql mysql-server nginx php
VPS
  • Install MySQL5.5 or MySQL5.1 yum –enablerepo=remi install mysql mysql-server # mysql5.5 yum install mysql mysql-server # mysql5.1

/etc/init.d/mysqld restart /usr/bin/mysql_secure_installation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!


In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...



All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!

mysql -uroot -p GRANT ALL PRIVILEGES ON . TO ‘admin’@‘%’ IDENTIFIED BY ‘db135792321’ with grant option;

  • Install nginx yum –enablerepo=remi install nginx /etc/init.d/nginx start ifconfig eth0 | grep inet | awk ‘{ print $2 }’

  • Install PHP5.4

yum –enablerepo=remi install php php-devel php-fpm php-mysql php-pecl-memcache php-pecl-gearman php-pecl-apc php-pecl-xhprof

yum –enablerepo=remi install php-fpm php-mysql

yum –enablerepo=remi install httpd php php-common yum –enablerepo=remi install php-pear php-pdo php-mysql yum –enablerepo=remi install php-pgsql php-pecl-memcache yum –enablerepo=remi install php-gd php-mbstring php-mcrypt php-xml

APC (php-pecl-apc) – APC caches and optimizes PHP intermediate code CLI (php-cli) – Command-line interface for PHP PEAR (php-pear) – PHP Extension and Application Repository framework PDO (php-pdo) – A database access abstraction module for PHP applications MySQL (php-mysql) – A module for PHP applications that use MySQL databases PostgreSQL (php-pgsql) – A PostgreSQL database module for PHP MongoDB (php-pecl-mongo) – PHP MongoDB database driver SQLite (php-sqlite) – Extension for the SQLite V2 Embeddable SQL Database Engine Memcache (php-pecl-memcache) – Extension to work with the Memcached caching daemon Memcached (php-pecl-memcached) – Extension to work with the Memcached caching daemon GD (php-gd) – A module for PHP applications for using the gd graphics library XML (php-xml) – A module for PHP applications which use XML MBString (php-mbstring) – A module for PHP applications which need multi-byte string handling MCrypt (php-mcrypt) – Standard PHP module provides mcrypt library support

/etc/init.d/php-fpm restart

Configure
  • Configure mysql vim /etc/my.cnf [mysqld] skip-character-set-client-handshake

character_set_client=utf8 character-set-server=utf8 collation-server=utf8_general_ci

#init-connect=‘SET NAMES utf8’

show variables like ‘char%’; show variables like ‘collation%’;

show variables like “%character%”; show variables like “%collation%”;

  • Configure php vi /etc/php.ini cgi.fix_pathinfo=0 session.save_path = “/tmp”

  • Configure nginx cd /usr/share/nginx mkdir htdocs chown nginx.nginx htdocs vi /etc/nginx/nginx.conf http { server_names_hash_bucket_size 64; }

vi /etc/nginx/conf.d/default.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#
 # The default server
#
server {
    listen       80;
    server_name example.com;

   
    location / {
        root   /usr/share/nginx/html;
        index index.php  index.html index.htm;
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
        root           /usr/share/nginx/html;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME   $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }
}

vi /etc/php-fpm.d/www.conf

1
2
3
4
5
6
7
8
9
[...]
; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
; will be used.
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
[...]

service php-fpm restart

  • Test Results vi /usr/share/nginx/html/info.php <?php phpinfo(); ?>

  • Set Up Autostart chkconfig –levels 235 mysqld on chkconfig –levels 235 nginx on chkconfig –levels 235 php-fpm on

Benchmark性能对比

wget http://akamaras.com/bench.sh sh bench.sh

CentOS 5

rpm -Uvh http://fedora.mirror.nexicom.net/epel//5/x86_64/epel-release-5-4.noarch.rpm rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-5.rpm rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm

yum –enablerepo=remi install mysql mysql-server yum –enablerepo=webtatic install php php-fpm php-mysql yum –enablerepo=webtatic install nginx

useradd demo passwd demo sudo vi /etc/ssh/sshd_config Port 25000 Protocol 2 PermitRootLogin no UseDNS no

/usr/sbin/visudo demo ALL=(ALL) ALL

/etc/init.d/sshd reload

Others

timezone scp -rp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime ln -sf /usr/share/zoneinfo/UTC /etc/localtime ## for Universal Coordinated Time ln -sf /usr/share/zoneinfo/EST /etc/localtime ## for Eastern Standard Time ln -sf /usr/share/zoneinfo/US/Central /etc/localtime ## for American Central time (including DST) ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime ## for American Eastern (including DST) ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime /etc/init.d/crond restart

Security

Update the server HISTTIMEFORMAT useadd user ssh-keygen chattr +i /etc/passwd; chattr +i /etc/shadow chattr +a /var/log/messages md5 file iptables fail2ban

ssh

PasswordAuthentication no ssh-keygen

DDos deflate

Denyhosts Fail2Ban

nginx module cc

iptables netstat

Nginx yum repo

cat /etc/yum.repos.d/nginx.repo [nginx] name=nginx repo baseurl=http://nginx.org/packages/centos/$releasever/$basearch/ gpgcheck=0 enabled=1

[nginx] name=nginx repo baseurl=http://nginx.org/packages/rhel/$releasever/$basearch/ gpgcheck=0 enabled=1

wget http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm rpm -ivh nginx-release-centos-6-0.el6.ngx.noarch.rpm

wget http://nginx.org/packages/rhel/6/noarch/RPMS/nginx-release-rhel-6-0.el6.ngx.noarch.rpm rpm -ivh nginx-release-rhel-6-0.el6.ngx.noarch.rpm

chkconfig mysqld on chkconfig nginx on chkconfig php-fpm on

SSH Tips

| Comments

rsa das密钥 公钥

生成密钥 [local] ssh-keygen -t dsa -b 2048 #根据提示输入回车或sshphrase密码

ssh-keygen -t rsa -f jack ssh-copy-id -i ~/.ssh/jack.pub user@server

scp ~/.ssh/jack.pub user@server:/home/user/.ssh/jack.pub ssh user@server “echo cat ~/.ssh/jack.pub >> ~/.ssh/authorized_keys”

ssh -i ~/.ssh/jack.pub user@server

vim ~/.ssh/config #server alias host server #ssh username user user #remote server address hostname server #remote server port port 22 #the public key filename(without .pub) identityfile ~/.ssh/jack

ssh server -v ssh server -vvv

server: chmod 600 ~/.ssh/authorized_keys chmod 700 ~/.ssh/

公钥id_dsa.pub放到remote-host:~/.ssh/authorized_keys [remote] cat id_dsa.pub >> .ssh/authorized_keys chmod 600 .ssh/authorized_keys

ssh-agent (ssh ForwardAgent yes) [local] eval ssh-agent ssh-add #输入sshphrase密码 echo ‘eval $(ssh-agent)’ >> ~/.bash_profile echo ‘ssh-add’ >> ~/.bash_profile eval ssh-agent -k 隐患 ssh-agent工作中创建socket文件 /tmp/ssh-sfisxD8489/agent.8489 SSH_AUTH_SOCK=/tmp/ssh-sfisxD8489/agent.8489; export SSH_AUTH_SOCK; ssh-add -l

keychain [local] vim .bash_profile keychain id_dsa . ~/.keychain/$HOSTNAME-sh

Man-in-the-moddle attack 中间人攻击

跳板机

client1———| |———–server1 |——->B(跳板机)——>| client2———| |———–server2

客户端 跳板机 工作机 在跳板机 ssh-keygen -p #修改ssh-keygen的passphrase口令,可改成空密码,安全考虑需对密钥加密 echo ‘eval keychain --eval id_dsa’ >> ~/.bash_profile eval $(keychain –eval –agents ssh -Q –quiet id_ecdsa)

security安全 chmod 400 ~/.ssh/authorized_keys sudo chattr +i ~/.ssh/authorized_keys sudo chattr +i ~/.ssh 工作机authorized_keys限制只能从跳板机ip登录 man sshd 工作机authorized_keys设置成无法被用户修改 chattr +i .ssh 需要更新时chartt -i .ssh 查看lsarrt -aR

vim /etc/ssh/sshd_config PasswordAuthentication no ChallengeResponseAuthentication no

上下键查找历史

vi ~/.inputrc “\e[A”: history-search-backward “\e[B”: history-search-forward set show-all-if-ambiguous on set completion-ignore-case on

Ctrl+r line

参考

Linode VPS

| Comments

Sign UP referral code: c829077325232e6920fd4e9ef5d4ae3d9133a736

Settings->Disk IO Rate 2000

tzselect vim /root/.bash_profile TZ=‘Asia/Shanghai’ export TZ

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime clock -w

cat /etc/sysconfig/clock ZONE=“Etc/UTC”

ZONE=“GMT” ZONE=“Asia/Shanghai” ZONE=“Brazil/East”

ntpdate asia.pool.ntp.org yum install ntpd /etc/init.d/ntpd start ntpq -p

Amazon Aws EC2

| Comments

Amazon EC2 EBS

$0.12 per 1 million I/O requests

CloudWatch

find the process io statistics

iotop (Kernel >= 2.6.20)

type: r o

io.py (Kernel < 2.6.20)

dstat

dstat --top-io -d --top-bio -l
dstat -cdlmnps
dstat 1
strace -f -p pid -e trace=write

atop

cat /proc/[PID]/io

pidstat

dmesg (rhel 5.7)

sysctl vm.block_dump=1
dmesg -c
dmesg -c | cut -d: -f1 | sort | uniq -c | sort -rn| head

参考

DDOS Attack

| Comments

DDOS: Distributed Denial of Service 分布式拒绝服务 利用合理的请求造成资源过载,导致服务不可用。将正常请求放大若干倍,通过若干个网络节点同时发起攻击,达成规范效应。

网络层DDOS TCP三次握手连接过程中发起 SYN flood、UDP flood、ICMP flood SYN flood: 伪造大量的源IP地址,分别向服务器端发送大量的SYN包,服务器端会返回SYN/ACK包,因源地址是伪造,伪造IP并不会应答,服务器端没收到伪造IP的回应,会重试3-5次且等待下一个SYN Time(30-120s) 对抗SYN flood主要措施 SYN Cookie/SYN Proxy…

应用层DDOS TCP三次握手已经完成 CC(Challenge Collapaser)攻击 对一些消耗资源较大的应用页面不断发起正常的请求,以达到消耗服务器资源的目的。 应用层DDOS攻击与正常业务的界线比较模糊 对抗手段: 优化服务器、应用代码性能缓解此类攻击;在应用中限制每个客户端请求频率;优化网络架构、利用负载均衡分流缓解网站压力 资源耗尽攻击

iptables iptables -A INPUT -p tcp –dport 80 –syn -m recent –name webpool –rcheck –seconds 60 –hitcount 10 -j LOG –log-prefix ‘DDOS:’ –log-ip-options iptables -A INPUT -p tcp –dport 80 –syn -m recent –name webpool –rcheck –seconds 60 –hitcount 10 -j DROP

tengine req_limit

主动防御 nginx模块防ddos抓取 ngx_http_limit_conn_module 限制单个ip连接数 ngx_http_limit_req_module 限制单个ip每秒请求数 nginx_limit_speed_module 对ip限速

fail2ban 通过扫描log来异步判断是否用iptable封掉,性能如何需观测 封掉每120秒超过120次访问的ip /etc/fail2ban/jail.conf [http-get-dos] enabled = true port = http,https filter = nginx-bansniffer logpath = /usr/local/nginx/logs/access.log maxretry = 120 findtime = 120 bantime = 3600 action = iptables[name=HTTP, port=http, protocol=tcp]

/etc/fail2ban/filter.d/nginx-bansniffer.conf #404 [Definition]

failregex = -.- .HTTP/1. . .*$ ignoreregex =

/etc/init.d/fail2ban restart

vim /etc/fail2ban/jail.conf [DEFAULT] ignoreip = 127.0.0.1 #这里最好加上办公区的ip ,呵呵,把自己封了就太傻B 了 bantime = 600 #ban 它多久 findtime = 600 #检查多长时间内的日志 maxretry = 3 #在监控的日志中,出现了多少次就要ban 了 backend = auto #好像是说读取日志文件的算法,这里让它自动吧

全局变量,下面可以覆盖的 有很多模块,定义一个自己的吧

[nginx-iptables] enabled = true # 启用模块的意思 filter = nginx # filter.d/nginx.conf , 必须在这里面写点东西 action = iptables[name=NGINX, port=80, protocol=tcp] # iptables 的设置 mail-whois[name=nginx, dest=you@admin.com] # 报警设置,写自己的邮箱吧,要起sendmail 噢 logpath = /data/nginx/logs/t.access.log # 监控的log maxretry = 100 #10 分钟使用100 次,ban 了你!

添加 /etc/fail2ban/filter.d/nginx.conf [Definition] failregex = ^ -.”(GET|POST) \/search.json. HTTP\/.*$ # 写自己的正则吧,符合这个正则的,就算一次fail ,10分钟内100 次(我的设置),就ban 了, 可以指代ip 或 hostname 什么的 ignoreregex =

配置完成,启动服务:/etc/init.d/fail2ban start chkconfig fail2ban on

shorewall:iptables防火墙配置工具 TCP Wrapper:限制连接来源的工具 Gamin:实时监视文件或文件夹变动的工具

从User-Agent阻止spider抓取 ### Block download agents ### if ($http_user_agent ~* WebZIP|wget) { return 403; } ###

if ($args ~* “action=lostpassword$”) { deny all; }

被动防御 iptables限制同一个ip同时连接数 限制同一ip一分钟最多建立15连接,超出的连接会被丢弃 /sbin/iptables -A INPUT -p tcp –dport 80 -i eth0 -m state –state NEW -m recent –set /sbin/iptables -A INPUT -p tcp –dport 80 -i eth0 -m state –state NEW -m recent –update –secondes 60 –hitcount 15 -j DROP service iptables save

用bash统计ip访问频率,超过频率的ip放进黑名单,黑名单的ip用iptables或nginx.conf封掉一定时间,或降低其许可的访问频率,另可加白名单功能 #!/bin/sh status=netstat -na|awk ‘$5 ~ /[0-9]+:[0-9]+/ {print $5}’ |awk -F “:” — ‘{print $1}’ |sort -n|uniq -c |sort -n|tail -n 1 NUM=echo $status|awk ‘{print $1}’ IP=echo $status|awk ‘{print $2}’ result=echo “$NUM > 150″ | bc if [ $result = 1 ] then echo IP:$IP is over $NUM, BAN IT! /sbin/iptables -I INPUT -s $IP -j DROP fi

根据特征码屏蔽请求(对CC攻击效果较好)

YDoD(Yahoo! Department of Defense) appche module -> taobao tdod

监测请求,打开页面频率太高或打开太多页面,请求会被服务器临时屏蔽,提示怀疑为机器人,要求输入验证码

购买机房流量清洗服务 ddos流量清洗设备,主要还是绿盟的黑洞和isp运营商的cisco guard为主 mrtg对带宽进行监视,配合ddos防火墙报警判断ddos攻击,通过对ddos抓包的流量协议进行分析以提取部分特殊的特征码进行过滤 流量来源进行检测,短时间内某个源地址连接数、请求数、发送数据流量超过限定阈值(有安全经验的人员制定)直接通过isp deny或路由牵引 纯流量DDOS到了出口网关就无解 纯暴力 安全宝或360网站卫士缓解DDOS攻击(基于CDN加入侵防御),阿里出类似服务 自建CDN防御DDoS(1):知己知彼,建设持久防线

大量数据包拥塞带宽 流量攻击 利用协议的缺陷,大量半连接耗尽资源 syn攻击

使用负载均衡分担流量 CDN 代理服务器引流 syn proxy 通过分析日志了解攻击规律 ip url User-Agent

包过滤技术:过滤对外开放端口 使用dns跟踪匿名攻击 使用ngrep来处理tfnsk(假冒源地址工具)攻击

OWASP

VPS VM Benchmark

| Comments

CPU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
yum install gcc gcc-c++ make libXext-devel
yum groupinstall "Development Tools"
yum install libX11-devel mesa-libGL-devel perl-Time-HiRes
wget -c http://byte-unixbench.googlecode.com/files/unixbench-5.1.3.tgz
tar xvzf unixbench-5.1.3.tgz
cd unixbench-5.1.3
make
./Run

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.2)

   System: AY1301120423032e49762: GNU/Linux
   OS: GNU/Linux -- 2.6.18-274.12.1.el5 -- #1 SMP Tue Nov 29 13:37:46 EST 2011
   Machine: x86_64 (x86_64)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (4800.5 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 1: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (4801.6 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   16:14:24 up 1 day,  1:19,  1 user,  load average: 0.02, 0.08, 0.11; runlevel 3

------------------------------------------------------------------------
Benchmark Run: Thu Jan 17 2013 16:14:24 - 16:42:24
2 CPUs in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       12414522.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     1311.3 MWIPS (10.0 s, 7 samples)
Execl Throughput                               3455.5 lps   (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        670464.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          202450.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1318858.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1470797.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  91449.2 lps   (10.0 s, 7 samples)
Process Creation                              12374.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   5983.4 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1026.3 lpm   (60.0 s, 2 samples)
System Call Overhead                        3195835.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   12414522.0   1063.8
Double-Precision Whetstone                       55.0       1311.3    238.4
Execl Throughput                                 43.0       3455.5    803.6
File Copy 1024 bufsize 2000 maxblocks          3960.0     670464.9   1693.1
File Copy 256 bufsize 500 maxblocks            1655.0     202450.0   1223.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1318858.5   2273.9
Pipe Throughput                               12440.0    1470797.5   1182.3
Pipe-based Context Switching                   4000.0      91449.2    228.6
Process Creation                                126.0      12374.4    982.1
Shell Scripts (1 concurrent)                     42.4       5983.4   1411.2
Shell Scripts (8 concurrent)                      6.0       1026.3   1710.4
System Call Overhead                          15000.0    3195835.3   2130.6
                                                                   ========
System Benchmarks Index Score                                        1022.8

------------------------------------------------------------------------
Benchmark Run: Thu Jan 17 2013 16:42:24 - 17:10:24
2 CPUs in system; running 2 parallel copies of tests

Dhrystone 2 using register variables       24106209.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     2600.2 MWIPS (9.9 s, 7 samples)
Execl Throughput                               7120.0 lps   (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        144642.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           43323.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        357576.4 KBps  (30.0 s, 2 samples)
Pipe Throughput                             2854960.4 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 664756.7 lps   (10.0 s, 7 samples)
Process Creation                              23556.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7865.5 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1324.0 lpm   (60.0 s, 2 samples)
System Call Overhead                        4307758.4 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   24106209.9   2065.7
Double-Precision Whetstone                       55.0       2600.2    472.8
Execl Throughput                                 43.0       7120.0   1655.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     144642.8    365.3
File Copy 256 bufsize 500 maxblocks            1655.0      43323.4    261.8
File Copy 4096 bufsize 8000 maxblocks          5800.0     357576.4    616.5
Pipe Throughput                               12440.0    2854960.4   2295.0
Pipe-based Context Switching                   4000.0     664756.7   1661.9
Process Creation                                126.0      23556.0   1869.5
Shell Scripts (1 concurrent)                     42.4       7865.5   1855.1
Shell Scripts (8 concurrent)                      6.0       1324.0   2206.6
System Call Overhead                          15000.0    4307758.4   2871.8
                                                                   ========
System Benchmarks Index Score                                        1189.1

Harddisk

1
2
yum install hdparm
yum install seeker

aws

1
2
3
4
5
6
7
hdparm -t /dev/xvda1
/dev/xvda1:
 Timing buffered disk reads:  212 MB in  3.00 seconds =  70.62 MB/sec
dd if=/dev/zero of=/tmp/output.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 57.8857 s, 37.1 MB/s

aliyun

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
hdparm -t /dev/hda

/dev/hda:
 Timing buffered disk reads:  466 MB in  3.00 seconds = 155.26 MB/sec

hdparm -t /dev/xvdb

/dev/xvdb:
 Timing buffered disk reads:  106 MB in  3.01 seconds =  35.17 MB/sec

dd if=/dev/zero of=/tmp/output.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 57.9623 seconds, 37.0 MB/s


[root@AY12081910044995c6294 ~]# dd if=/dev/zero of=/tmp/output.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 76.9027 s, 27.9 MB/s

seeker /dev/hda
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/hda [41943040 blocks, 21474836480 bytes, 20 GB, 20480 MB, 21 GiB, 21474 MiB]
[512 logical sector size, 512 physical sector size]
[1 threads]
Wait 30 seconds..............................
Results: 282 seeks/second, 3.545 ms random access time (510310 < offsets < 21474400900)

seeker /dev/xvdb
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/xvdb [482344960 blocks, 246960619520 bytes, 230 GB, 235520 MB, 246 GiB, 246960 MiB]
[512 logical sector size, 512 physical sector size]
[1 threads]
Wait 30 seconds..............................
Results: 1410 seeks/second, 0.709 ms random access time (5662600 < offsets < 246954366165)

IO

1
2
3
4
5
6
7
8
wget http://www.iozone.org/src/current/iozone3_414.tar
tar xvf iozone3_414.tar
cd iozone3_414/src/current
make
./iozone
./iozone  -i 0 -r 32 -s 2097152
./iozone -a -n 512m -g 4g -i 0 -i 1 -i 5 -f /app/iozone -Rb ./iozone.xls
./iozone -Mcew -i0 -i1 -i2 -s4g -r256k -f /app/io.tmp

Memcache

yum install mbw

Network

yum install iperf iperf -s iperf -c 192.168.1.23

DB

Vim Tips

| Comments

:set nu 打开显示行号

单行复制 :8t. -> 把第8行复制到当前行 :t8 把当前行复制到第8行下

多行复制 :8,10t. 把8到10行复制到当前行下

:E 浏览目录和文件 Ctrl-O跳回

:map < MiddleMouse > < Nop > 禁止单击 :map <2-MiddleMouse> < Nop > 禁止双击

:e :x :Q zz

注释某行 :14s/^/#/

Puppet Install Module Example42

| Comments

Pre-Install Installing Puppet Post-Install Use Puppet

Pre-Install

OS: CentOS 5.7 Agent/Master: Puppet 3.0

pre-puppet-2.6 post-puppet-2.6 puppetmasterd puppet master puppetd puppet agent puppet puppet apply puppetca puppet cert ralsh puppet resource puppetrun puppet kick puppetqd puppet queue filebucket puppet filebucket puppetdoc puppet doc pi puppet describe

Network:

Firewalls: master –8139/tcp-> agent master <-manifests|8140/tcp– agent file <-8140/tcp– agent

lokkit -p 8140:tcp
lokkit -p 8139:tcp

Name resolution: /etc/hosts

FQDN

fqdn.sh

On Master Agent Nodes

vi /etc/hosts
192.168.1.26 master master.test.com localhost
192.168.1.25 client client.test.com localhost

Master

hostname master.test.com

Agent

hostname client.test.com
ntp

On Master Agent Nodes

yum install ntp -y
chkconfig ntpd on
ntpdate pool.ntp.org
service ntpd start

Installing Puppet

1. Choose a Package Source

Using Puppet Labs’ Packages

wget http://yum.puppetlabs.com/el/5/products/x86_64/puppetlabs-release-5-6.noarch.rpm
gpg --recv-key 4BD6EC30
gpg --list-sigs 4BD6EC30
gpg --list-key --fingerprint 4BD6EC30
gpg -a --export 4BD6EC30 > /tmp/key
rpm --import /tmp/key
yum install puppetlabs-release-5-6.noarch.rpm -y

Using EPEL

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
yum install git

2. Install the Puppet Master

On Master Node

yum install puppet-server -y
chkconfig puppet on
service puppetmaster start

3. Install Puppet on Agent Nodes

On Agent Nodes

yum install puppet -y

Post-Install

Configure Puppet

Sign Node Certificates

On Agent Nodes [agent] server: puppet report: true pluginsync: true

vim /etc/puppet/puppet.conf
[agent]
server = master.test.com

ls /var/lib/puppet/
puppet agent --no-daemonize --onetime --verbose --debug
puppet agent --no-daemonize --onetime --verbose --debug --server=master.test.com

On Puppet Master

puppet cert list --all
puppet cert --sign client.test.com
puppet cert --debug --verbose --sign client.test.com
puppet cert list --all

Revoked Certificates

On Puppet Master

puppet cert revoke client.test.com
puppet cert --clean client.test.com
/etc/init.d/puppetmaster restart

On Puppet Nodes

rm -rf /var/lib/puppet/ssl

puppet agent -t

Autosign

On Puppet Master

vim /etc/puppet/puppet.conf
[main]
autosign=true
autosign = /etc/puppet/autosign.conf
cat > /etc/puppet/autosign.conf <<EOF
*.test.com
EOF
Test Manually

On Puppet Master

vim /etc/puppet/manifests/site.pp
node default {
        file {
                "/tmp/helloworld.txt": content => "hello, world\n";
        }
}

On Agent Node

puppet agent --test
puppet agent --test --server=master.test.com
cat /tmp/helloworld.txt
puppet agent --noop --test --server=master.test.com

Start and Enable the Puppet Services

puppet resource service puppet ensure=running enable=true
puppet resource service puppetmaster ensure=running enable=true

Using Puppet

Puppet Command

puppet --version
puppet config print modulepath
puppet agent -t --summarize 
puppet agent --no-daemonize --onetime --verbose --debug

Puppet Dashboard

yum install puppet-dashboard
/etc/init.d/mysqld start  #启动数据库
/usr/bin/mysqladmin -u root password 'new-password'
cd /usr/share/puppet-dashboard/
vim config/database.yml
production:
  database: dashboard_production
  username: dashboard
  password: 654321
  encoding: utf8
  adapter: mysql
  host: 192.168.1.26
  port: 63306
vim /usr/share/puppet-dashboard/config/environment.rb
#config.time_zone = 'UTC'
config.time_zone = 'Beijing'

rake RAILS_ENV=production db:create
rake RAILS_ENV=production db:migrate

On Puppet Master

vim /etc/puppet/puppet.conf
[main]
reports = store, http
reporturl = http://localhost:3000/reports/upload

/etc/init.d/puppetmaster start 
/etc/init.d/puppet-dashboard start
/etc/init.d/puppet-dashboard-workers start

On Agent Nodes

vim /etc/puppet/puppet.conf
[agent]
server = master.test.com
report = true
listen = true
runinterval = 600
vim /etc/puppet/auth.conf
path /run
method save
allow master.test.com
path /
auth any

/etc/init.d/puppet start

rake RAILS_ENV=production reports:import
rake RAILS_ENV=production reports:import EPORT_DIR=/path/to/your/reports
rake RAILS_ENV=production db:raw:optimize
rake RAILS_ENV=production reports:prune upto=1 unit=mon
rake RAILS_ENV=production FILE=/my/backup/file.sql db:raw:dump
rake RAILS_ENV=production FILE=/my/backup/file.sql db:raw:restore

Puppet Modules example42

apache, nginx, varnish php, ruby, tomcat mysql, pgsql, memcache

git clone --recursive -b 1.0 git://github.com/example42/puppet-modules.git
git clone --recursive git://github.com/example42/puppet-modules.git

On Puppet Master

cd /etc/puppet
git clone --recursive git://github.com/example42/puppet-modules-nextgen.git
vim puppet.conf
[master]
    modulepath = /etc/puppet/puppet-modules-nextgen
cat puppet-modules-nextgen/nginx/README.rdoc

vim manifests/site.pp 
node default {
        class { 'nginx':
        }        
}

node default {
        class { 'nginx':
        }
        class { 'apache':
                disable => true
        }
        class { 'php':
        }
        php::module { "pdo":
        }
        php::module { "gd":
        }
        php::module { "fpm":
        }
        php::module { "mysql":
        }
        php::module { "soap":
        }
        php::module { "zts":
        }
        php::module { "pecl-apc":
        }
        php::module { "pecl-memcache":
        }
        class { 'mysql':
        }
}

Install PHP-5.3 MySQL-5.5 Nginx-1.0

On Puppet Master

vim puppet-modules-nextgen/mysql/manifests/client.pp
 package { 'mysql55-client':
vim puppet-modules-nextgen/mysql/manifests/params.pp
    default => 'mysql55-server',

change httpd to nginx

vim puppet-modules-nextgen/php/params.pp
    default                   => 'nginx',

On Agent Node

rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
vim /etc/yum.repos.d/webtatic.repo
enabled=1
puppet agent --test --server=master.test.com

On Puppet Master

puppet kick -t c1.test.com

On Agent Node

/etc/init.d/nginx start
/etc/init.d/php-fpm start
/etc/init.d/mysql start

cat > /usr/share/nginx/html/phpinfo.php <<EOF
<?php phpinfo(); ?>
EOF

php /usr/share/nginx/html/phpinfo.php
php -m

Production Modules

base common firewall iptables ntp bind ssh users rsyslog sudo yum snmp puppet lvs haproxy hearbeat nginx php tomcat java mysql memcached redis cacti nagios nrpe vagrant

Specifying Puppet Nodes

cat /etc/puppet/manifests/site.pp
import "classes/*"

## Base Nodes

node default {
    include sudo
    include sshkeys
}

node appserverbasic {
    include django
    include apacheconf
    include app
}

node loadbalancer {
    include nginxlb
    include monitoring
}

## Specific Nodes

node 'fore.ducklington.org' inherits loadbalancer {
    include django
    include apacheconf
    include app
    include backups
}

node 'lb1.ducklington.org' inherits loadbalancer {
}

node 'lollipop.ducklington.org' inherits appserverbasic {
    include monitoring
    include backups
}

node 'test.lollipop.ducklington.org' inherits appserverbasic {
}

node 'monitoring1.ducklington.org', 'monitoring2.ducklington.org' {
    include monitoring
    include monitoringhub
}

CentOS REPO

| Comments

EPEL

EPEL Fedora 社区创建维护,含有大量软件,对官方标准源是很好的补充

Installing RHEL EPEL Repo on Centos 5.x or 6.x

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

rpm -Uvh http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

WIKI Fedora EPEL 下载 EPEL 下载

Remi

Remi最新稳定版 Remi 下载 rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-5.rpm

RPMForge

RPMForge CentOS系统下的软件仓库,CentOS社区认为最安全最稳定的一个软件仓库

rpm -Uvh http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm

rpm -Uvh http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

rpm -i http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el5.rf.x86_64.rpm rpm -i http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm yum –enablerepo=rpmforge-extras install git-1.7.12.4-1.el6.rfx.x86_64

yum –disablerepo=base,updates –enablerepo=rpmforge-extras install git

RPMFusion

RPMFusion Fedora 各种音频软件 稳定性不如rpmforge RPMFusion

Webtatic

PHP 5.4 on CentOS/RHEL 6 PHP 5.3 on CentOS/RHEL 5 MySQL 5.1 and 5.5 mpm-itk Git 1.7 Gearmand 0.14 and PHP Gearman pecl extension Facebook XHP

rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm yum install –enablerepo=webtatic package-name

rpm -Uvh http://repo.webtatic.com/yum/el6/latest.rpm yum install package-name

dell yum

wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash yum install srvadmin-all firmware-tools

源安装优先级

各软件库并不保证完全兼容且没有冲突 yum-priorities 设置priority 官方标准源优先级为1,最高,第三方推荐>10

priority=N (N为1到99的正整数,数值越小越优先) enabled=0 关闭 enabled=1 启用

常用命令

yum repolist yum –enablerepo=epel info zabbix

本地源

Learn Shell

| Comments

bash命令处理流程

                           +-------------+           单引号  
|------------------------->|             |--------------------------|  
|  ----------------------->| 1.分隔成记号|---- ---------------|     |  
|  |   ------------------->|             |      双引号        |     |  
|  |   |                   +-------------+                    |     |  
|  |   |                          ||                          |     |  
|  |   |读取下一个命令            \/                          |     |  
|  |   |     +-------------------------------------------+    |     |  
|  |   |     |                    2.                     |    |     |  
|  |   ------|              检验第一个记号               |    |     |  
|  |         |开放的关键字                    其他关键字 |    |     |  
|  |         |               非关键字                    |    |     |  
|  |         +-------------------------------------------+    |     |  
|  |                              ||                          |     |  
|  |                              \/                          |     |  
|  |            +-----------------------------+               |     |  
|  |  扩展别名  |           3. 检验第一个记号 |               |     |  
|  |------------|  别名                       |               |     |  
|               |              不是别名       |               |     |  
|               +-----------------------------+               |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|                           +--------------+                  |     |  
|                           | 4.大括号扩展 |                  |     |  
|                           +--------------+                  |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|                           +--------------+                  |     |  
|                           | 5.~符号扩展  |                  |     |  
|                           +--------------+                  |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|                           +--------------+       双引号     |     |  
|                           |  6.参数扩展  |<-----------------|     |  
|                           +--------------+                        |  
|                                 ||                                |  
|                                 \/                                |  
|                    +------------------------------+               |  
|                    |  7.命令替换(嵌套命令行处理)  |               |  
|                    +------------------------------+               |  
|                                 ||                                |  
|                                 \/                                |  
|                           +--------------+      双引号            |  
|                           |  8.算术扩展  |------------------|     |  
|                           +--------------+                  |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|                           +--------------+                  |     |  
|                           |  9.单词分割  |                  |     |  
|                           +--------------+                  |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|                           +--------------+                  |     |  
|                           | 10.路径名扩展|                  |     |  
|                           +--------------+                  |     |  
|                                 ||                          |     |  
|                                 \/                          |     |  
|               +----------------------------------------+    |     |  
|               | 11.命令查寻:函数,内置命令,可执行文件|<---|-----|  
|               +----------------------------------------+  
|                                 ||  
|                                 \/  
|将参数带入下一个命令        +-------------+  
|----------eval--------------| 12.运行命令 |  
                             +-------------+  

bash调试

cat -n test.sh

检查语法错误 bash -n test.sh

显示行号 export PS4=‘+[$LINENO]’ export PS4=‘+${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]}: ’ vim .bash_profile

跟踪整个脚本执行过程 bash -x test.sh

缩小排查错误范围 利用trap,调试钩子等输出关键调试信息,使用set -x及set +x对脚本片段进行重点跟踪

trap trap ‘command’ signal signal: EXIT ERR DEBUG trap ‘command’ EXIT trap ‘command’ 0

tee 将输入输出重定向某个文件进行查看调试

调试脚本片段 set -x 调试脚本 set +x

使用调试钩子

cat test.sh
#!/bin/bash
debug() {
    if [ "$_DEBUG" == "true" ]; then
        echo 1>&2 "$@"
    fi
}

debug "debuging…"
echo "test"

debug=true ./test.sh

DEBUG(){ [ "$DEBUG" = 0 ] && { echo "${BASH_LINENO}:$@ _"; $@; }; }
DEBUG set -x
DEBUG set +x

bash执行选项 -n 解析脚本,但不执行,可检查语法错误 -x 进行跟踪模式,显示执行的每一条命令 -c “string” 从string读取命令

-x选项增加 $LINENO 当前行号 $FUNCNAME 函数名称 ${FUNCNAME[0]} 当前正在执行的函数名称 ${FUNCNAME[1]} 调用函数${FUNCNAME[0]}的函数名称 $PS4 缺省是+

脚本复杂可使用开源调试器bashdb进行调试 bashdb

调试参考

学习资源

技巧

代码阅读