Saturday, March 23, 2013

Monitoring Performance in Solaris


This chapter describes procedures for monitoring system performance by using the vmstat, iostat, df, andsar commands. This is a list of the step-by-step instructions in this chapter.
How to Display Virtual Memory Statistics (vmstat)
The following example shows the vmstat display of statistics gathered at five-second intervals.
$ vmstat 5
 procs    memory            page             disk      faults     cpu
r b w  swap free re  mf  pi  po  fr de sr f0 s3 -- --  in  sy  cs us sy  id
0 0 8 28312  668  0   9   2   0   1  0  0  0  1  0  0  10  61  82  1  2  97
0 0 3 31940  248  0  10  20   0  26  0 27  0  4  0  0  53 189 191  6  6  88
0 0 3 32080  288  3  19  49   6  26  0 15  0  9  0  0  75 415 277  6 15  79
0 0 3 32080  256  0  26  20   6  21  0 12  1  6  0  0 163 110 138  1  3  96
0 1 3 32060  256  3  45  52  28  61  0 27  5 12  0  0 195 191 223  7 11  82
0 0 3 32056  260  0   1   0   0   0  0  0  0  0  0  0   4  52  84  0  1  99
Category
Field Name
Description
procs
  
Reports the following states:

r
The number of kernel threads in the dispatch queue

b
Blocked kernel threads waiting for resources

w
Swapped out LWPs waiting for processing resources to finish
memory

Reports on usage of real and virtual memory:

swap
Available swap space

free
Size of the free list
page

Reports on page faults and paging activity, in units per second:

re
Pages reclaimed

mf
Minor and major faults

pi
Kbytes paged in

po
Kbytes paged out

fr
Kbytes freed

de
Anticipated memory needed by recently swapped-in processes

sr
Pages scanned by page daemon (not currently in use). If sr does not equal zero, the page daemon has been running.
disk

Reports the number of disk operations per second, showing data on up to four disks
faults

Reports the trap/interrupt rates (per second):

in
Interrupts per second

sy
System calls per second

cs
CPU context switch rate
cpu

Reports on the use of CPU time:

us
User time

sy
System time

id
Idle time
How to Display System Event Information
Run vmstat -s to show the total of various system events that have taken place since the system was last booted.
        0 swap ins
        0 swap outs
        0 pages swapped in
        0 pages swapped out
409376480 total address trans. faults taken
  3075036 page ins
  2601555 page outs
  3812452 pages paged in
  6525552 pages paged out
 11007609 total reclaims
 10927650 reclaims from free list
        0 micro (hat) faults
409376480 minor (as) faults
  2957386 major faults
102738273 copy-on-write faults
 61711047 zero fill page faults
1002562077 pages examined by the clock daemon
     7881 revolutions of the clock hand
 16716370 pages freed by the clock daemon
  4999048 forks
  1138206 vforks
  5747009 execs
741660225 cpu context switches
736047593 device interrupts
528054538 traps
2496638575 system calls
430283487 total name lookups (cache hits 95%)
    81727 toolong
 10484677 user   cpu
  9528364 system cpu
443762786 idle   cpu
 16281790 wait   cpu
How to Display Swapping Statistics
Run vmstat -S to show swapping statistics.
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m1 m3 m4 m5   in   sy   cs us sy id
 0 0 0   8512   888   0   0 12 21 55  0 417 1  0  0  0  206 1040  308  2  2 96
si = Average number of LWPs swapped in per second
so = Number of whole processes swapped out
How to Display Disk Utilization Information (iostat)
You can display disk activity information by using the iostat command with a time interval. The following example shows disk statistics gathered every five seconds.
iostat 5
      tty          md1           md3           md4           md5          cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv  us sy wt id
   0    2  10   1   28    2   0   22    0   0    0    1   0   10   2  2  3 92
   0   47  58   7   39   16   2   34    0   0    0    0   0    0   0  2 19 78
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  1  0 98
   0   16   0   0    0    0   0    0    0   0    0    0   0    0   0  0  1 99
   0   16   2   0   22    0   0    0    0   0    0    0   0    0   2  3  1 95
   0   24   0   0    0    0   0    0    0   0    0    0   0    0   0  1  1 98
For Each ...
Field Name
Description
Terminal



tin
Number of characters in the terminal input queue

tout
Number of characters in the terminal output queue
Disk



bps
Blocks per second

tps
Transactions per second

serv
Average service time, in milliseconds
CPU



us
In user mode

sy
In system mode

wt
Waiting for I/O

id
Idle
How to Display Extended Disk Statistics
Run iostat -xtc to get extended disk statistics. This command displays a line of output for each disk.
                               extended device statistics      tty         cpu
device    r/s  w/s   kr/s   kw/s wait actv  svc_t  %w  %b  tin tout us sy wt id
md1       0.4  0.9    3.6    6.9  0.0  0.0   27.7   1   1    0    2  2  2  3 92
md3       0.1  0.2    1.0    1.3  0.0  0.0   21.7   0   0
md4       0.0  0.0    0.0    0.0  0.0  0.0    0.0   0   0
md5       0.0  0.0    0.7    0.0  0.0  0.0    9.9   0   0
md8       0.8  0.3    6.7   14.2  0.0  0.0   13.1   0   1
md10      0.2  0.9    1.8    6.8  0.0  0.0   15.5   0   1
md11      0.2  0.9    1.8    6.8  0.0  0.0   14.8   0   1
md30      0.0  0.2    0.5    1.3  0.0  0.0   11.4   0   0
md31      0.0  0.2    0.5    1.3  0.0  0.0   10.2   0   0
md40      0.0  0.0    0.0    0.0  0.0  0.0    0.0   0   0
md41      0.0  0.0    0.0    0.0  0.0  0.0    0.0   0   0
md50      0.0  0.0    0.4    0.0  0.0  0.0    9.4   0   0
md51      0.0  0.0    0.4    0.0  0.0  0.0    7.3   0   0
md80      0.4  0.3    3.3   14.2  0.0  0.0   10.3   0   0
md81      0.4  0.3    3.3   14.2  0.0  0.0   11.7   0   1
sd0       0.6  2.1    6.0   22.8  0.0  0.0   16.3   0   3
sd1       0.6  2.1    6.0   22.8  0.0  0.0   15.2   0   2
Field Name
Description
r/s
Reads per second
w/s
Writes per second
Kr/s
Kbytes read per second
Kw/s
Kbytes written per second
wait
Average number of transactions waiting for service (queue length)
actv
Average number of transactions actively being serviced
svc_t
Average service time, in milliseconds
%w
Percentage of time the queue is not empty
%b
Percentage of time the disk is busy
How to Check CPU Utilization (sar)
Display CPU utilization with the sar -u command. (The sar command without any options is equivalent to sar -u.) At any given moment, the processor is either busy or idle. When busy, the processor is in either user or system mode. When idle, the processor is either waiting for I/O completion or "sitting still" with no work to do.
Measure CPU utilization during 5 secs one time.
sar -u 5 1
Measure CPU utilization during 60 secs 1440 times and write result in file sar.log.
sar -u -o sar.log 60 1440 
To later review disk and tape activity from that period:
sar -d -f sar.log
Field Name
Description
%sys
Lists the percentage of time that the processor is in system mode
%user
Lists the percentage of time that the processor is in user mode
%wio
Lists the percentage of time the processor is idle and waiting for I/O completion
%idle
Lists the percentage of time the processor is idle and is not waiting for I/O
A high %wio generally means a disk slowdown has occurred.

Disable SSH login for individual users


1.)Edit the /etc/ssh/sshd_config file

Add the user names after “DenyUsers” string which you want disable .

Example :
DenyUsers ftpuser fdpuser2

2.) Restart the ssh service

For Solaris 10
--------------
svcadm refresh svc:/network/ssh:default

For Solaris 7,8 & 9
------------------
/etc/init.d/sshd restart

Solaris 10 - Increasing Number of Processes Per User


The below example is given for increasing the number of processes on Solaris 10 system on PER UID. The hardware used here is UltraSPARC T2 based system with Solaris 10 and 32 GB RAM.

We needed to increase the number of processesper user to more than current setting of 30000

bash-3.00# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 260000
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 29995
virtual memory (kbytes, -v) unlimited

Trying to increase the "max user processes" would fail with the following error:

bash-3.00# ulimit -u 50000
bash: ulimit: max user processes: cannot modify limit: Invalid argument
bash-3.00#

After going through the Solaris 10 Tunable Guide for Process sizing learned that there are 5 related parameters related to process sizing.

maxusers - The maximum number of processes on the system, The number of quota structures held in the system. The size of the directory name look-up cache (DNLC)
reserved_procs - Specifies the number of system process slots to be reserved in the process table for processes with a UID of root
pidmax - Specifies the value of the largest possible process ID. Specifies the value of the largest possible process ID. Valid for Solaris 8 and later releases.
max_nprocs - Specifies the maximum number of processes that can be created on a system. Includes system processes and user processes. Any value specified in /etc/system is used in the computation of maxuprc.
maxuprc - Specifies the maximum number of processes that can be created on a system by any one user

Looked at the current values for these parameter:

bash-3.00# echo reserved_procs/D | mdb -k
reserved_procs:
reserved_procs: 5

bash-3.00# echo pidmax/D| mdb -k
pidmax:
pidmax: 30000

bash-3.00# echo maxusers/D | mdb -k
maxusers:
maxusers: 2048
bash-3.00#

bash-3.00# echo max_nprocs/D | mdb -k
max_nprocs:
max_nprocs: 30000
bash-3.00#

bash-3.00# echo maxuprc/D| mdb -k
maxuprc:
maxuprc: 29995

So, in order to set the max per user processes in this scenario, we were required to make the changes to "pidmax" (upper cap), maxusers, max_nprocs & maxuprc
Sample entries in /etc/system & reboot


set pidmax=60000
set maxusers = 4096
set maxuprc = 50000
set max_nprocs = 50000

After making the above entries, we were able to increase the max user processes to 50000.
bash-3.00# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 260000
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 49995
virtual memory (kbytes, -v) unlimited
bash-3.00#

bash-3.00# echo reserved_procs/D |mdb -k
reserved_procs:
reserved_procs: 5
bash-3.00# echo pidmax/D |mdb -k
pidmax:
pidmax: 60000
bash-3.00# echo max_nprocs/D |mdb -k
max_nprocs:
max_nprocs: 50000
bash-3.00# echo maxuprc/D | mdb -k
maxuprc:
maxuprc: 50000
bash-3.00#

Note: If you are operating within the 30000 limit (default pidmax setting) the blog entry referred above seems to work fine. If you are looking at increasing the processes beyond 30000, it we need to make adjustment to other dependent parameters stated in this blog entry.

Add new printer queue on solaris server using lpadmin command


Add new printer queue on solaris server using lpadmin command

Add new printer queue on solaris server using lpadmin command
1.) Add printer name and port details on /etc/printers.conf file
Eg:
mtysap:\
:bsdaddr=mtyun118,mtysap,Solaris:

mtysapq:\
:bsdaddr=mtysun118,mtysapq,Solaris:

mtysapt:\
:bsdaddr=mtysun118,mtysapt,Solaris:

In above mtysap is Print queue name and mtysun118 is Server name.
2.) Copy the existing interface configuration file to new printer name
Eg:
cp /etc/lp/interfaces/mtysap /etc/lp/interfaces/mtysaptest

3.) Copy the existing configuration file from file to new printer name
Eg:
mkdir /etc/lp/printers/mtysaptest
cp –p /etc/lp/printers/ mtysap/* /etc/lp/printers/mtysaptest
ls –l /etc/lp/printers/mtysaptest
total 10
-rwxrwx--- 1 lp lp 1347 Aug 2 2006 alert.sh
-rw-rw---- 1 lp lp 4 Aug 2 2006 alert.vars
-rw-rw-r-- 1 lp lp 168 Aug 2 2006 configuration
-rw-rw-r-- 1 lp lp 16 Aug 2 2006 faultMessage
-rw-rw-r-- 1 lp lp 0 Aug 2 2006 users.deny

4.) Edit new printer configuration file and replace the IP address with IP of your network printer
Eg:
vi /etc/lp/printers/mtysaptest/configuration

# cat /etc/lp/printers/ mtysaptest /configuration
Banner: on
Content types: any
Device: /dev/null
Interface: /usr/lib/lp/model/netstandard
Printer type: unknown
Modules:
Options: protocol=tcp,dest=134.200.172.26:9100

5.) Change the ownership of files
chmod 775 /etc/lp/interfaces/ mtysaptest
chown lp:lp /etc/lp/interfaces/ mtysaptest
chown-R lp:lp /etc/lp/printers/mtysaptest

6.) Create and enable the printer
eg:
lpadmin -p mtysaptest -v /dev/null -i /etc/lp/interfaces/ mtysaptest
accept mtysaptest
enable mtysaptest

7.) Check the new printer status
Eg:
# lpstat -p mtysaptest

printer mtysaptest is idle. enabled since Fri 28 Jan 2011 01:47:21 PM GMT. available.

Note : Another way you can also run below script after adding printer in /etc/printers.conf

#!/bin/ksh
# execute the script followed by the printer name and IP.
/usr/sbin/lpadmin -p $1 -v /dev/null -A write -i /usr/lib/lp/model/ne
tstandard -o dest=$2 -o protocol=bsd -o nobanner -I simple,postscript -u allow:all
/usr/bin/enable $1
/usr/sbin/accept $1
echo "Printer "$1" created!"
echo "Printer configuration..."
/usr/bin/lpstat -lp $1
exit

Netmask Conversions


Netmask Conversions

If you have ever needed to know what a netmask looks like expressed in some other format this table of equivalents should help. It contains common IPv4 netmasks expressed in four different formats.



Add interface specific route


Add interface specific route

In solaris , you can add a route whose traffic should go out a specific interface by adding -ifp [ifname] to the route command line. For instance, suppose a host has two interfaces (eri0 and hme0) on the same IP subnet (10.4.2.9/24 with gateway 10.4.2.254), and traffic for just a few hosts needs to go out the secondary hme0 interface. One reason this setup may be needed is for monitoring both some firewalls and the apps that those firewalls protect from a single network management station. On the firewalls you would add host-specific routes for the network management station’s secondary interface via the firewall management network, allowing that interface to talk directly to the firewalls. The primary interface of the network management station gets routed normally, though, and so is able to talk to hosts protected by the same firewalls.


The following command makes this happen:

# route add -host 172.29.4.3 10.4.2.254 -ifp hme0

add host 172.29.4.3: gateway 10.4.2.254

# route add -host 172.29.4.4 10.4.2.254 -ifp hme0

add host 172.29.4.4: gateway 10.4.2.254

# route add -host 172.29.7.31 10.4.2.254 -ifp hme0

add host 172.29.7.31: gateway 10.4.2.254

# route add -host 172.29.7.32 10.4.2.254 -ifp hme0

add host 172.29.7.31: gateway 10.4.2.254

Now all traffic for the four hosts above will go out hme0 instead of eri0.

Wednesday, March 20, 2013

snmpxdmid: Error in adding row for subscription Table Entry




snmpxdmid: Error in adding row for subscription Table Entry

Solution :-


This is solaris 9 problem. The snmpdx startup scripts are in /etc/rc3.d

  If you are not using SNMP to monitor the system, disable them.  Particularly if you are not having completely up to date on security  patches for snmpdx, mibiisa, and worse, the snmpXdmid.  There
  are buffer overflow and other exploits in unpatched or  poorly configured daemons.

  If you are running snmp, you still don't need the snmpXdmid,  as that is a management interface that just happens to be  bundled in the free solstice enterprise agent and runs as  a sub agent under the snmpdx daemon.

Disable it...

   cd /etc/rc3.d
   ./S76snmpdx stop
   ./S77dmi stop
   mv S76snmpdx s76snmpdx_safe
   mv S77dmi s77dmi_safe

Tuesday, March 19, 2013

sendmail:Unable to qualify my own domain name in Solaris 10


How to resolve "Unable to qualify my own domain name" error


You may have received this error in /var/adm/messages

Symptom 1:

tail -f  /var/adm/messages
Mar 13 11:15:16 myserver sendmail[8420]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:15:16 myserver sendmail[8421]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:16:16 myserver sendmail[8421]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name
Mar 13 11:16:16 myserver sendmail[8420]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name

Symptom 2:

when you try to manually telnet localhost 25 or mailx, either you can send to a particular domain or the respond is very slow after entering the "mail from" command.

Resolution:

update /etc/hosts ( and /etc/inet/ipnodes if you are on solaris 10) to use the following.
root@myserver:>more /etc/hosts
# internet host table
#====================
127.0.0.1 localhost
10.106.127.105 myserver. myserver
10.106.63.32 myserver-rsc

Description:
Usually i would add the "myserver." with the dot behind to /etc/hosts but in Solaris 10, things work a little differently.

It turn out that in solaris 10, the OS will go through /etc/inet/ipnodes for IPv4 address before going to /etc/hosts.

In this case, since ldap does not have the entry, OS will go straight to /etc/inet/ipnodes.

This also mean that if you change the host IP of the solaris 10 server, please change in /etc/inet/ipnodes as well otherwise you have conflict of IP addresses.

Note that this is Solaris specific.

# man ipnodes
...
...
NOTES
IPv4 addresses can be defined in the ipnodes file or in the
hosts file. See hosts(4). The ipnodes file will be searched
for IPv4 addresses when using the getipnodebyname(3SOCKET)
API. If no matching IPv4 addresses are found in the ipnodes
file, then the hosts file will be searched. To prevent
delays in name resolution and to keep /etc/inet/ipnodes and
/etc/inet/hosts synchronized, IPv4 addresses defined in the
hosts file should be copied to the ipnodes file.
...
..

The other ways to disable the message is following :-


Disable the sendmail service in Solaris 10;

*1)* Check that SendMail is under SMF control (older Solaris 10 releases didn't have SMF, if I recall correctly)
root@blade ~# svcs -l svc:/network/smtp:sendmail
fmri         svc:/network/smtp:sendmail
name         sendmail SMTP mail transfer agent
enabled      true
state        online
next_state   none
state_time   Fri Jun 06 01:15:58 2008
logfile      /var/svc/log/network-smtp:sendmail.log
restarter    svc:/system/svc/restarter:default
contract_id  142 
dependency   require_all/refresh file://localhost/etc/mail/sendmail.cf (online)
dependency   require_all/refresh file://localhost/etc/nsswitch.conf (online)
dependency   optional_all/none svc:/system/filesystem/autofs (online)
dependency   require_all/none svc:/system/filesystem/local (online)
dependency   require_all/none svc:/network/service (online)
dependency   require_all/refresh svc:/milestone/name-services (online)
dependency   optional_all/refresh svc:/system/identity:domain (online)
dependency   optional_all/none svc:/system/system-log (online)
root@blade ~ # 
*2)* Disable the SendMail service;
root@blade ~ # ps -ef|grep sendmail
root  2669     1   0 01:16:59 ?           0:00 /usr/lib/sendmail -bd -q15m -C /etc/mail/local.cf
smmsp  2667     1   0 01:16:59 ?           0:00 /usr/lib/sendmail -Ac -q15m
root  2950  2788   0 01:56:18 pts/1       0:00 grep sendmail
root@blade ~ # svcadm disable svc:/network/smtp:sendmail
root@blade ~ # ps -ef | grep sendmail
root@blade ~ #
However, if your interested in stopping the error without disabling SendMail, you can attempt a few things;

*1)* Set the hostname to the desired FQDN;
root@blade ~ # hostname
blade
root@blade ~ # hostname blade.example.com
root@blade ~ # hostname
blade.example.com
root@blade ~ #
*2)* Set the default domain name;
root@blade ~ # echo "example.com" > /etc/defaultdomain
root@blade ~ # cat /etc/defaultdomain
example.com
root@blade ~ #
*3)* Use the "domainname" command to set the default domain
root@blade ~ # domainname example.com
root@blade ~ # domainname
example.com
root@blade ~ #
Note: Just using the "domainname" command is not persistent across reboots, the /etc/defaultdomain file is read by the system at boot up, and it should contain just the desired domain ("example.com"). Doing a "domainname" after a reboot when the /etc/defaultdomain file is set will return the domain name.

*4)* Check the entry in /etc/hosts;
A fresh install of Solaris 10 (Update 5) left me with the following (even though I setup DNS during the OS install);
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.1.200   blade   loghost
What fixed the SendMail error for me was changing it to the following;
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.1.200   blade.example.com blade loghost 
I hope this helps!