NMON performance: A free tool to analyze AIX and Linux performance

Usage notes: This nmon tool is NOT OFFICIALLY SUPPORTED. No warrantee is given or implied, and you cannot obtain help with it from IBM. If you have a question on nmon, please go on the Performance Tools Forum site (see Resources) so that others can find and benefit from the answers. To protect your email address from junk mail, you need to create a USER ID first (takes 20 seconds at most).

The nmon tool runs on:

• AIX® 4.1.5, 4.2.0 , 4.3.2, and 4.3.3 (nmon Version 9a: This version is functionally established and will not be developed further.)
• AIX 5.1, 5.2, and 5.3 (nmon Version 10: This version now supports AIX 5.3 and POWER5™ processor-based machines, with SMT and shared CPU micro-partitions.)
• Linux® SUSE SLES 9, Red Hat EL 3 and 4, Debian on pSeries® p5, and OpenPower™
• Linux SUSE, Red Hat, and many recent distributions on x86 (Intel and AMD in 32-bit mode)
• Linux SUSE and Red Hat on zSeries® or mainframe

The nmon tool is updated roughly every six months, or when new operating system releases are available. To place your name on the e-mail list for updates, contact Nigel Griffiths.

Use this tool together with nmon analyser (see Resources), which loads the nmon output file and automatically creates dozens of graphs.

Introduction

The nmon tool is designed for AIX and Linux performance specialists to use for monitoring and analyzing performance data, including:

• CPU utilization
• Memory use
• Kernel statistics and run queue information
• Disks I/O rates, transfers, and read/write ratios
• Free space on file systems
• Disk adapters
• Network I/O rates, transfers, and read/write ratios
• Paging space and paging rates
• CPU and AIX specification
• Top processors
• IBM HTTP Web cache
• User-defined disk groups
• Machine details and resources
• Asynchronous I/O — AIX only
• Workload Manager (WLM) — AIX only
• IBM TotalStorage® Enterprise Storage Server® (ESS) disks — AIX only
• Network File System (NFS)
• Dynamic LPAR (DLPAR) changes — only pSeries p5 and OpenPower for either AIX or Linux

Also included is a new tool to generate graphs from the nmon output and create .gif files that can be displayed on a Web site.

See the README file for more details.

Benefits of the tool

The nmon tool is helpful in presenting all the important performance tuning information on one screen and dynamically updating it. This efficient tool works on any dumb screen, telnet session, or even a dial-up line. In addition, it does not consume many CPU cycles, usually below two percent. On newer machines, CPU usage is well below one percent.

Data is displayed on the screen and updated once every two seconds, using a dumb screen. However, you can easily change this interval to a longer or shorter time period. If you stretch the window and display the data on X Windows, VNC, PuTTY, or similar, the nmon tool can output a great deal of information in one place.

The nmon tool can also capture the same data to a text file for later analysis and graphing for reports. The output is in a spreadsheet format (.csv).

Installing the tool

The tool is a stand-alone binary file (a different file for each AIX or Linux version) that you can install in five seconds, probably less if you type fast. Installation is simple:

• Copy the nmonXXX.tar.Z file to the machine. If using FTP, remember to use binary mode.
  Note: Version XXX replaces this example.
• To uncompress the file, run uncompress nmonXX.tar.Z.
• To extract the files, run tar xvf nmonXX.tar.
• Read the README file.
• To start the nmon tool, type nmon.
• If you are the root user, you might need to type ./nmon.

Extra notes for using nmon 9 for AIX 4 only

1. You must be the root user or allow regular users to read the /dev/kmemfile by typing the following command (as root):
   

   chmod ugo+r /dev/kmem

2. If you want the disk statistics, then also run (as root):
   

   chdev -l sys0 -a iostat=true

How to run the tool interactively

For running the tool interactively, read the front page of the file for a few hints. Then start the tool and use the one-key commands to see the data you want. For example, to get CPU, Memory, and Disk statistics, start nmon and type:

cmd

 

How to get help information while running interactively

Press the h key.

Additional help information

For additonal help information, try the following:

• Type the nmon -? command for brief details.
• Type the nmon -h command for full details.
• Read the README file.

How to capture the data to a file for later analysis and graphing

Run nmon with the -f flag. See nmon -h for the details. But as an example, try to run nmon for an hour capturing data snapshots every 30 seconds by using:       Â

nmon -f -s 30 -c 120 nmon -fT -s 30 -c 120

 

The second line also captures the top processes. Both of these create the output file in the current directory called:       Â

<hostname>_date_time.nmon

 

This file is in a comma-separated values (CSV) format and can be imported into a spreadsheet directly. If you are using Lotus® 1-2-3, the file needs to be sorted. (This is not required for the Excel version of the nmon analyser.) On AIX, follow this example:       Â

sort -A mymachine_311201_1030.nmon > xxx.csv

 

Notes to save you time:

• To load the nmon data capture file into a spreadsheet, check the spreadsheet documentation for loading CSV data files (.csv). Many spreadsheets accept this data as just one of the possible files to load or provide an import function to do this. Many spreadsheets have a fixed number of columns and rows. I suggest you collect a maximum of 300 snapshots to avoid hitting these issues.
• When you are capturing data to a file, nmon disconnects from the shell to ensure that it continues running, even if you log out. This means that nmoncan appear to crash, even though it’s still running in the background. To see if the process is still running, type:
  

  ps ?ef | grep nmon

• Read the README file for more information about which version of nmon to run on your particular operating system.
• nmon Version 10 for AIX 5 no longer uses /dev/kmem, but only public APIs. So, you don’t have to chage the permissions on /dev/kmem, and there is no need to have 32- and 64-bit versions of nmon.
• For AIX 5.1, 5.2, and 5.3, use nmon 10.
• On AIX, don’t report lslpp -Lcq bos.?p core dumps on AIX 5.1, about ML03 onwards. Also, WLM stats go missing after upgrading to AIX 5.2 ML5 to Nigel Griffiths, as these are AIX bugs. These are avoided by using nmon Version 10.
• Don’t use Microsoft® Windows® Telnet and use a larger window than 80 x 25 characters. Many developers use VNC and PuTTY to display nmon from a Windows machine — why not do the same!

New features for nmon on AIX Version 10

New Features	Description
Starting up	There is also now a small shell script called "nmon" that starts the right nmon version. Place this script and nmon binaries in your $PATH and type: nmon. This version is now only compiled in 32-bit mode. So, it runs on 32- and 64-bit hardware. The idea is to make it easier to install and run.
N = NFS	NFS is completely new for nmon 10.
p = Partitions	This is for shared CPU partitions information — the big p5/AIX5.3 feature.
C = CPU	This is for machines with 32 plus CPUs — up to 128 logical CPUs by demand.
c = CPU	Details your physical CPU use — if you are on a POWER5 with AIX 5.3 and in a shared CPU environment.
S = Subclass	This is for WLM subclasses — by request.
a = Disk adapters	Gives you details of the disk adapter — like their full type.
r = Resources	This includes your CPU speed in MHz.
k = Kernel	Gives some new fields.
L = Large pages	Gives you large-page stats — popular with high-performance guys.
D = Disk	Gives you more information about your disks, disk type sizes, free, volume groups, adapter, and so forth.
n = Network	Gives you information about your network adapters details, MTU, and errors.
m = Memory	Gives you more details on where your memory is going, system (kernel) and processes, and active virtual memory.
-B	This is a start-up option to remove the boxes.

Sample output for nmon 10 for AIX 5

Figure 1 below is a sample of the screen output. It shows the opening screen for AIX 5, with lots of useful information.
Figure 1. Sample output for nmon 10 for AIX 5

Figure 2 illustrates the details for CPU (this is a 4 CPU POWER5 machine with SMT switched on), memory use, kernel internal statistics, and disks statistics. Note: This logical partition (LPAR) is using six times its entitlement in half a CPU.
Figure 2. CPU details

Figure 3 shows the details of the network, NFS statistics, and journal filesystem use.
Figure 3. Network details

The details of the POWER5 shared processor micro-partitions statistics are shown in Figure 4 below.
Figure 4. LPAR details

Figure 5 illustrates the details of the Linux version of nmon, showing the CPU (this is a 2 CPU POWER5 machine with SMT switched on), LPAR statistics, memory use, network statistics, file system use, and disks statistics. Note: The physical CPU of this LPAR is only available with SUSE SLES9 Service Pack 1 and Red Hat EL 4 Update 1.
Figure 5. Linux version of nmon

Figure 6 shows the OS details of the machine, disk statistics (detailed mode), and the top processes.
Figure 6. Linux version of nmon continued

Obtaining the tool

The following download options are available:

• You can download nmon and its tools from IBM Wiki at http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon.
• Check out the Performance Tools forum for nmon questions and ideas at http://www-03.ibm.com/systems/p/community/.

 

Resources

Learn

• “nmon analyser — A free tool to produce AIX performance reports” (developerWorks, April 2006): Produce a wealth of report-ready graphs from nmon output.
• Check out the following IBM Redbooks for additional information on performance:
  • Understanding IBM pSeries Performance and Sizing, SG24-4810-01, Febraruary 2001
  • Database Performance on AIX in the DB2 UDB and Oracle Environments, SG24-5511, January 2003
  • AIX 5L Performance Tools Handbook,SG24-6039, August 2003
• Check out other articles and tutorials written by Nigel Griffiths:
  • AIX and UNIX zone
  • Across IBM and developerWorks
• “AIX 5 performance series: CPU monitoring and tuning“: Browse through this article to get rid of your CPU bottlenecks and improve performance.
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux
  • Open source
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
• New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
• AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
• IBM Virtual Innovation Center for Hardware: This site is the primary source for all System p AIX development.
• Safari bookstore: Visit this e-reference library to find specific technical resources.
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
• Podcasts: Tune in and catch up with IBM technical experts.

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
• nmon: Download nmon and its tools.

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
• Participate in the AIX and UNIX forums:
  • AIX 5L — technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools — technical
  • Virtualization — technical
  • More AIX and UNIX forums
• nmon questions: Check out the Performance Tools forum for nmon questions and ideas.

About the author

Nigel Griffiths works in the IBM eServer® pSeries Technical Support Advanced Technology Group. He specialises in performance, sizing, tools, benchmarks, and Oracle RDBMS. The nmon tool was developed to support benchmarks and performance tuning for internal use, but by popular demand is given away to deserving friends. If you have a question on nmon, please go on the Performance Tools Forum site (see Resources) so that others can find and benefit from the answers. To protect your email address from junk mail, you need to create a USER ID first (takes 20 seconds at most).

Greetings, please note if you used nmon_grapher script for producing nmon outputs on AIX 5.3 and now migrated to AIX 6.1 the script needs adjustment because of the change in TZ variable
AIX 5.3: CET-1CEST,M3.5.0,M10.5.0/03:00
AIX 6.1: Europe/Prague
As the script derives next starting time from the value -1 in older TZ format, it won’t work on 6.1 because it wouldn’t choose next starting time for data collection in the at job correctly.

from IBM.com

Top 99 Responsibilities of a DBA

Database administrators average salary and salary data for other jobs by Online Degrees

Database Architecture Duties

1. Planning for the database’s future storage requirements
2. Defining database availability and fault management architecture
3. Defining and creating environments for development and new release installation
4. Creating physical database storage structures after developers have designed an application
5. Constructing the database
6. Determining and setting the size and physical locations of datafiles
7. Evaluating new hardware and software purchase
8. Researching, testing, and recommending tools for Oracle development, modeling, database administration, and backup and recovery implementation, as well as planning for the future
9. Providing database design and implementation
10. Understanding and employing the optimal flexible architecture to ease administration, allow flexibility in managing I/O, and to increase the capability to scale the system
11. Working with application developers to determine and define proper partitioning

Backup and Recovery

12. Determining and implementing the backup/recovery plan for each database while in development and as the application moves through test and onto production
13. Establishing and maintaining sound backup and recovery policies and procedures
14. Having knowledge and practice of Oracle backup and recovery scenarios
15. Performing Oracle cold backups when the database is shut down to ensure consistency of the data
16. Performing Oracle hot backups while the database is operational
17. Performing Oracle import/export as a method of recovering data or individual objects
18. Providing retention of data to satisfy legal responsibilities of the company
19. Restoring database services for disaster recovery
20. Recovering the database in the event of a hardware or software failure
21. Using partitioning and transportable tablespaces to reduce downtime, when appropriate

Maintenance and Daily Tasks

22. Providing adjustment and configuration management of INIT.ORA
23. Adjusting extent size of rapidly growing tables and indexes
24. Administering database-management software and related utilities
25. Automating database startup and shutdown
26. Automating repetitive operations
27. Determining and setting critical thresholds for disk, tablespaces, extents, and fragmentation
28. Enrolling new users while maintaining system security
29. Filtering database alarm and alert information
30. Installing, configuring, and upgrading Oracle server software and related products installation
31. Logging Technical Action Reports (TARs); applying patches
32. Maintaining the “Database Administrator’s Handbook”
33. Maintaining an ongoing configuration for database links to other databases
34. Maintaining archived Oracle data
35. Managing contractual agreements with providers of database-management software
36. Managing service level agreements with Oracle consultants or vendors
37. Monitoring and advising management on licensing issues while ensuring compliance with Oracle license agreements
38. Monitoring and coordinating the update of the database recovery plan with the site’s disaster recovery plan
39. Monitoring and optimizing the performance of the database
40. Monitoring rollback segment and temporary tablespace use
41. Monitoring the status of database instances
42. Performing housekeeping tasks as required; purging old files from the Oracle database
43. Performing database troubleshooting
44. Performing modifications of the database structure from information provided by application developers
45. Performing monthly and annual performance reports for trend analysis and capacity planning
46. Installing new and maintaining existing client configurations
47. Performing ongoing configuration management
48. Performing ongoing Oracle security management
49. Performing routine audits of user and developer accounts
50. Performing translation of developer modeled designs for managing data into physical implementation
51. Performing correlation of database errors, alerts, and events
52. Planning and coordinating the testing of the new database, software, and application releases
53. Providing a focal point on calls to Oracle for technical support
54. Working as part of a team and providing 24×7 support when required

Methodology and Business Process

55. Coordinating and executing database upgrades
56. Coordinating upgrades of system software products to resolve any Oracle/operating system issues/conflicts
57. Creating error and alert processes and procedures
58. Creating standard entry formats for SQLNet files
59. Creating processes and procedures for functional and stress testing of database applications
60. Creating processes and procedures of application transport from DEV, to TEST, to PROD
61. Defining and maintaining database standards for the organization to ensure consistency in database creation
62. Defining database standards and procedures to cover the instance parameters, object sizing, storage, and naming. The procedures define the process for install/upgrade, corporate database requirements, security, backup/recovery, applications environment, source code control, change control, naming conventions, and table/index creation.
63. Defining the database service levels necessary for application availability
64. Defining methodology tasks for database software integration
65. Defining a methodology for developing and improving business applications
66. Creating a process to determine whether a new release is “stable” enough to be placed on the development system
67. Developing data-conversion processes for customization, testing, and production
68. Developing database test plans
69. Developing database administration procedures and responsibilities for production systems
70. Developing production migration procedures
71. Establishing and providing schema definitions, as well as tablespace, table, constraint, trigger, package, procedure, and index naming conventions
72. Facilitating design sessions for requirements gathering and defining system requirements
73. Providing database problem reporting, management, and resolution
74. Providing final approval for all technical architecture components that manage and exchange data, including database management software, serve hardware, data distribution management software, server hardware, data distribution management software, transaction processing monitors, and connecting client applications software
75. Providing processes for the setup of new database environments
76. Providing risk and impact analysis of maintenance or new releases of code
77. Providing standards and methods for database software purchasing
78. Providing standards and naming conventions
79. Handling multiple projects and deadlines

Education and Training

80. Attending training classes and user group conferences
81. Evaluating Oracle features and Oracle-related products
82. Understanding the Oracle database, related utilities, and tools
83. Understanding the underlying operating system as well as the design of the physical database
84. Understanding Oracle data integrity
85. Knowing the organization’s applications and how they map to the business requirements
86. Knowing how Oracle acquires and manages resources
87. Knowing enough about the Oracle tool’s normal functional behavior to be able to determine whether a problem lies with the tool or the database
88. Processing sound knowledge in database and system performance tuning
89. Providing in-house technical consulting and training
90. Staying abreast of the most current release of Oracle software and compatibility issues
91. Subscribing to database trade journals and web sources

Communication

92. Interfacing with vendors
93. Disseminating Oracle information to the developers, users, and staff
94. Training application developers to understand and use Oracle concepts, techniques, and tools that model and access managed data
95. Assisting developers with database design issues and problem resolutions, including how to run and understand the output from both TKProf and the Explain Plan utilities
96. Training interim DBAs and junior-level DBAs

Documentation

97. Creating and maintaining a database operations handbook for frequently performed tasks
98. Defining standards for database documentation
99. Creating documentation of the database environment

(Follow http://appsdbaportal.blogspot.com/)

What do you have to do to be a good DBA ?

From time to time, people on the Oracle forum, or comp.databases.oracle.server newsgroup ask the question: “What do you have to do to be a good DBA ?”

Joel Goodman and Harald van Breederode – a pair of highly skilled and very experienced instructors at Oracle University – recently asked themselves a slightly different question: “How do you have to change to stay a good DBA in the modern environment ?”

Historically, the Oracle DBA skill set was database-centric, usually limited to software installation, database creation, day-to-day maintenance, performance monitoring, tuning and most of all, backup and  recovery. Since  the arrival of Oracle 10g,  the  technology within  the DBA arena has changed, due to the increase in automated monitoring, tuning and manageability
features within  the Oracle  kernel,  and  also expanded  into  areas  that were  formerly  the  responsibility of OS, storage, and network administrators. Due to these changes, the knowledge required by the typical Oracle DBA has increased, requiring additional skills and in some cases job responsibilities. This shift from the traditional DBA, which we call DBA 1.0, toward the mod-
ern, post Oracle 9i DBA, which we call DBA 2.0, has occurred gradually over the past two major releases of the Oracle Database Server.

This expansion of the multi-skilled roles, DBA or otherwise, is countered by the ever-increasing strict partitioning of job roles and access restrictions in financial environments. It’s been two years since I was at a client or on a job where I had actually had any access to the OS for example or had any dba level access to the database (in theory at least). ..