.\" cvdb.8: auto-generated, DO NOT EDIT .\" .\" Copyright 2000-2018. Quantum Corporation. All Rights Reserved. .\" StorNext is either a trademark or registered trademark of .\" Quantum Corporation in the US and/or other countries. .\" .\" Code start macro .de Cs .sp .ft C .in +0.3i .nf .. .\" Code end macro .de Ce .fi .in -0.3i .ft R .. .TH CVDB 8 "January 2018" "Xsan File System" .SH NAME cvdb \- Xsan Client File System Debugger .SH SYNOPSIS .na .nh .HP .B cvdb .RI [ options ] .ad .hy .SH DESCRIPTION \fBcvdb\fR provides a mechanism for developers and system administrators to extract debugging information from the \fIXsan File System\fR client filesystem. It can be used by system administrators to change the level of system logging that the client filesystem performs. There is also a switch to retrieve various statistics. .SH USAGE \fBcvdb\fR is a multi-purpose debugging tool, performing a variety of functions. A rich set of options provide the user with control over various debug and logging functions. The main features of \fBcvdb\fR are as follows: .RS 0.5i .B "Control debug logging." .br .B "Control level and verbosity of \fIsyslog\fP logging." .br .B "Retrieve statistics." .RE .SH OPTIONS .IP \fB\-g\fR Retrieve the debug log from a running system. The log pointers are reset after this command, so that the next invocation of \fBcvdb -g\fR will retrieve new information from the buffer. .IP \fB\-C\fR Continuously snap the trace. (Only useful with the \fB-g\fR option.) .IP "\fB\-S\fP \fIstopfile\fP" Stop snapping the trace when the file \fIstopfile\fR appears. (Only useful when also using the \fB-g\fR and \fB-C\fR options.) .IP "\fB-D\fP \fImsec\fP" Delay \fImsec\fR milliseconds between trace snaps. The default is 1000 msec or one second. (Only useful when also using the \fB-C\fR and \fB-g\fR options.) .IP \fB\-F\fR Save the trace output to files named \fIcvdbout.000000\fR, \fIcvdbout.000001\fR, etc. instead of writing to standard output. These files will appear in the current working directory. (Only useful when also using the \fB-C\fR and \fB-g\fR options.) .IP "\fB\-n\fP \fIcnt\fP" After writing \fIcnt\fR files, overwrite the cvdbout out files starting with \fIcvdbout.000000\fR. This will essentially "wrap" the trace output. .IP "\fB\-N\fP \fIname\fP" Use \fIname\fR instead of \fBcvdbout\fR for the cvdb output files. (Only useful when also using the \fB-C\fR, \fB-g\fR, and \fB-F\fR options.) .IP \fB\-d\fR Disable debug logging. This is the initial (start-up) default. .IP \fB\-e\fR Enable debug logging. Disabled by default. Note: care should be taken when enabling logging in a production environment as this can significantly reduce file system performance. .IP "\fB-m modules=\fP\fIbitvector\fP \fBlogmask=\fP\fIbitvector\fP" Specify the trace points for a given module or modules. .IP \fB\-l\fR List the current trace points and their mask values. .IP \fB-L\fR List the available trace/debug points. .IP "\fB-s syslog=\fR{\fBnone\fR|\fBnotice\fR|\fBinfo\fR|\fBdebug\fR}" Set the \fIsyslog\fR logging value. The default at mount time is \fBnotice\fR. See .BR mount_acfs (8) for more information. .IP "\fB-R size=\fR[\fInbytes\fR[\fBk\fR|\fBm\fR|\fBg\fR]]" Resize the the debug log. By default, the size of the log is 4MB. The minimum allowed size is 32768 bytes. .IP \fB-v\fR Be verbose about the operations. .IP \fB-i\fR Print various statistics about the directory cache. If enabled and configured, the directory cache contains a number of buffers of directory contents. This cache is shared by all mounted Xsan file systems. Without \fB-v\fR, the following are printed: .RS .IP The number of directory buffers currently cached and the maximum number allowed. .IP The number of times a buffer has been "hit" in the cache. .IP The number of times a cache search missed and required an RPC to the MDC. .IP The number of times a read of the directory re-used the LAST buffer that was used on the previous read of the same directory (similar to a cache hit but doesn't probe the cache). .IP The number of times a read of a directory specified the EOF offset. .IP The number of times the directory cache for a specific directory was invalidated. For example, if the directory contents changed after it was read and a subsequent read directory was done thereby causing the invalidation. .PP If \fB-v\fP is also specified, \fB-i\fP displays more statistics. Note that there are 2 hashes in the directory cache: one for all buffers and one by directory and file system. .IP The number of entries in the hash used to find dir cache buffers. .IP The # of searches using the directory cache buffer hash. .IP The total # of probes searching the directory cache for buffers. This can be larger than searches in the hash since multiple buffers may hit the same hash bucket. .IP The maximum probes after hitting a particular hash bucket (for buffers). .IP The maximum probes in the hash by directory and file system. .IP \fB-b\fR Print various statistics about each buffer cache. The only other option that can be used with this is \fB-v\fR. There are buffer caches per \fBcachebufsize\fR, see .BR mount_acfs (8). For each buffer cache, the following is printed: .RS .IP # of mounted file systems using this buffer cache .IP # of buffers and total memory used .IP # of cache hits (and percentage) .IP # of cache misses (and percentage) .IP # of checks for write throttling to prevent over use by one file system. Write throttles only occur when more than 1 file system is using the cache. .IP # of times writes were throttled .PP If the \fB-v\fR option is also used with \fB-b\fR, the following additional statistics are printed for each buffer cache: .IP \fBbuffercachecap\fR, see .BR mount_acfs (8) .IP \fBbuffercachewant\fR (internal, means thread is waiting for a buffer) .IP \fBbufhashsize\fR (internal, # of entries in hash used to search buffers) .IP \fBbcdirtycnt\fR (internal, # of buffers with "dirty" data queued in cache) .IP \fBdirty_ndone\fR (internal, bcdirtycnt + buffers being written) .IP \fBflusheractive\fR (internal, flag indicating buffer flusher is active) .IP \fBdeferredflush\fR (internal, # of buffers deferred after files are closed) .IP \fBdirtywaiters\fR (internal, # of threads waiting due to throttling) .IP \fBrsvd max\fR (internal, maximum amount of reserved space seen) .IP \fBnon-zero rsvd min\fR (internal, minimum amount of reserved space seen > 0) .IP \fBsuccessful rsvd requests\fR (internal, # of times reserved space was needed) .IP \fBfailed rsvd requests\fR (internal, # of times reserved space not available) .RE .IP \fB-B\fR Print buffer cache statistics using a curses based display that refreshes every second. Statistics are maintained separately for reads and writes, for each cache segment, and each mount point. Statistics labeled \fBCumulative\fR are those representing the totals since the command was invoked or since the last reset. Those labeled \fBCurrent\fR represent the change in the last one second, roughly corresponding to the display refresh interval. .RS .PP Two keystrokes are interactively recognized on systems supporting curses. A \fBq\fR, quit, will cause the display to terminate. An \fBr\fR, reset, will reset the cumulative counters to zeros. .PP The \fB-B\fR option is intended to be used to to analyze performance of the buffer cache with various applications, I/O subsystems, and various configuration parameters. .PP The refreshing display is supported on clients that have a curses capability. Other clients will produce a line oriented output with similar content. .PP A deadman timer will terminate the display after 30 seconds with no file systems mounted. This is to avoid hanging during file system shutdown. .RE .IP \fB-x\fR Print distributed LAN proxy client and server statistics. The only other options that can be used with this are \fB-X\fR and \fB-f\fR. The proxy statistics are collected at both the client and server ends of each proxy connection. The client will have a connection entry for each path to a proxy server for each proxy client file system. A proxy server will have a connection entry for each path to each client which has the file system mounted. .RS .PP Note: The distributed LAN proxy options are only available on platforms which support the distributed LAN client or server. .TP The following information is displayed for each proxy connection: .RS .HP \fBClient/Server System ID\fR This IP address identifies the remote host. .HP \fBClient IP Addr\fR The IP address of the Client side of the connection. .HP \fBServer IP Addr\fR The IP address of the Server side of the connection. .HP \fBRead Bytes/Sec\fR Measured recent read performance of the connection. .HP \fBWrite Bytes/Sec\fR Measured recent write performance of the connection. .HP \fBFS Read Bytes/Sec\fR Measured recent read performance for all connections for this file system. .HP \fBFS Write Bytes/Sec\fR Measured recent write performance for all connections for this file system. .HP \fBQueued I/O\fR Outstanding I/O (backlog) for this connection. The backlog is meaningful for client side connections only. .RE .RE .IP "\fB-X\fP \fIoption\fP" Dump statistics for each path in comma separated value (CSV) format. (Only useful with the \fB-x\fR option.) The following \fIoptions\fR are available: .RS .IP \fB1\fR +0.3i Dump remote endpoint IP address and backlog in bytes. This option is only relevant for client mounts. .IP \fB2\fR +0.3i Dump remote endpoint IP address and read bytes per second. .IP \fB3\fR +0.3i Dump remote endpoint IP address and write bytes per second. .RE .IP "\fB-f\fP \fIfsname\fP" Specifies the file system name associated with an action option. For proxy statistics(\fB-x\fR option), filter on connections for the given file system. This parameter is required for the read/write statistics (\fB-y\fR or \fB-Y\fR) option. .sp .IP "\fB-U\fR" \fBNOTE: Not intended for general use. Only use when recommended by Quantum Support as a fault injection tool. \fP .sp This option resets the network connection to the proxy peer for all proxy connections on all file systems for which this node is either a proxy client or gateway. This simulates an unexpected network disconnect and reconnect. It is intended to test the robustness of the error handling and reconnect logic in the StorNext DLC proxy client and gateway systems. .IP "\fB-y\fR, \fB-Y\fR" Display the read/write statistics for the file system specified with the \fB-f\fR option (required). If \fB-Y\fR, also clear the stats. .IP "\fB-Z\fR" \fBNOTE: Not intended for general use. Only use when recommended by Quantum Support as a fault injection tool. \fP .sp This option resets the network connection to the file system manager for all active file systems. This simulates an unexpected network disconnect and reconnect. It is intended to test the robustness of the error handling and reconnect logic in the StorNext file system. .IP "\fB-z\fR" \fBNOTE: Not intended for general use. Only use when recommended by Quantum Support as a performance measuring tool. Setting this option could result in data corruption, loss of data, or unintended exposure of uninitialized disk data!! \fP .sp This option turns on the DEVNULL capability and only applies to linux clients. Once enabled this option will continue to be enabled until reboot. When this option is enabled, all I/O for files with the DEVNULL affinity is not performed at the lowest level. The code paths are all executed including the allocation of space, but the data is not read or written to disk. Instead, writes simply complete the I/O and return and reads zero out the "read" buffer and complete the I/O. .sp Files without the DEVNULL affinity are unaffected by this setting. .sp Before attempting to use this capability, make sure no one is already using DEVNULL as an affinity on any file system the client has access too. Then, modify the file system configuration file, \fBsnfs_config(5)\fP, for the file system under test to contain DEVNULL as an affinity on at least one stripe group that can hold data. Next, restart the fsm. Then, use \fBcvmkdir(1)\fP with -k DEVNULL to create a directory to hold files to be used for this test. Finally, enable the feature with this option, \fBcvdb -z\fP. .SH "DEBUG LOGGING" Developing code that runs in the kernel is very different than programming a user-level application. To assist plugin developers who may not be familiar with the kernel environment, Xsan provides a simple "tracepoint like" debugging mechanism. This mechanism allows developers to use printf-like statements to assist in debugging their code. .PP To use the debugging facility, each module (typically a ".c" file), must declare a structure of type \fIModuleLogInfo_t\fR. This structure defines the name of the module as it will appear in the debug statements, and indicates the debug level that is in effect for that module. .Cs ModuleLogInfo_t MyLogModule = { "mymodule_name", DEBUGLOG_NONE}; .Ce .PP To use the facility, each module must call the \fIAddLogModule()\fR routine. This is typically done when the module is first initialized (in the xxx_start() routine for a plugin). When logging is no longer required (as when the plugin is unloaded), the module should call \fIRemoveLogModule()\fR to free up the system resources. .PP Logging is not enabled by default. To enable logging at any time, specify the \fBenable\fR flag (\fB-e\fR) .Cs shrubbery %h: cvdb -e .Ce .PP To disable logging, specify the \fBdisable\fR flag. .Cs shrubbery %h: cvdb -d -v Disabling debug logging .Ce .PP The level of debugging is controlled via a 64-bit mask. This allows each module to have 64 different, discrete trace/log points. If the log point is enabled when the code is executed, the trace point will be dumped to the circular buffer. .PP A complete listing of all the pre-defined trace points can be obtained via: .Cs rabbit %h: cvdb -L Trace points: cvENTRY 0x0001 cvEXIT 0x0002 cvINFO 0x0004 cvNOTE 0x0008 cvWARN 0x0010 cvMEM 0x0020 cvNUKE 0x0040 cvLOOKUP 0x0080 cvGATE 0x0100 cvSTRAT 0x0200 cvRWCVP 0x0400 .Ce .PP These trace points would then be used to control the verbosity of logging. Using the example above, if the cvEXIT and cvINFO trace points are enabled, then only those trace points would be dumped to the log. .PP To enable the trace points, the first step is to determine the ID of the module. This is done with the \fBlist\fR command. .Cs shrubbery %h: cvdb -l Module 'cvfs_memalloc' module 0x000001 logmask 0x0000000000000000 Module 'cvfs_fsmsubr' module 0x000002 logmask 0x0000000000000000 Module 'cvfs_fsmdir' module 0x000004 logmask 0x0000000000000000 Module 'cvfs_fsmvfsops' module 0x000008 logmask 0x0000000000000000 Module 'cvfs_fsmvnops' module 0x000010 logmask 0x0000000000000000 Module 'cvfs_sockio' module 0x000020 logmask 0x0000000000000000 Module 'cvfs_subr' module 0x000040 logmask 0x0000000000000000 Module 'cvfs_vfsops' module 0x000080 logmask 0x0000000000000000 Module 'cvfs_vnops' module 0x000100 logmask 0x0000000000000000 Module 'cvfs_dmon' module 0x000200 logmask 0x0000000000000000 Module 'cvfs_rwlock' module 0x000400 logmask 0x0000000000000000 Module 'cvfs_rw' module 0x000800 logmask 0x0000000000000000 Module 'cvfs_fsmtokops' module 0x001000 logmask 0x0000000000000000 Module 'cvfs_extent' module 0x002000 logmask 0x0000000000000000 Module 'cvfs_plugin' module 0x004000 logmask 0x0000000000000000 Module 'cvfs_disk' module 0x008000 logmask 0x0000000000000000 .Ce .PP To enable the cvENTRY and cvEXIT trace points of the plugin, rwlock, vnops, and memalloc routines, use the \fBmodules\fR command. .Cs shrubbery %h: cvdb -m modules=0x4501 logmask=3 .Ce .PP The bit masks are additive, not replacement. This means that modules and trace points you do not specify are unaffected. To turn on all debugging on all trace points, specify minus one (-1). .Cs shrubbery %h: cvdb -m modules=-1 logmask=-1 .Ce .PP Once the module has been added to the system, log messages will then be dumped into a 1 meg circular buffer. Modules may find it convenient to declare a macro in each file so that the form of log messages will be the same in each file. For example, the following macro definition and following log function would dump information to the log buffer if the trace point is enabled: .Cs #define LOGINFO (&MyLogModule) LogMsg(LOGINFO, cvEXIT, "Plugin read return error %d bytes %llx", error, num_bytes); .Ce .PP To extract the messages from the log on a running system, use the \fB-g\fR option of \fBcvdb\fR. .SH SYSLOG The Xsan client file system can log certain events so that they show up on the system console and in the system log, \fI/var/adm/SYSLOG\fR. The verbosity of messages can be controlled via the \fIsyslog\fR parameter. The default is to log all messages. See .BR syslogd (1M) for more information of setting up system logging. .PP There are four log levels: \fInone, notice, info, \fR and \fIdebug \fR. The levels are prioritized so that the \fBdebug\fR level is the most verbose; setting the level to \fBnone\fR will turn off logging completely. The events that are logged at each level are as follows: .IP \fBnotice\fR .br \(bu reconnection with the FSM. .IP \fBinfo\fR .br \(bu all \fBnotice\fR messages, plus .br \(bu socket daemon termination .IP \fBdebug\fR .br \(bu Currently unused .PP The log level is set to \fIdebug\fR by default. .SH "BUSY UNMOUNTS" Occasionally, it will be impossible to unmount the Xsan volume even when it appears that all processes are no longer using the volume. The problem is that the processes are most likely in the \fIzombie\fR state; while they do not show up in \fBps\fR, then can be found using \fBcrash\fR. Usually, these processes are waiting on a lock in the Xsan file system, or waiting for a response from the FSM. .SH "DEBUG LOGGING EXAMPLES" .TP To enable logging: .B cvdb -e .TP To disable logging: .B cvdb -d .TP To retrieve (get) log information on a running system: .B cvdb -g > cvdbout .TP To continuously retrieve log information on a running system, snapping the trace once per second: .B cvdb -g -C > cvdbout .TP To continuously retrieve log information on a running system, snapping the trace once every two seconds and stopping when the file named \fBSTOP\fR appears: .B cvdb -g -C -D 2000 -S STOP > cvdbout .TP To continuously retrieve log information on a running system, and save the output to files named \fIcvdbout.000000\fR, \fIcvdbout.0000001\fR, etc. and wrapping after 100 files have been written: .B cvdb -g -C -F -n 100 .TP To continuously snap traces named \fI/tmp/snap.000000\fR, \fI/tmp/snap.000001\fR, etc.: .B cvdb -g -C -F -N /tmp/snap .TP To list all the modules and their enabled trace points: .B cvdb -l .TP To set trace points in individual modules: .B cvdb -m modules=\fIbitmask_of_modules\fP logmask=\fItracepoints\fP. .TP To resize the log to 12 megabytes: .B cvdb -R 12m .TP To dump out all the pre-defined trace points: .B cvdb -L .SH "SEE ALSO" .BR syslogd (1M), .BR umount (8), .BR cvdbset (8)