Research Division EED/Controls Software<P> Release Note 150.0<P> The Extraction and Plotting of Logged Data

Research Division EED/Controls Software

Release Note 150.0

The Extraction and Plotting of Logged Data

Robert E. West

August 14, 1995

Introduction

The EPICURE Data Logging System consists of three parts:

This document describes the third part, the EPICURE data logging utility for extracting and plotting the recorded data, which consists of two sections:

Data Extraction

A user first selects the location of the recorded data, the start and end of the time interval of interest, and the device names. The setting property is specified by dash E (-E) at the end of the device name. The status property is specified by dash T (-T) at the end of the device name. The reading property is the default if neither suffix is specified. The file of logged data is searched for all the data collected for the devices during the time interval. The data for each device is written into a separate file and then all the files are input to PAW for plotting.

A user interacts with the data extraction application via a menu interface implemented using the EPICURE Screen Management (ESM) software. Standard conventions of an ESM menu interface are the following:

The logging data extraction and plotting utility is initiated by selecting its entry from an EPICURE menu in an X-windows environment. Two new terminal windows are created, one for specifying and extracting the desired data and the other for displaying the generated plots. The window for the graphics output first appears as a completely blank rectangle in the upper left corner of the workstation screen. The window for specifying the various extraction parameters first appears in the lower right corner of the workstation display. This window may be partially obscured by the plotting window. To bring it to the forefront, place the mouse cursor on its title bar and then press mouse button #1.

When the extraction utility starts, it displays the values of the parameters last used. The logger name, the starting time, the ending time, and up to four device names are displayed in fields in the window. The logger name is the name of a VAX node which executes a data logging collector process. This name identifies the associated file of collected data.

New information may be specified for the interval times and the device names by placing the cursor in the field and then over-typing what is displayed. A field may also be modified by double clicking the mouse in the first column of the field. In a time field, a dialogue box appears requesting the input of the new time. In a device field which contains a name, the name is deleted. If the device field is blank or if the cursor is not in the first column, the menu of device names contained in the logging file appears for the selection of a new item. Over-typing is not permitted in the logger field. Double clicking anywhere in this field presents the menu of logger names. All the displayed fields may also be modified by selecting the corresponding item from the menu bar.

After specifying the logger file, time, and device parameters, plotting of the data is initiated by selecting the PLOT item from the menu bar. This menu contains two items: SELECTED and PREVIOUS. Double clicking on the item SELECTED results in the file of logged data being searched using the displayed selection parameters. Each device's extracted data is written into its own disk file. All these data files are then input to PAW to be plotted.

Selection of the item PREVIOUS from the PLOT popup menu allows a user to plot data files previously generated during this execution of the program. Assume a user specifies a device and then extracts and plots the data. He then specifies a second device and extracts and plots its data. If he then wants both these devices to be plotted on the same graph, he can specify the two devices together and then extract and plot them both. Since the previously extracted data resides in separate disk files, the extraction phase does not need to be repeated. Selecting PREVIOUS from the PLOT popup menu allows the user to choose one or more previously generated files of extracted data. These files are then input to PAW for plotting.

Plotting of the Extracted Data

The Physics Analysis Workstation (PAW) interactive utility developed at CERN is used to graph the files of extracted data. A FORTRAN routine and various PAW macros input the files of extracted data and draw the desired plots.

When the PAW graphics window is present, any window of a set of overlapping windows should be brought to the forefront by clicking the mouse on the window's title bar. Simply clicking somewhere in the desired window always activates the window but frequently does not result in it being displayed over the PAW window. Bringing the PAW window to the forefront by clicking on its title bar also insures that the mouse click is not incorrectly interpreted as an input to PAW.

When the PAW graphics window is brought to the forefront of a set of overlapping windows, the areas of the PAW graphics window which were overlaid may be blank, i.e., they weren't automatically refreshed. After the size of the graphics window is changed, the entire window may be totally blank. Positioning the mouse cursor in the window and clicking only button #3 causes the display to be refreshed.

The following text fields may appear in a column at the right of the PAW graphics window:

Placing the mouse cursor somewhere over one of these text strings, clicking button #1 to select, and then clicking button #3 to terminate results in execution of the selected option. When button #1 is pressed, an asterisk symbol should be placed at the position of the cursor. When button #3 is pressed, a box should be placed around the selected text string.

If the PAW graphics window is overlaid by other workstation windows, the first press of button #1 may only bring the PAW window to the forefront of the overlaid windows. If this is the case, no asterisk symbol will appear, so button #1 must be pressed a second time to initiate the selection, followed by pressing button #3.

Selection of EXIT exits the plotting section and returns control to the data extraction section. Selection of PRINT outputs a copy of the graphics window to the postscript printer queue specified by the logical symbol HIGZPRINTER. If this logical is not defined before requesting the EPICURE menu, the user is requested to make a selection from a list of the available postscript queues each time the logged plotting package is requested. Selection of REDRAW simply regenerates the graphics window from the data which was previous read from the disk files.

There are four plotting modes: SINGLE, OVERLAY, STACK, and AUTO. If only a single device is being graphed, the plotting mode is considered to be SINGLE and the other three text strings are not present. If two or more devices are being graphed, the text strings OVERLAY, STACK, and AUTO are all present on the display and any one may be selected.

The default mode is AUTO, which is used when the data first appears in the graphics window. Each device has its own plotting area and each plotting area appears in the graphics window under the one for the previous device. Each y-axis is scaled according to the particular device's data. All the x-axes are identical.

The STACK mode display is the same as that for the AUTO mode except that all the y-axes are the same. In this mode, the mouse may be used to expand a section of one of the device plots.

The OVERLAY mode displays all the device data in a single plot. Different symbols are used to designate the data points for the different devices. If the workstation has a color monitor, different colors are also used for the different devices.

Because it is much faster, individual plots are initially drawn with a line connecting successive data points. Selecting POINT draws a symbol at each data point with no connecting lines. Selecting LINE reverts to the initial mode. If all the devices are being overlaid on the same plotting field, the default is to draw a symbol at each data point, with a different symbol used for each device. The LINE option may be selected, but it may be difficult to distinguish the different devices unless a color monitor is being used, in which case different colors are used for the different devices.

Selecting RANGE allows zooming in on a section of a single plot. The rectangular subsection to be expanded is identified by repeatedly positioning the mouse cursor at the desired points and pressing button #1., finally terminating the selection process by pressing button #3. This zooming operation can be done repeatedly up to the resolution capabilities of the workstation windowing software. Selecting RESET reverts to using the min and max plotting intervals derived from the input data files.

Tutorial

The software which extracts and plots data previously recorded by the EPICURE data logging system can only be executed from a node on the network providing an X-windows environment. If a connection is established to the WARNER cluster using SET HOST, the local display must be specified via SET DISPLAY to the name of the local node. As an example, this section describes the extraction and plotting of several devices reporting weather-related information. For best results, a user should simultaneously read and execute this example while at a workstation node.

Startup

Data Extraction

The disk file of collected data is identified by selecting the name of the VAX node on which the associated data logging process is executing. The time interval of interest is identified by specifying the start and end of the interval. A maximum of four devices may be specified for data extraction and plotting.

Data Plotting

Numerous informational messages appear in the window as the PAW macros input the data and then prepare it for plotting. Eventually, a plot appears in the graphics window for each of the requested device names for which data was extracted from the logging file. If the current time is earlier than selected ending time, the label at the right end of the x-axis is the current time, not the specified ending time. It is assumed in this section that data was successfully extracted for the four devices BAROMETRIC_P, OUTSIDE_TEMP, HUMIDITY, and OPS_TEMP referenced in the previous section of the document.

Sections of the graphics window which are covered by overlapping windows on the workstation display are not automatically refreshed when the graphics window is brought to the forefront. Position the mouse cursor on the extraction window's title bar and click button #1. This window comes to the forefront so all of it is in view. Position the mouse cursor on the title bar of the graphic window and click button #1. The part of the graphics window previously covered by the extraction window is blank. Position the mouse cursor anywhere within the graphics window and press button #3. This causes the graphics window to be refreshed. Any previously overlapped sections are now restored.

Combinations and Expansions

Completion

Reference Section for the Data Logging Extraction Utility

The menu bar across the top of the workstation window contains the following items:

LOGGER

Presents a menu of the data loggers whose files of collected data may be accessed by this user.

TIMES

Presents a menu to specify the start and end of the interval over which data is to be obtained. There are two entries on the menu:

STARTING

Presents a dialogue box into which the user may type the earliest timestamp of interest. The timestamp format consists of date and time. The previously specified timestamp is presented for modification by using the arrow keys to place the cursor at the desired position and then over-typing what is there. Alternatively, control-U can be used to totally eliminate what is already there and then the new information can be entered. Unspecified fields of a date default to the current date. Unspecified fields of a time default to 0. A 24-hour clock is assumed unless P or A follows the time field. If the entered starting time is greater than the existing ending time, the ending time is set to one hour later than the specified starting time.

ENDING

Presents a dialogue box into which the user may type the oldest timestamp of interest. If the entered ending time is less than the existing starting time, the starting time is set to one hour earlier than the specified ending time.

DEVICE

Presents a menu for specifying the name of the device for which logged data is to be retrieved. There are two entries on the menu:

The devices are graphed in the same order as they are displayed on the screen. When two or more devices are plotted on the same graph, a different graphics symbol is used for each device's data points. If a color monitor is being used, a different color is also used for each device. If a particular symbol and/or color should be used for a certain device, then the device should be selected so that it resides in the corresponding entry in the display list.

READ LIST

Presents a menu of the device names contained in the most recently created directory of the logged data file. It is usually necessary to scroll this menu to see all the names.

NAME

Presents a dialogue box into which the user may type the name of the desired device. The asterisk character is used as the wildcard character. The asterisk may be entered as either the first character or the last character of a string. If the asterisk is the last character, a menu is presented listing all device names in the logged data file which start with the specified charaters. If the asterisk is the first character, a menu is presented listing all device names in the logged data file which contain the character sequence folling the asterisk. The device names appearing in these menus include the extensions for the setting and status properties. The sEtting property is indicated by a suffix of dash E (-T). The sTatus property is indicated by a suffix of dash T (-T). The reading property is the default if neither suffix is specified.

OPTIONS

Presents a menu of options affecting the content of the output files. There are seven entries:

FILENAME

The name of each intermediate file produced for a requested device defaults to the name of the specified device followed by an appropriate extension. Selection of this option enables a user to specify a name other than that of the selected device by modifying the default name in the dialogue box.

UNIQUE/ALL

UNIQUE reduces the number of data points written to the output files. Only unique data points are extracted. The timestamp value and the reading value both must be different from those of the previously extracted data point.

ALL specifies the extraction of all data points having a timestamp value within the specified interval.

SUMMARY/DATA

SUMMARY produces a text file which is a summary of the records contained in the logging file for the specified device within the specified time interval. No data files are produced for the plotting of a graph.

DATA generates the files required by the plotting process.

TEXT/NO TEXT

TEXT outputs an ASCII text string for the timestamp and the reading of each extracted data point. This mode of operation greatly increases the required processing time and the required amount of disk space for the text output file. The default is to not write an ASCII text representation of the data to the disk file. For this state, this menu entry appears as `Text' so text output may be selected. If text strings are being written to disk, then the menu entry appears as NO TEXT so no output of text strings may be selected.

FOUND BASE/ZERO BASE

The default mode for the extraction of data from the logging file is to subtract the specified starting time from each extracted timestamp before writing it to the output file. FOUND BASE disables that subtraction so the collected timestamp is output. The PAW graphics package is then used to normalize the data for all the specified devices, allowing PAW to process the output data totally independently of the extraction process. However, because a collected timestamp is a very large number, the roundoff errors seen in the PAW output may be quite noticeable.

If the collected timestamp is being written into the data file, then this menu entry appears as ZERO BASE so normalizing to zero may be selected.

WRITE LIST

Creates a new directory containing the names and starting times of all devices in the logging file. If the logging file contains a large number of data points, the search of the file for all device names may require many minutes.

EXTRACT

Starts searching the logging file to extract the data for the specified device(s) with timestamps in the specified time interval. A dialogue box is presented to verify that all search parameters are correct.

The TPU editor is invoked after the data collection phase has been performed so the intermediate text file may be examined. This text file always contains information regarding the number of data points examined and extracted and also timing statistics for the search process. If the text option was selected, the editor can be used to examine each extracted timestamp and data value.

PLOT

Presents a menu of items for requesting the plotting of extracted data. The are two entries:

SELECTED

Data extraction is done for each device which has been selected and the files of raw data are passed to PAW for the generation of graphical output.

PREVIOUS

Presents device names which have had data extracted since the user started this execution of the application. The user can then select up to four of these to be graphed by PAW. If the extracted timestamps are relative to the starting time, all the selected data files should reference the same starting time. Otherwise, the graphics output may be very confusing.

PRINT

Presents a menu of the available printer queues. Selecting an entry outputs a copy of the terminal display window to the printer queue.

EXIT

Exits the extraction and plotting processes. Several work files are created by the extraction application for each device. The PAW software package also creates various work files. All temporary work files are deleted upon exit.

Reference Section for the PAW Graphics Window

The following options may appear in a column at the right of the PAW graphics window:

EXIT

Exits the data plotting process and returns to the data extraction process.

PRINT

Transforms the displayed graph into a postscript file with the name PAW.PS and then outputs this file to the postscript printer queue specified by the logical symbol HIGZPRINTER. If this logical is not defined, the file is generated but not printed. The user's LOGIN.COM file may be used to define this logical to avoid repeatedly being asked for the name of the postscript queue.

REDRAW

Redraws the display using the device data which was input from the files of raw data. Since the display is redrawn from scratch using the original data, not simply refreshed as by pressing mouse button #3, any extraneous asterisk symbols resulting from pressing mouse button #1 will no longer be present.

SINGLE, AUTO, STACK, OVERLAY

If only a single device is being graphed, only SINGLE is present. Selection of SINGLE does not result in any action. If two or more devices are being graphed, switching to a different mode is accomplished by selecting the corresponding text string.

The default mode is AUTO, which is used when the data first appears in the graphics window. Each device has its own plotting area and each plotting area appears in the graphics window under the one for the previous device. Each y-axis is scaled according to the particular device's data. All the x-axes are identical.

The STACK mode display is the same as that for the AUTO mode except that all the y-axes are the same. In this mode, the mouse may be used to select a range of data in each of the individual device plots.

The OVERLAY mode displays all the device data in a single plot. Different symbols are used to designate the data points for the different devices. If the workstation has a color monitor, different colors are also used for the different devices.

POINT/LINE

LINE draws a line connecting successive data points. This is the default for a single device on a single plot. POINT displays a symbol for each data point with no connection of data points. This is the default when multiple devices have their data overlaid on the same plot.

RANGE

Initiates use of the mouse to select a subset of a plot. After the box appears around the text RANGE, position the mouse cursor at the desired point in a plot. When mouse button #1 is pressed, an asterisk symbol is placed at the cursor position. Move the cursor to another point in the same plot and press button #1 again. Another asterisk symbol appears and a line is drawn between the two asterisk symbols. Repeatedly doing this defines the subset of the x-axis and/or the y-axis which is to be displayed. Pressing mouse button #3 terminates the selection of the plot area which is to be displayed. Two or more points may be used to delineate the area of interest. A horizontal line selects a subsection of the x-axis. A vertical line selects a subsection of the y-axis. A diagonal line selects a section of both the x-axis and the y-axis. For any other combination of points, the area of interest is the rectangle enclosing all the designated points.

For the SINGLE or OVERLAY plotting mode, there is only one plot on which to identify a subsection. For the STACK mode, the selected subset of the y-axis applies only to the plot which is marked. The selected subset of the x-axis is applied to all the plots. RANGE is not selectable if the mode is AUTO.

RESET

Restores the axes of the plot to the full range of the extracted data values. For the STACK mode, the particular plot to be restored must be identified. This is done by placing the mouse cursor on the plot, pressing mouse button #1, and then terminating the selection by pressing mouse button #3.

Security, Privacy, Legal

rwest@fsus04.fnal.gov