Useful Features of Automated Test Systems in the R&D
Laboratory
by Joseph Czapski, rev. Oct. 2001
This article first appeared in Proceedings of IEEE AUTOTESTCON 2000, pp. 601-613. This online version contains the full text of the version presented at Autotestcon except for minor corrections and the addition of a clickable table of contents. Also, the figures have been changed from black and white LabVIEW 5 panels to color LabVIEW 6 panels.
©2000 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.
Abstract — This document lists, describes, and justifies features that a research and development engineer or technician would benefit from having in his or her automated test system. Features are presented relating to data acquisition, stimulus, control, user interface design, configuring measurements, documentation, storage, analysis, and system integrity. This document can serve as a guide in designing features of a laboratory test system.
Table of Contents
In the business of developing full-featured Automated Test Systems (ATSs), having a list of recommended system features is a boon. When the test engineer is designing the system with customer input, such a feature list can jog minds, and a design that meets the customer's needs fully will likely emerge.
This paper lists the features needed for a complete, robust, highly useful ATS for the research and development (R&D) laboratory, where flexible, powerful measurement tools are needed. Applicability to manufacturing and maintenance test is briefly addressed at the end of the paper.
The list evolved from several years of the author's building and using ATSs. It also had as input some fine papers addressing ATS architecture [1]–[4] and some fine, full-featured test & measurement software [5].
An Inexpensive, PC-Based ATS Approach
Powerful test & measurement programming languages, such as National Instruments's LabVIEW, and commercial PC-based data acquisition boards make it possible to develop powerful ATSs rapidly and smoothly [2], [6]. Such test systems may consist of:
The test system software is best contained all in one application, which runs automatically at PC boot-up. The single application should be highly modular, consisting of many individual, changeable routines, and should be easily upgradeable.
R&D Lab Activity and Personnel
The R&D laboratory is dynamic and unpredictable, where lab personnel cannot know what type of measurement they will want to do next. Take, for example, the characterizing of a semiconductor device:
Lab personnel may want to test a device's performance over ranges of variables such as input or supply voltage, lead current, switching frequency, and die temperature. Also, they may test samples having ranges of processing parameters such as anneal time and temperature, etching time, doping level, implant energy, and deposited materials. Some parameters are swept over their ranges during a measurement, while others are constants that change between measurements.
Testing performance may require measuring several different voltages, waveforms, and currents. Lab personnel would like to acquire measurements vs. several different swept parameters, and under different conditions or states. Then, they want to be able to analyze the results in many ways: graph any measured quantity vs. any parameter, make a table of same, perform curve arithmetic and curve fitting, and generate statistics such as histograms. The measurement data should go into a database which can be browsed or searched for any combination of parameters. The data can also be immediately published to the world-wide-web for viewing from any location.
For tests that are repeated many times (the usual case), or are performed by less skilled engineers or technicians, certain conveniences can be included. The procedures and screens should be user-friendly, with set-up instructions and pictures presented and with obvious buttons to click. There can be engineer-programmable automatic sequencing of measurements. And, there should be thorough automatic error catching, including operator errors, values out of range, connection errors, and hardware malfunctions.
The role of the test technician is explored well in [7]. The paper explains the benefits of designing the ATS software, particularly the status messages, descriptions, and Help screens, to teach the technician why a certain test is being run, how it is being run, and how to troubleshoot problems. Giving the technician full access to information and to low-level diagnostic tests benefits everyone. The technician will likely be more involved, productive, and satisfied.
Every feature put into an ATS costs money to develop. A customer must decide which features are worth the price. Following are the typical costs of not designing in robustness and flexibility: missed deadlines, lower quality, lost customers, and employees frustrated by lack of tools and information, lack of automation of tedious data-crunching, and frequent system down time.
It is difficult to foresee the sometimes explosive response of the laboratory to powerful automated measurement, analysis, and presentation tools. Research progress speeds up immeasurably, and a company can go in entirely new directions when people are able to see data trends that were formerly inextractable.
Data Acquisition
Analog Voltage Measurement
Measuring analog voltages is the usual way of acquiring data. Sometimes the voltage is generated from the circuitry of the DUT, but most of the time it is generated by a sensor, or transducer. Examples of quantities sensed:
Often, the voltage is measured in a repeated, timed manner, at a sample rate anywhere from 10 hertz to 10 megahertz. The samples may form a waveform that needs to be stored in its entirety and analyzed. If there is not enough RAM in the computer to store the waveform, advanced streaming-to-disk methods may have to be used.
In many cases the samples do not need to be stored but are analyzed in real time. Simply averaging the samples, and storing the averages, improves the accuracy of the measurement. Other common real-time analyses are noise (standard deviation) and frequency spectrum (FFT) calculations.
When samples need to be taken within nanoseconds of an event, advanced hardware triggering is used. Triggering capability is available on some data acquisition boards, but it may need to be augmented with custom interface circuitry. For sampling within several milliseconds of an event, as opposed to nanoseconds, software triggering can be employed. The software waits for a particular level on one channel before sampling another channel.
Signal conditioning is often necessary and can be done with custom circuitry or commercial conditioning modules. Some examples of conditioning circuits are:
Digital Input
Data is sometimes available from the DUT in binary form, and can be read serially or in parallel through a digital interface. Common interfaces are the PC's serial and parallel ports and commercial digital I/O boards. Data can be read in a timed, handshaked manner or asynchronously with a simple handshake protocol.
The input pins of a digital I/O board can usually be read individually and can be wired to conveniently read the states of switches and digital lines on the DUT.
Counter inputs are often available on I/O boards. A hardware counter can count the number of pulses on a line occurring in a given interval, even when the pulses are, say, 100 nanoseconds wide. Counters enable measurement of pulse frequency. A custom frequency-to-voltage circuit can also be used, particularly for non-TTL-level waveforms.
Analog Stimulus
Digital Output
Complementing Digital Input, explained previously, the DUT sometimes will read configuration and waveform data in binary form through a digital interface. Also, the output pins of a digital I/O board can usually be wired to control individual switches in the DUT.
An advanced type of digital stimulus is a "pattern generator." Some DUTs require complex, synchronized timing signals on a bank of digital inputs. These signals can be provided by the PC through a digital I/O board, in which the timing patterns are loaded into the PC's RAM and continuously, cyclically played out to the DUT at a high data rate. This capacity is usually too much for the PC to handle and is better implemented in custom interface circuitry containing a programmable logic device (PLD).
Control
The DUT often needs to be controlled in some way automatically during the measurement. Two common capabilities are temperature and position control. Temperature control is often achieved with dedicated feedback control hardware of the proportional-integral-derivative (PID) type or fuzzy-logic type. Position is usually controlled with an electric motor, which can be conveniently driven by a PC-based motor-control board.
Equipment for Acquisition, Stimulus, and Control
A variety of powerful multi-purpose I/O boards for a PC's PCI bus are commercially available. They are often the best choice because of their low cost, compactness, availability of options, and simplicity.
Concerns of the PC's noisy environment, length of analog cables in the system, robustness, power supply, and number of slots, may point an engineer toward an external chassis for multi-purpose I/O boards. Common chassis classes are VXI, PXI, and SCXI. Their use raises the cost of the system because of the higher cost of the boards and the significant cost of the chassis.
Certain high-precision measurements and special capabilities (such as control of a fluid pump) require external, high-priced, commercial test equipment. Automation of this equipment requires communicating with it somehow, commonly through GPIB bus, RS-232, or simple analog voltage control. External instruments are usually the costliest part of the test system, and learning the communication language is time- consuming for the programmer.
Commonly, the prime function of an ATS in an R&D laboratory is sweeping the value of one parameter while measuring the effect on another parameter. It is hard to predict which of the many different stimuli and settings of a DUT the R&D user will want to sweep. With the appropriate user interface (Figures 1 and 2), any n variables can be swept. Then, each measured channel will form an n-dimensional array of readings.
The sweep of each variable is defined either with Start, Stop, and Interval values or with a table of individual points. As the measurement runs, the swept variables change as nested loops, and the looping order is user settable. To minimize test time, it makes sense for the innermost loop to be the parameter that takes the shortest time to change (and for the system to settle after it changes) and the outermost loop to be the parameter that takes the longest time to change.
There are basically two categories of R&D measurements: the above-described multi-variable sweep and the simpler continuous monitoring. In continuous monitoring, all measured channels are sampled continuously, updating on the screen numeric indicators, magnitude bars, and/or strip charts [4]. Monitoring runs continuously or for a desired finite time, with optional continuous logging of every measured point.
Using panel selection tabs as shown in Figure 1, the user can switch at will between the Measure panel and Monitor panel. The user can pause in the middle of a measurement, switch to the Monitor panel to check values on the strip charts, then switch back to the Measure panel and continue the measurement. Sometimes it is useful to set up the measurement so that, between every sweep point, the monitor screen is shown, and the user must verify that the measured parameters are settled and valid before clicking Accept to store the data point and continue the test.
While monitoring, the user can interactively change stimuli or controls and see the effect on the measured channels. He or she can wiggle connections, move the DUT or equipment around, and experiment in other ways to check for noise sources and troubleshoot problems. He or she can also monitor and record channels at a high sample rate for a fixed time, then switch to the analysis screen to plot samples vs. time, histograms, and frequency spectra. These real-time analysis features are very useful in troubleshooting noise.
Screen Layout
Figures 1 - 3 show an idea for the layout of the PC screen user interface.
Figure 1. "Measure"
panel. Displayed while a measurement is running. These three
figures were optimized for this paper format and just give the
general user-interface idea. Screens in practical use are larger,
include more, and have a nicely spaced layout.
Title Bar of Window
At the top of the screen, the title bar is the standard and appropriate place to display, from left to right, the name of the software, the name of the measurement, and the lot and serial numbers of the DUT, all separated by hyphens.
Panel Selection Tabs
In Figure 1, notice the row of 9 tabs near the top of the panel. It is useful to be able to switch among the different panels with a single click. This is done with tabs or buttons, along the bottom or top of the panel. Each panel contains the displays, indicators, and controls required for that functional area. In Figure 2, the Configure panel, another row of tabs has appeared to allow navigation among the 7 different configuration panels.
Pull-down Menus
Traditional pull-down menus in the usual location (the menu bar just below the title bar) are the place to put all the commands that won't be done by buttons, knobs, and numeric entry. Pull-down menus are practically infinitely expandable and can contain any functions a test engineer can come up with.
Standard pull-down headings that are in every menu bar are File, Edit, and Help. The File menu contains commands for file writing and reading, and for printing and "sending" elsewhere. The Edit menu contains commands such as Cut, Copy, Paste, Clear, and Undo, as well as formatting and preferences dialogs. The Help pull-down accesses all the user documentation, manuals, and help screens.
Status Message Bar
At the top of the panel, just under the menu bar, is the Status bar. Whenever the user is wondering what's going on in the software, he or she reads the status message. The status message will tell:
When an error message is displayed, the text can be colored red, the whole status bar can flash (but not too much to be readable), and an initial series of beeps can sound.
Description Bar
In the lower left of the panel is the description bar. When the mouse pointer passes over a control or indicator on the panel, a description of the control or indicator appears in the description bar. The description tells the user just what an indicator means, or just what a control will do. Pop-up "tip strips" can also provide this function.
Measurement Control Buttons
In the lower right of the panel are buttons labeled Start, Pause, Abort, and Power Off. The abort and power-off buttons are explained in a later section. Using colors with these buttons, such as green and red, is recommended.
Measurement Progress Bar and Elapsed Time Indicator
In the lower left of Figure 1, Measure panel, are a measurement progress bar labeled Completed and a numeric indicator showing how many minutes have elapsed since the measurement began. Users with tight schedules find these handy.
Curve Family Graphing during Measurement
When sweeping one or more parameters in a measurement, it is ideal for the user to be able to watch graphs growing as the measurement progresses. These graphs will each have a measured parameter as the y axis and a swept parameter as the x axis.
When two parameters are swept, the graph is a family of curves, with each curve representing a different value of the outermost loop. The value for a curve is shown as either a number just next to the curve or in a separate legend to the side of the graph. If more than two parameters are swept, the curve family graph can simply clear and begin anew for each new value of the third loop.
An alternative type of graph for three or more swept parameters is the 3D surface plot, employing fine shading and point-source lighting. Each new third loop begins to draw a new, translucent surface. The translucency allows lower surfaces to be seen to some extent. Mouse-directed, omnidirectional rotation allows flexibility in viewing the surfaces. Be aware that not every three-parameter sweep will form a 3D surface plot that is meaningful to an observer.
Numeric Indicators during Measurement
On the panel during the measurement, several numeric indicators should be displayed and updated as the measurement progresses. These include:
When tests are run repeatedly, or when in an environment where a mouse and keyboard are inconvenient to use, tangible controls are a nice feature. These controls can include:
Sliders are nice controls to use but are an interesting case because of initialization issues. When a measurement or configuration is loaded, all controls need to go to their saved positions. There are two choices: 1) the computer can tell the operator where to position the slider, or 2) a computer-controlled electric motor can position the slider.
Figure 2. "Configure: Sweep" panel. By
selecting a parameter in the listbox on the left, then clicking
an Enter button, the user can select which parameters to sweep
and in what order.
Save and Recall Ability for All Test Configurations
Figure 2 shows an example of one of seven configuration panels. All the settings on every panel can be saved to and recalled from a configuration file. Appropriate settings are also saved with every measurement data file. The goal is to be able to recall from disk any setting one makes, by finding either a desired configuration file or desired data file.
Backwards compatibility is a key feature of configuration files and data files. When the test system software is upgraded, and the formats of these files change, the older formats must still be able to be read. Version numbers in the file headers help to accomplish this backwards compatibility.
Events Associated with Settings
Changes in certain settings may require hardware or communication events, such as running a motor. Associating this event as part of the setting data structure makes good system architecture [4].
The DUT configuration panel contains controls and indicators for:
The Channels configuration panel displays an editable table of every stimulus and measurement channel. For each channel, the user can set signal hardware source (chassis, board, or separate test equipment), signal type and direction, voltage and current limits, connector and pin numbers, and averaging and normalizing to be performed when measuring.
Equipment and Laboratory Identification
In the Equipment and Laboratory configuration panels, the user enters:
A very useful tool is the ability to program and execute a sequence of different kinds of measurements. To do this, the user employs a test sequence editor, either a commercial variety such as National Instruments's Test Stand or a custom editor like that found in [5]. The test sequencing feature can be very complex, and can be time-consuming to develop and integrate with the rest of the software. Some of the advanced features of test sequences are:
Measurement Results Database
Measured data is stored in a highly structured database for ease of browsing, searching, and retrieval. The measurement can be configured to automatically save the data every time the measurement is run. Other saving options are:
In addition to the data, much other information is saved in the data file. This information includes all measurement configurations and related items, including:
Regardless of the format in which the data is routinely stored, any data file should be exportable to most of the common data formats, especially tab-delimited text spreadsheet format. Users will invariably want to import the data into other computer applications.
Flexible, thorough report printing is an important feature. Printed copy is the traditional means of conveying data to the interested parties (customers and bosses). The report printing features of the test system should provide neat, flexible layouts that include graphs (line curves, bar graphs, 3D surface plots) and tables. Measurement reports should include as much of the measurement configuration information as practical, preferably the whole configuration. If this information will not fit on the same page as the graph, a separate page just for configuration information is acceptable.
The world-wide-web has emerged as the most powerful way to convey test results [9]. Measurement results can be published to the web automatically or upon the user's command to Publish. The web pages have the same information as described in Report Printing above, but the hypertext, graphical web interface allows more flexibility and thoroughness.
Web publishing is truly useful in conveying results immediately to interested parties. A lot of miscommunication and aggravation can be avoided when the measurement results are promptly available to a customer or a boss, be they anywhere in the world (provided they know the access password).
More challenging to program, but quite doable with tools that have emerged in the last few years, is running, controlling, and real-time monitoring of the test system remotely via a web browser [10].
Results Database Browsing and Searching
The ATS should structure the database, and give the user the proper interactive tools, to enable flexible browsing and searching of stored results. Results should be searchable by ranges of measured values, by stimuli and settings, and by type of measurement (e.g. sweep settings). And, when the system software is upgraded, old data file formats should still be able to be searched, a feature called "backwards compatibility."
Figure 3. "Analyze"
panel. Data sets listed in the box at the left can be selected
and added to the graph or put into the equation above the graph
for arithmetic manipulation. The Slice button allows the user to
choose which curve of a two-dimensional data set, or which curve
family of a 3D data set, to plot. The Edit button allows editing
of data, such as selecting only a portion of the data to plot.
The ability to display multiple curves in different ways is an important R&D tool. An engineer often needs to compare one measurement with another, to look for differences between devices or between measurements at different settings and to look for correlation among different measured parameters.
Curves should be displayable in separate graphs stacked one on top of another or as overlaid plots on one graph. Overlaid plots are often best on a graph that has two or more different Y axes to show plots with different scales or units. Two moveable cursors, with a "cursor deltas" display, enable examination of separation between points (Figure 3).
The configuration of the Analysis panel should be saveable as an analysis set-up file for possible future recall.
The user should be able to add, subtract, multiply, and divide measured curves [5], and to add or multiply any constant with a curve. It is useful to be able to select just a portion of the curve for an arithmetic operation.
The resulting calculated curve should be displayed on a graph and added to the list of Data Sets shown on the analysis screen (Figure 3). At any time, the newly calculated data sets can be saved, all together or selectively. Later, these calculated data sets can be recalled to the analysis list, possibly replacing items of the same name, merging old calculations with new as desired.
A useful feature is to be able to fit the data with a straight line, a polynomial curve, or any desired linear or non-linear equation, and to be able to view and store the best-fit curve and equation coefficients.
Curve Construction or Manual Data Entry
At times, a user needs to display or perform arithmetic with a curve that is not the result of a system measurement. A feature allowing manual entry of a curve, whether by keyboard or by mouse on a graph, is useful.
Statistics, Histograms, and FFTs
With many stored measurements of the same type, or with continuous logged data points of the same quantity, a need arises for statistical and frequency spectrum analysis. Histograms are very useful, with accompanying calculations of average, peak value, and standard deviation. For histograms, it is especially useful to be able to clip the data and then to display words to the effect that, for example, 7 data points are outside the bounds of the histogram.
"Integrity" refers to those features that are not necessary for the test system to function at the simplest level but that are necessary for the results to represent the truth, for the system to be practical and useful for the laboratory, and for the system to be reliable and in operating condition most of the time.
Operator Instructions and Help Screens
Thorough, organized operator instructions and help screens are wonderful features that are time-consuming to develop but well worth the time and effort, especially if less-skilled personnel will be running the test system.
Instruction and help screens of an ATS ideally include not just text, but also photographs, diagrams, audio, and/or video [11]. These screens are best viewable with a web browser, and then can be viewed remotely [7]. Instructions and help can pop up on the computer display in several circumstances:
Especially effective is giving the operator the ability to annotate the instruction and help screens. By adding text or audio notes, the operator can pass lessons learned to future users.
It's often useful to require the system user to "log on" with a username and password. Each username is assigned one of four levels: Operator, Technician, Engineer, and Administrator, from lowest to highest authority. The Administrator is the only person with the ability to set up usernames, passwords, and levels.
In most systems with user levels, the lower-level users are denied access to certain menu items, usually preventing them from making any changes in the system configuration or stimulus and measurement parameters. In an R&D lab test system, however, denying access to menu items is contrary to the desired role of the test technician [7]. A better approach is that, if a user wants to change any configuration items, the user will not be able to change the configuration file on disk unless he or she is the same or higher level than the user who first created the file. If the user is a lower level, he or she must save the configuration under a new filename. And only the Administrator can set which configurations are loaded at system boot-up.
Calibration of Stimulus and Measurement Channels
Routine, automated calibration of stimulus and measurement channels has been emphasized by many as important to an ATS [4], [12]. Using standard reference sources and meters in an official calibration procedure helps to assure the accuracy and precision of a test system's measurements.
Distinction is made between full equipment calibration and simple drift calibration. While full calibration need only be done once a year or even less often [12], drift calibration should be an automated part of the measurements. Automated drift calibration should be performed as often as warranted by the statistics of the drift or "random walk" [13]. In the case of significant, frequent drift, autozeroing should be performed before and after each measurement [4].
Ability to store calibration results is a good feature. The history of calibration offsets and gains can be viewed to examine the long-term integrity of a stimulus or measurement channel.
Abort, Restart, and Emergency Stop
Being able to abort the measurement at any point is a necessary feature. Figures 1 - 3 show the Abort button in the lower right of the screen, but it is best that the abort button be a big, external, tangible, red button.
For tests that take a long time, values are appended to the data file as they are measured. If the test is aborted in the middle, or even if the power goes out, the data up to that point is intact. The user should be able to restart the test where it left off if he or she would rather not start it over from the beginning.
An Emergency Stop button is needed for systems that can short, arc, overheat, or crush themselves from motor-driven motion. Hitting the obvious Emergency Stop button will stop the test and kill power to the equipment.
Flexible Hardware-Software Interface
A flexible hardware "abstraction" layer has been discussed in [2], [14]–[16]. This interface layer between the hardware and software allows painless portability of the software to different hardware. It also allows a piece of hardware, such as a PC I/O board or separate test instrument, to be exchanged with another piece of hardware of similar functionality but with possibly a different manufacturer and communication protocol without requiring much software modification. The interface is realized largely through the stimulus and measurement channels configuration table discussed previously.
Some standard classes of portable drivers are:
Hardware Connection and Function Checking
Automatically, before a measurement is run, or interactively when the operator selects Diagnostics, the test system hardware can be checked by the software for properly working connections and functionality. This hardware checking should include polling of test equipment and circuit boards to see if they are turned on, connected, and communicating. It should also include detection of faults in the signal path [12].
Signal path fault detection and isolation is a powerful feature that is complex to design. A sequence of separating connections and inserting loop-back assemblies can determine exactly which section of the circuitry, connectors, or cabling contains a fault. Once a fault is isolated, the test system can be operable again shortly if spare hardware has been stocked. Stocking spares is therefore a very worthwhile investment.
The test system software should catch errors, display an error message on the screen, and take appropriate action for a variety of error conditions [3], including:
Many ATSs have the characteristic that if they are missing any of their hardware, they don't run. An ATS should be able to at least run analyses on stored data without doing anything with hardware. Even better, a measurement should be able to run with a simulated DUT [3]. Simulation enables system testing and operator training to be done before real measurements take place. And, if a problem is suspected, simulated data aids in troubleshooting the software without worrying of its being corrupted by hardware.
Applicability to Manufacturing or Maintenance Test
This paper has presented recommended features for ATSs in the R&D laboratory. However, most ATSs that have been fielded perform manufacturing or maintenance test. Manufacturing and maintenance test systems usually do not have most of these R&D features, but they certainly benefit from their inclusion. These features give a test system beneficial flexibility. They allow an interested user, of any level, to analyze a DUT further, probing for hidden performance issues or for causes of failure. As discussed previously and in [7], performing deeper analysis produces a more effective test technician.
Manufacturing ATSs require certain additional features:
The usual product development sequence is R&D followed by transfer to manufacturing. The corresponding ATS development process is explained in [17]. R&D tests can be designed with future factory needs in mind. The R&D test modules can be reused by manufacturing. And, while performing R&D measurements, statistics on measured values can be generated to arrive at pass/fail limits for manufacturing test.
Maintenance ATSs require certain additional features as well:
A list of useful features of an R&D test system has been presented with, we hope, the right amount of explanation. The list is ever-evolving. Please e-mail the author if you have anything to add to it.
The author thanks the following for most of his experience that led to this paper: Steve Temme and the SoundCheck software of Listen, Inc., Boston, Massachusetts; the Applications Group of Photoelectron Corp., Lexington, Mass.; the infrared focal-plane-array test lab of Lockheed Martin IR Imaging Systems (formerly Loral), Lexington, Mass.; and the infrared detector development group of Raytheon, Research Division, Lexington, Mass.