NAME xyz2pipe, pipe2xyz - Read and Write 3D and 4D Spectral Data SYNOPSIS xyz2pipe -in inTemplate > outStream [-out outName] [-fdata loc1 parm1...] [-x | -y | -z | -a] [-ri2c] [-nofs] [-hdr] [-par sListName] [-cpu cpuID] [-first firstPlane] [-last lastPlane] [-mem memMB] [-verb] [-help] pipe2xyz -out outTemplate < inStream [-in inName] [-fdata loc1 parm1...] [-x | -y | -z | -a | -nv] [-ri2c] [-nodir] [-nofs] [-hdr] [-first firstPlane] [-last lastPlane] [-mem memMB] [-mt maxTries] [-to uSecTimeOut] [-inPlace] [-verb] [-help] DESCRIPTION The xyz2pipe program reads a series of one-dimensional vec- tors from the specified axis of a multidimensional spectral data set. It is used in conjunction with the programs nmrPipe and pipe2xyz to process multidimensional data. The input required by xyz2pipe is a series of 2D files in the NMRPipe data format. This input series will usually have been created by a spectrometer format conversion pro- gram such as bruk2pipe, var2pipe or bin2pipe, or by a previ- ous processing scheme. In order to understand the use of xyz2pipe and its comple- mentary program pipe2xyz, it might be helpful to review the details of the nmrPipe processing program first. Details are presented in the nmrPipe manual pages. A typical nmrPipe 3D or 4D processing scheme consists of several programs connected together by UNIX pipes; a sequence of programs connected this way is called a pipe- line. At the head of the processing pipeline, xyz2pipe reads data from the input files, and sends the input vectors down the pipeline. In the interior of the pipeline, the nmrPipe program applies processing functions to the stream of vectors. At the end of the pipeline, the program pipe2xyz gathers the processed vectors and writes them to an output data series. The xyz2pipe program serves three important functions in a processing scheme: It provides a convenient way to construct a single stream of data from a multi-file 3D or 4D input, so that the pipeline technique of nmrPipe can be used for spectral processing. It provides access to data vectors from any dimension of a 3D or 4D data set, without the need to specify anything other than the source of data and the dimen- sion to use. It automatically provides the correct combination of real and imaginary data for the specified dimension, so that dimensions can be conveniently processed or re-processed in any order. SPECIFYING THE INPUT TO PROCESS As mentioned above, the multidimensional input dataset for xyz2pipe is a series of files, each of which is a 2D plane. The files in the series have a systematic set of names which includes the plane number as part of the name. For example, a time-domain input series of 64 files might have names such as hnca001.fid, hnca002.fid, ... hnca064.fid. Rather than specifying the entire series of input files explicitly, xyz2pipe uses an input template to specify the input series; the template is a single name that stands for the entire series. The template name contains an integer format field which shows where the plane number will be inserted into the actual file name. The integer field is specified according to the rules of the C Programming Language. According to these rules, the template for the hnca example above would be "hnca%03d.fid". In this tem- plate, the integer field is specified by the "%03d" part of the template: The "%" indicates the beginning of the field where the plane number will be inserted into the file name. The "d" indicates the end of the field where the plane number will be inserted into the file name. The "3" indicates that a minimum of three digits should be used to represent the plane number. The "0" indicates that the plane number will be padded on the left with zeros when needed, to give the speci- fied number of digits. So, for example, the integer value for plane 1 becomes "001". It is recommended that template names should not have numerals immediately before or after the integer field, as in "hn15%03d.fid". SPECIFYING THE DIMENSION TO PROCESS In the xyz2pipe nomenclature, a 3D input consists of an X-Axis, a Y-Axis, and a Z-Axis, while 4D data also includes an A-Axis. The X-Axis refers to data vectors from rows in the 2D file planes, while the Y-Axis refers to data vectors from columns in the 2D file planes. The Z-Axis and A-Axis refer to data vectors taken from across the appropriate series of files. Each axis corresponds to one detection dimension in the spectrum. The xyz2pipe program will read vectors from a given dimen- sion of an input series according to its axis argument, which can be either -x, -y, -z, or -a. Correspondingly, the pipe2xyz program will write vectors to a given dimension according to similar axis arguments. Both programs read and write data in a predictable, sequential order. For example, consider a Triple Resonance 3D HNCO FID, where the X-Axis is the directly-detected HN (amide proton) dimen- sion, the Y-Axis is the CO 13C (carbonyl) detection dimen- sion, and the Z-Axis is the 15N (amide nitrogen) detection dimension. The following simple processing scheme would apply a window function, a zero fill, and a Fourier Transform to all the vectors in the HN dimension, while extracting the left half of each HN vector and deleting the HN-imaginary data along the way. In the example, the input data comes from the file series hnco001.fid ... in the sub-directory fid, and the output result is written to the series hnco001.ft1 ... in the sub-directory ft: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | pipe2xyz -out ft/hnco%03d.ft1 -x In the above example, xyz2pipe reads HN vectors from the X-Axis, one plane at a time; the vectors are processed by instances of nmrPipe, and the processed HN vectors are writ- ten by pipe2xyz. In this case, since both xyz2pipe and pipe2xyz have the same axis argument -x, the order of data in the input and the output remains exactly the same. So, the X-Axis of the output is HN, the Y-Axis is CO, and the Z-Axis is 15N. The following scheme could be used to process the CO dimen- sion of the previous result: xyz2pipe -in ft/hnco%03d.ft1 -y \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft2 -y Since the -y argument is used, xyz2pipe now supplies CO vec- tors from the Y-Axis of the input; it does this by reading the input data one plane at a time, and exchanging the X-Axis and Y-Axis vectors within each plane before sending the data to the pipeline. This means that the processing functions in the pipeline receive the data as: X=CO, Y=HN, and Z=15N. Correspondingly, pipe2xyz receives data at the end of the pipeline one plane at a time, and since it is also used with the -y argument, it exchanges the X-Axis and Y-Axis vectors back again before writing the result, restoring the original dimension order. To finish the 3D transform, the 15N vectors from the Z-Axis could be processed with a similar scheme: xyz2pipe -in ft/hnco%03d.ft2 -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft3 -z In this case, xyz2pipe now supplies 15N vectors from the Z-Axis of the input; the mechanism is more complicated than the -x or -y cases, but the result is that the data is sent to the pipeline in the order X=15N, Y=HN, and Z=CO. At the end of the pipeline, pipe2xyz reorganizes the data in a com- plementary way before writing the result, to restore the original dimension order. The above three schemes taken together form a basic but com- plete 3D Fourier Transform, where the dimensions are pro- cessed one at a time in turn, and the intermediate results are saved on the disk at each stage. Most 3D processing schemes will include use of additional processing functions such as solvent filtering, phasing, and baseline correction. But, importantly, they will also include changes to the nature of the scheme itself, such as the following elements described below: Process two or more dimensions in one pass by the use of the 2D Transpose function TP, or the 3D Transpose function ZTP; this saves disk space and improves pro- cessing speed. Process data in-place when possible to conserve disk space. Process the dimensions in a different sequence, for instance by processing the Z-Axis before processing the Y-Axis. This is usually done to accommodate special signal processing methods such as Linear Pred- iction. Specifically re-order the dimensions in the data out- put, in support of a particular post-processing analysis plan. USING 2D TRANSPOSE IN 3D SCHEMES The nmrPipe Transpose processing function TP exchanges the X-Axis and Y-Axis vectors of its input planes, one plane at a time. So, it can be used to simplify and improve the per- formance of a 3D or 4D processing scheme, by avoiding the need to save an intermediate result on the disk. For exam- ple, our three stage processing scheme above could be replaced with the following more efficient scheme, which still produces exactly the same final result: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft2 -y xyz2pipe -in ft/hnco%03d.ft2 -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft3 -z Besides the use of the Transpose function TP, this two-stage example differs from the previous examples in another respect: it includes use of xyz2pipe and pipe2xyz together with different axis arguments, in this case -x and -y. The axis arguments have been chosen so that the input and output dimension orders will remain unchanged; the Transpose func- tion TP exchanges X-Axis and Y-Axis vectors during process- ing, while pipe2xyz -y exchanges them again before writing the result. The net outcome is that the data order of the output stays the same. Of course, there is no special reason that the data order needs to stay the same before and after processing. So, for most purposes, the following version of the processing scheme would be acceptable, and perhaps a little faster: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft2 -x xyz2pipe -in ft/hnco%03d.ft2 -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft3 -z Note that the only difference between the two examples is the use of pipe2xyz -x instead of pipe2xyz -y. In the latter example, the dimension order of data in the output will be changed, since the Transpose function TP will exchange the X-Axis and Y-Axis vectors, but output via pipe2xyz -x will not exchange them back again. The end result will therefore have X=CO, Y=HN, and Z=15N. In other words, the 2D file planes of the final result will be tran- sposed relative to the original data. IN-PLACE PROCESSING In all of the examples so far, each processing pipeline reads data from one series of files, and writes data to a different series of files. However, it is possible in many cases to use the same files for both input and output, so that the input data is replaced with the processed result directly; this is called in-place processing. In cases where the intermediate results are not needed, it can sub- stantially reduce disk space used during processing. In order to perform in-place processing correctly, the argument -inPlace must be used with pipe2xyz at the end of any pro- cessing pipeline where the input is the same as the output, for example: xyz2pipe -in data/hnco%03d.dat -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out data/hnco%03d.dat -y -inPlace xyz2pipe -in data/hnco%03d.dat -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out data/hnco%03d.dat -z -inPlace There are some restrictions to the use of in-place process- ing: Input from xyz2pipe -x or xyz2pipe -y must be output to either pipe2yxz -x or pipe2yxz -y. Input from xyz2pipe -z must be output only to pipe2yxz -z. Input from xyz2pipe -a must be output only to pipe2yxz -a. 4D data file series must use the two-field template names, described in the section on 4D Processing below. CHANGING THE ORDER OF PROCESSING In the examples so far, each dimension was processed in order; first the X-Axis, then the Y-Axis, and finally the Z-Axis. However, in principal, the dimensions can be pro- cessed in any order. In practice, certain orders of pro- cessing may be more advantageous than others. In most cases, the directly-detected dimension will be processed first, especially in situations where solvent filtering is needed, or in cases where a large part of the acquisition spectral window has no interesting signal and can be dis- carded during processing. Use of Linear Prediction (LP) to extend a given dimension will often motivate a particular order of processing. LP will work best when the time-domain vectors to extend have the fewest possible signals. In order to achieve this, all of the other dimensions of the spectrum should be Fourier Transformed first; this will localize the spectral signals in the transformed dimensions and simplify the remaining time-domain dimension. For example, to apply Linear Prediction to the CO dimension of the HNCO data above, the HN (X-Axis) and 15N (Z-Axis) dimensions will be processed first: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | pipe2xyz -out ft/hnco%03d.ft1 -x xyz2pipe -in fid/hnco%03d.fid -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | pipe2xyz -out ft/hnco%03d.ft1 -z xyz2pipe -in fid/hnco%03d.fid -y \ | nmrPipe -fn LP -fb \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | pipe2xyz -out ft/hnco%03d.ft1 -y CHANGING THE ORDER OF DIMENSIONS Depending on the programs and approaches which will be used after processing to analyze a spectrum, it may be desirable to change the data order in a specific way, either during processing or afterward. For example, the HNCO schemes above all provide results where the output 2D file planes have HN and CO dimensions. But, if the 2D file planes with HN and 15N dimensions are needed instead, xyz2pipe and pipe2xyz can be used together with different combinations of axis arguments: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft2 -y xyz2pipe -in ft/hnco%03d.ft2 -z \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft3 -x In this example, the intermediate result hnco001.ft2 ... has a data order the same as the original, X=HN, Y=CO, Z=15N. In the second stage, xyz2pipe -z reorganizes the data so that the X-Axis is 15N, the Y-Axis is HN, and the Z-Axis is CO; at the end of this pipeline, pipe2xyz -x saves this data without changing the order, so that the 2D output planes have 15N and HN dimensions. Note that data can also be re-organized without intermediate processing steps, by sending the output of xyz2pipe directly to pipe2xyz: xyz2pipe -in ft/hnco%03d.ft3 -z \ | pipe2xyz -out ft/hnco%03d.dat -x SUMMARY OF AXIS ORDERS As shown in the previous examples, xyz2pipe and pipe2xyz re-organize the spectral data according to the specified axis arguments. A summary of the data order produced by each option is given here as a reference: xyz2pipe -x: leaves the data order unchanged when sending vectors to the pipeline; this is represented by the notation XYZ->XYZ for 3D data, or XYZA->XYZA for 4D data. xyz2pipe -y: exchanges the X-Axis and Y-Axis data before sending vectors to the pipeline. This is represented by the notation XYZ->YXZ for 3D data, or XYZA->YXZA for 4D data. xyz2pipe -z: shifts the position of the the X-Axis, Y-Axis, and Z-Axis before sending vectors to the pipe- line (circular right-shift); the original Z-Axis becomes the X-Axis, the original Y-Axis becomes the Z-Axis, and the original X-Axis becomes the Y-Axis. This is represented by the notation XYZ->ZXY for 3D data, and XYZA->ZXYA for 4D data. xyz2pipe -a: shifts the position of the the X-Axis, Y-Axis, Z-Axis, and A-Axis of 4D data before sending vectors to the pipeline (also a circular right-shift); the original A-Axis becomes the X-Axis, the original Z-Axis becomes the A-Axis, the original Y-Axis becomes the Z-Axis, and the original X-Axis becomes the Y-Axis. This is represented by the notation XYZA->AXYZ. pipe2xyz -x: leaves the data order unchanged when sav- ing the result; this can be represented by the nota- tion XYZ->XYZ for 3D data, or XYZA->XYZA for 4D data. pipe2xyz -y: exchanges the X-Axis and Y-Axis data before saving the result. This can be represented by the notation XYZ->YXZ for 3D data, or XYZA->YXZA for 4D data. As a reminder that this is complementary to xyz2pipe -y, it can also be represented by YXZ->XYZ or YXZA->XYZA. pipe2xyz -z: shifts the position of the the X-Axis, Y-Axis, and Z-Axis before saving the result (this time, a circular left-shift); the original Z-Axis becomes the Y-Axis, the original Y-Axis becomes the X-Axis, and the original X-Axis becomes the Z-Axis. This can be represented by the notation XYZ->YZX for 3D data, and XYZA->YZXA for 4D data. As a reminder that this is complementary to xyz2pipe -z, it can also be represented by ZXY->XYZ or ZXYA->XYZA. pipe2xyz -a: shifts the position of the the X-Axis, Y-Axis, Z-Axis, and A-Axis of 4D data before saving the data (also a circular left-shift); the original A-Axis becomes the Z-Axis, the original Z-Axis becomes the Y-Axis, the original Y-Axis becomes the X-Axis, and the original X-Axis becomes the A-Axis. This can be represented by the notation XYZA->YZAX. As a rem- inder that this is complementary to xyz2pipe -a, it can also be represented by AXYZ->XYZA. USING THE 3D TRANSPOSE FUNCTION Several examples above include use of the 2D Transpose func- tion TP as a way to avoid saving an intermediate result; in these examples, 3D transform schemes were reduced to only two stages, rather than three. The nmrPipe program also includes a 3D Transpose function, ZTP, which can be used in a similar way. The ZTP function exchanges the X-Axis vec- tors and the Z-Axis vectors; this makes it possible to con- duct a 3D Fourier Transform in a single pass, without any intermediate result: xyz2pipe -in fid/hnco%03d.fid -x \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn EXT -left -di \ | nmrPipe -fn YTP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | nmrPipe -fn ZTP \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/hnco%03d.ft3 -z Interestingly, in the above example, the dimension order of both the input and the output is the same. The initial data order is X=HN, Y=CO, and Z=15N; after the 2D Transpose YTP (another name for the TP function), the X-Axis and Y-Axis vectors are exchanged, giving X=CO, Y=HN, and Z=15N. The ZTP function then exchanges the X-Axis and Z-Axis vectors (XYZ->ZYX), giving X=15N, Y=HN, and Z=CO. At the end of the pipeline, use of pipe2xyz -z shifts the dimensions XYZ -> YZX before the result is saved, restoring the original order. Note that the ZTP function performs a similar but not ident- ical or complementary re-ordering of the data as xyz2pipe -z and pipe2xyz -z. Note also that the ZTP function should be used carefully, since it requires enough memory to fit an entire 3D spec- trum. PROCESSING 4D DATA Access to the forth dimension of 4D data is provided by xyz2pipe -a and pipe2xyz -a, in the same manner as for the other dimensions. But in the case of 4D data, its helpful to use a somewhat different naming convention for the results. In previous 3D examples, the file templates used included a single three-digit integer field for the plane index. In the case of 4D data, the templates will usually have two different integer fields, the first one for the index number of the A-Axis, and the second one for the index number of the Z-Axis. This is accommodated by a two-digit field "%02d" followed by a three digit field "%03d": xyz2pipe -in ft/n15c13noe%02d%03d.DAT -a -verb \ | nmrPipe -fn SP -off 0.5 \ | nmrPipe -fn ZF -size 64 \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -90 -p1 180 -di \ | pipe2xyz -out ft/n15c13noe%02d%03d.DAT -a -ov -inPlace In this example, the file series will have names such as n15c13noe01001.DAT, where the first "01" corresponds to the A-Axis index, and the following "001" corresponds to the Z-Axis index. Complete 4D processing schemes are given in the EXAMPLES section below. XYZ2PIPE OPTIONS -in inTemplate This option specifies the source of the input data. The inTemplate argument can be one of the following three: Single-field Template: this is a file name with one integer-field specification inserted. The field specification is made according to the rules of the C Programming Language; "%03d" is the most common template for 3D data. Example: test%03d.dat. Single-field template names are also called "Z For- mat Templates". Two-field Template: this is a file name with two different integer-field specifications inserted. It is used for 4D data, where the first field can hold the A-Axis index, and the second field can hold the Z-Axis index. The field specification is made according to the C Programming Language. Example: test%02d%03d.dat. Two-field template names are also called "AZ Format Templates". File-Name List: this is the name of a text file which explicitly lists the series of 2D file planes in the input, one file name per line. -out outName This option can be used to write the output stream to a single file rather than to the standard output. It is provided for use with other programs which may require single-file data. -fdata loc1 parm1... The nmrPipe data format includes a header array of 512 four-byte floating point values at the beginning of the data stream. The -fdata option allows adjustment of individual file header values in the output, specified as header location/header value pairs; the locations are specified as origin=1, and the values are floating point. The locations, their meanings, and values are outlined in sethdr. -x This option is used with xyz2pipe to provide a data stream of 1D vectors from the X-Axis of the input. It leaves the data order unchanged: XYZ->XYZ. -y This option is used with xyz2pipe to provide a data stream of 1D vectors from the Y-Axis of the input. It exchanges the X and Y dimensions of the input: XYZ->XYZ. -z This option is used with xyz2pipe to provide a data stream of 1D vectors from the Z-Axis of the input. It shifts the X, Y, and Z dimensions of the input: XYZ->ZXY. -a This option is used with xyz2pipe to provide a data stream of 1D vectors from the A-Axis of the input. It shifts the X, Y, Z, and A dimensions of the input: XYZA->AXYZ. -ri2c In the multi-file input format used by xyz2pipe, com- plex data in the Z-Axis or A-Axis is interleaved, whereas in the X-Axis complex data is separated into real and imaginary vectors. Using this flag with the -z or -a options will force the X-Axis data to be interleaved before the data is re-arranged to give vec- tors from the Z-Axis or A-Axis. It is intended for use in schemes to extract single X-Axis/Z-Axis or X-Axis/A-Axis planes from time-domain data. Future versions of xyz2pipe may perform this action automati- cally when needed. -nofs This flag will suppress any file-size checking of input or output files used by xyz2pipe. By default, only the first 2D file plane of the input series will be tested, along with the output file if one has been specified. -nodir This flag will suppress automatic directory creation for the output files written by pipe2xyz. It is included to accommodate cases where automatic creation may fail. -hdr This option enables a check of all the file parameter headers in the input series to ensure that they are consistent. It also enables file-size checking of the whole file series, rather than just the first plane. The checking procedures can be time consuming, espe- cially for 4D data, so they are limited by default. -first firstPlane -last lastPlane These options are used to create an output stream from a sub-set of the input data, rather than the entire 2D series. If processing was not performed in-place, the options can be used to continue an interrupted process- ing session. The firstPlane and lastPlane arguments specify the first and last input planes in the range of planes to process. When the -first or -last options are used, information about the range of planes taken from the input is tem- porarily recorded in the data parameter header, for interpretation by other programs. In particular, this means that program pipe2xyz can automatically handle processed results from a sub-set of the data by writing the sub-set into the correct position of the overall result. By default, it is the first plane of the data, plane 1. The range of planes used as input depends on which axis argument is used: -x or -y: the input plane range extends from 1 to the total number of input file planes (ZSIZE*ASIZE). -z or -a: the input plane range extends from 1 to the number of vectors in a given 2D file plane (YSIZE). -par serverListName This option is used to provide for a simple mode of parallel processing with static load balancing. This means that there is no compensation for changing speeds of each computer during processing. To use this mode, a given processing scheme is executed simultaneously on two or more computers, each of which will process a different sub-set of the data in a given dimension. Each computer decides which sub-set to process accord- ing to the information in the serverListName text file. The argument serverListName is the name of a text file which lists the hostnames of the computers to use, along with arbitrary numbers indicating their estimated relative speeds. In this example, the three computers named spin, spam, and spite are listed; according to the listing, computer spite is 4 times faster than com- puter spin, while computer spam is 2 times faster than computer spin: spite 4 spam 2 spin 1 Programs nmrShell and waitFor are provided to execute and synchronize parallel processing schemes. See the nmrShell manual page for for more details. -cpu cpuID This argument is used with the "-par serverListName" option to select a particular CPU on a multi-processor computer. In such cases, each CPU of the multi-processor machine has its own entry in the server list file, but all of these entries have the same host- name. The cpuID specifies which of these entries applies for the current process. The first entry for a given hostname has a cpuID of 0, the next entry has a cpuID of 1, etc. The default value of cpuID is 0. Programs nmrShell and waitFor are provided to execute and synchronize parallel processing schemes. See the nmrShell manual page for for more details. -mem memMB This option can be used to provide a suggested value for the amount of memory in megabytes to allocate for re-organizing the data. This option is provided for testing and performance tuning. By default, the memory size will be set automatically: For the -x mode, the memory size will be fixed at the size of one 2D file plane. For the -y mode, the memory size will be fixed at twice the size of one 2D file plane. For the -z mode, the memory size will be set to four times the size of an X-Axis/Z-Axis plane from the input. For the -a mode, the memory size will be set to four times the size of an X-Axis/A-Axis plane from the input. -verb If this option is used, xyz2pipe will print status mes- sages during processing; the status messages are sent to standard error. -help This option will cause xyz2pipe to print a terse list of command-line options. This help text is sent to standard error, and it can be re-directed to a file with the ">&" idiom: xyz2pipe -help >& help.out EXAMPLES HNCO Example 1: (X=HN Y=CO Z=N) the following example shows the most often-used scheme for 3D Fourier processing. This amide-detected data has signals only on the left side of transformed HN vectors, so only the left side is retained during processing. #!/bin/csh xyz2pipe -in fid/test%03d.fid -x -verb \ | nmrPipe -fn SOL \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 43 -p1 0.0 \ | nmrPipe -fn EXT -left -di -sw \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 1.0 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -135 -p1 180 -di \ | pipe2xyz -out ft/test%03d.ft2 -y xyz2pipe -in ft/test%03d.ft2 -z -verb \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -z HNCO Example 2: shows a somewhat different processing than the data above. In this case, the POLY function is used in the time-domain to fit and remove the solvent. It is used again to apply an automated baseline correction to the HN vectors, but only after the planes have been transformed in both dimensions. In addition, a specific range of chemical shifts is extracted from the HN-vectors, rather than just the left side. #!/bin/csh xyz2pipe -in fid/test%03d.fid -x -verb \ | nmrPipe -fn POLY -time \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 43 -p1 0.0 \ | nmrPipe -fn EXT -x1 5.5ppm -xn 11ppm -di -sw \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 1.0 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -135 -p1 180 -di \ | nmrPipe -fn TP \ | nmrPipe -fn POLY -auto \ | pipe2xyz -out ft/test%03d.ft2 -x xyz2pipe -in ft/test%03d.ft2 -z -verb \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -z Extracting an XZ Plane: the first XY plane of an FID is often processed before the complete 3D, to confirm process- ing parameters and inspect the data. It is also possible to use xyz2pipe to extract the first XZ plane so that it too can be processed for testing: #!/bin/csh xyz2pipe -in fid/test%03d.fid -z -ri2c -first 1 -last 1 \ | nmrPipe -fn TP -hyper \ | pipe2xyz -out xz%03d.fid -x -ov nmrPipe -in xz001.fid \ | nmrPipe -fn POLY -time \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 43 -p1 0.0 -verb \ | nmrPipe -fn EXT -left -di -sw \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 \ | nmrPipe -fn ZF \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di -verb \ -out test.ft2 -ov N15 NOE Example 1: (X=HN Y=H Z=N) in this example, the indirectly-detected H dimension was acquired with a one-point delay time; this would normally require a P0=- 180,P1=360 phase shift. As an alternative, this scheme shifts the time-domain data to the right by one point, eliminating the need for the P0=-180,P1=360 phase shift. However, since the first point of the time-domain is effectively missing, there will be a zero-order baseline distortion in the corresponding frequency-domain data. This problem is solved by applying an automatic zero-order baseline correction. #!/bin/csh xyz2pipe -in fid/test%03d.fid -x -verb \ | nmrPipe -fn SOL \ | nmrPipe -fn SP -off 0.35 -end 0.99 -pow 2 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 \ | nmrPipe -fn EXT -di -x1 5ppm -xn 10.5ppm -sw \ | nmrPipe -fn TP \ | nmrPipe -fn ZF -pad 1 \ | nmrPipe -fn RS -rs 1 -sw \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | nmrPipe -fn POLY -auto -ord 0 \ | pipe2xyz -out ft/test%03d.ft3 -y xyz2pipe -in ft/test%03d.ft3 -z -verb \ | nmrPipe -fn SP -off 0.35 -end 0.95 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/n15noe%03d.ft3 -z -inPlace N15 NOE Example 2: in this version of the scheme above, Linear Prediction (LP) is used to replace the missing first point in the time-domain, rather than using baseline correc- tion. Note that the arguments to "LP -before -x1 2 -pred 1" are included for completeness; by default, "LP -before" alone will replace the first point. This LP approach is slower than the baseline correction method above, and it generally doesn't work as well. #!/bin/csh xyz2pipe -in fid/test%03d.fid -x -verb \ | nmrPipe -fn SOL \ | nmrPipe -fn SP -off 0.35 -end 0.99 -pow 2 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 \ | nmrPipe -fn EXT -di -x1 5ppm -xn 10.5ppm -sw \ | nmrPipe -fn TP \ | nmrPipe -fn ZF -pad 1 \ | nmrPipe -fn RS -rs 1 -sw \ | nmrPipe -fn LP -before -x1 2 -pred 1 \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -y xyz2pipe -in ft/test%03d.ft3 -z -verb \ | nmrPipe -fn SP -off 0.35 -end 0.95 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/n15noe%03d.ft3 -z -inPlace LP in Two Dimensions: in this example, mirror image Linear Prediction (LP) is used in two dimensions of a double constant-time CBCA(CO)NH experiment (X=HN Y=CACB Z=N). This same example would instead apply forward-backward LP if the use "LP -ps0-0" is replaced by "LP -fb". In the approach detailed below, the processing scheme is arranged so that the 1D time-domain vectors to be extended by LP are as simple as possible (i.e. have the fewest possible signals). To achieve this, LP is only used on a given dimension when all the remaining dimensions have been processed. This localizes the spectral signals as much as possible, since in the time-domain, a given signal can extend significantly across the spectrum, but in the frequency-domain the signal will be "concentrated" into a peak. In this example, the HN and N dimensions are transformed, then the CACB dimension is Linear Predicted and then transformed. Finally, the N dimension is inverse transformed, Linear Predicted, then re-transformed. Note the use of "-inv" to undo the effects of not only the Fourier Transform, but also the phasing (if any), the zero filling (if any) and the window function. #!/bin/csh xyz2pipe -in fid/test%03d.fid -x -verb \ | nmrPipe -fn SOL \ | nmrPipe -fn SP -off 0.45 -end 1.00 -pow 2 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 125.0 -p1 0.0 \ | nmrPipe -fn EXT -left -di -sw \ | pipe2xyz -out ft/test%03d.ft3 -x xyz2pipe -in ft/test%03d.ft3 -z \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -z -inPlace xyz2pipe -in ft/test%03d.ft3 -y \ | nmrPipe -fn LP -ps0-0 \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -y -inPlace xyz2pipe -in ft/test%03d.ft3 -z \ | nmrPipe -fn HT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -inv \ | nmrPipe -fn FT -inv \ | nmrPipe -fn ZF -inv \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 -c 0.5 -inv \ | nmrPipe -fn LP -ps0-0 \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -auto \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 0.0 -p1 0.0 -di \ | pipe2xyz -out ft/test%03d.ft3 -z -inPlace 4D Examples 4D Example 1: in this example, the X-Axis and the Y-Axis (HN and H) are processed together in one pass, then the Z-Axis (13C) and A-Axis (15N) are each processed separately. #!/bin/csh xyz2pipe -in fid/n15c13noesy%05d.fid -x -verb \ | nmrPipe -fn EXT -time \ | nmrPipe -fn POLY -time \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 2 -c 0.5 \ | nmrPipe -fn ZF -size 1024 \ | nmrPipe -fn FT -di \ | nmrPipe -fn EXT -x1 10.9ppm -xn 6.1ppm -sw \ | nmrPipe -fn YTP \ | nmrPipe -fn SP -off 0.45 -end 1.0 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -size 128 \ | nmrPipe -fn FT -di \ | pipe2xyz -out ft/n15c13noesy%02d%03d.DAT -y -ov -verb xyz2pipe -in ft/n15c13noesy%02d%03d.DAT -z -verb \ | nmrPipe -fn SP -off 0.4 -end 0.95 -pow 1 -c 1.0 \ | nmrPipe -fn ZF -size 64 \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -90 -p1 180 -di \ | pipe2xyz -out ft/n15c13noesy%02d%03d.DAT -z \ -ov -verb -inPlace xyz2pipe -in ft/n15c13noesy%02d%03d.DAT -a -verb \ | nmrPipe -fn SP -off 0.4 -end 0.95 -pow 1 -c 1.0 \ | nmrPipe -fn ZF -size 64 \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -90 -p1 180 -di \ | pipe2xyz -out ft/n15c13noesy%02d%03d.DAT -a \ -ov -inPlace 4D Example 2: in this version of the scheme above, the ZTP 3D Transpose Function is used so that the Z-Axis can be pro- cessed along with the X-Axis and Y-Axis. #!/bin/csh xyz2pipe -in fid/n15c13noesy%05d.fid -x -verb \ | nmrPipe -fn EXT -time \ | nmrPipe -fn POLY -time \ | nmrPipe -fn SP -off 0.45 -end 0.95 -pow 2 -c 0.5 \ | nmrPipe -fn ZF -size 1024 \ | nmrPipe -fn FT -di \ | nmrPipe -fn EXT -x1 10.9ppm -xn 6.1ppm -sw \ | nmrPipe -fn YTP \ | nmrPipe -fn SP -off 0.45 -end 1.0 -pow 1 -c 0.5 \ | nmrPipe -fn ZF -size 128 \ | nmrPipe -fn FT -di \ | nmrPipe -fn ZTP -verb \ | nmrPipe -fn SP -off 0.4 -end 0.95 -pow 1 -c 1.0 \ | nmrPipe -fn ZF -size 64 \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -90 -p1 180 -di \ | pipe2xyz -out ft/n15c13noesy%02d%03d.DAT -z -ov -verb xyz2pipe -in ft/n15c13noesy%02d%03d.DAT -a -verb \ | nmrPipe -fn SP -off 0.4 -end 0.95 -pow 1 -c 1.0 \ | nmrPipe -fn ZF -size 64 \ | nmrPipe -fn FT \ | nmrPipe -fn PS -p0 -90 -p1 180 -di \ | pipe2xyz -out ft/n15c13noesy%02d%03d.DAT -a \ -ov -inPlace SEE ALSO bin2pipe(1), bruk2pipe(1), pipe2xyz(1), sethdr(1), showhdr(1), var2pipe(1), xyz2pipe(1) DIAGNOSTICS All help text, informational messages, and error messages are sent to standard error. Many error messages are intended to be self-explanatory, but some are included for development and testing purposes. Some common problems include: C-Shell: processing scripts must be explicitly identi- fied as C-shell scripts in order to run properly. They should always begin with the following text as the first characters of the first line: #!/bin/csh Continuation Characters: probably the most common problem encountered when using pipeline processing schemes concerns the use of the backslash character "\" to continue a command onto the next line of a shell-script. This use of the backslash character will only work correctly if there are no trailing blank characters at the end of the line. Input: problems here include use of an input file or input directory that doesn't exist, or use of an ordi- nary input file name instead of an input template. LEGAL NOTICE This program and its related software is provided "as is". The NIH, the Laboratory of Chemical Physics, NIDDK, and the authors make no warranties, either express or implied, as to any matter whatsoever with respect to the software. In particular, any and all warranties of merchantability and fitness for any particular purpose are expressly excluded. In no event will the NIH, the Labora- tory of Chemical Physics, NIDDK, or the authors be liable for any loss of profits, any incidental, special, exem- plary, or consequential damages of any nature whatsoever (including without limitation, loss of use or other com- mercial or research loss) arising out of or relating to the use or performance of the software.