Syntax of Each Keyword

Syntax:

	filtyp: (type).

Where type is

text	- Ordinary text file with each record having a defined separator.
fixed	- fixed record sizes
variable	- variable record sizes
xml	- xml files
csv	- csv files - normal csv with single records on a single line
nlcsv	- csv files where end-of-lines can be part of the data too

If the filtyp:f or v, you will need to specify the size (or for variable maximum size) of each record.

For most applications, filtyp:text means you have to also define the record separator, 'recsep' too.

Syntax for 'recsep' and 'fldsep'
Syntax:
```
	recsep: (FipSeq string)
	fldsep: (FipSeq string)
eg:		recsep:\036
```
Normally the separator will be \n or \r\n for NewLine or Carriage NewLine.
Note that if you just put '\n', 'ipformat' will automatically take any combination of CR and NL.
Syntax for 'reckey' and 'fldkey' - Define the record or field key
This defines the type and size of either the record or the field key.
Normally keys are positioned at the beginning of the record/field but optionally these can be at the end or at an offset from the beginning or end.
Syntax:
```
	reckey: (length) : (type) : (posn) : (EndChr) : (delY/N)
	fldkey: (length) : (type) : (posn) : (EndChr) : (delY/N)
```
where

length or size of key - This can be 0 for any length
type (optional) can be

a-alphabetic
u-uppercase
l-lowercase
n-number
p-printable
s-space(or tab, CR, NL or FF)
b-binary (ie anything)
x-alphanumeric
c-control (ie < 040 or >= 0177),
z-anpa hdr field (ie alnum plus non-quad/format punct.
t-punctuation

posn (optional) is the offset on the key from the start of the zone If negative count from the end of the zone
endchr (optional) is a single chr which terminates the key

The separator can be any punctuation chr as long as the same chr is used for each field. eg the following two are equal :
```
	fldkey  3,n,,|
	fldkey  3|n||\174
```
Ie the end Chr is a pipe but as that is used as a separator, use the octal value
What is a key ? The key is used really as a TYPE of PROCESSING flag for the output section.
It can be a unique record key - such as a stock code - but if you have several thousand, it is going to be unwieldy specifying all of them.
So generally we are trying to classify records into general types. For example of a text file containing schools results like :
```
	School	Pinky High
	Head	James Pinky
	Pupil	Ramsay		Macdonald	31.6
	Pupil	Thatcher	Margret		77.3
	Pupil	U-Dones		Helen		99.3
```
We can use the first field, which is alpha and variable length followed by a space.
```
	reckey:0:a
```
This can be signalled in the output section as :
```
	r="school"	(do the school bit)
	r="head"	(do the head bit)
	r="pupil"	(do the pupil bit)
```
Syntax for 'recsiz', 'fldsiz' - Define the length of fixed size records & fields
Syntax :
```
	recsiz: (length)
	fldsiz: (length)
```
where length is the size of key
Syntax for 'keycasesens' - Field and Record keys can be upper AND lower
Normally all keys - record or field - are considered to be case insensitive. So a key r = 'aaa' will pick up both AAA and aaa.
Use this command to force the difference.
Syntax :
```
	keycasesens:yes
```
Syntax for 'number' - Change the number system for specifying non-printable chrs
When a non-printable chr is specified in the form '\012' the 'number' system can be changed to decimal or hex.
The default number system is octal.
Syntax:
```
		number:dec
	or	number:hex
	or	number:oct
```
The change takes effect for all lines in the parameter file lower down until changed again by another 'number' keyword (Why you would specify different number systems for different parts, I have no idea).
So a New Line chr will be
```
		\012		octal
		\010		decimal
		\0a		hex
```
Syntax for 'wild' - Allow wild card strings using a particular chr
Syntax :
```
	wild: (Chr to use to signify a wild string)
	wild:*
```
This allows a wild string to be used when specifying record keys.
Note there is NO automatic wild string chr - you always have to specify it.
For example:
```
	wild:$
```
Allows us to specify in the output section :
```
	r=$	"This is done for all records" 
```
Syntax for 'wchr' - Allow a wild card character using a particular chr
Syntax :
```
	wchr: (Chr to use to signify a wild character)
	wchr:?
```
This allows a wild character to be used when specifying record keys.
Note there is NO automatic wild chr - you always have to specify it.
For example :
```
	wchr:?
```
Allows us to specify in the output section :
```
	r="abc?e"	"This is done for all records abc(something)e" 
```
Syntax for 'stripeol' - Do NOT strip multiple blank records
Syntax :
```
	stripeol:no
```
Where the 'recsep' is some combination of CR and NL, normally blank lines or multiple occurances of CR and NL are stripped.
This command is used to turn that option OFF and to treat all lines as valid records, even ones with no data.
Syntax for 'startkey' - Force the record Key or type of the BEFORE-FIRST record
Syntax :
```
	startkey:yes
```
This forces the key BEFORE the first record to be 'x1594' which can be used in the 'ifprv' test - if previous key.
Syntax for 'set' - Short forms or format names
Syntax :
```
	set	(name)	(any fixed text) 
	set	pagehdr	Stock<t>Close<tr>High<tr>Low<tr>Prev<qr>#\n
```
Set lets us specify easy-to-remember names to reference strings
Sets can NOT be split over several lines.
All leading and trailing spaces are stripped. So use wither double quotes to embed or the '\s' escape string. To specify a double quote, use the octal string.
FipSeq strings are useable but note that 'set' are parsed when the parameter file is chnaged/created so that any Variable data will be of that time and any FIP hdr data meaningless.
```
	eg	set	timedate	\$d-\$m-\$y \$h:\$n
```
will produce the date and time of when the parameter file was last changed.
To get run time date/time, specify the same string in the output section record line NOT the 'set'
The name of the set - timedate in our example - is case-INsensitive. So it may be called as TIMEDATE or TimeDate or any other variation known to man.
Syntax for 'include' - Include another file of instructions
Syntax :
```
	include	(filename)
	include	quark.tags
```
This will include another text file in tables/form/text with more Data Formatting commands.
The filename is force UPPERCASE as normal so in the above case the file will be :
```
	/usr/fip/tables/form/text/QUARK.TAGS
```
Generally include files will have commands such as 'set', 'calc' which are common to a series of Data Formats - like a quark styles for example.
Note that if you update the include file, ipformat will NOT rebuild its binary so you need to 'touch' all the other text files in form/text to get the new version. Normally of course ipformat realises a change has been made to the main text file and rebuilds its binary automatically. :
```
	ipt form/text
	touch *
```
Syntax for 'calc' - Define a Calculation or Choose output style for a number
Syntax :
```
	calc:(name)	(The calculation) 
	calc:percent	100*(c1/c2)
```
Define a calculation where cX are used as variables. The 'name' is that used in the 'output' section.
Calculations can NOT be split over several lines.
Variables are loaded at run-time using savnum, savbyte, savint, savswint, savlong or savswlong.
The default precision for a calculation is 2 decimal places. This can be overridden using the syntax :
```
	calc:percent:0	100*(c1/c2)
```
where the '0' after the name is the number of decimal places in the range 0-6.
Calc can also be used to change the output format of a number by specifying the precision. If the raw data is in 4 dec places and you only want 2:
```
	calc:dec2:2	c22
```
and in the output section, load c22 from data - in this case field 3 :
```
	savnum=22 f3
```
Operators can be :

+ plus
- minus
* multiply
/ divide

Take care when dividing by zero !
Use round brackets to denote how the calculation should be worked out. Ie the deepest level is calculated first, and then the calculation is worked from left to right.
For example :
```
	(c1/c2)*(c3/(c5+c4))/100
will add run through the following order :
	Step 1	(c5+c4)
	Step 2	(c1/c2)
	Step 3	(c3/result of Step 1)
	Step 4	(result of Step 2 * result of Step 3)
	Step 5	(result of Step 4 / 100)
```
Syntax for 'fraction' - Define a Fraction
Syntax :
```
	fraction:(name):(precision) (Output style WITH & WITHOUT fraction)
```
where

name is the function to be used in the output
precision is smallest denomiator - 2, 4, 8, 16, 32, 64, 128 or 256 - default is eighths
+ is used as separator - which can be any punctuation
first style is parse string used if THERE IS A FRACTION and integer
+ is used as separator
second style is parse string used if THERE IS ONLY AN INTEGER and NO FRACTION
+ is used as separator
```
		fraction:star:16	+\ZI \ZD/\ZN+\ZI.0+
		fraction:stox:32	|\ZI (\ZD-\ZN)|\ZI|
```
Fraction takes a field, partial field, saved field or calculation and split into 3 portions which can then be used as the normal FipSeq.

ZS is the sign + or -
ZI is the integer
ZD is the fraction amount
ZN is the fraction denominator

Specify two styles - the first for if there is a non-zero fraction amount and the second if there is none.
Syntax for 'base' - Define a Base number
Syntax :
```
	base:(name):(precision) (Output style WITH & WITHOUT fraction) 
```
Base is exactly the same as fraction except base will NOT attempt to 'reduce' the fraction.

ie 2/4 is left as ZD=2, ZN=4 while fraction will force ZD=1 ZN=2
Syntax for 'date' - Generate a date and/or time from a number
Syntax :
```
	date:(name):(data order)	(Output style)
	date:mono:dm    +Last \ZW was \ZD-\ZN-\ZZ+
```
where mono is the name which is used in the output section
The Order of the raw data is important. So we divide it up into a series of 0 or more 2-digit numbers (or space digit) whose order is specified using :

d - day -- 1-31
m - month -- 1-12
y - year -- (last 2 digits only, must be after 1970)
c - century -- 19 or 20 only
h - hour -- 0-23
t - minute -- 0-59
x - padding -- ignore 1 or 2 numbers at this position. Only 1 padding is allowed.

So for the 21st March 2010, the incoming data must be :

if it is : 21032010 data order is : dmcy
if it is : 210310 data order is : dmy
if it is : 032110 data order is : mdy

Obviously only one format of data can be handled by a single 'date' but you can have as many 'date's as needed.
Note that spaces, letters and punctuation are stripped and/or used as field delimitors. So the following are all equivalent :

200896
20th 8-August, 96
This day 20th August (the 8th month) in the Year 96

The Output format is defined in normal FipSeq between two deliminators ('+' in our example above, but can be any punctuation except semicolon ';').
The new data is added to a series of extra FipHdr fields :

ZD - 2 digit day of month
ZM - 2 digit month
ZY - 2 digit year e.g. 92
ZZ - 4 digit year e.g. 1992
ZW - Day of week e.g. Monday, Tuesday etc
ZS - 3 character day of week e.g. Mon, Tue etc
ZN - Full name of month e.g. January, February
ZL - 3 character month e.g. Jan, Feb, Mar etc
ZJ - Julian day of year
ZH - Hour 00-23
ZI - Hour 00-12
ZT - Minute 00-59
ZP - am/pm
ZA - Week Number 00-53
ZB - Week Number 01-53
ZC - Day of the week 0-6

(see also manual page for 'strftime' for slightly more information)
Note that actual Day and Month names depend on the LOCALE of your shell/computer.
The Default output format, if none is specfied, is :
```
	+\ZW, \ZD \ZN \ZZ+
```
Which for (order dmcy), (data) 16111997 English LOCALE gives
```
	Sunday, 16 November 1997
```
Note that if any information is NOT supplied, the run time date/time is used.
Syntax for 'partial' - Further subdivide a record or field
Syntax :
```
	partial:(name)	(type):(length):(startchr):(endchr)
```
where

type can be

a : alphabetic
u : upper case alpha
l : lower case alpha
n : numeric
t : punctuation
p : printable (generally Spc to~ })
s : space ( tab, space, ff, cr, nl, lf)
z : anpa type string (alphanumeric plus hyphen)
b : binary

length can be zero for unlimited.
startchr and endchr are optional.

There can be up to 100 partial fields as required.
The contents of a Partial field are accessed by specifying pX where X is the sequential no from the start of the partial - ie first is p1, then p2 etc
Syntax to partial a field in the output section record line is
```
	(partial name) (field name)
```
and then you can use any partial fields. For example:

To pull apart a data in the form : 26th March 1997
```
		; comment	   26  th      March   1997 
		partial:pdate	s:0 n:2 a:0 s:0 a:0 s:0 n:4 
```
In the output section, assuming a field 3 of record type 99 contains the date, we can out put just the year by :
```
		r=99	pdate f3	p7
```
will produce "1997"
Syntax for 'revpartial' - Further subdivide a record or field
The 'revpartial' syntax is exactly the same as for 'partial'.
The only difference is that the data for revpartial is processed from RIGHT to LEFT while partial processes it LEFT to RIGHT.
Syntax for 'match' - search and replace on a single field
Syntax :
```
	match:(matchname)	/(search string)/(replacement)
	match:(matchname):c	/(search string)/(replacement)
```
where

matchname is a unique alphabetic name
c (optional) is for case-SENSITIVE searches
the delimiters - '/' in the above example - can be any punctuation as long as they are are the same for that match line

This is a localized search and replace or search and zap function which is applied ONLY to a record, field, partial field or save area.
Normally the search is case-INSENSITIVE but can be forced so with the ':c' after the matchname.
No wild chr or wild strings are permissable at present.
To specify in the output section :
```
		(matchname) (zone)
```
where

matchname is the SAME as specified in the 'match'
zone is the record, field, partial or save area.

Up to thirty matches can be specified for a single zone.
For example :
```
	match:jan	/1/January/ 
	match:feb	/2/February/ 
	match:mar	 /3/March/ 
	etc .. etc 
	match:dec	/12/December/
```
In the output section - let's pretend field 4 of record type 23 contains a number we want to translate to a month :
```
	r=23	dec nov oct sep aug jul jun may apr mar feb jan f4 
```
Note that dec nov and oct are done first else if 'f4' was "11" then the jan match will replace "11" with "JanJan" !
'match' complements the character xchg in IPXCHG. However they are slightly different in that 'match' is localised in that you apply it ONLY to a single field whereas IPXCHG works on a whole file (or using flags, to a selected paragraph, line or section of the file).
Syntax for 'firstmatch' - search and replace on a single field
The syntax for firstmatch is exactly the same as match.
The only difference is that 'match' does all matches while 'firstmatch' stops after the first successful search/replace.
Syntax for 'style' - Reformat a data zone
Syntax :
```
 	style: (name)   (a single printf conversion syntax)
        style:twonum    %.02d
```
- Uses printf which is nasty but a standard (of sorts). Do a man printf for fuller information if you need.
- Always starts '%'
- If the expression does not end with an 's' ('d' for integer for example), then the string in the header field is first converted to that type.
- Specify One and ONLY one expression (can not have %s%d%f) - as it takes the first only
- do NOT use for fixed data, ONLY the conversion string.
- Types are :

string : s
char : c
long : d,i,o,p,u,x,X
float : f,e,E,g,G
% : print a % !
type n is ignored ??

Examples

- to trim a string, use a dot : %.5s
- To pad a string with spaces : %5s
- To pad a string with spaces (left justified) : %-5s
- To pad a number with leading zeros : %.06d
Syntax for 'abbrev' - abbreviate strings, such as stock names, to fit a defined width
Syntax :
```
        abbrev: (name) : width:(size) [(optional other keywords)...]
```
abbrev:mangleStox width:30 strip-chr-order:eaiou\s,.:;
always need width
optionals pointsize:
use-afm-file: (name)
use-generic-widths:
strip-chr-order:
strip-type-order:
if NOT specified, that type of chr is NOT stripped
R - strip multiple spaces
H - strip Unprintables - high control chrs
C - strip Unprintables - low control chrs
S - strip Spaces
T - strip punctuation
Y - strip Uppercase vowels
V - strip Lowercase vowels
l/u - strip lower/upper case letters
n - strip numbers

Syntax for 'lookup' - lookup a vlaue from a file of key, values

Syntax :

        lookup: (name) file:(lookup-filename) [(optional other keywords)...]

lookup:name file:/fip/tables/setup/subjectcodes.fip key-field:3 value-field:1 sep:\042 always need file:(filename) - no default optionals key-field: (field number of the key field - starting at 1 - default 1 for 1st) value-field: (field number of the data or value field - starting at 1 - default 2 for 2nd) default-value: (FipSeq - default - none) separator: (Fipseq chr - default is '|') type-format: (filetype - Fixed, CSV, TAB-sep - default is blank for TEXT) comment-line: (character which signifies a comment line - default is ';') numeric-key: signifies the key is numeric. sorted: yes/no case-sensitive-key: yes/no allow-spaces: Syntax for 'name' - Overwrite the default name of the output file Syntax : name: (Fip Hdr strings)Remember that the 'name' is a list of Fip Hdr fields which will probably include the SN field which is the original name. The default name on output is : #SN:(original filename)#DU:(name of the paramfile)s#SC:FORM#DF:FORMWhere DU is forced lowercase. The 'hdr', 'outque' and 'name' keywords have the same syntax and roughly the same use so further information is in under 'hdr'. Remember 'name' is done AFTER all processing of the data - it is the last thing done before the file is sent on for further distribution. So information gleaned from the input, perhaps left in Save Areas, can be used in the name. 'hdr' however is done FIRST, BEFORE any data is touched. Syntax for 'outque' - Overwrite the default output folder Syntax : outque: (Fip Hdr strings)This is not often used as the output folder 'spool/2go' or the contents of the -o Input switch is normally enough. This parameter is used where the actaul output folder is decided by the data ! The 'hdr', 'outque' and 'name' keywords have the same syntax and roughly the same use so further information is in under 'hdr'. Remember 'outque' is done AFTER all processing of the data - it is the last thing done before the file is sent on for further distribution. So information gleaned from the input, perhaps left in Save Areas, can be used in the name. 'hdr' however is done FIRST, BEFORE any data is touched. Syntax for 'hdr' - Add extra fields to the FIP header of the output file Syntax : hdr: (Fip Hdr strings)Remember that although the specification MUST be kept on a single line, HASH may be used as a delimitor for Header fields. Note also that as the output file is a completly NEW file and has no physical connection to the input file, NO Fip Hdr fields are transfered from input to output UNLESS specified in the 'hdr' or 'name'. Generally use 'hdr' to preserve fields you need as for 'name' there is a limit to the number of characters - and the type of characters : no meta characters or slashes '/'etc - such as the Source Header for example : hdr:#SH:\SH#SN:\SN#DF:rogerwill transfer the SH and SN fields and force DF to roger. As 'hdr' is processed BEFORE the data, no information generated by IPFORMAT during processing is available. However the 'name' keyword is processed AFTER so such data may be added then. The default Fip Hdr put on an output file has the following fields : SU:form - ie Source is form HS:form_0_95-3-9_17:41:40_4_67 - for tracking the file HT:794792500 - date and time !! 'hdr' supplements but does not replace these. Other useful fields can be : DF - output format for ipedsys, ipgtwy, ipout, ipprint etc. This must be in the 'name' parameter as by default it contains 'DF:form' eg : DF:albertwill pickup tables/print/ALBERT if ipprint is the program sending to the final destination. CX - force the xchg name to be used to the contexts of this field. eg CX:STOCKSwill override the SC2DC fields normally used for 'ipxchg'. Syntax for 'nohdr' - do NOT add a Fip header to the output file Syntax : nohdr:This is used when the output file is to be used immediately by a Unix program which does not understand the Fip Hdr - sort for example. Syntax for 'chrset' - force the SC or Source Character set Syntax : chrset: (name)This fills in the SC: Fip Hdr field. The default is FORM. Syntax for 'before' - Insert data BEFORE any output from the input file. Syntax : before: (Fip Seq strings and Record processing commands) Syntax for 'after' - Insert data AFTER ALL output. Syntax : after: (Fip Seq strings and Record processing commands)Both 'before' and 'after' can have tests, builtins, contents of save areas etc although obviously for 'before', most of these may have nothing in.

In the Output section - Record processing lines The actual data is processed and output using the Record processing lines. The Syntax of each line is that the first bit specifies which record type or key the rest of the line applies to : r=(key) (output) For example : r=abc "Fried fish starts " s1 spc f4 " and the rest ..." There can be multiple lines for the same record type. The following two lines will give the same result as the one above : r=abc "Fried fish starts " s1 r=abc spc f4 " and the rest ..." For lines where you want to process for all EXCEPT a particular record type/key, use teh syntax 'r#' : r#35 "Nobody wants record 35s !" How to specify you want to use, format and/or output zones ? records r3 or r="abc" if not numeric fields f99 or f=Z if not numeric partial fields p22 - always numeric save zones s1 - always numeric flags x199 - always numeric calculations c4 - always numeric tr> counters z4 - always numeric blocks b77 - always numeric xml start tags t=tagname - always starts with a letter xml end tags e=tagname - always starts with a letter xml continuation tags u=tagname - always starts with a letter xml attribute tags a=attname - always starts with a letter set name --- specify name as in the 'set' fixed text --- " some fixed text " A '*' can be used in certain cases to signify 'ALL' zones ie clrflag=* clear all flags. f* output all fields from this record. Use double quotes for alphabetic keys and those with embedded spaces. You should try not to use 'sets', 'partial's or 'match's with names in the form 'z999' where z is one of the single letters above and 999 is a number in the rangle 1-999. Note that blocks are 'super records' but should be ignored for now. Note that case is IGNORED in keys in the current version . When 'ipformat' finds a name in the record processing line, it does the following sequence : - check to see if it is an already specified constant 'set' name. - if not, is it a zone - record, field, partial or block (eg p44, f3) - if not, is it a builtin command (see below) - if not, is it a save zone, flag or counter (eg s1, x33, c7) - otherwise it is considered some FipSeq string and saved as such. Spaces, End-Of-Lines and Double Quotes in the output One common failing when putting together a new parameter file is to completely forget about spaces (or other separators) and end-of-lines (CR or NL or CR NL or whatever) in the output file. The point is - you have to specify them as NOTHING is implicit in the output file. There is no hidden magic which suddenly realises that you want an end-of-line when you need it. You have to state where and when you want them. Generally this will be done by either putting them as constant/'set's or specifying them in the record processing line. The following are exactly the same : Either set spc \s set ql \n output: r="BIG" f5 spc f3 spc f99 spc f5 ql Or output: r="BIG" f5 \s f3 " " p99 \s f5 \nAs you can specify a space as either '\s' or in double quotes, to output a double quote character, you need to specify it as an number : \000. Builtins There are a number of builtin conversion routines for formatting zones - records, fields, save areas etc. These are called by placing the name of the conversion BEFORE the name of the zone eg : zapspcextra p5 which means : 'zap all leading, trailing and multiple spaces from partial field 5' A single zone can be subject to several builtins : zappunc zapspc caps f=Z which means : take field "Z" and zap all punctuation and zap all spaces and force uppercase before outputting. Builtins for case conversion : caps - force zone uppercase lwrcase - force zone lowercase idicase - force zone idiot upper and lowercase upper1 - force first letter of every word uppercase initial - force first letter of the first word uppercase Builtins for removing spaces: zapspc - remove all spaces from zone zapspcextra - remove all leading, trailing and multiple spaces from zone zapspclead - remove all leading spaces from zone zapspctrail - remove all trailing spaces from zone Builtins for removing punctuation: zappunc - remove all punctuation from zone zappuncextra - remove all leading, trailing and multiple punctuation from zone zappunclead - remove all leading punctuation from zone zappunctrail - remove all trailing punctuation from zone Builtins for Counters: setctr - set a counter incctr - add one to a counter decctr - subtract one from a counter clrctr - clear a counter or set it to zero Builtins for Calculations: savnum - save a printable number in a variable savbyte - save a single byte in a variable savint - save a binary integer (2 bytes) in a variable savswint - save a binary integer (2 bytes swapped) in a variable savlong - save a binary long (4 bytes) in a variable savswlong - save a binary long (4 bytes swapped) in a variable Miscellaneous: strlen - returns the length of the string which can be output or saved or tested zapleadzero - removes leading zeros from zone zapctl - remove all control characters from zone incfile - include standing file at this point r=99 incfile /home/standing/ s4 inctemplate - include standing file of FipSeq (and expand all the fields) at this point r=99 inctemplate /home/standing/ s4 newfile - finish this file, send it and start another r=abc newfile if any more information is specified AFTER the 'newfile' on the record processing line, it will be added to the FIP Hdr unless 'nohdr' has been specified. eg: r=abd newfile #DF:newform#QQ:\$Z log - log message in the Item Log continue - ignore all other tests for this record and continue with the next data record stop! - stop processing now. If there is an 'after' section it is done before the program finishes. (please note the exclamation mark !) reckey - output the actual record key. This is useful where wild cards are used for all records but you still need to output what the key was. chksum - output a LRC based on the field presented Tests There is a further selection of tests which can be made one zones inside the date. These enable you to select even finer some processing depending on actual data. If and ONLY if the test is true is the rest of the line continued with. Syntax for Tests (ifxxx) (first string) (second string if required) where strings can be fields, partials, saves or fixed text Actual tests can be : ifprv/ifnprv - test previous record type/key or not ifnxt/ifnnxt - test next record type/key or not (2006plus only) ifeq/ifne - test if 2 zones are equal or not ifgt/iflt - test if a zone is greater than another or not ifflag/ifnflag - test if a flag is ON or OFF ifnul/ifnnul - test if a zone is empty or not ifspc/ifnspc - test if a zone only contains spaces or not ifalpha - test if a zone only contains letters a-Z or not ifnum - test if a zone only contains number/digits 0-9 or not ifcon/ifncon - test if a string is (not) found within another iffile/ifnfile - test if a file exists or not ifpunct - test if a zone only contains punctuation or not Note that sequence is important for comparing two fields that may be different lengths as ifeq will be true if the first field is complete ie : 1st=AAA 2nd=AAABC will be true 1st=AAABC 2nd=AAA will be false Example 1 : r=24 ifprv r=35 "Last record was type 35 and this is 24"Only if the previous record type was "35" will the string be output Example 2 : r=24 f3 ifnul f3 " _ " x99For record type 24, output field 3 and if there was nothing in it, output a (spc) (dash) (spc). Flag 99 will also be set if there was nothing there. Example 3 : When using numeric data, please ensure that all extraneous characters are stripped from the zone before the test. In particular strip commas, plus signs, currency symbols etc. For example, if field 7 has data like p9300.0007 and save field 9 has 10,000 compare the two by : match:mnop /p// match:mnocomma /,// output: r=99 ifgt mnop mnocomma f7 mnop mnocomma s9 "Field 7 > Save 9" Using Flags Flags are a really useful means for deciding type of processing to do - or NOT to do. Commands for setting, clearing and testing flags are : - To set a flag : x999 where 999 is the flag number - To clear a flag : clrflag=999 - To clear all flags : clrflag=* - To test a flag is ON : ifflag x3 (rest of the commands on line are done ONLY if true) - To test a flag is OFF : ifnflag x5 (rest of the commands on line are done ONLY if false) For example, let's use flag 3 to test if record type 'abc' has Richard, Helen or George in the first field. Print out 'New name is (name) (newline)' if it does : r=abc clrflag=3 r=abc ifeq "Richard" f1 x3 r=abc ifeq "Helen" f1 x3 r=abc ifeq "George" f1 x3 r=abc ifflag x3 "New name is " f1 nl Save areas Save areas may be used to store strings - either in their original state or after conversion/formatting by other builtins. Commands for setting, clearing and testing save areas are : - To output a save area : s999 -- where 999 is the save number - To clear a save area : clrsave=999 - To clear all saves : clrsave=* - To save data in a save area : save=999 (string) eg save=1 f3 save=5 caps f7 save the contents of field 7 in save area 5 AFTER forcing to Uppercase - To append data to a save area : savcat=88 (string) eg save=77 "ABC" save zone 77 holds ABC savcat=77 "DEF" save zone 77 now holds ABCDEF There is also a special case for saving data in a FipHdr field - savefiphdr=AB "123" Save areas may be used in the normal 'if' tests, eg : ifeq "AAA" s1 x88 if the contents of save area 1 starts "AAA., set flag 88 ON Putting Data into/Out of Fiphdrs savefiphdr : Saving data in a FipHdr field eg savefiphdr=AB t=acorn puts the contents of the tag into FipHdr field AB In order to save any data at the end of formatting, use the hdr: command. Using FipSeq Any FipSeq process is available in DataFormatting by adding 'fipseq-' before the syntax. please see the FipSeq doc for more information fiphdr-fixed fiphdr-partial fiphdr-combie fiphdr-option fiphdr-repeat fiphdr-style fiphdr-replace fiphdr-newdate fiphdr-lookup fiphdr-unique fiphdr-valid fiphdr-perl fiphdr-merge fiphdr-sum Using Counters Counters are integers (ie proper numbers with no decimals or fractions in the range -32000 to +32000. They are signalled by 'zX' where X is a number. They can be used to count the number of occurences of a record or field or even types of data and act accordingly. All counters are set to zero when the program starts and by using the builtins : incctr decctr setctr clrctr in the Record processing lines, you can manipulate them. For example, to add some random markup every 10th line of a record type AB using counter 26 : r="AB" incctr=26 ifeq 10 z26 clrctr=26 "[pt9][font99]"ie : For all records type AB, add 1 to counter 26, then test if ctr 26 is equal to 10; if so reset ctr 26 back to zero and output string '[pt9][font99]'. The syntax for 'setctr' is setctr=99 345 - set ctr 99 to a fixed number 345 setctr=297 p3 - set ctr 297 to the contents of partial field 3. In the second example, if the p3 is NOT a number, ctr 297 is set to zero. Also if p3 is a decimal number like '123.456', only the main number is saved. Using Calculations Calculations are defined in the first part of the parameter file and used in the record processing part : For example : calc:mktcap c1*c2 output: r=BC savnum=1 f5 savnum=2 f7 mktcapIn this example we define 'mktcap' to be variables 1 and 2 multiplied together. Then in the output section, for record type BC. field 5 is saved in variable 1 and field 7 in variable 2 before we do the calculation and output the result. A quick word about BINARY numbers. Normally fields will hold printable data - such as in the example above - and we use the builtin 'savnum' to take that number for use in the calculation(s). However some data is already in a binary form. Use builtins 'savbyte, savint, savswint, savlong and savswlong' to load these numbers. Often these will be derived from a partial field using the 'b' for binary field type. eg: partial:bindata b:2 b:4 b:2 b:4What is a swapped integer or long ? Some computers - like the PDP-11 and most Intel 16+ bit chips - hold the data in reverse byte mode. - So if the data has been generated on a SPARC OR rs6000 or a Mac the data is 'normal' - use savint or savlong. - While data from PDP-11s or Intel based PCs could well need to be swapped. Loading Variables : Save a printable zone as a number variable - use 'savnum'savnum=5 p4- save the contents of p4 as a number. So if p4 held the string '789', c5 would be the number '789. Save a fixed number in a number variable - use 'savnum' againsavnum=7 1234- loads the number '1234' into c7. Save the contents of a single byte - use 'savbyte'savbyte=33 p7 Note that the contents of the variables, c1, c2 etc are not amended by the calculation UNLESS you specifically save it, eg : r=BC savnum=1 f5 savnum=2 f7 savnum=3 mktcapwill load c3 with the result of the 'mktcap' calculation. XML files Processing XML files is really simple ! In the FILE DEFINITION section, state filtyp:XML and no recsep, rectyp, filsep or filtyp In the OUTPUT section, instead of records 'r', use : t for start tag - triggered on a start or standalone tag e for end tag - triggered on an end tag u for continuation tag - neither start nor end but somewhere in the middle ! And instead of fields 'f=3', use a for attributes 'a=city' or tags 't=name' to extract the data eg: names of TAGs are like Records ie <Eventplace>Theatro Moderno</Eventplace> t=Eventplace "This event takes place at " t=EventPlace will display "This event takes place at Theatro Moderno" names of ATTRIBUTES are like Fields 'a=city' ie <Property city="PocketNook"> t=Property "city is " a=city will display "city is PocketNook" Just a refresher ... An XML structure like <SoccerScore> - start tag <Game time="1330" /> - attribute 'time' on standalone tag 'Game' <HomeTeam>Liverpool</HomeTeam> - start tag with data and end tag beat - data in a continuation tag <AwayTeam>Chelsea</AwayTeam> - start tag with data and end tag by - data in a continuation tag <HomeGoals>8</HomeGoals> - start tag with data and end tag to - data in a continuation tag <AwayGoals>1</AwayGoals> - start tag with data and end tag </SoccerScore> - end tag 1. endtags use 'e=' when you need to trigger something at the END of a tag Note end tags can NEVER have data or attributes - they are used soley to 'cleanup' things as they signal the end of a tag or block of tags eg <Metadata> <MetadataType FormalName="Securities Identifier"/> <Property FormalName="Company Name" Value="NovaBay Corp"/> <Property FormalName="Ticker Symbol" Value="NBY"/> <Property FormalName="Exchange" Value="TSX"/> <Property FormalName="SlugLine Display Order" Value="2"/> </Metadata> In the Output section : output: .... ; on start tag, clrsaves 100-102 t=Metadata clrsave=100-102 t=Property ifnnul a=FormalName ifeq a=FormalName "Company Name" save=100 a=Value t=Property ifnnul a=FormalName ifeq a=FormalName "Ticker Symbol" save=101 a=Value t=Property ifnnul a=FormalName ifeq a=FormalName "Exchange" save=102 a=Value e=Metadata s100 " has ticker " s101 " on the " s102 " stock exchange" ... gives 'NovaBay Corp has ticker NBY on the TSX stock exchange' 2. continuations Pretend you have this data .... <para>start of para <Ital>italic</Ital> continuation of para</para> In the Output section : output: .... ; start para by resetting font to default t=para "<f$>" t* ; start of italic.. t=Ital "<f:Bodoni-Italic>" t* ; end italic e=Ital "<f$>" ; rest of para is in the continuation bit u=p t* Useful Hints and Examples DEFAULT KEYS The default record key is 'x1594' Repeatable fields - In addition to multiple records 'r=* lines 'hdr', 'name', 'before', 'after' are now repeatable - so you can specify several lines of each. eg : hdr: SN:\SN\n hdr: SU:FORM\n hdr: SC:form\n hdr: DU:\IQ\n after: ifflag x44 "Flag 44 is still on\n" after: ifnnul s3 ifcon "GREED" caps s3 "\n I am a " s3 "person\n" after: "That's all Folks!\n" Examples of Builtins : STRLEN ; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2) r=HH ifnnul f2 iflt 44 strlen f2 "Big Field 2 here over 44 chrs long" \n r=KK "Save Field for Name (s55) is " strlen s55 " chrs long" ZAPLEADZERO ; data - field 99 is 00000330303, field 101 is 00000000.00 r=3 "This outputs 330303=" zapleadzero f99 ", while this is 0.00=" zapleadzero f101

if it is : 21032010	data order is : dmcy
if it is : 210310	data order is : dmy
if it is : 032110	data order is : mdy

- to trim a string, use a dot	: %.5s
- To pad a string with spaces	: %5s
- To pad a string with spaces (left justified)	: %-5s
- To pad a number with leading zeros	: %.06d

The default Fip Hdr put on an output file has the following fields :
SU:form	- ie Source is form
HS:form_0_95-3-9_17:41:40_4_67	- for tracking the file
HT:794792500	- date and time !!
'hdr' supplements but does not replace these.

How to specify you want to use, format and/or output zones ?
records	r3	or r="abc" if not numeric
fields	f99	or f=Z if not numeric
partial fields	p22	- always numeric
save zones	s1	- always numeric
flags	x199	- always numeric
calculations	c4	- always numeric tr>	counters	z4	- always numeric
blocks	b77	- always numeric
xml start tags	t=tagname	- always starts with a letter
xml end tags	e=tagname	- always starts with a letter
xml continuation tags	u=tagname	- always starts with a letter
xml attribute tags	a=attname	- always starts with a letter
set name	---	specify name as in the 'set'
fixed text	---	" some fixed text "

A '*' can be used in certain cases to signify 'ALL' zones ie
clrflag=*	clear all flags.
f*	output all fields from this record.

Builtins for case conversion :
caps	- force zone uppercase
lwrcase	- force zone lowercase
idicase	- force zone idiot upper and lowercase
upper1	- force first letter of every word uppercase
initial	- force first letter of the first word uppercase
Builtins for removing spaces:
zapspc	- remove all spaces from zone
zapspcextra	- remove all leading, trailing and multiple spaces from zone
zapspclead	- remove all leading spaces from zone
zapspctrail	- remove all trailing spaces from zone
Builtins for removing punctuation:
zappunc	- remove all punctuation from zone
zappuncextra	- remove all leading, trailing and multiple punctuation from zone
zappunclead	- remove all leading punctuation from zone
zappunctrail	- remove all trailing punctuation from zone
Builtins for Counters:
setctr	- set a counter
incctr	- add one to a counter
decctr	- subtract one from a counter
clrctr	- clear a counter or set it to zero
Builtins for Calculations:
savnum	- save a printable number in a variable
savbyte	- save a single byte in a variable
savint	- save a binary integer (2 bytes) in a variable
savswint	- save a binary integer (2 bytes swapped) in a variable
savlong	- save a binary long (4 bytes) in a variable
savswlong	- save a binary long (4 bytes swapped) in a variable
Miscellaneous:
strlen	- returns the length of the string which can be output or saved or tested
zapleadzero	- removes leading zeros from zone
zapctl	- remove all control characters from zone
incfile	- include standing file at this point r=99 incfile /home/standing/ s4
inctemplate	- include standing file of FipSeq (and expand all the fields) at this point r=99 inctemplate /home/standing/ s4
newfile	- finish this file, send it and start another r=abc newfile if any more information is specified AFTER the 'newfile' on the record processing line, it will be added to the FIP Hdr unless 'nohdr' has been specified. eg: r=abd newfile #DF:newform#QQ:\$Z
log	- log message in the Item Log
continue	- ignore all other tests for this record and continue with the next data record
stop!	- stop processing now. If there is an 'after' section it is done before the program finishes. (please note the exclamation mark !)
reckey	- output the actual record key. This is useful where wild cards are used for all records but you still need to output what the key was.
chksum	- output a LRC based on the field presented

Actual tests can be :
ifprv/ifnprv	- test previous record type/key or not
ifnxt/ifnnxt	- test next record type/key or not (2006plus only)
ifeq/ifne	- test if 2 zones are equal or not
ifgt/iflt	- test if a zone is greater than another or not
ifflag/ifnflag	- test if a flag is ON or OFF
ifnul/ifnnul	- test if a zone is empty or not
ifspc/ifnspc	- test if a zone only contains spaces or not
ifalpha	- test if a zone only contains letters a-Z or not
ifnum	- test if a zone only contains number/digits 0-9 or not
ifcon/ifncon	- test if a string is (not) found within another
iffile/ifnfile	- test if a file exists or not
ifpunct	- test if a zone only contains punctuation or not

Commands for setting, clearing and testing flags are :
- To set a flag	: x999 where 999 is the flag number
- To clear a flag	: clrflag=999
- To clear all flags	: clrflag=*
- To test a flag is ON	: ifflag x3 (rest of the commands on line are done ONLY if true)
- To test a flag is OFF	: ifnflag x5 (rest of the commands on line are done ONLY if false)

Commands for setting, clearing and testing save areas are :
- To output a save area	: s999 -- where 999 is the save number
- To clear a save area	: clrsave=999
- To clear all saves	: clrsave=*
- To save data in a save area	: save=999 (string)
eg save=1 f3 save=5 caps f7
save the contents of field 7 in save area 5 AFTER forcing to Uppercase
- To append data to a save area	: savcat=88 (string)
eg save=77 "ABC"
save zone 77 holds ABC
savcat=77 "DEF"
save zone 77 now holds ABCDEF
There is also a special case for saving data in a FipHdr field - savefiphdr=AB "123"

setctr=99 345	- set ctr 99 to a fixed number 345
setctr=297 p3	- set ctr 297 to the contents of partial field 3.