Syntax of Each Keyword

In the Output section - Record processing lines

The actual data is processed and output using the Record processing lines.

The Syntax of each line is that the first bit specifies which record type or key the rest of the line applies to :

		r=(key)		(output)
For example :
	r=abc	"Fried fish starts " s1 spc f4 " and the rest ..."
There can be multiple lines for the same record type. The following two lines will give the same result as the one above :
		r=abc	"Fried fish starts " s1 
		r=abc	spc f4 " and the rest ..."
For lines where you want to process for all EXCEPT a particular record type/key, use teh syntax 'r#' :
		r#35	"Nobody wants record 35s !"

How to specify you want to use, format and/or output zones ?
records r3or r="abc" if not numeric
fieldsf99or f=Z if not numeric
partial fieldsp22- always numeric
save zoness1 - always numeric
flagsx199 - always numeric
calculationsc4- always numeric tr>countersz4- always numeric
blocksb77- always numeric
xml start tagst=tagname- always starts with a letter
xml end tagse=tagname- always starts with a letter
xml continuation tagsu=tagname- always starts with a letter
xml attribute tagsa=attname- always starts with a letter
set name---specify name as in the 'set'
fixed text---" some fixed text "

A '*' can be used in certain cases to signify 'ALL' zones ie
clrflag=*clear all flags.
f*output all fields from this record.

Use double quotes for alphabetic keys and those with embedded spaces.

You should try not to use 'sets', 'partial's or 'match's with names in the form 'z999' where z is one of the single letters above and 999 is a number in the rangle 1-999.

Note that blocks are 'super records' but should be ignored for now.

Note that case is IGNORED in keys in the current version .

When 'ipformat' finds a name in the record processing line, it does the following sequence :

- check to see if it is an already specified constant 'set' name.
- if not, is it a zone - record, field, partial or block (eg p44, f3)
- if not, is it a builtin command (see below)
- if not, is it a save zone, flag or counter (eg s1, x33, c7)
- otherwise it is considered some FipSeq string and saved as such.

Spaces, End-Of-Lines and Double Quotes in the output

One common failing when putting together a new parameter file is to completely forget about spaces (or other separators) and end-of-lines (CR or NL or CR NL or whatever) in the output file.

The point is - you have to specify them as NOTHING is implicit in the output file. There is no hidden magic which suddenly realises that you want an end-of-line when you need it. You have to state where and when you want them.

Generally this will be done by either putting them as constant/'set's or specifying them in the record processing line. The following are exactly the same : Either

	set	spc	\s
	set	ql	\n
	r="BIG"		f5 spc f3 spc f99 spc f5 ql
	r="BIG"		f5 \s f3 " " p99 \s f5 \n
As you can specify a space as either '\s' or in double quotes, to output a double quote character, you need to specify it as an number : \000.


There are a number of builtin conversion routines for formatting zones - records, fields, save areas etc.

These are called by placing the name of the conversion BEFORE the name of the zone eg :

		zapspcextra p5
which means :
'zap all leading, trailing and multiple spaces from partial field 5'

A single zone can be subject to several builtins :

zappunc zapspc caps f=Z
which means :
take field "Z" and zap all punctuation and zap all spaces and force uppercase before outputting.
Builtins for case conversion :
caps - force zone uppercase
lwrcase - force zone lowercase
idicase - force zone idiot upper and lowercase
upper1 - force first letter of every word uppercase
initial- force first letter of the first word uppercase
Builtins for removing spaces:
zapspc - remove all spaces from zone
zapspcextra - remove all leading, trailing and multiple spaces from zone
zapspclead - remove all leading spaces from zone
zapspctrail - remove all trailing spaces from zone
Builtins for removing punctuation:
zappunc- remove all punctuation from zone
zappuncextra - remove all leading, trailing and multiple punctuation from zone
zappunclead - remove all leading punctuation from zone
zappunctrail - remove all trailing punctuation from zone
Builtins for Counters:
setctr- set a counter
incctr- add one to a counter
decctr- subtract one from a counter
clrctr- clear a counter or set it to zero
Builtins for Calculations:
savnum- save a printable number in a variable
savbyte- save a single byte in a variable
savint- save a binary integer (2 bytes) in a variable
savswint- save a binary integer (2 bytes swapped) in a variable
savlong- save a binary long (4 bytes) in a variable
savswlong- save a binary long (4 bytes swapped) in a variable
strlen- returns the length of the string which can be output or saved or tested
zapleadzero- removes leading zeros from zone
zapctl- remove all control characters from zone
incfile - include standing file at this point
	r=99	incfile /home/standing/ s4
inctemplate - include standing file of FipSeq (and expand all the fields) at this point
	r=99	inctemplate /home/standing/ s4
newfile - finish this file, send it and start another
	r=abc	newfile
if any more information is specified AFTER the 'newfile' on the record processing line, it will be added to the FIP Hdr unless 'nohdr' has been specified. eg:
	r=abd	newfile	#DF:newform#QQ:\$Z
log- log message in the Item Log
continue- ignore all other tests for this record and continue with the next data record
stop!- stop processing now. If there is an 'after' section it is done before the program finishes. (please note the exclamation mark !)
reckey- output the actual record key. This is useful where wild cards are used for all records but you still need to output what the key was.
chksum- output a LRC based on the field presented


There is a further selection of tests which can be made one zones inside the date.

These enable you to select even finer some processing depending on actual data. If and ONLY if the test is true is the rest of the line continued with.

Syntax for Tests

	(ifxxx) (first string) (second string if required) 
where strings can be fields, partials, saves or fixed text
Actual tests can be :
ifprv/ifnprv - test previous record type/key or not
ifnxt/ifnnxt - test next record type/key or not (2006plus only)
ifeq/ifne - test if 2 zones are equal or not
ifgt/iflt - test if a zone is greater than another or not
ifflag/ifnflag - test if a flag is ON or OFF
ifnul/ifnnul - test if a zone is empty or not
ifspc/ifnspc - test if a zone only contains spaces or not
ifalpha- test if a zone only contains letters a-Z or not
ifnum- test if a zone only contains number/digits 0-9 or not
ifcon/ifncon- test if a string is (not) found within another
iffile/ifnfile- test if a file exists or not
ifpunct- test if a zone only contains punctuation or not

Note that sequence is important for comparing two fields that may be different lengths as ifeq will be true if the first field is complete ie :

		1st=AAA		2nd=AAABC	will be true 
		1st=AAABC	2nd=AAA		will be false 
Example 1 :
	r=24	ifprv r=35	"Last record was type 35 and this is 24"
Only if the previous record type was "35" will the string be output

Example 2 :

	r=24	 f3 ifnul f3 " _ " x99
For record type 24, output field 3 and if there was nothing in it, output a (spc) (dash) (spc). Flag 99 will also be set if there was nothing there.

Example 3 : When using numeric data, please ensure that all extraneous characters are stripped from the zone before the test. In particular strip commas, plus signs, currency symbols etc. For example, if field 7 has data like p9300.0007 and save field 9 has 10,000 compare the two by :

	match:mnop	/p//
	match:mnocomma	/,//
	r=99	ifgt mnop mnocomma f7 mnop mnocomma s9      "Field 7 >  Save 9"

Using Flags

Flags are a really useful means for deciding type of processing to do - or NOT to do.
Commands for setting, clearing and testing flags are :
- To set a flag: x999 where 999 is the flag number
- To clear a flag: clrflag=999
- To clear all flags: clrflag=*
- To test a flag is ON: ifflag x3 (rest of the commands on line are done ONLY if true)
- To test a flag is OFF: ifnflag x5 (rest of the commands on line are done ONLY if false)

For example, let's use flag 3 to test if record type 'abc' has Richard, Helen or George in the first field. Print out 'New name is (name) (newline)' if it does :

	r=abc	clrflag=3 
	r=abc	ifeq "Richard" f1	x3 
	r=abc	ifeq "Helen" f1		x3 
	r=abc	ifeq "George" f1	x3 
	r=abc	ifflag x3		"New name is " f1 nl 

Save areas

Save areas may be used to store strings - either in their original state or after conversion/formatting by other builtins.
Commands for setting, clearing and testing save areas are :
- To output a save area: s999 -- where 999 is the save number
- To clear a save area: clrsave=999
- To clear all saves: clrsave=*
- To save data in a save area: save=999 (string)
	eg	save=1	f3 
		save=5	caps f7
save the contents of field 7 in save area 5 AFTER forcing to Uppercase
- To append data to a save area: savcat=88 (string)
	eg	save=77 "ABC"
save zone 77 holds ABC
		savcat=77 "DEF"
save zone 77 now holds ABCDEF
There is also a special case for saving data in a FipHdr field - savefiphdr=AB "123"

Save areas may be used in the normal 'if' tests, eg :

	ifeq "AAA" s1	x88 
if the contents of save area 1 starts "AAA., set flag 88 ON

Putting Data into/Out of Fiphdrs

savefiphdr : Saving data in a FipHdr field
eg savefiphdr=AB t=acorn
puts the contents of the tag into FipHdr field AB

In order to save any data at the end of formatting, use the hdr: command.

Using FipSeq

Any FipSeq process is available in DataFormatting by adding 'fipseq-' before the syntax.

please see the FipSeq doc for more information

Using Counters

Counters are integers (ie proper numbers with no decimals or fractions in the range -32000 to +32000.

They are signalled by 'zX' where X is a number.

They can be used to count the number of occurences of a record or field or even types of data and act accordingly.

All counters are set to zero when the program starts and by using the builtins :

in the Record processing lines, you can manipulate them.

For example, to add some random markup every 10th line of a record type AB using counter 26 :

	r="AB"	incctr=26	ifeq 10 z26	clrctr=26	"[pt9][font99]"
ie : For all records type AB, add 1 to counter 26, then test if ctr 26 is equal to 10; if so reset ctr 26 back to zero and output string '[pt9][font99]'.
The syntax for 'setctr' is
setctr=99 345- set ctr 99 to a fixed number 345
setctr=297 p3- set ctr 297 to the contents of partial field 3.

In the second example, if the p3 is NOT a number, ctr 297 is set to zero. Also if p3 is a decimal number like '123.456', only the main number is saved.

Using Calculations

Calculations are defined in the first part of the parameter file and used in the record processing part :

For example :

	calc:mktcap	c1*c2
	r=BC	savnum=1 f5	savnum=2 f7	mktcap
In this example we define 'mktcap' to be variables 1 and 2 multiplied together. Then in the output section, for record type BC. field 5 is saved in variable 1 and field 7 in variable 2 before we do the calculation and output the result.

A quick word about BINARY numbers.

Normally fields will hold printable data - such as in the example above - and we use the builtin 'savnum' to take that number for use in the calculation(s).

However some data is already in a binary form. Use builtins 'savbyte, savint, savswint, savlong and savswlong' to load these numbers. Often these will be derived from a partial field using the 'b' for binary field type. eg:

	partial:bindata	b:2 b:4 b:2 b:4
What is a swapped integer or long ? Some computers - like the PDP-11 and most Intel 16+ bit chips - hold the data in reverse byte mode.

- So if the data has been generated on a SPARC OR rs6000 or a Mac the data is 'normal' - use savint or savlong.

- While data from PDP-11s or Intel based PCs could well need to be swapped.

Loading Variables :

Save a printable zone as a number variable - use 'savnum'
savnum=5 p4
- save the contents of p4 as a number. So if p4 held the string '789', c5 would be the number '789.
Save a fixed number in a number variable - use 'savnum' again
savnum=7 1234
- loads the number '1234' into c7.
Save the contents of a single byte - use 'savbyte'
savbyte=33	p7

Note that the contents of the variables, c1, c2 etc are not amended by the calculation UNLESS you specifically save it, eg :

	r=BC	savnum=1 f5	savnum=2 f7	savnum=3 mktcap
will load c3 with the result of the 'mktcap' calculation.

XML files

Processing XML files is really simple !

In the FILE DEFINITION section, state filtyp:XML and no recsep, rectyp, filsep or filtyp

In the OUTPUT section, instead of records 'r', use :
t for start tag - triggered on a start or standalone tag
e for end tag - triggered on an end tag
u for continuation tag - neither start nor end but somewhere in the middle !

And instead of fields 'f=3', use a for attributes 'a=city' or tags 't=name' to extract the data

        names of TAGs are like Records
		ie	<Eventplace>Theatro Moderno</Eventplace>
                	t=Eventplace "This event takes place at " t=EventPlace
		will display
                	"This event takes place at Theatro Moderno"

        names of ATTRIBUTES are like Fields 'a=city'
        	ie	<Property city="PocketNook">
                        t=Property "city is " a=city
		will display
			"city is PocketNook"

Just a refresher ...

An XML structure like
	<SoccerScore>				- start tag
		<Game time="1330" />		- attribute 'time' on standalone tag 'Game'
		<HomeTeam>Liverpool</HomeTeam>	- start tag with data and end tag
		beat				- data in a continuation tag
		<AwayTeam>Chelsea</AwayTeam>	- start tag with data and end tag
		by				- data in a continuation tag
		<HomeGoals>8</HomeGoals>	- start tag with data and end tag
		to				- data in a continuation tag
		<AwayGoals>1</AwayGoals>	- start tag with data and end tag
	</SoccerScore>				- end tag

1. endtags
 use 'e=' when you need to trigger something at the END of a tag
Note end tags can NEVER have data or attributes - they are used soley to 'cleanup' things as they signal the end of a tag or block of tags

               <MetadataType FormalName="Securities Identifier"/>
               <Property FormalName="Company Name" Value="NovaBay Corp"/>
               <Property FormalName="Ticker Symbol" Value="NBY"/>
               <Property FormalName="Exchange" Value="TSX"/>
               <Property FormalName="SlugLine Display Order" Value="2"/>

In the Output section :
; on start tag, clrsaves 100-102
t=Metadata	clrsave=100-102
t=Property	ifnnul a=FormalName ifeq a=FormalName "Company Name"	save=100 a=Value
t=Property	ifnnul a=FormalName ifeq a=FormalName "Ticker Symbol"	save=101 a=Value
t=Property	ifnnul a=FormalName ifeq a=FormalName "Exchange"	save=102 a=Value
e=Metadata	s100 " has ticker " s101 " on the " s102 " stock exchange"

... gives 'NovaBay Corp has ticker NBY on the TSX stock exchange'

 2. continuations

Pretend you have this data ....
  <para>start of para <Ital>italic</Ital> continuation of para</para>

In the Output section :
; start para by resetting font to default
t=para     "<f$>" t*
; start of italic..
t=Ital     "<f:Bodoni-Italic>" t*
; end italic
e=Ital     "<f$>"
; rest of para is in the continuation bit
u=p     t*

Useful Hints and Examples

The default record key is 'x1594'

Repeatable fields

        - In addition to multiple records 'r=* lines
		'hdr', 'name', 'before', 'after' are now repeatable - so you can specify several lines of each.
          eg :
		hdr: SN:\SN\n
		hdr: SU:FORM\n
		hdr: SC:form\n
		hdr: DU:\IQ\n

                after:  ifflag x44      "Flag 44 is still on\n"
                after:  ifnnul s3       ifcon "GREED" caps s3           "\n  I am a " s3 "person\n"
                after:  "That's all Folks!\n"

Examples of Builtins :

	; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2)
	r=HH    ifnnul f2       iflt 44 strlen f2       "Big Field 2 here over 44 chrs long" \n
	r=KK    "Save Field for Name (s55) is " strlen s55 " chrs long"


	; data - field 99 is 00000330303, field 101 is 00000000.00
	r=3     "This outputs 330303=" zapleadzero f99 ", while this is
0.00=" zapleadzero f101