0.Introduction to FipSeq


1.Six Types of FipSeq

Ordinary visible data

Control or Unprintable characters.

Unicode characters (UTF-8)

Common escaped characters.

Contents of FIP header fields

System variables


2.Reformatting FIP header field data

Using FipSeq to format your output - Overview


3. In More Detail


0. Introduction to FipSeq

Quite often the information in FipHdr fields needs to be presented, formatted or styled differently on output.

Sometimes you need to find out times and dates when the files were processed.

Maybe you need to embeded non-printable characters at the begining or end of a file to start so anterior process.

FipSeq is the notation used to select and format these types of actions.

If you are testing difficult bits of FipSeq, we recommend you use the program 'sffparse' which reads a parameter file of FipSeq, an data file with a FipHdr or a FipHdr string plus an output expression. All you would ever want in the world of FipSeq !

1.Six Types of FipSeq

There are six basic types of FipSeq which can all be mixed and matched where necessary :

  1. Ordinary visible data : The sort of stuff you can type and see !

The Slow-challenged coloured cunning-quadruped jumped over ... etc etc etc

  1. Control or unprintable characters are preceded by backslash and are 3 numbers such as \040 \333

Note that Hex numbers can also be specified as \xA9 at any time - (that is little ‘x’ after the ‘\’ and two hex digits)

The number system used can be octal (default), decimal or hex depending on the 'number’ keyword specified in your parameter file.

eg a space is \040 (octal), \032 (decimal) and \020 (hex) Jimmy \321lsen

  1. Unicode Characters (UTF-8) - backslash and 4 hex numbers : \uXXXX eg \u12BC
  2. (UTF-8) - backslash and 4 dec numbers : \e9999 eg \e8194
  3. Common Escaped control characters - single letter following a backslash:

Valid ones are :



Octal 015



Octal 012



Octal 015 012



Octal 040



Octal 013



Octal 011



Octal 010



Octal 134


5.      Contents of FIP header fields – using a ‘\’ with two UPPERCASE letters (the second may be a number)


\SN - output the filename

\SU - output the Wire Agency name


Remember that the Source Header field (SH) is a special case and to get the data specify \X* where * is the single SH field (note that actual FipHdr fields starting 'X' like XS are NOT accessible).

For example:

SH::Sreu:N1234:P3:CSSS:KSports Roundup

In this case \XK is ‘Sports Roundup‘

\XP is ‘3’

6.      System Variables





J11 style NAME;dd/mm



3 figure sequence number
This is normally increments on each file



4 figure sequence number
This is normally increments on each file



4 figure sequence number incrementing on each call



year (99)



century (19 or 20)



month (mm)



month (Mmm)



day (99)



hour (99)



min (99)



offset from GMT



GMT date and time in CCYYMMDDHHMMSS format



sec (99)



julian day of the year



Day of the week (abbreviated to 3 letters)



No Of seconds since 1st Jan 1970 - UnixTime or C Time



random alpha numeric character between a-z, 0-9



strip trailing spaces from this point to the last non-space



if Optional field failed, strip to this point (see below)



no of lines for file

(only in some programs)


no of words for file

(only in some programs)


no of chrs for file

(only in some programs)



Plus for certain programs :

Note that in the old versions, if you use a sequence number several times for the same file, (\$Z or \$S) then the seqno was incremented EACH time you call it. This is no longer true. – use \$z or \$s for incrementing on each file and \$v for incrementing on each time it is used within a file.


2.Reformatting FIP header field data

Using and abusing FipSeq !


Note that all the different types of FipSeq can be mixed in whatever you are outputting.


For example, IPPRINT page heading might look like :


header:File:\SN\tFrom:\SU\r\nPrinted at:\t\$h:\$n \$d-\$m-\$c\$y\r\n\r\n


For a file AFP0223 from Source AFP would give 3 lines at the top of the page :


File:AFP0223 From:AFP

Printed at: 12:20 23-Mar-1996

Part 2 But what happens when :

So now we have to start playing with the FipHdr data to create new fipHdr fields for internal use only.

There are seven functions for reformatting the data:

fixed:QZ 1234543\$j..\SN

partial:QT ST,3,2,U,<,>

- based on a field delimiter like a dash :

repeat:QR XK,-,3

- or based on a type like NOT a number or a SPACE

repeat:QR XK,,3,U,N

combie:QZ ep|na,(0000000)a

option:QT ep,11,7,s

style:QH XM,%.02s

replace:LC XC SPO=s INL=i UDL=u ECO=f

newdate:JS min-38 hour+8 day+\MD "\ZD-\ZI-\ZZ"


--- Note these FipHdr fields are ONLY for use internally to that one program and are NOT added to the existing FipHdr.

--- They can be used in exactly the same way as any other FipHdr field.

--- If an existing FipHdr field has the same name as a newly created fixed/partial/repeat/option/style, the OLD is ignored and the NEWLY created one used.

--- So please make sure you do NOT use existing FipHdr field names with valid data in them as they will NOT be accessible. To be sure, use fields starting Qx, Nx, Jx where x is A-Z, 0-9

--- The new fields can be linked together up to 20 times or levels. Example of a snippet of parameter file for IPEDSYS to generate a consistent filename which can be sorted easily on the day of the year :

Lets have a FipHdr containing 3 normal FipHdr fields : SN for storyname, HS for History and SH for the Source Header




..and the relevant part of the IPEDSYS parameter file ..

; Create a new field called QJ using REPEAT

; Get the Day of the year from the HS field (which is the 6th field)

; Create a new field called QR using COMBIE

; this will be the same as QJ or 0 if either HS did not exist or there was no 6th field

; Create a new field called QE using STYLE

; which will force it to 3 digits padded zero.


; Use QE to create a new filename for the output file which will be :

; 3 digit Julian data, Storyname, Category code (C field in SH) and Keyword (K in SH)

repeat:QJ HS,_,6

combie:QR QJ,0

style:QE QR,%.03d


; Filename


For our example FipHdr, this would generate a filename of :


3. In More Detail ...........

The syntax used is :

(keyword:) (New FipHdr field) (tabs/spaces) (parameters) (eoln)

·         For FIXED fields :

fixed:QT 1234543

ie If \QT is specified, replace with 1234543

Syntax fixed: [newfield] [tab/space] [fixed text]

use escape chrs like \s to embed spaces, tabs, newlines etc

If you use a sequence number several times for the same file, (\$Z or \$S) then normally the seqno is incremented EACH time you call it.

However if you specify it once in a fixed field, this new fipHdr field can then be used as many times as you wish !

Example :

before:\$Z : is the seqno

after:\$Z : is now the next seqno

if the seqno is 24, this will stuff :

"24 : is the seqno" at the top of the file

and "25 : is now the next seqno" at the bottom

Probably NOT what you wanted!

Using fixed we get the same result for this file, next seqno for the next etc.:

fixed:QZ \$Z

before:\QZ : is the seqno

after:\QZ : is still the seqno

if the seqno is 24, this will stuff :

"24 : is the seqno" at the top of the file

and "24 : is still the seqno" at the bottom

·         For PARTIAL fields. An example:

partial:QT ST,3,2,U,+,|

ie Create QT using the contents of the ST header field starting at byte 3 for 2 characters and, if alphabetic, force UPPERCASE.

Syntax partial: [newfield] [tab/space]

[existing field] [comma] [startposn]

[opt comma] [opt length]

[opt comma] [opt processing]

[opt comma] [opt start character]

[opt comma] [opt end character]

where : Start and Length are counters from 1 not 0.

Length can be zero or not defined for all characters in the field

Processing can be one or two chrs (or none for no change) :

If you want to filter a TYPE of chr :

a - alphabetic letters

x - alphanumeric - letters and numbers

e - alphanumeric plus dot

z - anpa type - alphanumeric plus dash

d - decimal numbers plus optional dot, commas

m - money plus optional dot, commas, plus, minus and dollar

n - number 0,1,2,3,4,5,6,7,8,9

t - punctuation (!"\043$%^&*()_+=-:;@'~#<>,.|\)

s - space (space, tab, ff, cr, nl but NOT backspace)

p - printable (including space)

w - printable (excluding space)

c - control

Filters on negatives are also valid using a '#' before the type. eg partial:ZE SN,,,#c will give all the chrs which are NOT control

This will ONLY allow characters matching this type.

For alphabetic chrs, you may also force the case :

u - uppercase

l - lowercase

i - idicase - or camel-case where the

f - upperFirst or initial case

eg partial:QE SN,1,6,#sL

will give the first 6 NONspace chrs forced lower

Start chr is a start of valid data chr

End chr is an end of string when this chr is encounted

·         For COMBINATIONS :

combie:QZ ep|na,(0000000)a

ie Stuff QZ with the contents of the EP header field

- if EP does not exist or has no data then there use the NA field

- if NA does not exist or has no data then there use the fixed text '(0000000)a'.

Syntax combie: [newfield] [tab/space]

[existing field1] [|] [existing field2]

[opt comma] [opt default fixed text]

PIPE '|' is used for separating HdrFlds while the first comma means that the rest of the data is fixed text to use as a default.

Note that if a FipHdr field is present BUT has no data, it is considered NOT present. eg


Hdr field BB 'BB:' will fail 'BB:1' will pass

·         For OPTIONAL fields (used in conjuction with the \$O flag):

option:QT ep,11,7,s

ie If EP header field exists and has a space in the 7th position, send this text else strip text until the \$O flag.

Syntax option: [newfield] [tab/space]

[newfield] [?] [existing field] [comma] [size]

[opt comma] [opt posn of test chr]

[opt comma] [opt 'S' to output data otherwise nothing is output]

[opt comma] [opt String to test for]

where size is minimum size of field.

Normally this is just a TEST - ie nothing is output - but the 'send' parameter can be used to output the contents from the position specified IF THERE.

Note that both size and test are counters from 1 not 0.

A single chr can be tested to be non-space as in the example above.

If either the size or the test is FALSE, all text and subsequent data whether fixed or variable (including more Optionals) is ignored until the EndOpt flag is met - '\$O' (oh, not zero).

How is this used ?

option:JN SN,10

before:StoryName is \SN \JN..and it is 10 or more chrs long at \$h:\$n \$Owhile the ..

If the SN field is less than 10 chrs in length - say SN:lily - the 'before' string will output :

StoryName is lily while the ..

ie JN fails so all (text, FipHdr, Sys Variables etc) is stripped to the \$O

However an SN:bigbigbigbig gives a 'before' string of :

StoryName is bigbigbigbig ..and it is 10 or more chrs long at 12:22 while the ..

Also see a more involved example at the end of the REPEAT section.

Taking four cases where an example field XB is :

1. NOT present

2. present with no data

3. present with data

4. present with data including a space at position 3

Specify (no XB) XB: XB:111 XB:11 11

option:QH XB fail fail pass pass

option:QH XB,1 fail fail pass pass

option:QH XB,4 fail fail fail pass

option:QH XB,1,1 fail fail pass pass

option:QH XB,,1 fail fail pass pass

option:QH XB,,3 fail fail fail fail

option:QH XB,,4 fail fail fail pass

In a further example we can test the content of a field

eg option:QP PR,,,,21

Here we do not care about size, position and output; only whether the field starts with the value "21".
Case (lower or upper) is always ignored.
To test fields with spaces, special chrs etc, put double quotes around.

eg option:PF XF,,,,\023\021

option:PZ XR,,,,"PASNAP \$d"

Note that the data is compared ONLY for the length of the string specifi ed in the 'option'. So if the FipHdr 'SS' contains "12345678"


option:PS SS,,,,1 - will be TRUE

option:PS SS,,,,12345678 - will be TRUE

option:PS SS,,,,123456789 - will be FALSE

So to test for an EXACT string, add a comma at the end of the field. (if you want a comma as part of the string, use FipSeq to encode it - \054 (octal))

If the string to test is NOT the first chr of the FipHdr field, put the stat posn in the 'test' field.

option:PS SS,,3,,34567

Test a string is NOT equal using the following syntax:

option:PS #SS,,,,12345

ie is SS is NOT 12345, PS will be valid

Test is a number is greater or less than a given amount - use 6th (a gretaer numer) and 7th parameters (less than) to check

option:Q5 N1,,,,,21,14
N1 must be between 14 and 21 for Q5 to be true

·         For REPEAT fields, There are two different types of syntax :

1. with a particular chr as separator :

repeat:QR XK,-,3

find FipHdr field XK, using '-' as sep, find the 3rd sub-field

Use this for FipHdr fields like :


2. using a type of chr (or NOT a type of chr)

in this case the sep is left blank, and valid types are

u - uppercase

l - lowercase

x - alphanumeric

e - alphanumeric plus dot

t - punctuation

s - space (space, tab, ff etc)

n - number

p - printable

c - control


repeat:QR XK,,3,U

To negate, use the '#' before the type

repeat:QR XK,,3,#X

Ie QR is the 3rd field in the K field of SH separated by a non-alphanumeric chr

Use this for FipHdr fields like :

repeat:QR RR,,5,S

RR:a319 Boeing-747 TigerMoth MothsBros *&^%$ BingBang


There is actually a third type of REPEAT code for use with OPTION where it is possible to output all codes :

In IPEDSYS, the keyword 'before' adds a String at the top of the data file :

before:<MODVER ID="1:00" VER="03">\n\ZU<DEST ID="1:05">\QU</DEST>\n\$O<FIFO ID="1:20"FFNO="25">\n

; add our option/repeat with a star ..

; if Fip Hdr field \ZU is specified,

; ..look for field \QU has a least 1 chr in it

; .. if there is, output to the \$O flag and then start again with a test for the next field

; .. if there is nothing, strip until the \$O.

option:ZU QU,1

; Hdr Field QU is really EACH sub-field of \NU separated by a comma.

repeat:QU NU+,+*


----So if Fip Hdr field NU is :



----The top of the file will have :

<MODVER ID="1:00" VER="03">

<DEST ID="1:05">HOK</DEST>

<DEST ID="1:05">DDD</DEST>

<DEST ID="1:05">NNR</DEST>

<DEST ID="1:05">MSV</DEST>

<DEST ID="1:05">SMF</DEST>

<DEST ID="1:05">XXX</DEST>

<FIFO ID="1:20" FFNO="25">

... and the rest of the data

·         For STYLE :

- This uses the C 'printf' which is nasty but a standard (of sorts).

- Do a man printf for fuller information if you need.

- Note it ALWAYS starts with a '%'

- If the expression does not end with an 's' ('d' for integer for example), then the string in the header field is first converted to that type.

- Specify One and ONLY one expression (can not have %s%d%f) - as is takes the first only

- Do NOT use for fixed data - use keyword 'fixed:' (as explained above)

- Types are :

string : s

char : c

long : d,i,o,p,u,x,X

float : f,e,E,g,G

% : print a % !

type n is ignored ??


- to trim a string, use a dot : %.5s

- To pad a string with spaces : %5s

- To pad a string with spaces (left justified) : %-5s

- To pad a number with leading zeros : %.06d

·         For REPLACE :

Copy the contents of one FipHdr field into another and then search and

replace characters or strings.



replace:(New FipHdr) (spc/tab) (Existing FipHdr)

[optional ,(ForceCase/IgnoreCase)]

(spc/tab) (Search String) (punctuation) (Replace)


eg replace:LC XC SPO=s INL=i UDL=u ECO=f


There can be up to 50 searches.

The If the 'C' flag is NOT specified (to Ignorecase), the Search String MUST always be in the right case. So always


specify all combinations :

SPO=s spo=s

The output can be forced upper or lowercase by specifying 'U' or 'L'

after the existing FipHdr field.


eg replace:LC XC,u SPO=s INL=i UDL=u ECO=f


Strings with spaces and punctuation MUST be enclosed by double quotes.



eg replace:K2 YR,u ".Y"=yellow ".C"=cyan ".M"=magenta ".K"=black

The replacement string may be in FipSeq - but NOT the search.

You must take care to only use small amounts of data - 1000 chrs max

plus Never call a FipHdr field from itself !!

·         For NEWDATE :

Create a new internal FipHdr field with a date relative to when the program runs

Syntax :

newdate: (new FipHdr field) (differences) (style)

where differences is a series of keywords and the varying amount

and Style MUST be in double quotes. Style can contain both ordinary words and FipSeq.

The syntax for differences is

years + or - a number






ie hours+3 for plus 3 hours

days-7 for the same day a week ago

If you need to use a date which is relative to a date/time which is NOT the current date, put 'basedate=YYYYMMDDHHNNBB'.

There are no spaces between the number and the hours etc.

Several differences can be specified, each spearated by a SPC or TAB.

The contents of ANOTHER fiphdr field may be used for the number :


Only the first (or for 'm' first two) letters are considered.

The default is a PLUS sign meaning in the future.

ie 'h3' is the same as 'hours+3'


The Style MUST be in double quotes. They can contain both ordinary words and FipSeq. Relevant header fields are

ZD - 2 digit day of month (leading space)

ZG - 2 digit day of month (leading zero)

ZM - 2 digit month

ZY - 2 digit Year 92

ZZ - 4 digit Year 1992

ZW - Day of week as in Monday, Tuesday etc

ZS - 3 chr Day of week as in Mon, Tue etc

ZN - Month as in January, February

ZT - 3 chr Month as in Jan, Feb, Mar etc

ZJ - Julian day of year

ZH - Hour 00-23

ZI - Hour 00-12

ZF - Minute 00-59

ZE - Second 00-59

ZU - 1st, 2nd, 3rd, 24th for the day of

ZA,ZB,ZK - The three different week-of-year numbers (ZA is 0-53 starting Sunday, ZB is 0-53 starting Monday, ZK is ISO8601 1-53 where the first week has 4 or more days starting Monday!)

ZC - Day of week

ZP - AM or PM

Note that actual Day and Month names depend on your LOCALE

Default is "\ZW, \ZD \ZN \ZZ"



newdate:JS min-38 hour+8 day+\MD "xx\ZD-\ZS-\ZZ or \ZY \ZH:\ZTxx"

Where MD is a FipHdr field in the incoming file.

Gives a result like xx01-Sat-2000 or 00 00:17xx


·         For UNIQUE :

Control the contents of a single FipHdr field so that :

- each element is unique

- each element is separated by one and only one separator chr


unique:ZZ F1,|


unique:(new FipHdr Fld) (space) (old FipHdr Field)

[(opt) (comma) (separator) ]

[(opt) (comma) (force Upper or Lower case) ]


The Separator defaults to a plus sign '+'.

eg unique:Q1 XC,=

So if XC (ie the C field of SH) was :

SH:N1234:Sabc:Ccat1 cat1 cat3 cat2 cat3 cat4:P3

Q1 now becomes


The individual elements of are considered different if a space,

plus '+' or comma ',' is found.

Multiple spaces etc are ignored.

If you want the Sep to be a comma, use another punctuation chr

eg unique:Q1 XC+,


·         For VALID :

- create a new FipHdr by checking the contents of another against a standing list of valid entries. If there is no match, the first entry is used as the default.

valid:(new FipHdr Fld) (space) (old FipHdr Field)

[(opt) (comma) (force Upper or Lower case) ]

(list of values using one or more spaces or tabs as separator)


If any of the values contain a space or comma, it must be inside double quotes

eg valid:Q1 XP 4,1,2,3,5

Create Q1 from XP which must be a value of 1 to 5 and defaults to 4


·         For LOOKUP :

- create a new FipHdr by matching the contents of another against a standing lookup file. Note this is for simple, small lookup files NOT massive tables of thousands of entries.
Use fipseq REPLACE for a small number of fixed entries - say up to 30.
Use fipSeq LOOKUP for between 30 and 100 entries (or if the file needs to be maintained externally)
. Use program IPLOOKUP for anything else.

lookup:(new FipHdr Fld) (space) (old FipHdr Field)

file:(filename - no default,; this field is mandatory)

key-field: (field number of the key field - starting at 1 - default 1 for 1st)

value-field: (field number of the data or value field - starting at 1 - default 2 for 2nd)

default-value: (FipSeq - default - none)

separator: (Fipseq chr - default is '|')

type-format: (filetype - CSV, TAB-sep, Fixed width - default is blank for TEXT)

comment-line: (character which signifies a comment line - default is ';')

numeric-key: signifies the key is numeric.

sorted: yes/no is the file sorted ?

case-sensitive-key: yes/no ie Cardiff or cardiff

allow-spaces: normally leading and trailing spaces in both the key and searched-for zone are removed before testing.

missing-msg:"Missing value " log message if there is no entry for this value

only-allow:(char which starts line of only those lines to check) This is the opposite of comment-line as ONLY lines starting with this chr will be scanned.

wild:(chr) allow keys to end with this wildcard chr This allows the data to have short keys. ie wild:% In the file car%|4wheels|petrol will match data with 'car alarm', 'carriage', 'cart'

rpt-sep:(FipSeq chr - no default) Separator between multiple input keys

The old FipHdr field will have the key to search for.

eg lookup:Y1 YP file:Categories sep:| comment:# key:3 value:5

Create Y1 from the entry in file /fip/tables/setup/Categories which matches the contents of XP where the key is in field 3 and the value to use is in field 5.

If using Fixed width fields, the width of each field follows the 'F' in type-format.

eg lookup:Y2 SU file:services key:1 value:4 type-format:F,6,6,2,2,3

Create Y2 from the entry in file /fip/tables/setup/services which matches the contents of SU where the key is in field 1 (which are characters 1 to 6) and the value to use is in field 4 (characters 15 and 16) .

Where the old FipHdr has more than one key - and you need to lookup ALL of them, then specify the Rpt-Separator - usually a spc (as it is unprintable, use \s) The result is a single FipHdr with all the new codes withe the same separator.

eg F1:AAA BBB CCC DDD and the lookup file maps ; comment here AAA^ZZZ BBB^YYY DDD^XXX

lookup:F2 F1 file:/fip/tables/setup/MYMAP sep:^ comment:; key:1 value:2 rpt-sep:\s gives F2:ZZZ YYY XXX (note CCC is missing, so only 3 are returned)


·         For PERL :

- create a new FipHdr from a Perl Regular expression.

Take care that the syntax is correct and does not have an infinite loop!

Take care to respect Perl quotes - especially for Winnt where for some bizarre reason we need two double quotes around it !

eg for Winnt

perl:Z4 ""@hoho = 29 * \F4;""

Note that all '\' should be doubled as the RegExp string is considered to be FipSeq, so other FipHdr fields may be included.

The RegExp MUST be on a single line and you are advised to use a 'print' to display the result you want.

Any NewLines and/or Carriage Returns are mapped to Spaces.

perl:(new FipHdr Fld) (space) (Perl RegExp)

eg perl:E1 $seqno = "\VD"; $bigSeq = 24*$seqno; print "Seqno is $bigSeq";

Create E1 with (if FipHdr VD:3) "Seqno is 72".


·         For MERGE :

- create a new FipHdr from a on or more occurances of another FipHdr field.

Normally, it a FipHdr field is repeated, only the last FipHdr fields is accessible - which allows you to overwrite entries quite happily.

But there are some (few) occasions where in fact you want the first or second or maybe all of them concatenated together in one long string. This could be when you run 'ipxchg' against another formatted file which has duplicated entries for some fields.

merge:(new FipHdr Fld) (space) (old FipHdr Fld)

[(optional) , Separator]

[(optional) , First-occurance]

[(optional) , NumberOfTimes]

In the above syntax the comma may be replaced by any other punctuation except '$'. The Separator is to divide two or more fields and can be in FipSeq and can be a TAB (\t), SPC (\s) or any printable chr. The First occurance is counted from the begining or top of the FipHdr. The NumberOfTimes is the number of fields to take into account.

eg if the FipHdr had :

merge:BC AB,+

gives a new Fiphdr field of BC:blue+green+red


·         For SUM :

- create a new FipHdr from the result of a calculation or change between decimal, hex or octal

sum:(new FipHdr Fld) [: optional flags] (space) (the calculation)

The calculation is in FipSeq - so can include other FipHdr fields Calculations can NOT be split over several lines. The default precision for a calculation is 2 decimal places. This can be overridden using the syntax : sum:AB 100*(\Q1/\Q2)

Flags can be