ipxchg
IPXCHG translates the Character Set and any Style changes that are needed to
process any incoming file. It uses a parameter file, in tables/xchg, to define
what the processing should be and how it should be done.
The incoming file is normally passed on by 'ipwheel' as part of the pipline to
get the data to the destination. IPXCHG can also be run as a once-hit against a
particular file.
In order to determine what parameter file to use, 'ipxchg' uses progs.FipHdr fields,
if they exist.
This can be overridden by the '-D' input switch which will force the xchg to be
that name.
Otherwise the 'CX' field is first looked for.
If there is no 'CX', the 'SC' field is compared to the 'DC' field. If they are
different, the xchg file used will be SC2DC.
ie if 'SC:AP' and 'DC:HTML', then the xchg parameter file will be 'AP2HTML'.
If there is NO SC, DC or CX and NO -D input switch, a string at the beginning
of the filename (up to the first HASH/POUND SIGN) can be used.
The actual parameter file is in tables/xchg and the filename will be forced
Uppercase.
Syntax for the xchg parameter file held in tables/xchg :
one parameter per line
; Comments lines may be inserted at any time and are preceeded by a ';'.
: Comment lines can also be preceeded by a colon in this case
(type) [ (flag) ] (delimiter) (1st parameter) [ (delimiter) (2nd parameter)
...]
where items in [] are optional
- type is a CASE SENSITIVE type of parameter - c, D, x etc
There should be NO leading spaces/tabs. To aid legibility,
other letters and spaces may be put AFTER the type but take
care NOT to use letters or punctuation, eg these are equivalent:
chrset :ascii
and c:ascii
- A flag is any number between 1 and 100. It allows conditional
xchgs to be done. Ie the flag must have be set ON using the 'o'
command beforehand or the line is ignored.
- Delimiter can be any punctuation EXCEPT backslash or Hash.
it must also be consistent :
z:FSPC:after
z_FSPC_after
- Parameter string is any FIP sequence of fixed text,
Octal numbers (\000 format) and Unix escape chrs (\n etc).
In some cases - eg 'r' for Add progs.FipHdr - the data can be any
progs.FipSeq string including System Variables and existing progs.FipHdr
fields.
The first parameter string is called the TO or SEARCH field.
The second parameter (if there) is FROM or REPLACE field.
There are no mandatory parameters - the defaults are :
c: ASCII Chrset is ASCII
b: 8 : 8 Input stream is 8 bit chr and output is 8 bit
Default is : No escapes, xchgs or other mutilations.
List of Types
_____________
Syntax Example
------ -------
x, X - An xchg; from is the source string, to is the replacement string.
The From or search string can be 1-255 chrs long; the replacement string can be
0-255 chrs long.
x:fromString:toString x:+++:bong
The difference between 'x' and 'X' is :
'x' the search string is case SENSITIVE
'X" the search string is case INsensitive
ie X:head:hhh
will replace 'HeAd' or 'HEAD' or 'head' etc with 'hhh'
c -This refers to parameter file tables/chrset/PA
(see below for format of this external chrset file)
c:(main chrset) c:PA
There are a number of hard-coded chrsets which do NOT need to be in chrset:
c:ascii 7 bit ascii
c:isoascii 8 bit ascii
c:ascii2baudot automatic conversion from ascii to baudot
c:c2let,c2num the two baudot sets
c:utf8 8 bit unicode - see doc on utf
c:utf16 16 bit unicode - see doc on utf
c:ucs2 16 bit unicode (Microsoft's notepad/Sql Server version)
Multiple chrsets can be defined as comma separated names :
c:PA,PA2,PA3
There can be up to 10 chrsets.
or if only one alternate set needs to be defined :
C - This is the alternate chrset - for 5 or 6 bit sets for example
C:alt chrset C:c2num
So these two are equivalent :
c:c2let,c2num c:c2let
C:c2num
h - This is the header chrset. Flag z:chghdr must be specified for the fields
to translate (see below). If there is no 'h:' the main chrset is used.
h:hdr chrset h:swhdr
Please do NOT use characters NL, Tilde (octal 176) and Colon in Header fields
They will be moved to octal 371, 372 and 373 respectively if you do !
e, E - Escape chrs to the alternate and back to the main chrset for
multi-chrset
For example where multi-layer chrsets such as BAUDOT and TTS are used,
e:escape chr from main e:\033
E:escape chr from alt E:\037
Note the default is to pass these chrs through - use z:ZAPESC to strip.
OR these can be specified as
e:(Esc chr):(Number of chrset)
where the number is the chrset in the 'c:' line above :
c:PA,PA2,PA3 PA is 1, PA2 is 2 and PA3 is 3
there can be up to 20 escape chrs. The actual chr MUST be unique so octal 005
cannot
escape to both PA and PA2 at the same time. But more than one escape chr
can point to the same chrset.
Escapes MUST always be specified lower down the parameter file than Chrsets.
s, S - Escape chrs for a single Chr from the alternate and then revert back to
the main chrset for multi-chrset
For example where multi-layer chrsets such as ISO-7901 in its many guises.
s:single chr escape from main s:&
S:single chr escape from alt S:&
Where we want to escape to a secondary chrset for a single chr.
Note the default is to pass these chrs through - use z:ZAPESC to strip.
d, D - Delete all between from and to strings.
Lowercase d strips BOTH FROM and TO strings, Upper D strips only the FROM.
d:ignore from:to d:<:>
D:ignore from:to D::
u, U - UNYank or unsave text previously saved using 'y' or 'Y'
Lowercase u adds the text at the BEGINNING of the FROM field,
while Uppercase U adds it AFTER the FROM. The TO field is ignored.
u:
U:=AddAfter=
w, W - Allow wild character and strings
The first character of the Lowercase w is the wild CHARACTER ie
a single character, while the first character of the Uppercase W
is the wild string.
w:? wild chr is '?'
W:* wild string is '*'
Note that only one wild string is allowed in the TO or SEARCH string
of any test and it can NOT be the first chr of the SEARCH string.
There is no restriction on the number of wild Chrs in either search
replace strings. When used in REPLACE the saved chrs are output
in the same order as found.
The maximum number of chrs and string is 1024.
; wild CHR is '?'
w:?
; wild STRING is '*'
W:*
x:ABCD???HIJ:ab?fg*
An incoming string of 'ABCDEFGHIJ' will be stripped to 'abEfgFG'
Note that you should make sure that the two wilds are not in
any other SEARCH or REPLACE string anywhere else in the
parameter file or you will get funny results :
W .*
x .[p*].
x .@*&%$.abcde
will probably NOT give the required values for the second, fixed string.
Note that by default the wild card/string is positional - use it and lose it.
To use the Wild string as a full string each time - as in the example :
x .[cp*].[cp*][cl*]
use the option 'z:ALLWILD'
To use the contents of the wild buffer in a later xchg line,
use the option 'z:KEEPWILD'
z - Setup Option
z:option z:FCC:
Options include:
FCC idiot upper and lower-case. Note you may force an
Uppercase letter by using a '~' as a precedent chr.
NUL do NOT strip NULs ie binary zero chrs
NULS are normally deleted from the output automatically
TOPSPC strip all spaces, carriage return and line feeds at the top of the
file
BOTTOMSPC strip all spaces, carriage return and line feeds at the bottom of
the file
SPC or SPC:before strip DUPLICATE spaces BEFORE xchgs
SPC:after strip DUPLICATE spaces AFTER xchgs
LINE or LINE:before strip DUPLICATE newlines/line feeds BEFORE xchgs
LINE:after strip DUPLICATE newlines/line feeds AFTER xchgs
IGNORE-DATA Ignore the data part of the file.
Any fipHdr and before/after are processed but all data from the file is
stripped.
FIPHDR-IS-DATA-TOO Allow xchgs on the whole of the file
including the progs.FipHdr part!
NOFIPHDR or NOHDR Do NOT add the FIP header to the data.
CHGHDR Also replace header fields with this chrset. This
defaults to the Source Header only (SH). Other fields
can be specified by listing after this flag. eg :
z:chghdr:QA,SH,DI
The header chrset can be specified by the 'h:' parameter
or will default to the same as the primary chrset as
in the 'c:' parameter. Note that if the fields also
exist in the filename, they are also changed.
Note you cannot replace individual subfields of the SH -
like XP, XF etc. Use NEWHDR to make a copy of that
subfield first and then chghdr it.
z:newhdr:AP \XP
h:anpapriority
z:chghdr:ap
HDRONLY Only process the progs.FipHdr fields - leave the data alone - see below.
WORDMAC If the file is in the Mac version of the Microsoft Word
format, the header and trailer are stripped, leaving
just the text section. This works for Versions 5 and 6 only.
WORDPC If the file is in the Mac version of the Microsoft Word
format, the header and trailer are stripped, leaving
just the text section. This works for Versions 6 only.
ZAPESC For escaped chrsets - such as 5 level baudot or 6 level
TTS, delete/ignore the escape chrs.
ALLWILD Use the whole of the wild string in a REPLACE, not just
the remainder.
KEEPWILD:
Normally the wild card buffer is only available for
the one Xchg line and then zapped after use.
This allow it to be used later on.
PARTIAL new FIP header field created from existing field
z:partial:qc XK,3,8,u
ie To make QC, take K header field from the Source Header (SH)
posn 3 for 8 chrs, UPPERCASE.
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
FIXHDR additional FIP header information
z:fixhdr:QZ 1234543
ie If QZ is specified, replace with 1234543
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
OPTION test a progs.FipHdr field before outputting
z:option:QR ZZ,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
REPEAT create a new progs.FipHdr field from an existing field
z:repeat:QR ZZ,-,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
COMBIE create a new progs.FipHdr field from one of two existing fields
or use a default
z:combie:QR ZZ|ZR,(defaultpls)
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
STYLE: create a new progs.FipHdr field from an existing field and
pad it with spaces, zeros or justify it using
a 'printf' type style
z:style:QZ XZ,%.04f
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
REPLACE: create a new progs.FipHdr field from an existing field and
replace one or more strings/chrs.
z:replace:qK XK "BC-"=""
(refer to progs.SysAdmin progs.FipSeq doc for more information)
NEWDATE: create a new progs.FipHdr field containing the date offset from
the current datee and time
z:newdate:QD days+3 "\ZH - \ZD"
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
SUM:, VALID:, LOOKUP:, UNIQUE:,
(refer to progs.SysAdmin progs.FipSeq doc for more information)
NEWHDR: create and SAVE a new progs.FipHdr field.
Any progs.FipSeq can be added.
z:newhdr:JF FormatIs.\XF
z:newhdr:JN \XN+\XS=\SN
This field is added to the progs.FipHdr at the start of
processing and can therefore be 'hdrXchged' etc.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
Please note that any progs.FipSeq (z:REPLACE: z:COMBIE: etc) MUST be
specified first BEFORE any NEWHDR lines in the parameter file.
TEXT2FIPHDR: Extract a line at the top of the file and make it a
progs.FipHdr field (and normally/optionally remove from text).
A line is ended by any combination of CR and NL
This is done on the RAW, incoming data,; not the output.
At the top of text :
This is done on the RAW, incoming data,; not the output.
At the bottom of text :
Specify with a negitive line number : -1 for the last, -2 2nd last etc
Lines are counted AFTER z:bottomspc and and Strip from (I:)
If there is a trailing NL, CR or CRNL, they are ignored
syntax z:text2fiphdr: (line number):(fiphdr):(optional flags)
eg z:text2fiphdr:1:DI:no
The optional flags can be :
stripfromdata - also delete that data
or xml - ignore any xml tags and attributes and only get data
The default strip is YES. There can be 100 z:text2fiphdr lines
(note you cannot strip XML)
REDUCEBEFORE: (chr)
REDUCEAFTER: (chr)
Reduce all multiple occurances of a single chr to a one
The character can be in progs.FipSeq - \031
reducebefore acts After the 'chrset' and Before any xchgs
reduceafter acts after all processing.
There can be up to 100 reducesbefores and afters.
WRAPTEXT: (line length) : (End-of-Line string)
Wrap text after all processing to a specified number
of chrs per line. Add the End-of-Line string for long
lines.
The default line length is 64.
The default end of line is NL or \n
IGNORE-XML-IN-WRAP: yes/no
Ignore any tags within <> in calculating wordwrap
default: no for leave as part of the calculation
WRAP-LONG-WORDS:(length)
add a space after this number of characters to wrap long words
eg z:wrap-long-words:77
default: no
NUMBER: (octal/decimal/hex)
Change the number system for progs.FipSeq numbers to this
The default is octal.
Note this takes effect for ALL parameters in the file.
FILEBEFORE: (complete pathname to a progs.FipSeq file)
BINARY-FILEBEFORE: (complete pathname to a progs.FipSeq file)
Add the contents of this file AFTER the progs.FipHdr (if there
is no 'z:nohdr' and before the 'a:' text string before
and before the data.
There can be up to 100 file befores.
The file can be contain any text and progs.FipSeq except the binary
version will copy the data byte for byte with no checking or
parsing for FipSeq.
z:filebefore:/fip/tables/customer/Header.fipseq
z:binary-filebefore:/fip/spool/savefiles/\SN
z:filebefore:/fip/spool/savefiles/ThirdFile
FILEAFTER: (complete pathname to a progs.FipSeq file)
BINARY-FILEAFTER: (complete pathname to a progs.FipSeq file)
ditto for after.
The data from this file is add AFTER data and AFTER the
'A' text string after.
There can be up to 100 file afters.
INPUT-FILEBEFORE:
INPUT-FILEAFTER:
Special case of before and after files where the data
is the binary contents of the input data file with NO
changes. This is rarely used - as an extra attachment
with a preceeding mimetype, for example.
LOCALE: (valid locale)
Change the locale from the System Locale to this
The locale MUST be valid !
CONVERT-TO-UCS2:
Convert the incoming data to the UCS2 variant of Unicode.
This process is done on ALL xchged data as an absolutely last stage.
CONVERT-TO-ENTITIES: (dec|hex|quark)
Convert all non-basic ascii chars to the ϧ notation.
eg a pound sign (xA6 or octal 236) becomes ž
Use this for XML feeds that do not want UTF8.
The option can be :
dec - decimal notation ì
hex - hex notation ¦
quark - quark express notation <\#236>
indesign - adobe indesign notation <0x00A6>
Default is decimal
ENTITY-HASH:(fipseq chr)
Replacement chr for the HASH '#' in an entity created by CONVERT-TO-ENTITIES
Use this if any progs.FipHdr fields need converting too.
CONVERT-TO-UTF8:
Convert the incoming data to the UTF8 variant of Unicode.
This process is NOT done on any xchged data.
If you need the reverse process, you will have to find
all the funny chrs and map them to whatever eg
x +\u2034+(euro)
READ-ENTITY-FILE: (name of file in tables/xchg)
Many websites/xml feeds encode certain chrs as 'entities' - either named or
numbered:
ie ' for a ' € for the euro sign
numbers like £ for a progs.PoundSign
This uses a progs.W3C entity file to automatically resolve Html/Xml entities
before the rest of the xchg is run
If the entity is NOT in the file, it is left in the data and not converted.
so &russiandoll; would be left alone
These are exactly the same format/files as used by ipsgml and
are 'borrowed' from w3c http://www.w3.org/TR/xhtml1/DTD/ appendix 2
They look like
Multiple files can be specified :
z:read-entity-file:xhtml-lat1.ent
z:read-entity-file:xhtml-special.ent
z:read-entity-file:xhtml-symbol.ent
CONVERT-FROM-BASE64:
Convert the data to or from Base64
when converting TO base64,
- Adjust line lengths with the optional number (default 72)
- Only the data part is converted - which allows befores
and/or filebefores to add lines like:
Content-Transfer-Encoding: base64
- Note that if you are encoding PDF files, you nearly
always need to preserve NULs - using z:nul
See the example below.
UNMATCHED-UNICODE: (progs.FipSeq String)
Where the chrset of the incoming data is UTF8 or UTF16 or UCS2, use this to
map any hi-byte unicode chr to a replacement.
z:unmatched-unicode:*
FLAG-IF-FIPHDR: (flag number) : (progs.FipHdr field)
Turn on a flag if a progs.FipHdr exists and has a least one character in.
z:flag-if-fiphdr:42:XC
There is an optional string after the progs.FipHdr which is a further test that
the progs.FipHDr equals this string.
z:flag-if-fiphdr:44:SU:AFP
or use either != or # to signify NOT
z:flag-if-fiphdr:44:SU#AFP
USA-DATE-FORMAT \$A date format should be MM/DD (default is DD/MM)
usa-data-format:no will also force it DD/MM
UNICODE-MAP (filename)
This will use a Unicode Mapping table - such as BIG5 or CNS and map to
Unicode
Always use in conjunction with 'z:convert-to-utf8' and
'c:isoascii'. For standard maps - look at
http://www.unicode.org/Public/MAPPINGS/
or http://tcl.apache.org/sources/tcl/tools/encoding
In these lookup tables, '#' is a comment.
Each chr is on a separate line:
(raw hex chr) (spc) (unicode chr) spc [ opt comment]
eg from KOI-8.TXT
0x9F 0x00F7 # DIVISION SIGN
0xA0 0x2550 # BOX DRAWINGS DOUBLE HORIZONTAL
0xA1 0x2551 # BOX DRAWINGS DOUBLE VERTICAL
0xA2 0x2552 # BOX DRAWINGS DOWN SINGLE AND RIGHT DBL
0xA3 0x0451 # CYRILLIC SMALL LETTER IO
UNICODE-MAP-IS-7BIT:yes
Some code maps - like BIG5 - are 7 bit codes not 8
as the mapping start at xA1 not xFF
default is NO (or unspecified) for 8 bit
UNICODE-MAP-BASE: (ascii | isoascii | none)
fill in any missing elements with the basic ASCII set
ascii - basic 7 bit from 0 to x7F
isoascii- basic 8 bit from 0 to xFF
none - ignore it !
default - none
FLIPLINES
Flip a line of data from left2right to right2left
A line end is CR, NL or CRNL
CONVERT-FIPHDR-TO-QP: (orig progs.FipHdr field):(new):(language):(length):(progs.WrapChr)
Convert a single progs.FipHdr to Quoted-Printable for use
in a Mime header (as used by the 'Subject' line in smtp)
If no language is specified, iso-8859-1 is the default.
If no length is specified, the whole field is converted otherwise
this is the maximum allowed on the line (normally 76).
If length is specified and the language is 'utf-8', the chrs are
converted to and from beforehand so that complete chrs are included.
The Optional progs.WrapChr is a progs.FipSeq chr which signals to ipsmtp to wrap here
rather than trimming the Fiphdr to that size.
eg
z:convert-fiphdr-to-qp:DI:DJ:koi-8
DI:DJIA at 3pm is 9999.3
generates
DJ:=?koi-8?Q?=DJIA=20at=203pm=20is=209999.3?=
CONVERT-ENCODED-WORDS: (all|hdr|data)
This converts the progs.FipHdr or the Data or Both from Mime Encoded Words format
Use this to convert Email subject lines like
Subject: =?windows-1255?B?5uQg8uXjIPDx6eXvIOH54ensIODw4/jl?=
An Encoded Word line (usually an email field) has the format '=? (chrset)
'?' (type) '?' (data) '?='
This only works within text, so if the data is in a progs.FipHdr field, use this
and
Z:FIPHDR-IS-DATA-TOO in the xchg parameter file (and make sure you have no
other xchg lines which could
damage the progs.FipHdr !).
ANCHOR:
Search for common web urls and wrap an tag around them
Urls starting with 'http://', https://, ftp://', 'www.',
'ftp.' and 'mailto:' are converted.
eg www.fippo.org becomes
www.fippo.org
This works for Text only and can have a negetive effect on wordwrap.
PRESERVE-FILENAME:
The input filename will be used as the output filename.
Normally the output filename has a few fipHdr fields added.
This is the same as the -P input switch, but will only be done
for files using this xchg.
REPLACE-TAG: (original xml tag) (spaces) (replacement tag)
used to replace a tag with its attributes
Also use 'z:replace-tag-attX' (where X is 1-9) to put the attributes into a
temporary, private progs.FipHdr area, so you can use progs.FipSeq to reouput them in the
new tag.
EG - Incoming data has format :
Reed Elsevier
But we need the style of :
Manpower MAN shares led the gainers, rising 2.5
percent to $44.52.
z:replace-tag:ticker \T0 \T6
z:replace-tag-att1:companyLongName
z:replace-tag-att2:companyShortName
; Sedol is T3
z:replace-tag-att3:sedol
z:replace-tag-att4:isin
z:replace-tag-att5:countryCodeId
z:replace-tag-att6:zzcode
LOG-LINE: (extra information for logging in progs.FipSeq)
z:log-line:\SN company=\A1 indcode=\A2 isin=\A3
The default is \SN \CX for filename and xchg(s)
ipxchg needs the '-l' input switch for this to print
An Example of an xchg :
-----------------------
; use the MAC chrset ie /fip/tables/chrset/MAC
c:mac
; strip all text from notes mode to the next mode change
D:::
x::
x::
x::
; use a different delimiter for changes sake
x-\300e-e
; strip duplicate spaces both before and after the xchgs
; default is BEFORE the xchgs
z:SPC:before
z:SPC:after
Multiple Passes
---------------
From version 47b you can run multiple xchgs in the same run by specifying in
the CX Fiphdr file (usually in the USERS file)
stocks DP:edsys1 DQ:junk-edp CX:Stocks1,Stocks2,Stocks3 SC:NO DC:YES
DF:stocks
This will run the same file through 3 xchgs successively - tables/xchg/STOCKS1,
STOCKS2 and STOCKS3 - before sending to the Editorial System.
Previously the OLD method was ...
Note that only one pass of the file can be made in one go. So to run multiple
passes, add dummy addresses in the tables/sys/USERS file eg :
stocks DP:localhost DQ:2go CX:Stocks1 SC:NO DC:YES DU:stocks1
stocks1 DP:localhost DQ:2go CX:Stocks2 SC:NO DC:YES DU:stocks2
stocks2 DP:edsys1 DQ:junk-edp CX:Stocks3 SC:NO DC:YES DF:stocks
How to use flags
----------------
- ON 'o' and OFF 'f' and progs.FlipFlop 'p' lines do not affect the data themselves.
- The flag is turned on/off AFTER the string.
- add spaces around the flag for better legibility, These are the same :
x77:\n\n\n:\r\n
x 77 :\n\n\n:\r\n
- NOT all lines are affected :
x, X, I, o, f, r, k, K, d, D
- you can use flags within flags :
o 22 +\n+32
Only if flag 22 is ON, a newline will turn 32 ON.
- Test for a flag being OFF with a '#' before the number :
f # 19 :abc:33
if flag 19 is OFF, string 'abc' will turn flag 33 OFF (if 33 was ON)
o # 19 |abc|33
if flag 19 is OFF, string 'abc' will turn flag 33 ON (ignoring if 33 was ON or
OFF)
- Flags 1 to 100 are acceptable.
- You want to set flags on and off before any xchgs which use them as the xchg
may zap the data you need to key off. ie Turn the first 'A' after a NL to a
'B':
o:\n:32
x 32 :A:B
f 32 :A:32
will NEVER turn flag 32 OFF as the preceeding line will have modified it !
Better do :
o:\n:32
f 32 :A:32
x 32 :A:B
From version 44j, you may specifiy multiple flags on/off by putting a
non-number between :
o +\n+1,2,3,4
Current limitations are :
-------------------------
Max size of an XCHG parameter file is 64k chrs.
Max size of a search OR a replacement string is 255 chrs each.
Max number of flags is 100 (1-100).
Running IPXCHG from a Script
----------------------------
The program can be used in a script to run against a single file using the '-1'
input switch. In this case, if the output file exists, it is OVERWRITTEN. If
the output queue is the same as the input, the input FILE is overwritten. The
'-D/-F/-o' switches are also advised in this case to be sure where the output
file will end up. Eg
/fip/bin/ipxchg -1 formwork/SHARES.RUBY -F -o formsave -D crunchy
will process (and delete after) the file from spool/formwork using the xchg
file tables/xchg/CRUNCHY and leave the xchged file in formsave.
or ipxchg -1 Cruddo -D mangle
will process Cruddo in the current queue using xchg file MANGLE.
Note that the output queue is ALWAYS under /fip/spool.
Special Case of Running an Xchg against the progs.FipHdr
--------------------------------------------------
Using 'z:hdronly' allows you to ONLY modify data in the progs.FipHdr and leave the
data part completely untouched.
Only a subset of commands are useable - some are obviously ignore like Word
format. Others not taken into account are :
- 'h' for hdr field translate (use normal 'c' for chrset)
- 'a' and 'A' for adding before and after the data part
You may find it advisable to use flags to limit the damage an xchg can do to a
particular progs.FipHdr field.
For example - we want to xchg LAG, SOC and ORO in the 'NC' progs.FipHdr field and and
word progs.ManU in the 'NZ' :
; no chrset chg here
c:isoascii
; turn ON progs.HdrOnly
z:hdronly
; Flag 1 is ON during the NC field only - turned off at the end of the line
; note we use pipes as delimiters as we have a colon ':' as part of the data
o |\nNC:| 1
f |\n| 1
; Only search for these if flag 1 is on
x 1 |LAG|PAR:12000000:LAG
x 1 |SOC|PAR:21000000:SOC
x 1 |ORO|PAR:21000000:ORO
; Flag 2 is ON during the NZ field only - turned off at the end of the line
o |\nNH:| 2
f |\n| 2
; Only search for these if flag 2 is on
x 2 |ManU|Liverpool
Example of using ipxchg to encode mail attachments
--------------------------------------------------
Use this for sending PDFs for example therough 'ipsmtp'
; main parameter file
; default chrset
c:isoascii
; preserve NULs
z:nul
; convert to base64 with a line length of 75 chrs
z:convert-to-base64:75
; read and parse the mime-header template beforehand
z:filebefore:/fip/tables/xchg/MIMEHDR.BEFORE
; At the end - finish the last part
A+\r\n--180191Fip\SN180191--\r\n
... and MIMEHDR.BEFORE could look like:
Content-Type: multipart/mixed;\r
boundary="180191Fip\SN180191"\r\n\r
--180191Fip\SN180191\r
Content-Type: text/plain\r\n\r
The following is a PDF from the Fippo
--180191Fip\SN180191\r
Content-Disposition: attachment; filename="\SN"\r
Content-Type: application/pdf\r
Content-Transfer-Encoding: base64\r\n\r
Picking up data in the file and reusing as a progs.FipHdr
---------------------------------------------------
Say you want to pick up the funny date/time stamp at the end of a wire file..
;
; create TI field from wire file
; needs ipxchg.44d plus (13nov00)
stuff the lot into the progs.FipHdr 'TI'
r-TI-*
; a '*' is a wild string
W:*
; do not zapt the wild buffer after the 'x' line
z:keepwild
; strip from the ETX to the EST
x:\003*EST:
; ** ENDS **
---------------------------------------------------
Input flags
-----------
Input options are (all optional) :
Either -i : queue in spool to scan default: xchg
Or -1 : name of a single file to xchg default: none
In this case the input file is NOT deleted and
the output is forced to the folder '-o' field if there.
or the filename '-N' if specified
-o : default destination queue if no CQ field is present
Normally a FIP hdr field, CQ, determines the output queue.
If that is missing, this field overwrites the default
If this does NOT starts with a '/', it is assumed to
be under /fip/spool. default: 2atex
-O : display the resultant, xchged file and do NOT save to disk.
Use this inside a CGI for example default: disk
-d : name of the default xchg default: no default
-D : force the xchg file to this def: use progs.SC2DC or CX
-e : Supplementary xchg filename default: none
This name is appended to the CX or progs.SC2DC file name
This enables a 2nd group of xchges to be used.
-E : name of a single entity file that should be run on ALL data. def: none
see Z:READ-ENTITY-FILE: above
-f : file creep time default: 0
For files arriving from a network, this is the wait time
Generally this should be 5 secs for RCPed files etc.
-F : force all output to the output destination queue and ignore
the CQ field. default: use CQ
-G : WINNT/2k - grab the first file only default: no
-h : Extra progs.FipHdrs to be added to the input fiphdr default: none
-I : ignore any include-files in the parameter file default: allow
ie ignore z:FILEBEFORE:abc.pl
-l : log items through default: no
-L : do NOT complain/log files requesting non-existent xchgs
- just use the default default: complain
-N : output filename (or path and filename) for single files (-1) only.
default: overwrite input file
-P : Preserve the filename for the output file default: if spooled, chged.
-r : move the original file to this queue default: delete it
-R : move the original file to this queue default: delete it
and Change the name to CCYYMMDD.HHMMSS.(name)
-S : display any message that is being logged. default: log only.
-t : scan time of directory default: 2 secs
-x : do NOT delete the original default: delete it
This only applies to spooled files; running single mode
will NOT delete the input.
-X : do NOT allow multiple xchgs in the CX progs.FipHdr field . default: allow
-W : 10 ths of a second - MUST wait if nothing processed default: 0
-Y : one or more flags to set ON at the start of the run default: none
-Y 33,44
-9 : do NOT run speedyFip with named pipes default: do if available
-v : print version number and exit default: no
Version Control
;048d3 ; 17apr07 added convert-to-entities:indesign
;a-b 14may07 added Z:WRAP-LONG-WORDS:(length)
;c-d2 05nov07 added xml for progs.TEXT2FIPHDR
(;d WINNT bugette ;d1-2 poss bug if single chr X and already-zapped data ;d3
bugette with R:A3:abc)
;047z bugette with multiple xchgs losing CQ
;a-b 19apr06 added z:replace-tag
;c 15may06 bugette if flag set on input or FLAG-IF-FIPHDR without other 'o'
and 'f'
;de 25may06 added z:convert-to-entities at last and -N newname for single
;fg 28may06 added z:ignore-xml-in-wrap
;h-k 22jun06 added z:convert-to-entities:quark and ENTITY-HASH
;l 25jul06 allow flags in i/I strips and fixed bugette in -Y startflags
;m-n 14aug06 wrinkle in convert264 one too many '=' at end
;o 21sep06 added progs.WrapChr to CONVERT-FIPHDR-TO-QP
;p 23oct06 bugette with c:utf8/z:convert-to-utf8
;q-s 13nov06 added convert-encode-words:
;t 08dec06 added <@\> to quark-entities and removed TAB
;u-v 08jan07 allow ^ and % in parameter file names (v bugette in wild)
;w-y 08feb07 added z:FLIPLINES
;z 17mar07 added -3 extraFH3 plus Winnt -O is now binary
;046z3 mod for unicode-map (single bytes bugette)
;b 26jan04 added INPUT-FILEBEFORE/AFTER for original, raw file
added CONVERT-FIPHDR-TO-QP
;c 30mar04 chrset can now be hex too \x4E
;d-e 19apr04 bugettes with unicode in hdr and include_file
;f-i 28apr04 added z:anchor
;j 29jun04 added max length to Convert-to-QP
;k-l 16aug04 added Multiple xchgs..
;m-q 07sep04 added -9 speedy and x/m speedup and -I
;r-s 01apr05 add unicode-map-is-7bit for CP850 and KOI7
;t 06may05 bugette with wildchrs (emmettlanebug)
and the hex in CONVERT-FIPHDR-TO-QP should be UCASE
;u 30jun05 bugette in map-unicode - sometimes making a mapped chr NUL
;v 02aug05 bugette in i/I if string > 120 chrs
;w-z3 22sep05 added CONVERT-TO-UCS2 and read-entity-file - z3 thsxhg->STDBUF
;045z 16apr02 added UNMATCHED-UNICODE
;a/b 18apr02 added 'm' for multibyte chr as in m+\u20ac+EURO\s
;c 25jun02 added -G for WINNT/2k grab first file
;d 31jul02 z:NEWHDR - new fipHdr field may also have '$', '-' and/or '_'
;e 09aug02 z:FIPHDR-IS-DATA-TOO added
;f/g 15aug02 ':' is only matched/moved to 372 in progs.FipHdr SH
;h 30oct02 bugette in flags where files has thousands of flags
;i 21dec02 woops - allow h:utf8 too where UTF8 chrs are in the fiphdr
;j 22dec02 allow z:text2fiphdr for lines at the END of text
;k 14jan03 bugette with h:utf8
;m 16jan03 CHANGE - d:<:> will now be ignored if the end is not found
;n 03mar03 bottomspc now zaps replaced copy too
;o 10mar03 added IGNORE-DATA
;p-q 27mar03 better filenames for Win2k plus bugette in NEWHDR with NEWDATE
;r-t 10apr03 added FLAG-IF-FIPHDR
;u-w 14apr03 added USA-DATE-FORMAT/minors
;x-y 25sep03 bottomspc now zaps beforehand too
;z 31oct03 timings
;044z 18aug00 allow multiple filebefores/afters and -h for fiphdr and -O for
stdout
;a/b 19aug00 added text2fiphdr
;c 10oct00 finally got the repeated chr bug
;d 03nov00 ..but that blew Yank/Unyank
;e 12nov00 allow SN length to be variable and allow wilds in 'r'
;f 20nov00 bug - stripped progs.FipHdr for zero length data part
;g 06mar01 possible bug with flags and big files.
;h 27mar01 bugette - strips to/from (i/I) were parsed twice
;i 16apr01 bugette with flags again
;j/k 05jul01 allowed multiple flags on/off o +\n+1,2,3,4
;m 03sep01 bugette with a/A and no data
;o/p/q 13sep01 added binary-filebefore/after and move_it added
;r 03oct01 bugette with progs.FileBefore and no data in file.
;s 25oct01 bugette with text2fiphdr and stripping.
;t 14nov01 added FlipFlipFlags.
;u 19nov01 added z:bottomspc.
;v 26nov01 added c:utf8, c:utf16 and z:convert-to-utf8
;w 29nov01 added flags to A-add after
;x 29jan02 added convert-to/from-base64
;y 25feb02 bugette in d and D where 2nd string may be matched early
;z 15apr02 c:utf8 not working brilliantly.
© progs.FingerPost Ltd. 2008 and years before
ipxchg
ipxchg
IPXCHG translates the Character Set and any Style changes that are needed to
process any incoming file. It uses a parameter file, in tables/xchg, to define
what the processing should be and how it should be done.
The incoming file is normally passed on by 'ipwheel' as part of the pipline to
get the data to the destination. IPXCHG can also be run as a once-hit against a
particular file.
In order to determine what parameter file to use, 'ipxchg' uses progs.FipHdr fields,
if they exist.
This can be overridden by the '-D' input switch which will force the xchg to be
that name.
Otherwise the 'CX' field is first looked for.
If there is no 'CX', the 'SC' field is compared to the 'DC' field. If they are
different, the xchg file used will be SC2DC.
ie if 'SC:AP' and 'DC:HTML', then the xchg parameter file will be 'AP2HTML'.
If there is NO SC, DC or CX and NO -D input switch, a string at the beginning
of the filename (up to the first HASH/POUND SIGN) can be used.
The actual parameter file is in tables/xchg and the filename will be forced
Uppercase.
Syntax for the xchg parameter file held in tables/xchg :
one parameter per line
; Comments lines may be inserted at any time and are preceeded by a ';'.
: Comment lines can also be preceeded by a colon in this case
(type) [ (flag) ] (delimiter) (1st parameter) [ (delimiter) (2nd parameter)
...]
where items in [] are optional
- type is a CASE SENSITIVE type of parameter - c, D, x etc
There should be NO leading spaces/tabs. To aid legibility,
other letters and spaces may be put AFTER the type but take
care NOT to use letters or punctuation, eg these are equivalent:
chrset :ascii
and c:ascii
- A flag is any number between 1 and 100. It allows conditional
xchgs to be done. Ie the flag must have be set ON using the 'o'
command beforehand or the line is ignored.
- Delimiter can be any punctuation EXCEPT backslash or Hash.
it must also be consistent :
z:FSPC:after
z_FSPC_after
- Parameter string is any FIP sequence of fixed text,
Octal numbers (\000 format) and Unix escape chrs (\n etc).
In some cases - eg 'r' for Add progs.FipHdr - the data can be any
progs.FipSeq string including System Variables and existing progs.FipHdr
fields.
The first parameter string is called the TO or SEARCH field.
The second parameter (if there) is FROM or REPLACE field.
There are no mandatory parameters - the defaults are :
c: ASCII Chrset is ASCII
b: 8 : 8 Input stream is 8 bit chr and output is 8 bit
Default is : No escapes, xchgs or other mutilations.
List of Types
_____________
Syntax Example
------ -------
x, X - An xchg; from is the source string, to is the replacement string.
The From or search string can be 1-255 chrs long; the replacement string can be
0-255 chrs long.
x:fromString:toString x:+++:bong
The difference between 'x' and 'X' is :
'x' the search string is case SENSITIVE
'X" the search string is case INsensitive
ie X:head:hhh
will replace 'HeAd' or 'HEAD' or 'head' etc with 'hhh'
c -This refers to parameter file tables/chrset/PA
(see below for format of this external chrset file)
c:(main chrset) c:PA
There are a number of hard-coded chrsets which do NOT need to be in chrset:
c:ascii 7 bit ascii
c:isoascii 8 bit ascii
c:ascii2baudot automatic conversion from ascii to baudot
c:c2let,c2num the two baudot sets
c:utf8 8 bit unicode - see doc on utf
c:utf16 16 bit unicode - see doc on utf
c:ucs2 16 bit unicode (Microsoft's notepad/Sql Server version)
Multiple chrsets can be defined as comma separated names :
c:PA,PA2,PA3
There can be up to 10 chrsets.
or if only one alternate set needs to be defined :
C - This is the alternate chrset - for 5 or 6 bit sets for example
C:alt chrset C:c2num
So these two are equivalent :
c:c2let,c2num c:c2let
C:c2num
h - This is the header chrset. Flag z:chghdr must be specified for the fields
to translate (see below). If there is no 'h:' the main chrset is used.
h:hdr chrset h:swhdr
Please do NOT use characters NL, Tilde (octal 176) and Colon in Header fields
They will be moved to octal 371, 372 and 373 respectively if you do !
e, E - Escape chrs to the alternate and back to the main chrset for
multi-chrset
For example where multi-layer chrsets such as BAUDOT and TTS are used,
e:escape chr from main e:\033
E:escape chr from alt E:\037
Note the default is to pass these chrs through - use z:ZAPESC to strip.
OR these can be specified as
e:(Esc chr):(Number of chrset)
where the number is the chrset in the 'c:' line above :
c:PA,PA2,PA3 PA is 1, PA2 is 2 and PA3 is 3
there can be up to 20 escape chrs. The actual chr MUST be unique so octal 005
cannot
escape to both PA and PA2 at the same time. But more than one escape chr
can point to the same chrset.
Escapes MUST always be specified lower down the parameter file than Chrsets.
s, S - Escape chrs for a single Chr from the alternate and then revert back to
the main chrset for multi-chrset
For example where multi-layer chrsets such as ISO-7901 in its many guises.
s:single chr escape from main s:&
S:single chr escape from alt S:&
Where we want to escape to a secondary chrset for a single chr.
Note the default is to pass these chrs through - use z:ZAPESC to strip.
d, D - Delete all between from and to strings.
Lowercase d strips BOTH FROM and TO strings, Upper D strips only the FROM.
d:ignore from:to d:<:>
D:ignore from:to D::
u, U - UNYank or unsave text previously saved using 'y' or 'Y'
Lowercase u adds the text at the BEGINNING of the FROM field,
while Uppercase U adds it AFTER the FROM. The TO field is ignored.
u:
U:=AddAfter=
w, W - Allow wild character and strings
The first character of the Lowercase w is the wild CHARACTER ie
a single character, while the first character of the Uppercase W
is the wild string.
w:? wild chr is '?'
W:* wild string is '*'
Note that only one wild string is allowed in the TO or SEARCH string
of any test and it can NOT be the first chr of the SEARCH string.
There is no restriction on the number of wild Chrs in either search
replace strings. When used in REPLACE the saved chrs are output
in the same order as found.
The maximum number of chrs and string is 1024.
; wild CHR is '?'
w:?
; wild STRING is '*'
W:*
x:ABCD???HIJ:ab?fg*
An incoming string of 'ABCDEFGHIJ' will be stripped to 'abEfgFG'
Note that you should make sure that the two wilds are not in
any other SEARCH or REPLACE string anywhere else in the
parameter file or you will get funny results :
W .*
x .[p*].
x .@*&%$.abcde
will probably NOT give the required values for the second, fixed string.
Note that by default the wild card/string is positional - use it and lose it.
To use the Wild string as a full string each time - as in the example :
x .[cp*].[cp*][cl*]
use the option 'z:ALLWILD'
To use the contents of the wild buffer in a later xchg line,
use the option 'z:KEEPWILD'
z - Setup Option
z:option z:FCC:
Options include:
FCC idiot upper and lower-case. Note you may force an
Uppercase letter by using a '~' as a precedent chr.
NUL do NOT strip NULs ie binary zero chrs
NULS are normally deleted from the output automatically
TOPSPC strip all spaces, carriage return and line feeds at the top of the
file
BOTTOMSPC strip all spaces, carriage return and line feeds at the bottom of
the file
SPC or SPC:before strip DUPLICATE spaces BEFORE xchgs
SPC:after strip DUPLICATE spaces AFTER xchgs
LINE or LINE:before strip DUPLICATE newlines/line feeds BEFORE xchgs
LINE:after strip DUPLICATE newlines/line feeds AFTER xchgs
IGNORE-DATA Ignore the data part of the file.
Any fipHdr and before/after are processed but all data from the file is
stripped.
FIPHDR-IS-DATA-TOO Allow xchgs on the whole of the file
including the progs.FipHdr part!
NOFIPHDR or NOHDR Do NOT add the FIP header to the data.
CHGHDR Also replace header fields with this chrset. This
defaults to the Source Header only (SH). Other fields
can be specified by listing after this flag. eg :
z:chghdr:QA,SH,DI
The header chrset can be specified by the 'h:' parameter
or will default to the same as the primary chrset as
in the 'c:' parameter. Note that if the fields also
exist in the filename, they are also changed.
Note you cannot replace individual subfields of the SH -
like XP, XF etc. Use NEWHDR to make a copy of that
subfield first and then chghdr it.
z:newhdr:AP \XP
h:anpapriority
z:chghdr:ap
HDRONLY Only process the progs.FipHdr fields - leave the data alone - see below.
WORDMAC If the file is in the Mac version of the Microsoft Word
format, the header and trailer are stripped, leaving
just the text section. This works for Versions 5 and 6 only.
WORDPC If the file is in the Mac version of the Microsoft Word
format, the header and trailer are stripped, leaving
just the text section. This works for Versions 6 only.
ZAPESC For escaped chrsets - such as 5 level baudot or 6 level
TTS, delete/ignore the escape chrs.
ALLWILD Use the whole of the wild string in a REPLACE, not just
the remainder.
KEEPWILD:
Normally the wild card buffer is only available for
the one Xchg line and then zapped after use.
This allow it to be used later on.
PARTIAL new FIP header field created from existing field
z:partial:qc XK,3,8,u
ie To make QC, take K header field from the Source Header (SH)
posn 3 for 8 chrs, UPPERCASE.
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
FIXHDR additional FIP header information
z:fixhdr:QZ 1234543
ie If QZ is specified, replace with 1234543
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
OPTION test a progs.FipHdr field before outputting
z:option:QR ZZ,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
REPEAT create a new progs.FipHdr field from an existing field
z:repeat:QR ZZ,-,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
COMBIE create a new progs.FipHdr field from one of two existing fields
or use a default
z:combie:QR ZZ|ZR,(defaultpls)
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
STYLE: create a new progs.FipHdr field from an existing field and
pad it with spaces, zeros or justify it using
a 'printf' type style
z:style:QZ XZ,%.04f
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
REPLACE: create a new progs.FipHdr field from an existing field and
replace one or more strings/chrs.
z:replace:qK XK "BC-"=""
(refer to progs.SysAdmin progs.FipSeq doc for more information)
NEWDATE: create a new progs.FipHdr field containing the date offset from
the current datee and time
z:newdate:QD days+3 "\ZH - \ZD"
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
SUM:, VALID:, LOOKUP:, UNIQUE:,
(refer to progs.SysAdmin progs.FipSeq doc for more information)
NEWHDR: create and SAVE a new progs.FipHdr field.
Any progs.FipSeq can be added.
z:newhdr:JF FormatIs.\XF
z:newhdr:JN \XN+\XS=\SN
This field is added to the progs.FipHdr at the start of
processing and can therefore be 'hdrXchged' etc.
(refer to progs.SysAdmin progs.FipSeq doc for more information)
Please note that any progs.FipSeq (z:REPLACE: z:COMBIE: etc) MUST be
specified first BEFORE any NEWHDR lines in the parameter file.
TEXT2FIPHDR: Extract a line at the top of the file and make it a
progs.FipHdr field (and normally/optionally remove from text).
A line is ended by any combination of CR and NL
This is done on the RAW, incoming data,; not the output.
At the top of text :
This is done on the RAW, incoming data,; not the output.
At the bottom of text :
Specify with a negitive line number : -1 for the last, -2 2nd last etc
Lines are counted AFTER z:bottomspc and and Strip from (I:)
If there is a trailing NL, CR or CRNL, they are ignored
syntax z:text2fiphdr: (line number):(fiphdr):(optional flags)
eg z:text2fiphdr:1:DI:no
The optional flags can be :
stripfromdata - also delete that data
or xml - ignore any xml tags and attributes and only get data
The default strip is YES. There can be 100 z:text2fiphdr lines
(note you cannot strip XML)
REDUCEBEFORE: (chr)
REDUCEAFTER: (chr)
Reduce all multiple occurances of a single chr to a one
The character can be in progs.FipSeq - \031
reducebefore acts After the 'chrset' and Before any xchgs
reduceafter acts after all processing.
There can be up to 100 reducesbefores and afters.
WRAPTEXT: (line length) : (End-of-Line string)
Wrap text after all processing to a specified number
of chrs per line. Add the End-of-Line string for long
lines.
The default line length is 64.
The default end of line is NL or \n
IGNORE-XML-IN-WRAP: yes/no
Ignore any tags within <> in calculating wordwrap
default: no for leave as part of the calculation
WRAP-LONG-WORDS:(length)
add a space after this number of characters to wrap long words
eg z:wrap-long-words:77
default: no
NUMBER: (octal/decimal/hex)
Change the number system for progs.FipSeq numbers to this
The default is octal.
Note this takes effect for ALL parameters in the file.
FILEBEFORE: (complete pathname to a progs.FipSeq file)
BINARY-FILEBEFORE: (complete pathname to a progs.FipSeq file)
Add the contents of this file AFTER the progs.FipHdr (if there
is no 'z:nohdr' and before the 'a:' text string before
and before the data.
There can be up to 100 file befores.
The file can be contain any text and progs.FipSeq except the binary
version will copy the data byte for byte with no checking or
parsing for FipSeq.
z:filebefore:/fip/tables/customer/Header.fipseq
z:binary-filebefore:/fip/spool/savefiles/\SN
z:filebefore:/fip/spool/savefiles/ThirdFile
FILEAFTER: (complete pathname to a progs.FipSeq file)
BINARY-FILEAFTER: (complete pathname to a progs.FipSeq file)
ditto for after.
The data from this file is add AFTER data and AFTER the
'A' text string after.
There can be up to 100 file afters.
INPUT-FILEBEFORE:
INPUT-FILEAFTER:
Special case of before and after files where the data
is the binary contents of the input data file with NO
changes. This is rarely used - as an extra attachment
with a preceeding mimetype, for example.
LOCALE: (valid locale)
Change the locale from the System Locale to this
The locale MUST be valid !
CONVERT-TO-UCS2:
Convert the incoming data to the UCS2 variant of Unicode.
This process is done on ALL xchged data as an absolutely last stage.
CONVERT-TO-ENTITIES: (dec|hex|quark)
Convert all non-basic ascii chars to the ϧ notation.
eg a pound sign (xA6 or octal 236) becomes ž
Use this for XML feeds that do not want UTF8.
The option can be :
dec - decimal notation ì
hex - hex notation ¦
quark - quark express notation <\#236>
indesign - adobe indesign notation <0x00A6>
Default is decimal
ENTITY-HASH:(fipseq chr)
Replacement chr for the HASH '#' in an entity created by CONVERT-TO-ENTITIES
Use this if any progs.FipHdr fields need converting too.
CONVERT-TO-UTF8:
Convert the incoming data to the UTF8 variant of Unicode.
This process is NOT done on any xchged data.
If you need the reverse process, you will have to find
all the funny chrs and map them to whatever eg
x +\u2034+(euro)
READ-ENTITY-FILE: (name of file in tables/xchg)
Many websites/xml feeds encode certain chrs as 'entities' - either named or
numbered:
ie ' for a ' € for the euro sign
numbers like £ for a progs.PoundSign
This uses a progs.W3C entity file to automatically resolve Html/Xml entities
before the rest of the xchg is run
If the entity is NOT in the file, it is left in the data and not converted.
so &russiandoll; would be left alone
These are exactly the same format/files as used by ipsgml and
are 'borrowed' from w3c http://www.w3.org/TR/xhtml1/DTD/ appendix 2
They look like
Multiple files can be specified :
z:read-entity-file:xhtml-lat1.ent
z:read-entity-file:xhtml-special.ent
z:read-entity-file:xhtml-symbol.ent
CONVERT-FROM-BASE64:
Convert the data to or from Base64
when converting TO base64,
- Adjust line lengths with the optional number (default 72)
- Only the data part is converted - which allows befores
and/or filebefores to add lines like:
Content-Transfer-Encoding: base64
- Note that if you are encoding PDF files, you nearly
always need to preserve NULs - using z:nul
See the example below.
UNMATCHED-UNICODE: (progs.FipSeq String)
Where the chrset of the incoming data is UTF8 or UTF16 or UCS2, use this to
map any hi-byte unicode chr to a replacement.
z:unmatched-unicode:*
FLAG-IF-FIPHDR: (flag number) : (progs.FipHdr field)
Turn on a flag if a progs.FipHdr exists and has a least one character in.
z:flag-if-fiphdr:42:XC
There is an optional string after the progs.FipHdr which is a further test that
the progs.FipHDr equals this string.
z:flag-if-fiphdr:44:SU:AFP
or use either != or # to signify NOT
z:flag-if-fiphdr:44:SU#AFP
USA-DATE-FORMAT \$A date format should be MM/DD (default is DD/MM)
usa-data-format:no will also force it DD/MM
UNICODE-MAP (filename)
This will use a Unicode Mapping table - such as BIG5 or CNS and map to
Unicode
Always use in conjunction with 'z:convert-to-utf8' and
'c:isoascii'. For standard maps - look at
http://www.unicode.org/Public/MAPPINGS/
or http://tcl.apache.org/sources/tcl/tools/encoding
In these lookup tables, '#' is a comment.
Each chr is on a separate line:
(raw hex chr) (spc) (unicode chr) spc [ opt comment]
eg from KOI-8.TXT
0x9F 0x00F7 # DIVISION SIGN
0xA0 0x2550 # BOX DRAWINGS DOUBLE HORIZONTAL
0xA1 0x2551 # BOX DRAWINGS DOUBLE VERTICAL
0xA2 0x2552 # BOX DRAWINGS DOWN SINGLE AND RIGHT DBL
0xA3 0x0451 # CYRILLIC SMALL LETTER IO
UNICODE-MAP-IS-7BIT:yes
Some code maps - like BIG5 - are 7 bit codes not 8
as the mapping start at xA1 not xFF
default is NO (or unspecified) for 8 bit
UNICODE-MAP-BASE: (ascii | isoascii | none)
fill in any missing elements with the basic ASCII set
ascii - basic 7 bit from 0 to x7F
isoascii- basic 8 bit from 0 to xFF
none - ignore it !
default - none
FLIPLINES
Flip a line of data from left2right to right2left
A line end is CR, NL or CRNL
CONVERT-FIPHDR-TO-QP: (orig progs.FipHdr field):(new):(language):(length):(progs.WrapChr)
Convert a single progs.FipHdr to Quoted-Printable for use
in a Mime header (as used by the 'Subject' line in smtp)
If no language is specified, iso-8859-1 is the default.
If no length is specified, the whole field is converted otherwise
this is the maximum allowed on the line (normally 76).
If length is specified and the language is 'utf-8', the chrs are
converted to and from beforehand so that complete chrs are included.
The Optional progs.WrapChr is a progs.FipSeq chr which signals to ipsmtp to wrap here
rather than trimming the Fiphdr to that size.
eg
z:convert-fiphdr-to-qp:DI:DJ:koi-8
DI:DJIA at 3pm is 9999.3
generates
DJ:=?koi-8?Q?=DJIA=20at=203pm=20is=209999.3?=
CONVERT-ENCODED-WORDS: (all|hdr|data)
This converts the progs.FipHdr or the Data or Both from Mime Encoded Words format
Use this to convert Email subject lines like
Subject: =?windows-1255?B?5uQg8uXjIPDx6eXvIOH54ensIODw4/jl?=
An Encoded Word line (usually an email field) has the format '=? (chrset)
'?' (type) '?' (data) '?='
This only works within text, so if the data is in a progs.FipHdr field, use this
and
Z:FIPHDR-IS-DATA-TOO in the xchg parameter file (and make sure you have no
other xchg lines which could
damage the progs.FipHdr !).
ANCHOR:
Search for common web urls and wrap an tag around them
Urls starting with 'http://', https://, ftp://', 'www.',
'ftp.' and 'mailto:' are converted.
eg www.fippo.org becomes
www.fippo.org
This works for Text only and can have a negetive effect on wordwrap.
PRESERVE-FILENAME:
The input filename will be used as the output filename.
Normally the output filename has a few fipHdr fields added.
This is the same as the -P input switch, but will only be done
for files using this xchg.
REPLACE-TAG: (original xml tag) (spaces) (replacement tag)
used to replace a tag with its attributes
Also use 'z:replace-tag-attX' (where X is 1-9) to put the attributes into a
temporary, private progs.FipHdr area, so you can use progs.FipSeq to reouput them in the
new tag.
EG - Incoming data has format :
Reed Elsevier
But we need the style of :
Manpower MAN shares led the gainers, rising 2.5
percent to $44.52.
z:replace-tag:ticker \T0 \T6
z:replace-tag-att1:companyLongName
z:replace-tag-att2:companyShortName
; Sedol is T3
z:replace-tag-att3:sedol
z:replace-tag-att4:isin
z:replace-tag-att5:countryCodeId
z:replace-tag-att6:zzcode
LOG-LINE: (extra information for logging in progs.FipSeq)
z:log-line:\SN company=\A1 indcode=\A2 isin=\A3
The default is \SN \CX for filename and xchg(s)
ipxchg needs the '-l' input switch for this to print
An Example of an xchg :
-----------------------
; use the MAC chrset ie /fip/tables/chrset/MAC
c:mac
; strip all text from notes mode to the next mode change
D:::
x::
x::
x::
; use a different delimiter for changes sake
x-\300e-e
; strip duplicate spaces both before and after the xchgs
; default is BEFORE the xchgs
z:SPC:before
z:SPC:after
Multiple Passes
---------------
From version 47b you can run multiple xchgs in the same run by specifying in
the CX Fiphdr file (usually in the USERS file)
stocks DP:edsys1 DQ:junk-edp CX:Stocks1,Stocks2,Stocks3 SC:NO DC:YES
DF:stocks
This will run the same file through 3 xchgs successively - tables/xchg/STOCKS1,
STOCKS2 and STOCKS3 - before sending to the Editorial System.
Previously the OLD method was ...
Note that only one pass of the file can be made in one go. So to run multiple
passes, add dummy addresses in the tables/sys/USERS file eg :
stocks DP:localhost DQ:2go CX:Stocks1 SC:NO DC:YES DU:stocks1
stocks1 DP:localhost DQ:2go CX:Stocks2 SC:NO DC:YES DU:stocks2
stocks2 DP:edsys1 DQ:junk-edp CX:Stocks3 SC:NO DC:YES DF:stocks
How to use flags
----------------
- ON 'o' and OFF 'f' and progs.FlipFlop 'p' lines do not affect the data themselves.
- The flag is turned on/off AFTER the string.
- add spaces around the flag for better legibility, These are the same :
x77:\n\n\n:\r\n
x 77 :\n\n\n:\r\n
- NOT all lines are affected :
x, X, I, o, f, r, k, K, d, D
- you can use flags within flags :
o 22 +\n+32
Only if flag 22 is ON, a newline will turn 32 ON.
- Test for a flag being OFF with a '#' before the number :
f # 19 :abc:33
if flag 19 is OFF, string 'abc' will turn flag 33 OFF (if 33 was ON)
o # 19 |abc|33
if flag 19 is OFF, string 'abc' will turn flag 33 ON (ignoring if 33 was ON or
OFF)
- Flags 1 to 100 are acceptable.
- You want to set flags on and off before any xchgs which use them as the xchg
may zap the data you need to key off. ie Turn the first 'A' after a NL to a
'B':
o:\n:32
x 32 :A:B
f 32 :A:32
will NEVER turn flag 32 OFF as the preceeding line will have modified it !
Better do :
o:\n:32
f 32 :A:32
x 32 :A:B
From version 44j, you may specifiy multiple flags on/off by putting a
non-number between :
o +\n+1,2,3,4
Current limitations are :
-------------------------
Max size of an XCHG parameter file is 64k chrs.
Max size of a search OR a replacement string is 255 chrs each.
Max number of flags is 100 (1-100).
Running IPXCHG from a Script
----------------------------
The program can be used in a script to run against a single file using the '-1'
input switch. In this case, if the output file exists, it is OVERWRITTEN. If
the output queue is the same as the input, the input FILE is overwritten. The
'-D/-F/-o' switches are also advised in this case to be sure where the output
file will end up. Eg
/fip/bin/ipxchg -1 formwork/SHARES.RUBY -F -o formsave -D crunchy
will process (and delete after) the file from spool/formwork using the xchg
file tables/xchg/CRUNCHY and leave the xchged file in formsave.
or ipxchg -1 Cruddo -D mangle
will process Cruddo in the current queue using xchg file MANGLE.
Note that the output queue is ALWAYS under /fip/spool.
Special Case of Running an Xchg against the progs.FipHdr
--------------------------------------------------
Using 'z:hdronly' allows you to ONLY modify data in the progs.FipHdr and leave the
data part completely untouched.
Only a subset of commands are useable - some are obviously ignore like Word
format. Others not taken into account are :
- 'h' for hdr field translate (use normal 'c' for chrset)
- 'a' and 'A' for adding before and after the data part
You may find it advisable to use flags to limit the damage an xchg can do to a
particular progs.FipHdr field.
For example - we want to xchg LAG, SOC and ORO in the 'NC' progs.FipHdr field and and
word progs.ManU in the 'NZ' :
; no chrset chg here
c:isoascii
; turn ON progs.HdrOnly
z:hdronly
; Flag 1 is ON during the NC field only - turned off at the end of the line
; note we use pipes as delimiters as we have a colon ':' as part of the data
o |\nNC:| 1
f |\n| 1
; Only search for these if flag 1 is on
x 1 |LAG|PAR:12000000:LAG
x 1 |SOC|PAR:21000000:SOC
x 1 |ORO|PAR:21000000:ORO
; Flag 2 is ON during the NZ field only - turned off at the end of the line
o |\nNH:| 2
f |\n| 2
; Only search for these if flag 2 is on
x 2 |ManU|Liverpool
Example of using ipxchg to encode mail attachments
--------------------------------------------------
Use this for sending PDFs for example therough 'ipsmtp'
; main parameter file
; default chrset
c:isoascii
; preserve NULs
z:nul
; convert to base64 with a line length of 75 chrs
z:convert-to-base64:75
; read and parse the mime-header template beforehand
z:filebefore:/fip/tables/xchg/MIMEHDR.BEFORE
; At the end - finish the last part
A+\r\n--180191Fip\SN180191--\r\n
... and MIMEHDR.BEFORE could look like:
Content-Type: multipart/mixed;\r
boundary="180191Fip\SN180191"\r\n\r
--180191Fip\SN180191\r
Content-Type: text/plain\r\n\r
The following is a PDF from the Fippo
--180191Fip\SN180191\r
Content-Disposition: attachment; filename="\SN"\r
Content-Type: application/pdf\r
Content-Transfer-Encoding: base64\r\n\r
Picking up data in the file and reusing as a progs.FipHdr
---------------------------------------------------
Say you want to pick up the funny date/time stamp at the end of a wire file..
;
; create TI field from wire file
; needs ipxchg.44d plus (13nov00)
stuff the lot into the progs.FipHdr 'TI'
r-TI-*
; a '*' is a wild string
W:*
; do not zapt the wild buffer after the 'x' line
z:keepwild
; strip from the ETX to the EST
x:\003*EST:
; ** ENDS **
---------------------------------------------------
Input flags
-----------
Input options are (all optional) :
Either -i : queue in spool to scan default: xchg
Or -1 : name of a single file to xchg default: none
In this case the input file is NOT deleted and
the output is forced to the folder '-o' field if there.
or the filename '-N' if specified
-o : default destination queue if no CQ field is present
Normally a FIP hdr field, CQ, determines the output queue.
If that is missing, this field overwrites the default
If this does NOT starts with a '/', it is assumed to
be under /fip/spool. default: 2atex
-O : display the resultant, xchged file and do NOT save to disk.
Use this inside a CGI for example default: disk
-d : name of the default xchg default: no default
-D : force the xchg file to this def: use progs.SC2DC or CX
-e : Supplementary xchg filename default: none
This name is appended to the CX or progs.SC2DC file name
This enables a 2nd group of xchges to be used.
-E : name of a single entity file that should be run on ALL data. def: none
see Z:READ-ENTITY-FILE: above
-f : file creep time default: 0
For files arriving from a network, this is the wait time
Generally this should be 5 secs for RCPed files etc.
-F : force all output to the output destination queue and ignore
the CQ field. default: use CQ
-G : WINNT/2k - grab the first file only default: no
-h : Extra progs.FipHdrs to be added to the input fiphdr default: none
-I : ignore any include-files in the parameter file default: allow
ie ignore z:FILEBEFORE:abc.pl
-l : log items through default: no
-L : do NOT complain/log files requesting non-existent xchgs
- just use the default default: complain
-N : output filename (or path and filename) for single files (-1) only.
default: overwrite input file
-P : Preserve the filename for the output file default: if spooled, chged.
-r : move the original file to this queue default: delete it
-R : move the original file to this queue default: delete it
and Change the name to CCYYMMDD.HHMMSS.(name)
-S : display any message that is being logged. default: log only.
-t : scan time of directory default: 2 secs
-x : do NOT delete the original default: delete it
This only applies to spooled files; running single mode
will NOT delete the input.
-X : do NOT allow multiple xchgs in the CX progs.FipHdr field . default: allow
-W : 10 ths of a second - MUST wait if nothing processed default: 0
-Y : one or more flags to set ON at the start of the run default: none
-Y 33,44
-9 : do NOT run speedyFip with named pipes default: do if available
-v : print version number and exit default: no
Version Control
;048d3 ; 17apr07 added convert-to-entities:indesign
;a-b 14may07 added Z:WRAP-LONG-WORDS:(length)
;c-d2 05nov07 added xml for progs.TEXT2FIPHDR
(;d WINNT bugette ;d1-2 poss bug if single chr X and already-zapped data ;d3
bugette with R:A3:abc)
;047z bugette with multiple xchgs losing CQ
;a-b 19apr06 added z:replace-tag
;c 15may06 bugette if flag set on input or FLAG-IF-FIPHDR without other 'o'
and 'f'
;de 25may06 added z:convert-to-entities at last and -N newname for single
;fg 28may06 added z:ignore-xml-in-wrap
;h-k 22jun06 added z:convert-to-entities:quark and ENTITY-HASH
;l 25jul06 allow flags in i/I strips and fixed bugette in -Y startflags
;m-n 14aug06 wrinkle in convert264 one too many '=' at end
;o 21sep06 added progs.WrapChr to CONVERT-FIPHDR-TO-QP
;p 23oct06 bugette with c:utf8/z:convert-to-utf8
;q-s 13nov06 added convert-encode-words:
;t 08dec06 added <@\> to quark-entities and removed TAB
;u-v 08jan07 allow ^ and % in parameter file names (v bugette in wild)
;w-y 08feb07 added z:FLIPLINES
;z 17mar07 added -3 extraFH3 plus Winnt -O is now binary
;046z3 mod for unicode-map (single bytes bugette)
;b 26jan04 added INPUT-FILEBEFORE/AFTER for original, raw file
added CONVERT-FIPHDR-TO-QP
;c 30mar04 chrset can now be hex too \x4E
;d-e 19apr04 bugettes with unicode in hdr and include_file
;f-i 28apr04 added z:anchor
;j 29jun04 added max length to Convert-to-QP
;k-l 16aug04 added Multiple xchgs..
;m-q 07sep04 added -9 speedy and x/m speedup and -I
;r-s 01apr05 add unicode-map-is-7bit for CP850 and KOI7
;t 06may05 bugette with wildchrs (emmettlanebug)
and the hex in CONVERT-FIPHDR-TO-QP should be UCASE
;u 30jun05 bugette in map-unicode - sometimes making a mapped chr NUL
;v 02aug05 bugette in i/I if string > 120 chrs
;w-z3 22sep05 added CONVERT-TO-UCS2 and read-entity-file - z3 thsxhg->STDBUF
;045z 16apr02 added UNMATCHED-UNICODE
;a/b 18apr02 added 'm' for multibyte chr as in m+\u20ac+EURO\s
;c 25jun02 added -G for WINNT/2k grab first file
;d 31jul02 z:NEWHDR - new fipHdr field may also have '$', '-' and/or '_'
;e 09aug02 z:FIPHDR-IS-DATA-TOO added
;f/g 15aug02 ':' is only matched/moved to 372 in progs.FipHdr SH
;h 30oct02 bugette in flags where files has thousands of flags
;i 21dec02 woops - allow h:utf8 too where UTF8 chrs are in the fiphdr
;j 22dec02 allow z:text2fiphdr for lines at the END of text
;k 14jan03 bugette with h:utf8
;m 16jan03 CHANGE - d:<:> will now be ignored if the end is not found
;n 03mar03 bottomspc now zaps replaced copy too
;o 10mar03 added IGNORE-DATA
;p-q 27mar03 better filenames for Win2k plus bugette in NEWHDR with NEWDATE
;r-t 10apr03 added FLAG-IF-FIPHDR
;u-w 14apr03 added USA-DATE-FORMAT/minors
;x-y 25sep03 bottomspc now zaps beforehand too
;z 31oct03 timings
;044z 18aug00 allow multiple filebefores/afters and -h for fiphdr and -O for
stdout
;a/b 19aug00 added text2fiphdr
;c 10oct00 finally got the repeated chr bug
;d 03nov00 ..but that blew Yank/Unyank
;e 12nov00 allow SN length to be variable and allow wilds in 'r'
;f 20nov00 bug - stripped progs.FipHdr for zero length data part
;g 06mar01 possible bug with flags and big files.
;h 27mar01 bugette - strips to/from (i/I) were parsed twice
;i 16apr01 bugette with flags again
;j/k 05jul01 allowed multiple flags on/off o +\n+1,2,3,4
;m 03sep01 bugette with a/A and no data
;o/p/q 13sep01 added binary-filebefore/after and move_it added
;r 03oct01 bugette with progs.FileBefore and no data in file.
;s 25oct01 bugette with text2fiphdr and stripping.
;t 14nov01 added FlipFlipFlags.
;u 19nov01 added z:bottomspc.
;v 26nov01 added c:utf8, c:utf16 and z:convert-to-utf8
;w 29nov01 added flags to A-add after
;x 29jan02 added convert-to/from-base64
;y 25feb02 bugette in d and D where 2nd string may be matched early
;z 15apr02 c:utf8 not working brilliantly.
© FingerPost Ltd. 2008 and years before