API reference
High-level API
High-level compression / decompression API.
These are the functions most applications should use:
compress()/decompress()operate on in-memory bytes.compress_file()/decompress_file()operate on paths and add conveniences such as output-name selection and (for binary output) file-type detection via libmagic.read_header()inspects a TERSE file without decompressing it.
Example:
import terselib
data = terselib.compress(b"HELLO WORLD\n", text=True)
result = terselib.decompress(data, text=True)
assert result.data == b"HELLO WORLD\n"
- class terselib.api.DecompressResult(data, header, text)[source]
Outcome of a decompression call.
- Variables:
data – The decompressed bytes.
header – The parsed
TerseHeader.text – Whether text-mode rendering was applied.
- Parameters:
data (bytes)
header (TerseHeader)
text (bool)
- header: TerseHeader
- terselib.api.enable_verbose_logging(level=10)[source]
Attach a stderr handler to the
terseliblogger.The library is silent by default (a
logging.NullHandleris installed). Call this — or configure theterseliblogger yourself — to see progress and diagnostic messages.- Parameters:
level (int) – Logging level to enable, default
logging.DEBUG.- Return type:
None
- terselib.api.read_header(source)[source]
Parse the header of a TERSE file without decompressing it.
- Parameters:
source (bytes | str | PathLike) – A path to a TERSE file, or its first bytes (at least 12).
- Returns:
The parsed and validated header.
- Raises:
TerseFormatError – If the header is invalid.
- Return type:
- terselib.api.decompress(data, text=None, codepage='internal', rdw=True)[source]
Decompress a TERSE file image.
- Parameters:
data (bytes) – The complete TERSE file bytes.
text (bool | None) – Rendering mode.
Trueconverts EBCDIC to ASCII and renders logical records as LF-terminated lines;Falseyields the exact original bytes (with a 4-byte RDW before each record of a variable-format file).None(the default) auto-detects: the file is decoded in binary mode and, if the record payloads look like EBCDIC text, re-rendered as text.codepage (str) – Code page for text-mode conversion —
"internal"(the AMATERSE-compatible spec tables) or any EBCDIC codec name such as"cp037"or"cp1047".rdw (bool) – Emit RDW record prefixes in binary-variable mode (spec behaviour). Set to
Falseto get bare concatenated records.
- Returns:
A
DecompressResultwith the output bytes and the parsed header.- Raises:
TerseFormatError – If the header is invalid.
TerseCorruptError – If the compressed payload is corrupt.
- Return type:
- terselib.api.compress(data, method='SPACK', host=True, text=False, variable=True, record_length=None, codepage='internal', rdw_input=False, mvs_header=True)[source]
Compress bytes into a TERSE file image.
- Parameters:
data (bytes) – The source bytes.
method (str) –
"PACK"(LZW) or"SPACK"(LZMW, the default — it generally compresses better).host (bool) –
True(default) writes a host file (12-byte header, interoperable with AMATERSE on z/OS);Falsewrites a native SPACK file (4-byte header, raw byte stream, no record model).text (bool) – Treat data as ASCII text: split it into lines, translate ASCII→EBCDIC and store each line as a logical record. Binary mode (default) stores the bytes untranslated.
variable (bool) – Use variable (V/VB) record format — the default — rather than fixed (F/FB). Ignored for native files.
record_length (int | None) – Fixed LRECL, variable-mode segment size, or (text-variable) maximum line length recorded in the header. Sensible defaults are chosen when omitted; see
terselib.records.build_codepoints().codepage (str) – Code page for text-mode ASCII→EBCDIC conversion.
rdw_input (bool) – In binary-variable mode, treat data as RDW-framed records (as produced by FTP
SITE RDWtransfers or by binary decompression) instead of segmenting it every record_length bytes.mvs_header (bool) – Write the MVS-annotated header form AMATERSE produces (default);
Falsewrites the minimal all-zero form.
- Returns:
The complete TERSE file bytes.
- Raises:
TerseParameterError – On invalid parameter combinations.
- Return type:
- terselib.api.decompress_file(source, dest=None, text=None, codepage='internal', rdw=True, detect_type=True)[source]
Decompress a TERSE file on disk.
- Parameters:
dest (str | PathLike | None) – Output path. When omitted, the name is derived from source (its
.trs-style suffix stripped) and, for binary output with detect_type enabled, an extension chosen by sniffing the decompressed bytes with libmagic.text (bool | None) – Rendering mode, as for
decompress().codepage (str) – Code page for text-mode conversion.
rdw (bool) – Emit RDWs in binary-variable mode (spec behaviour).
detect_type (bool) – Use file-magic content detection when choosing a default output extension for binary output.
- Returns:
(output_path, result).- Return type:
Exceptions
Exception hierarchy for terselib.
- exception terselib.exceptions.TerseFormatError[source]
The input is not a valid TERSE file (bad header, bad validation bytes, inconsistent record lengths, …).
Headers (Layer 0)
Layer 0 — parsing and building TERSE file headers (spec §2).
Two header layouts exist:
Native (version
0x01/0x07): 4 bytes — the version byte plus three fixed validation bytes. Always SPACK, binary, fixed-record.Host (version
0x02= PACK,0x05= SPACK): 12 bytes carrying the record format (F/V), the record length, and informational MVS fields.
- terselib.header.MVS_BLOCK_CAPACITY = 27998
BlockSize used by the MVS-annotated header form: the usable half-track capacity of a 3390 DASD volume, as written by AMATERSE.
- class terselib.header.TerseHeader(version, method, host, variable, record_length, flags=0, ratio=0, block_size=0, size=12)[source]
Parsed representation of a TERSE file header.
- Variables:
version – The version byte (
0x01,0x02,0x05or0x07).method – Compression method,
"PACK"or"SPACK".host – True for host files (12-byte header), False for native.
variable – True when the record format is variable (V/VB).
record_length – Effective record length. For fixed files this is the LRECL; for variable files it is the maximum record data length (LRECL − 4).
0for native files, which carry none.flags – Raw flags byte (host only).
ratio – Compression-ratio hint (informational, host only).
block_size – Block size hint (informational, host only).
size – Header size in bytes (4 or 12).
- Parameters:
- terselib.header.parse_header(data)[source]
Parse and validate the header at the start of data (spec §2, §12).
- Parameters:
data (bytes) – The file image (only the first 12 bytes are examined).
- Returns:
A validated
TerseHeader; itssizegives the offset at which the 12-bit code stream begins.- Raises:
TerseFormatError – If the header is not a valid TERSE header.
- Return type:
- terselib.header.build_header(method, host, variable, record_length, mvs_style=True)[source]
Build a header for a file being compressed (spec §2, §8.3).
- Parameters:
method (str) –
"PACK"or"SPACK".host (bool) – True for a host file (12-byte header), False for native (4-byte header; forces SPACK).
variable (bool) – True for variable record format (host only).
record_length (int) – Record length to record in the header — the LRECL for fixed files, LRECL − 4 (the maximum record data length) for variable files. Ignored for native files.
mvs_style (bool) – When true (the default), write the MVS-annotated header form AMATERSE itself produces (
FLAGMVS | FLAGVSand a 3390 half-track block size); when false, write the minimal form withFlags = Ratio = BlockSize = 0.
- Returns:
The encoded header bytes.
- Raises:
TerseParameterError – On an invalid method / mode combination.
- Return type:
Records and rendering (Layer 1)
Layer 1 — the record / codepoint model (spec §4, §6.1, §7).
- Decompression side
RecordSinkimplements the spec’sPutChar/endRecord/closemachinery: it receives codepoints (0..257) from a Layer-2 decoder and reconstructs the output byte stream (lines, RDW-framed records or raw bytes depending on the mode).- Compression side
build_codepoints()converts source bytes into the stream of codepoints (1..257) consumed by the Layer-2 encoders, applying text-mode line parsing, ASCII→EBCDIC translation, record segmentation and fixed-record padding.
- class terselib.records.RecordSink(host, text, variable, record_length, ebc_to_asc, emit_rdw=True)[source]
Reconstructs decompressed output from a stream of codepoints.
Implements
PutChar/endRecord/closefrom spec §6.1. The behaviour matrix (spec §9):host
text
variable
rendering
host
text
fixed
EBCDIC→ASCII, line break every record_length
host
text
variable
EBCDIC→ASCII, line break at each RECORDMARK
host
binary
fixed
raw bytes, record marks discarded
host
binary
variable
RDW-framed records (4-byte RDW + payload)
native
binary
fixed
raw byte stream
- Parameters:
host (bool) – True for host files.
text (bool) – True to render in text mode (EBCDIC→ASCII + lines).
variable (bool) – True for variable record format.
record_length (int) – Fixed record length (used in host-text-fixed mode only).
ebc_to_asc (bytes) – 256-byte EBCDIC→ASCII table used in text mode.
emit_rdw (bool) – When False, suppress the 4-byte RDW prefix normally written for each record in binary-variable mode (a convenience deviation from the spec; the default True follows spec §6.1).
- put(x)[source]
Process one codepoint
X(0..257) — specPutChar.- Parameters:
x (int)
- Return type:
None
- close()[source]
Flush any trailing partial record and return the output bytes.
Only a non-empty trailing record is flushed. (Spec §6.1 as published also forced a final empty
endRecordin text + variable mode; testing against AMATERSE-produced data shows that this adds a spurious trailing line, so it is not done — see the specification’s erratum note.)- Return type:
- terselib.records.build_codepoints(data, host, text, variable, record_length, asc_to_ebc, rdw_input=False, pad_byte=0)[source]
Convert source bytes into the Layer-1 codepoint stream (spec §7).
- Parameters:
data (bytes) – The source file bytes.
host (bool) – True for host output, False for native.
text (bool) – True to treat data as ASCII text (line-split it and translate ASCII→EBCDIC); False for binary.
variable (bool) – True for variable record format (host only; native is always fixed/raw).
record_length (int | None) – For fixed format, the LRECL; for variable format, the record data length used to segment binary input (and the upper bound recorded for text input). When
None, defaults are chosen: text-variable uses the longest line, text-fixed uses 80, binary-fixed uses 1 and binary-variable segments at 4096.asc_to_ebc (bytes) – 256-byte ASCII→EBCDIC table (text mode only).
rdw_input (bool) – In binary-variable mode, parse data as RDW-framed records instead of segmenting it at record_length.
pad_byte (int) – Byte used to pad the final short fixed binary record.
- Returns:
(codepoints, header_record_length)— the codepoint stream as anarray('H')and the record length to place in the header (LRECL for fixed, max data length for variable).- Raises:
TerseFormatError – If RDW parsing of the input fails.
TerseParameterError – On invalid parameter combinations.
- Return type:
Codecs (Layer 2)
Layer 2 — the PACK codec (LZW with LRU slot recycling; spec §6.2, §8.1).
This is not the classic growing-code-width LZW found in generic compression libraries: codes are always 12 bits, the dictionary starts full (pre-seeded so every phrase slot expands to EBCDIC spaces) and slots are recycled with a least-recently-used policy instead of resetting. Existing LZW packages therefore cannot be reused.
The decoder follows spec §6.2 exactly. The encoder (spec §8.1) drives an identical copy of the decoder’s dictionary machinery with each code it emits, so its model stays bit-identical to what the decoder will rebuild.
- terselib.lzw.decode(get_code, putchar)[source]
Decode a PACK (LZW) code stream (spec §6.2).
- Parameters:
get_code (Callable[[], int]) – Zero-argument callable returning successive 12-bit codes;
0terminates (seeterselib.bitpack.make_unpacker()).putchar (Callable[[int], None]) – Callable receiving each output codepoint
0..257(seeterselib.records.RecordSink).
- Raises:
TerseCorruptError – If a dictionary chain is longer than the dictionary itself (a cycle caused by corrupt input).
- Return type:
None
- terselib.lzw.encode(codepoints)[source]
Encode a codepoint stream with the PACK (LZW) method (spec §8.1).
Greedy longest-match selection over a dictionary model advanced in lockstep with the decoder. Two rules keep the emitted stream safe for the reference decoder:
a match is never extended into the slot that is currently the LRU victim (emitting the victim would make the decoder corrupt its LRU list while recycling that same slot), and
a newly learned phrase only becomes matchable once its extension codepoint is fixed, i.e. from the following emission on.
Layer 2 — the SPACK codec (LZMW; spec §6.3, §8.2).
LZMW grows its dictionary by concatenating the two most recently emitted
phrases, so phrases can double in length each step. Phrases are stored as
binary-tree nodes (Left/Right); reference counting keeps a phrase
alive while it is a child of another phrase, and unreferenced phrases are
recycled least-recently-used once all 3838 slots are in use.
No reusable LZMW implementation exists on PyPI, and the slot / LRU / refcount discipline must match the reference decoder bit-for-bit anyway, so the codec is implemented here from the specification.
The encoder (spec §8.2) drives an identical copy of the decoder’s dictionary machinery and keeps a phrase trie for greedy longest-match lookup.
- terselib.lzmw.decode(get_code, putchar)[source]
Decode a SPACK (LZMW) code stream (spec §6.3).
- Parameters:
- Raises:
TerseCorruptError – If expansion meets a negative child or the expansion stack exceeds
STACKSIZE.- Return type:
None
- terselib.lzmw.encode(codepoints)[source]
Encode a codepoint stream with the SPACK (LZMW) method (spec §8.2).
Greedy longest-match selection using a trie of the phrases currently live in the shared dictionary model. When the dictionary is full, the phrase that is the current LRU victim is never emitted: the decoder evicts that slot before expanding the incoming code, so emitting it would make the decoder recycle the very slot it is about to expand.
Bit packing (Layer 3)
Layer 3 — packing and unpacking of 12-bit codes (spec §3).
The compressed payload of a TERSE file is a stream of 12-bit codes packed
MSB-first into bytes; two codes occupy exactly three bytes. The stream is
terminated by the explicit code 0, after which host files are padded
with EBCDIC spaces to a 1024-byte block boundary.
- class terselib.bitpack.CodePacker(out)[source]
Packs 12-bit codes into bytes, MSB-first (spec §3.1).
- terselib.bitpack.pack_codes(codes)[source]
Pack an iterable of 12-bit codes into a payload byte string.
The caller must include the terminating
0code in codes, and — for host files — apply the 1024-byte block padding to the complete file (header included) afterwards, e.g. viapad_file_to_block().
- terselib.bitpack.pad_file_to_block(file_image, block=1024, pad_byte=0)[source]
Pad a complete host file image to a 1024-byte boundary (spec §3.2).
The padding counts the whole file, header included: AMATERSE writes host files whose total size is a multiple of 1024 bytes.
- terselib.bitpack.make_unpacker(data, start=0)[source]
Build a
GetBlokfunction over data (spec §3.3).The returned zero-argument callable yields successive 12-bit codes and returns
ENDOFFILE(0) once the input bytes are exhausted.- Parameters:
- Returns:
A callable
get_code() -> int.- Raises:
TerseCorruptError – If the stream ends after 8 of the 12 bits of a fresh code (a truncated file).
- Return type:
Translation tables and code pages
EBCDIC ⇄ ASCII translation tables and code-page support.
Host-mode TERSE files always store EBCDIC bytes. Text-mode decompression maps EBCDIC→ASCII and text-mode compression maps ASCII→EBCDIC using a pair of 256-byte translation tables.
Two sources of tables are supported:
"internal"(the default) — the tables printed in Section 5 of the format specification. These reproduce, byte for byte, the translation used by the reference TERSE tooling and are the right choice for interoperating with AMATERSE.Any Python codec name for an EBCDIC code page (
"cp037","cp1047","cp500", …). Installing the third-partyebcdicpackage makes additional code pages such ascp1141available; it is imported automatically when present.
- terselib.translate.INTERNAL_CODEPAGE = 'internal'
Name of the built-in spec translation, accepted by
get_tables().
- terselib.translate.ASCII_SUB = 26
Substitute byte used when a character has no mapping in the target encoding: ASCII SUB.
- terselib.translate.EBCDIC_SUB = 63
Substitute byte used when a character has no EBCDIC mapping: EBCDIC SUB.
- terselib.translate.EBC_TO_ASC = b'\x00\x01\x02\x03\xcf\t\xd3\x7f\xd4\xd5\xc3\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\xc7\xb4\x08\xc9\x18\x19\xcc\xcd\x83\x1d\xd2\x1f\x81\x82\x1c\x84\x86\n\x17\x1b\x89\x91\x92\x95\xa2\x05\x06\x07\xe0\xee\x16\xe5\xd0\x1e\xea\x04\x8a\xf6\xc6\xc2\x14\x15\xc1\x1a \xa6\xe1\x80\xeb\x90\x9f\xe2\xab\x8b\x9b.<(+|&\xa9\xaa\x9c\xdb\xa5\x99\xe3\xa8\x9e!$*);^-/\xdf\xdc\x9a\xdd\xde\x98\x9d\xac\xba,%_>?\xd7\x88\x94\xb0\xb1\xb2\xfc\xd6\xfb`:#@\'="\xf8abcdefghi\x96\xa4\xf3\xaf\xae\xc5\x8cjklmnopqr\x97\x87\xce\x93\xf1\xfe\xc8~stuvwxyz\xef\xc0\xda[\xf2\xf9\xb5\xb6\xfd\xb7\xb8\xb9\xe6\xbb\xbc\xbd\x8d\xd9\xbf]\xd8\xc4{ABCDEFGHI\xcb\xca\xbe\xe8\xec\xed}JKLMNOPQR\xa1\xad\xf5\xf4\xa3\x8f\\\xe7STUVWXYZ\xa0\x85\x8e\xe9\xe4\xd10123456789\xb3\xf7\xf0\xfa\xa7\xff'
EBCDIC → ASCII table from spec §5.1, indexed by the EBCDIC byte.
- terselib.translate.ASC_TO_EBC = b'\x00\x01\x02\x037-./\x16\x05%\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13<=2&\x18\x19?\'"\x1d5\x1f@Z\x7f{[lP}M]\\Nk`Ka\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9z^L~no|\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xad\xe0\xbd_my\x81\x82\x83\x84\x85\x86\x87\x88\x89\x91\x92\x93\x94\x95\x96\x97\x98\x99\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xc0O\xd0\xa1\x07C !\x1c#\xeb$\x9bq(8I\x90\xba\xec\xdfE)*\x9dr+\x8a\x9agVdJShYF\xea\xda,\xde\x8bUA\xfeXQRHi\xdb\x8e\x8dstu\xfa\x15\xb0\xb1\xb3\xb4\xb5j\xb7\xb8\xb9\xcc\xbc\xab>;\n\xbf\x8f:\x14\xa0\x17\xcb\xca\x1a\x1b\x9c\x044\xef\x1e\x06\x08\twp\xbe\xbb\xacTcefb0BGW\xee3\xb6\xe1\xcd\xed6D\xce\xcf1\xaa\xfc\x9e\xae\x8c\xdd\xdc9\xfb\x80\xaf\xfdxv\xb2\x9f\xff'
ASCII → EBCDIC table from spec §5.2, indexed by the ASCII byte.
- terselib.translate.get_tables(codepage='internal')[source]
Return the EBCDIC→ASCII and ASCII→EBCDIC tables for a code page.
- Parameters:
codepage (str) –
"internal"for the specification’s built-in tables (the AMATERSE-compatible default), or any Python/ebcdiccodec name such as"cp037"or"cp1047".- Returns:
Tuple
(ebc_to_asc, asc_to_ebc)of two 256-byte tables, each indexed by the source byte and yielding the target byte.- Raises:
TerseParameterError – If codepage names no known codec.
- Return type:
File-type detection
File-type detection for decompressed output.
When a TERSE file is decompressed in binary mode the original file name
and type are not recorded in the container, so detect_extension()
sniffs the decompressed bytes to pick a sensible file extension. It uses
the magic package (libmagic bindings) when available and falls back
to a small built-in signature table otherwise.
- terselib.filetype.detect_extension(data, default='.bin')[source]
Guess an appropriate file extension for decompressed bytes.
Constants
Format constants for the TERSE / SPACK container.
All values follow the TERSE / SPACK File Format — Implementation
Specification (TERSE_SPACK_FORMAT_SPEC.md), Section 14 (“constant quick
reference”).
- terselib.constants.VERSION_NATIVE_SPACK = 1
Version byte: native SPACK (LZMW), 4-byte header.
- terselib.constants.VERSION_NATIVE_SPACK_ALT = 7
Alternate version byte for native SPACK (LZMW), 4-byte header.
- terselib.constants.VERSION_HOST_PACK = 2
Version byte: host PACK (LZW), 12-byte header.
- terselib.constants.VERSION_HOST_SPACK = 5
Version byte: host SPACK (LZMW), 12-byte header.
- terselib.constants.NATIVE_VALIDATION = b'\x89i\xa5'
The three validation bytes that follow the version byte in a native header.
- terselib.constants.TREESIZE = 4096
Dictionary index space: 12-bit codes 0..4095.
- terselib.constants.CODESIZE = 257
Largest “atom” code; codes 258..4095 are learned phrases.
- terselib.constants.BASE = 0
LRU / free-list sentinel index.
- terselib.constants.NONE_NODE = -1
“No node” marker used in the SPACK tree.
- terselib.constants.ENDOFFILE = 0
Sentinel returned by the 12-bit unpacker at end of input, and the explicit end-of-stream code written by compressors.
- terselib.constants.RECORDMARK = 257
Codepoint marking the end of a logical record in variable-format streams.
- terselib.constants.STACKSIZE = 2047
Maximum expansion-stack depth for SPACK phrase expansion (0x07FF).
- terselib.constants.LZW_SEED = 65
LZW dictionary pre-seed value: EBCDIC space (0x40) + 1.
- terselib.constants.HOST_PAD_BYTE = 0
Host files are padded with this byte to a 1024-byte boundary after the end-of-stream terminator. (The published spec said EBCDIC space 0x20; files produced by AMATERSE are observed to pad with 0x00. A decoder never reads the padding, so the value only matters cosmetically.)
- terselib.constants.HOST_PAD_BLOCK = 1024
Host file block-padding boundary.
- terselib.constants.FLAGUNDEF = 128
Header flag bits (host header, offset 4).
- terselib.constants.METHOD_PACK = 'PACK'
Method names.