TEST.
``````
:::
:::{note}
Custom markup was introduced in EmPy version 3.3 and transformed into
extension markup in EmPy version 4.1.
:::
#### Finalizers
Every interpreter must shutdown (by the `shutdown` method being called
either implicitly or explicitly). As part of this process, the
interpreter runs any **finalizers** that may have been registered with
the interpreter.
Finalizers are callable objects which take zero arguments. The
interpreter contains a list of finalizers which are called
sequentially in order. If a finalizer raises an exception, any
remaining finalizers will not be called that exception will be
propagated up.
To add a finalizer, call either the `appendFinalizer` or
`preprendFinalizer` interpreter methods. To clear all finalizers,
call the `clearFinalizers` method.
:::{note}
`atExit` is an alias for `appendFinalizer` for backward compatibility.
:::
:::{admonition} Example 55: Finalizers
_Source_:
``````
@# These are printed in reverse order.
@empy.appendFinalizer(lambda: empy.write("This is the last line.\n"))@
@empy.appendFinalizer(lambda empy=empy: empy.write("This is the penultimate line.\n"))@
@{
import em
class Finalizer:
def __init__(self, interp):
self.interp = interp
def __call__(self):
self.interp.write("This is the third to last line.\n")
finalizer = Finalizer(empy)
empy.appendFinalizer(finalizer)
}@
This is the first line.
``````
_Output_:
``````
This is the first line.
This is the third to last line.
This is the penultimate line.
This is the last line.
``````
:::
:::{tip}
The finalizers are callables which take no arguments when they are
called, so if they need a reference to the interpreter to do their
work (as the example above), they need to arrange it with a closure,
an implicit argument, or by implementing it as an instance which
contains a reference to the interpreter. All three of these
approaches are illustrated in the example above.
:::
:::{warning}
When using the `expand` method or standalone function, the `StringIO`
object which is used to capture the output of the ephemeral
interpreter is processed before finalizers are handled. Thus
attempting to use finalizers to generate output will not work as
expected. Instead, create a dedicated interpreter:
```python
import em
source = ... # the EmPy source to expand
with em.StringIO() as file:
with em.Interpreter(output=file, ...) as interp:
interp.string(source)
output = file.getvalue()
```
:::
:::{note}
Finalizers were introduced in EmPy version 2.1.
:::
### Diversions
EmPy supports an extended form of **diversions**, which are a
mechanism for deferring and playing back output on demand, similar to
the functionality included in [m4](https://www.gnu.org/software/m4/).
Multiple "streams" of output can be diverted (deferred) and played
back (undiverted) in this manner. A diversion is identified with a
name, which is any immutable object such an integer or string.
Diversions can be played back multiple times ("replayed") if desired.
When recalled, diverted code is *not* resent through the EmPy
interpreter (although a [filter](#filters) could be set up to do
this).
By default, no diversions take place. When no diversion is in effect,
processing output goes directly to the specified output file. This
state can be explicitly requested at any time by calling the
`empy.stopDiverting` function. It is always legal to call this
function, even when there is currently no active diversion.
When diverted, however, output goes to a deferred location which can
then be recalled later. Output is diverted with the
`empy.startDiversion` function, which takes an argument that is the
name of the diversion. If there is no diversion by that name, a new
diversion is created and output will be sent to that diversion; if the
diversion already exists, output will be appended to that preexisting
diversion.
Output send to diversions can be recalled in two ways. The first is
through the `empy.playDiversion` function, which takes the name of the
diversion as an argument. This plays back the named diversion, sends
it to the output, and then erases that diversion. A variant of this
behavior is the `empy.replayDiversion`, which plays back the named
diversion but does not eliminate it afterwards; `empy.replayDiversion`
can be repeatedly called with the same diversion name, and will replay
that diversion repeatedly. `empy.createDiversion` will create a
diversion without actually diverting to it, for cases where you want
to make sure a diversion exists but do not yet want to send anything
to it.
The diversion object itself can be retrieved with
`empy.retrieveDiversion`. Diversions act as writable file-objects,
supporting the usual `write`, `writelines`, `flush`, and `close`
methods. The data that has been diverted to them can be manually
retrieved in one of two ways; either through the `asString` method,
which returns the entire contents of the diversion as a single string,
or through the `asFile` method, which returns the contents of the
diversion as a readable (not writable) file-like object.
Diversions can also be explicitly deleted without playing them back
with the `empy.dropDiversion` function, which takes the desired
diversion name as an argument.
Additionally there are three functions which will apply the above
operations to all existing diversions: `empy.playAllDiversions`,
`empy.replayAllDiversions`, and `empy.dropAllDiversions`. The
diversions are handled in lexicographical order by their name. Also,
all three will do the equivalent of a `empy.stopDiverting` call before
they do their thing.
The name of the current diversion can be requested with the
`empy.getCurrentDiversionName` function; also, the names of all
existing diversions (in sorted order) can be retrieved with
`empy.getAllDiversionNames`. `empy.isExistingDiversionName` will
return whether or not a diversion with the given name exists.
When all processing is finished, the equivalent of a call to
`empy.playAllDiversions` is done. This can be disabled with the
`--no-auto-play-diversions` (_configuration variable:_ `autoPlayDiversions`) option.
:::{admonition} Example 3: Diversions sample
_Source_:
``````
This text is output normally.
@empy.startDiversion('A')@
(This text was diverted!)
@empy.stopDiverting()@
This text is back to being output normally.
Now playing the diversion:
@empy.playDiversion('A')@
And now back to normal output.
``````
_Output_:
``````
This text is output normally.
This text is back to being output normally.
Now playing the diversion:
(This text was diverted!)
And now back to normal output.
``````
:::
:::{note}
Diversions were introduced in EmPy version 1.0.
:::
### Filters
EmPy also supports dynamic **filters**. Filters are put in place
immediately before the final output file, and so are only invoked
after all other processing has taken place (including interpreting and
diverting). Filters take input, remap it, and then send it to the
output. They can be chained together where a series of filters point
to each other in series and then finally to the output file.
The current top-level filter can be retrieved with `empy.getFilter`
(or `empy.getFirstFilter`). The last filter in the chain (the one
just before the underlying file) can be retrieved with
`empy.getLastFilter`. The filter can be set with `empy.setFilter`
(which allows multiple arguments to constitute a chain). To append a
filter at the end of the chain (inserting it just before the
underlying output file), use `empy.appendFilter`. To prepend it to
the top of the chain, use `empy.prependFilter`. A filter chain can be
set directly with `empy.setFilterChain`. And a filter chain can be
reset with `empy.resetFilter`, removing all filters.
Filters are, at their core, simply file-like objects (minimally
supporting `write`, `flush`, and `close` methods that behave in the
usual way) which, after performing whatever processing they need to
do, send their work to the next file-like object or filter in line,
called that filter's "sink." That is to say, filters can be "chained"
together; the action of each filter takes place in sequence, with the
output of one filter being the input of the next. The final sink of
the filter chain will be the output file. Additionally, filters
support a `_flush` method (note the leading underscore) which will
always flush the filter's underlying sink; this method should be not
overridden.
Filters also support two additional methods, not part of the
traditional file interface: `attach`, which takes as an argument a
file-like object (perhaps another filter) and sets that as the
filter's "sink" -- that is, the next filter/file-like object in line.
`detach` (which takes no arguments) is another method which flushes
the filter and removes its sink, leaving it isolated. Finally,
`next`, if present, is an attribute which references the filter's sink
-- or `None`, if the filter does not yet have a sink attached.
To create your own filter, you can create an object which supports the
above described interface, or simply derive from the `Filter` class
(or one of its subclasses) in the `emlib` module and override the
relevant methods.
:::{admonition} Example 4: Filters sample
_Source_:
``````
@{
# For access to the filter classes.
import emlib
}@
This text is normal.
@empy.appendFilter(emlib.FunctionFilter(lambda x: x.upper()))@
This text is in all uppercase!
@empy.appendFilter(emlib.FunctionFilter(lambda x: '[' + x + ']'))@
Now it's also surrounded by brackets!
(Note the brackets are around output as it is sent,
not at the beginning and end of each line.)
@empy.resetFilter()@
Now it's back to normal.
``````
_Output_:
``````
This text is normal.
THIS TEXT IS IN ALL UPPERCASE!
[NOW IT'S ALSO SURROUNDED BY BRACKETS!
(NOTE THE BRACKETS ARE AROUND OUTPUT AS IT IS SENT,
NOT AT THE BEGINNING AND END OF EACH LINE.)
]Now it's back to normal.
``````
:::
:::{note}
Filters were introduced in EmPy version 1.3.
:::
### Hooks
The EmPy system allows for the registration of **hooks** with a
running EmPy interpreter. Hooks are objects, registered with an
interpreter, whose methods represent specific hook events. Any number
of hook objects can be registered with an interpreter, and when a hook
is invoked, the associated method on each one of those hook objects
will be called by the interpreter in sequence. The method name
indicates the type of hook, and it is called with a keyword list of
arguments corresponding the event arguments.
To use a hook, derive a class from `emlib.Hook` and override the
desired methods (with the same signatures as they appear in the base
class). Create an instance of that subclass, and then register it
with a running interpreter with the `empy.addHook` function. A hook
instance can be removed with the `empy.removeHook` function.
More than one hook instance can be registered with an interpreter; in
such a case, the appropriate methods are invoked on each instance in
the order in which they were appended. To adjust this behavior, an
optional `prepend` argument to the `empy.addHook` function can be used
dictate that the new hook should placed at the *beginning* of the
sequence of hooks, rather than at the end (which is the default).
Also there are explicit `empy.appendHook` and `empy.prependHook`
functions.
All hooks can be enabled and disabled entirely for a given
interpreter; this is done with the `empy.enableHooks` and
`empy.disableHooks` functions. By default hooks are enabled, but
obviously if no hooks have been registered no hooks will be called.
Whether hooks are enabled or disabled can be determined by calling
`empy.areHooksEnabled`. To get the list of registered hooks, call
`empy.getHooks`. All the hooks can be removed with `empy.clearHooks`.
Finally, to invoke a hook manually, use `empy.invokeHook`.
For a list of supported hooks, see [Hook methods](#hook-methods) or
the `Hook` class definition in the `emlib` module. (There is also an
`AbstractHook` class in this module which does not have blank stubs
for existing hook methods in case a user wishes to create them
dynamically.)
For example:
:::{admonition} Example 5: Hooks sample
_Source_:
``````
@# Modify the backquote markup to prepend and append backquotes
@# (say, for a document rendering system, cough cough).
@{
import emlib
class BackquoteHook(emlib.Hook):
def __init__(self, interp):
self.interp = interp
def preBackquote(self, literal):
self.interp.write('`' + literal + '`')
return True # return true to skip the standard behavior
empy.addHook(BackquoteHook(empy))
}@
Now backquote markup will render with backquotes: @
@`this is now in backquotes`!
``````
_Output_:
``````
Now backquote markup will render with backquotes: `this is now in backquotes`!
``````
:::
:::{note}
Hooks were originally introduced in EmPy version 2.0, much improved in
version 3.2, and revamped again in version 4.0.
:::
#### Hook methods
##### Hook `at...` methods
These hooks are called when a self-contained event occurs.
`atInstallProxy(proxy, new)`
: A `sys.stdout` proxy was installed. The Boolean value `new`
indicates whether or not the proxy was preexisting.
`atUninstallProxy(proxy, done)`
: A `sys.stdout` proxy was uninstalled. The Boolean value `done`
indicates whether the reference count went to zero (and so the proxy
has been completely removed).
`atStartup()`
: The interpreter has started up.
`atReady()`
: The interpreter has declared itself ready for processing.
`atFinalize()`
: The interpreter is finalizing.
`atShutdown()`
: The interpreter is shutting down.
`atParse(scanner, locals)`
: The interpreter is initiating a parse action with the given scanner
and locals dictionary (which may be `None`).
`atToken(token)`
: The interpreter is expanding a token.
`atHandle(info, fatal, contexts)`
: The interpreter has encountered an error. The `info` parameter is a
3-tuple error (error type, error, traceback) returned from
`sys.exc_info`, `fatal` is a Boolean indicating whether the
interpreter should exit afterwards, and `contexts` is the context
stack.
`atInteract()`
: The interpreter is going interactive.
##### Hook context methods
`pushContext(context)`
: This context is being pushed.
`popContext(context)`
: This context has been popped.
`setContext(context)`
: This context has been set or modified.
`restoreContext(context)`
: This context has been restored.
##### Hook `pre...`/`post...` methods
The `pre...` hooks are invoked before a token expands. The hook can
return a true value to indicate that it has intercepted the expansion
and the token should cancel native expansion. Not explicitly
returning anything, as in standard Python, is equivalent to returning
`None`, which is a false value, which continues expansion:
:::{admonition} Example 56: Hook `pre...` methods
_Source_:
``````
@{
import emlib
import sys
class Hook(emlib.Hook):
def __init__(self, interp):
self.interp = interp
def preString(self, string):
self.interp.write('[' + string + ']')
return True
empy.addHook(Hook(empy))
}@
@# Now test it:
@"Hello, world!"
``````
_Output_:
``````
["Hello, world!"]
``````
:::
:::{tip}
It's typical to want to have an instance of the
interpreter/pseudomodule available to the hook, but it is neither done
automatically nor is it required.
:::
The `post...` hooks are invoked after a non-intercepted token finishes
expanding. Not all `pre...` hooks have a corresponding `post...`
hook. The `post...` hooks take at most one argument (the result of
the token expansion, if applicable) and their return value is ignored.
`preLineComment(comment)`, `postLineComment()`
: The line comment `@#... NL` with the given text.
`preInlineComment(comment)`, `postInlineComment()`
: The inline comment `@*...*` with the given text.
`preWhitespace(whitespace)`
: The whitespace token `@ WS` with the given whitespace.
`prePrefix()`
: The prefix token `@@`.
`preString(string)`, `postString()`
: The string token `@'...'`, etc. with the given string.
`preBackquote(literal)`, `postBackquote(result)`
: The backquote token `` @`...` `` with the given literal.
`preSignificator(key, value, stringized)`, `postSignificator()`
: The significator token `@%... NL`, etc. with the given key, value
and a Boolean indicating whether the significator is stringized.
`preContextName(name)`, `postContentName()`
: The context name token `@?...` with the given name.
`preContextLine(line)`, `postContextLine()`
: The context line token `@!...` with the given line.
`preExpression(pairs, except, locals)`, `postExpression(result)`
: The expression token `@(...)` with the given if-then run pairs, the
except run, and the locals dictionary (which may be `None`).
`preSimple(code, subtokens, locals)`, `postSimple(result)`
: The simple expression token `@word` (etc.) with the given code,
subtokens and locals.
`preInPlace(code, locals)`, `postInPlace(result)`
: The in-place expression token `@$...$...$` with the given code
(first section) and locals (which may be `None`).
`preStatement(code, locals)`, `postStatement()`
: The statement token `@{...}` with the given code and locals (which
may be `None`).
`preControl(type, rest, locals)`, `postControl()`
: The control token `@[...]` of the given type, with the rest run and
locals (which may be None).
`preEscape(code)`, `postEscape()`
: The control token `@\...` with the resulting code.
`preDiacritic(code)`, `postDiacritic()`
: The diacritic token `@^...` with the resulting code.
`preIcon(code)`, `postIcon()`
: The icon token `@|...` with the resulting code.
`preEmoji(name)`, `postEmoji()`
: The emoji token `@:...:` with the given name.
`preExtension(name, contents, depth)`, `postExtension(result)`
: An extension with the given name, contents and depth was invoked.
`preCustom(contents)`, `postCustom()`
: The custom token `@<...>` with the given contents.
##### Hook `before...`/`after...` methods
The `before...` and `after...` hooks are invoked before and after (go
figure) mid-level expansion activities are performed. Any `locals`
argument indicates the locals dictionary, which may be `None`.
If the expansion returns something relevant, it is passed as a
`result` argument to the corresponding `after...` method.
`beforeProcess(command, n)`, `afterProcess()`
: The given command (with index number) is being processed.
`beforeInclude(file, locals, name)`, `afterInclude()`
: The given file is being processed with the given name.
`beforeExpand(string, locals, name)`, `afterExpand(result)`
: `empy.expand` is being called with the given string and name.
`beforeTokens(tokens, locals)`, `afterTokens(result)`
: The given list of tokens is being processed.
`beforeFileLines(file, locals)`, `afterFileLines()`
: The given file is being read by lines.
`beforeFileChunks(file, locals)`, `afterFileChunks()`
: The given file is being read by buffered chunks.
`beforeFileFull(file, locals)`, `afterFileFull()`
: The given file is being read fully.
`beforeString(string, locals)`, `afterString()`
: The given string is being processed.
`beforeQuote(string)`, `afterQuote(result)`
: The given string is being quoted.
`beforeEscape(string)`, `afterEscape(result)`
: The given string is being escaped.
`beforeSignificate(key, value, locals)`, `afterSignificate()`
: The given key/value pair is being processed.
`beforeCallback(contents)`, `afterCallback()`
: The custom callback is being processed with the given contents.
`beforeAtomic(name, value, locals)`, `afterAtomic()`
: The given atomic variable setting with the name and value is being
processed.
`beforeMulti(names, values, locals)`, `afterMulti()`
: The given complex variable setting with the names and values is
being processed.
`beforeImport(name, locals)`, `afterImport()`
: A module with the given name is being imported.
`beforeFunctional(code, lists, locals)`, `afterFunctional(result)`
: A functional markup is with the given code and argument lists (of
EmPy code) is being processed.
`beforeEvaluate(expression, locals, write)`, `afterEvaluate(result)`
: An evaluation markup is being processed with the given code and a
Boolean indicating whether or not the results are being written
directly to the output stream or returned.
`beforeExecute(statements, locals)`, `afterExecute()`
: A statement markup is being processed.
`beforeSingle(source, locals)`, `afterSingle(result)`
: A "single" source (either an expression or a statement) is being
compiled and processed.
`beforeFinalizer(final)`, `afterFinalizer()`
: The given finalizer is being processed. If the `beforeFinalizer`
hook returns true for a particular finalizer, then that finalizer
will not be called.
:::{seealso}
The list of hook methods is available in the `hooks` help topic and is
summarized [here](http://www.alcyone.com/software/empy/HELP.html#hook-methods-summary).
:::
## Customization
The behavior of an EmPy system can be customized in various ways.
### Command line options
EmPy uses a standard GNU-style command line options processor with
both short and long options (_e.g._, `-p` or `--prefix`). Short
options can be combined into one word, and options can have values
either in the next word or in the same word separated by an `=`. An
option consisting of only `--` indicates that no further option
processing should be performed.
EmPy supports the following options:
`-V/--version`
: Print version information exit. Repeat the option for more details
(see below).
`-W/--info`
: Print additional information, including the operating system, Python
implementation and Python version number.
`-Z/--details`
: Print all additional details about the running environment,
including interpreter, system, platform, and operating system
release details.
`-h/--help`
: Print basic help and exit. Repeat the option for more extensive
help. Specifying `-h` once is equivalent to `-H default`; twice to
`-H more`, and three or more times to `-H all` (see below).
`-H/--topics=TOPICS`
: Print extended help by topic(s). Topics are a comma-separated list
of the following choices:
| Topic | Description |
| --- | --- |
| `usage` | Basic command line usage |
| `options` | Command line options |
| `markup` | Markup syntax |
| `escapes` | Escape sequences |
| `environ` | Environment variables |
| `pseudo` | Pseudomodule attributes and functions |
| `variables` | Configuration variable attributes |
| `methods` | Configuration methods |
| `hook` | Hook methods |
| `controls` | Named escapes (control codes) |
| `diacritics` | Diacritic combiners |
| `icons` | Icons |
| `emojis` | User-specified emojis (optional) |
| `hints` | Usage hints |
| `topics` | This list of topics |
| `default` | `usage,options,markup,hints` and `topics` |
| `more` | `usage,options,markup,escapes,environ,hints` and `topics` |
| `all` | `usage,options,markup,escapes,environ,pseudo,config,controls,diacritics,icons,hints` |
As a special case, `-H` with no topic argument is treated as `-H
all` rather than error.
`-v/--verbose`
: The EmPy system will print debugging information to `sys.stderr` as
it is doing its processing.
`-p/--prefix=CHAR` (_environment variable:_ `EMPY_PREFIX`, _configuration variable:_ `prefix`)
: Specify the desired EmPy prefix. It must consistent of a single
Unicode code point (or character), or an empty string for no prefix
(see below). Defaults to `@`.
`--no-prefix`
: Specify that EmPy use no prefix. In this mode, will only process
text and perform no markup expansion. This is equivalent to
specyfing `-p ''`.
`-q/--no-output`
: Use a null file for the output file.
`-m/--pseudomodule=NAME` (_environment variable:_ `EMPY_PSEUDO`, _configuration variable:_ `pseudomoduleName`)
: Specify the name of the EmPy pseudomodule/interpreter. Defaults to
`empy`.
`-f/--flatten` (_environment variable:_ `EMPY_FLATTEN`, _configuration variable:_ `doFlatten`)
: Before processing, move the contents of the
`empy` pseudomodule into the globals, just
as if `empy.flattenGlobals()` were executed immediately after
starting the interpreter. This is the equivalent of executing `from
empy import *` (though since the pseudomodule is not a real module
that statement is invalid). _e.g._, `empy.include` can be referred
to simply as `include` when this flag is specified on the command
line.
`-k/--keep-going`
: Don't exit immediately when an error occurs. Execute the error
handler but continue processing EmPy tokens.
`-e/--ignore-errors`
: Ignore errors completely. No error dispatcher or handler is
executed and token processing continues indefinitely. Implies
`-k/--keep-going`.
`-r/--raw-errors` (_environment variable:_ `EMPY_RAW_ERRORS`, _configuration variable:_ `rawErrors`)
: After logging an EmPy error, show the full Python traceback that
caused it. Useful for debugging.
`--brief-errors`
: When printing an EmPy error, show only its arguments and not its
keyword arguments. This is in contrast to the default (verbose)
where keyword arguments are shown.
`--verbose-errors`
: When printing an EmPy error, show both its arguments and its
(sorted) keyword arguments. This is the default.
`-i/--interactive` (_environment variable:_ `EMPY_INTERACTIVE`, _configuration variable:_ `goInteractive`)
: Enter interactive mode after processing is complete by continuing to
process EmPy markup from the input file, which is by default
`sys.stdin`); this can be changed with the `input` interpreter
attribute. This is helpful for inspecting the state of the
interpreter after processing.
`-d/--delete-on-error` (_environment variable:_ `EMPY_DELETE_ON_ERROR`, _configuration variable:_ `deleteOnError`)
: If an error occurs, delete the output file; requires the use of the
one of the output options such as `-o/--output=FILENAME`. This is
useful when running EmPy under a build systemn such as GNU Make. If
this option is not selected and an error occurs, the output file
will stop when the error is encountered.
`-n/--no-proxy` (_environment variable:_ `EMPY_NO_PROXY`, _configuration variable:_ `useProxy`)
: Do not install a proxy in `sys.stdout`. This will make EmPy thread
safe but writing to `sys.stdout` will not be captured or processed
in any way.
`--config=STATEMENTS`
: Perform the given configuration variable assignments. This option
can be specified multiple times.
`-c/--config-file=FILENAME` (_environment variable:_ `EMPY_CONFIG`)
: Read and process the given configuration file(s), separated by the
platform-specific path delimiter (`;` on Windows, `:` on other
operating systems). This option can be specified multiple times.
`--config-variable=NAME` (_configuration variable:_ `configVariableName`)
: Specify the variable name corresponding to the current configuration
when configuration files are processed. Defaults to
`_`.
`-C/--ignore-missing-config`
: Ignore missing files while reading and processing configurations.
By default, a missing file is an error.
`-o/--output=FILENAME`
: Specify the file to write output to. If this argument is not used,
final output is written to the underlying `sys.stdout`.
`-a/--append=FILENAME`
: Specify the file to append output to. If this argument is not used,
final output is appended to the underlying `sys.stdout`.
`-O/--output-binary=FILENAME`
: Specify the file to write output to and open it as binary.
`-A/--append-binary=FILENAME`
: Specify the file to append output to and open it as binary.
`--output-mode=MODE`
: Specify the output mode to use.
`--input-mode=MODE`
: Specify the input mode to use. Defaults to `'r'`.
`-b/--buffering` (_environment variable:_ `EMPY_BUFFERING`, _configuration variable:_ `buffering`)
: Specify the buffering to use. Use an integer to specify the maximum
number of bytes to read per block or one of the following string
values:
| Name | Value | Description |
| --- | --- | --- |
| `full` | -1 | Use full buffering |
| `none` | 0 | Use no buffering |
| `line` | 1 | Use line buffering |
| `default` | 16384 | Default buffering |
If the choice of buffering is incompatible with other settings, a
`ConfigurationError` is raised. This option has no effect on
interactive mode, as `sys.stdin` is already open. Defaults to
16384.
`--default-buffering`
: Use default buffering.
`-N/--no-buffering`
: Use no buffering.
`-L/--line-buffering`
: Use line buffering.
`-B/--full-buffering`
: Use full buffering.
`-P/--preprocess=FILENAME`
: Process the given EmPy (not Python) file before main document
processing begins.
`-Q/--postprocess=FILENAME`
: Process the given EmPy (not Python) file after main document
processing begins.
`-I/--import=MODULES`
: Import the given Python (not EmPy) module(s) into the interpreter
globals before main document processing begins.
`-D/--define=DEFN`
: Define the given variable into the interpreter globals before main
document processing begins. This is executed as a Python assignment
statement (`variable = ...`); if it does not contain a `=`
character, then the variable is defined in the globals with the
value `None`.
`-S/--string=STR`
: Define the given string variable into the interpreter globals before
main document processing begins. The value is always treated as a
string and never evaluated; if it does not contain a `=` character,
then the variable is defined as the empty string (`''`).
`-E/--execute=STATEMENT`
: Execute the given arbitrary Python (not EmPy) statement before main
document processing begins.
`-F/--file=FILENAME`
: Execute the given Python (not EmPy) file before main document
processing begins.
`-G/--postfile=FILENAME`
: Execute the given Python (not EmPy) file after main document
processing begins.
`-w/--pause-at-end`
: Prompt for a line of input after all processing is complete. Useful
for systems where the window running EmPy would automatically
disappear after EmPy exits (_e.g._, Windows). By default, the input
file used is `sys.stdin`, so this will not work when redirecting
stdin to an EmPy process. This can be changed with the `input`
interpreter attribute.
`-l/--relative-path` (_configuration variable:_ `relativePath`)
: Prepend the location of the EmPy script to Python's `sys.path`.
This is useful when the EmPy scripts themselves import Python .py
modules in that same directory.
`--replace-newlines`
: Replace newlines in (Python) expressions before evaluation.
`--no-replace-newlines`
: Don't replace newlines in (Python) expressions before evaluations.
This is the default.
`--ignore-bangpaths`
: Treat bangpaths as comments. By default, bangpaths (starting lines
that begin with the characters `#!`) are treated as comments and
ignored.
`--no-ignore-bangpaths`
: Do not treat bangpaths as comments. This is the opposite of the
default.
`--none-symbol` (_configuration variable:_ `noneSymbol`)
: The string to write when expanding the value `None`. Defaults to
`None`, which will result in no output.
`--no-none-symbol`
: Do not write any preset string when expanding `None`; equivalent to
setting `noneSymbol` to `None`.
`--expand-user`
: Expand user constructions (`~user`) in configuration file pathnames.
This is the default.
`--no-expand-user`
: Do not expand user constructions (`~user`) in configuration file
pathnames. By default they are expanded.
`--auto-validate-icons`
: Auto-validate icons when an icon markup is first used. This is the
default. See below.
`--no-auto-validate-icons`
: Do not auto-validate icons when an icon markup is first used. See
below.
`--starting-line` (_configuration variable:_ `startingLine`)
: Specify an integer representing the default starting line for
contexts. Default is 1.
`--starting-column` (_configuration variable:_ `startingColumn`)
: Specify an integer representing the default starting column for
contexts. Default is 1.
`--emoji-modules` (_configuration variable:_ `emojiModuleNames`)
: A comma-separated list of emoji modules to try to use for the emoji
markup (`@:...:`). See below. Defaults to
`emoji,emojis,emoji_data_python,unicodedata`.
`--no-emoji-modules`
: Only use `unicodedata` as an emoji module; disable all other emoji
modules.
`--disable-emoji-modules`
: Disable all emoji module usage; just rely on the `emojis` attribute
of the configuration. See below.
`--ignore-emoji-not-found`
: When using emoji markup (`@:...:`), do not raise an error when an
emoji is not found; just pass the `:...:` text through.
`-u/--binary/--unicode` (_environment variable:_ `EMPY_BINARY`, _configuration variable:_ `useBinary`)
: Operate in binary mode; open files in binary mode and use the
`codecs` module for Unicode support. This is necessary in older
versions of Python 2._x_.
`-x/--encoding=E`
: Specify both input and output Unicode encodings. Requires
specifying both an input and an output file.
`--input-encoding=E` (_environment variable:_ `EMPY_INPUT_ENCODING`, _configuration variable:_ `inputEncoding`)
: Specify the input Unicode encoding. Requires specifying an input
file rather than `sys.stdout`.
:::{note}
Specifying a non-default encoding when using interactive mode
(`sys.stdin`) raises a `ConfigurationError`.
:::
`--output-encoding=E` (_environment variable:_ `EMPY_OUTPUT_ENCODING`, _configuration variable:_ `outputEncoding`)
: Specify the output Unicode encoding. Requires specifying an output
file rather than `sys.stdout`.
:::{note}
Specifying a non-default encoding when using `sys.stdout` raises a
`ConfigurationError`.
:::
`-y/--errors=E`
: Specify both [input and output Unicode error
handlers](https://docs.python.org/3/library/functions.html#open).
`--input-errors=E` (_environment variable:_ `EMPY_INPUT_ERRORS`, _configuration variable:_ `inputErrors`)
: Specify the [input Unicode error
handler](https://docs.python.org/3/library/functions.html#open).
:::{note}
Specifying a non-default error handler when using interactive mode
(`sys.stdin`) raises a `ConfigurationError`.
:::
`--output-errors=E` (_environment variable:_ `EMPY_OUTPUT_ERRORS`, _configuration variable:_ `outputErrors`)
: Specify the [output Unicode error
handler](https://docs.python.org/3/library/functions.html#open).
:::{note}
Specifying a non-default error handler when using `sys.stdout`
raises a `ConfigurationError`.
:::
`-z/--normalization-form=F` (_configuration variable:_ `normalizationForm`)
: Specify the Unicode normalization to perform when using the
diacritics markup (`@^...`). Specify an empty string (`''`) to
skip normalization. Defaults to `NFKC`
for modern versions of Python and `''` for very old versions of
Python 2._x_.
`--no-auto-play-diversions` (_configuration variable:_ `autoPlayDiversions`)
: Before exiting, do not automatically play back any remaining
diversions. By default such diversions are played back.
`--no-check-variables` (_configuration variable:_ `checkVariables`)
: When modifying configuration variables, normally the existence and
types of these variables is checked and if it doesn't exist or it is
attempting to be assigned to an incompatible type, it will raise a
`ConfigurationError`. To override this behavior, use this flag.
`--context-format` (_configuration variable:_ `contextFormat`)
: Specify the format for printing contexts. See below.
`--success-code=N` (_configuration variable:_ `successCode`)
: Specify the exit code for the Python interpreter on success.
Defaults to 0.
`--failure-code=N` (_configuration variable:_ `failureCode`)
: Specify the exit code for the Python interpreter when a processing
error occurs. Defaults to 1.
`--unknown-code=N` (_configuration variable:_ `unknownCode`)
: Specify the exit code for the Python interpreter when an invalid
configuration (such as unknown command line options) is encountered.
Defaults to 2.
:::{seealso}
The list of command line options is available in the `options` help
topic and is summarized
[here](http://www.alcyone.com/software/empy/HELP.html#command-line-options-summary).
:::
### Environment variables
The following environment variables are supported:
`EMPY_OPTIONS`
: Specify additional command line options to be used. These are in
effect added to the start of the command line and parsed before
any explicit command line options and processing begins.
For example, this will run the EmPy interpreter as if the `-r` and
`-d` command line options were specified:
% export EMPY_OPTIONS='-r -d'; em.py ...
`EMPY_CONFIG` (_command line option:_ `-c/--config-file=FILENAME`)
: Specify the configuration file(s) to process before main document
processing begins.
`EMPY_PREFIX` (_command line option:_ `-p/--prefix=CHAR`, _configuration variable:_ `prefix`)
: Specify the prefix to use when processing.
`EMPY_PSEUDO` (_command line option:_ `-m/--pseudomodule=NAME`, _configuration variable:_ `pseudomoduleName`)
: Specify the name of the pseudomodule/interpreter to use when
processing.
`EMPY_FLATTEN` (_command line option:_ `-f/--flatten`, _configuration variable:_ `doFlatten`)
: If defined, flatten the globals before processing.
`EMPY_RAW_ERRORS` (_command line option:_ `-r/--raw-errors`, _configuration variable:_ `rawErrors`)
: If defined, after an error occurs, show the full Python tracebacks
of the exception.
`EMPY_INTERACTIVE` (_command line option:_ `-i/--interactive`, _configuration variable:_ `goInteractive`)
: If defined, enter interactive mode by processing markup from
`sys.stdin` after main document processing is complete.
`EMPY_DELETE_ON_ERROR` (_command line option:_ `-d/--delete-on-error`, _configuration variable:_ `deleteOnError`)
: If defined, when an error occurs, delete the corresponding output
file.
`EMPY_NO_PROXY` (_command line option:_ `-n/--no-proxy`, _configuration variable:_ `useProxy`)
: If defined, do not install a `sys.stdout` proxy.
`EMPY_BUFFERING` (_command line option:_ `-b/--buffering`, _configuration variable:_ `buffering`)
: Specify the desired file buffering.
`EMPY_BINARY` (_command line option:_ `-u/--binary/--unicode`, _configuration variable:_ `useBinary`)
: If defined, use binary mode.
`EMPY_ENCODING`
: Specify the desired input and output Unicode encodings.
`EMPY_INPUT_ENCODING` (_command line option:_ `--input-encoding=E`, _configuration variable:_ `inputEncoding`)
: Specify the desired input Unicode encoding only.
`EMPY_OUTPUT_ENCODING` (_command line option:_ `--output-encoding=E`, _configuration variable:_ `outputEncoding`)
: Specify the desired output Unicode encoding only.
`EMPY_ERRORS`
: Specify the desired input and output Unicode error handler.
`EMPY_INPUT_ERRORS` (_command line option:_ `--input-errors=E`, _configuration variable:_ `inputErrors`)
: Specify the desired input Unicode error handler.
`EMPY_OUTPUT_ERRORS` (_command line option:_ `--output-errors=E`, _configuration variable:_ `outputErrors`)
: Specify the desired output Unicode error handler.
:::{seealso}
The list of environment variables is available in the `environ` help
topic and is summarized
[here](http://www.alcyone.com/software/empy/HELP.html#environment-variables-summary).
:::
:::{note}
Environment variables were first introduced in EmPy version 2.2, and
revamped in version 4.0.
:::
### Configuration
**Configurations** are objects which determine the behavior of an EmPy
interpreter. They can be created with an instance of the
`Configuration` class and have a set of attributes (**configuration
variables**) which can be modified. Most configuration variables
correspond to a command line option. The configuration instance also
contains supporting methods which are used by the interpreter which
can be overridden.
When configuration variables are modified, they are by default checked
to make sure have a known name and that they have the correct type; if
not, a `ConfigurationError` will be raised. This behavior can be
disabled with `--no-check-variables` (_configuration variable:_ `checkVariables`).
When a configuration is assigned to an interpreter, it exists as a
`config` attribute of the `empy` pseudomodule and can be modified by a
running EmPy system. Configurations can be shared between multiple
interpreters if desired.
:::{admonition} Example 57: Configuration instances
_Source_:
``````
@{
empy.config.prefix = '$'
}$
${
print("The EmPy prefix is now $, not @!")
}$
``````
_Output_:
``````
The EmPy prefix is now $, not @!
``````
:::
:::{tip}
This example shows a quirk of changing configurations in the middle of
processing an EmPy document; the prefix changes from a `@` to a `$`
by the end of the first statement markup, so a `$` and a newline is
required to suppress the trailing newline; a `@` would have been sent
to the output unchanged since it is no longer the prefix. Use
[command line options](#command-line-options), [environment variables](#environment-variables) or [configuration
files](#configuration-files) to avoid
issues like this, as they are processed before any EmPy document.
For changing the prefix, use `-p/--prefix=CHAR` (_environment variable:_ `EMPY_PREFIX`, _configuration variable:_ `prefix`).
:::
#### Configuration files
**Configuration files** are snippets of Python (not EmPy) code which
can be executed under an EmPy system to modify the current
configuration. By convention they have the extension .conf. though
this is not a requirement. Configuration files are processed before
any expansion begins and are specified with the `-c/--config-file=FILENAME` (_environment variable:_ `EMPY_CONFIG`) command line option; a list of configuration files can be
specified with a `:` delimiter (`;` on Windows); the delimiter can be
specified with `--path-separator` (_configuration variable:_ `pathSeparator`). A nonexistent
configuration file specified in this way is an error unless
`-C/--ignore-missing-config` is specified.
When a configuration file is processed, its contents are executed in a
Python (not EmPy) interpreter and then any resulting variable
assignments are assigned to the configuration instance. So:
``````python
prefix = '$'
``````
is a simple configuration file which will change the EmPy prefix to
`$`.
Any resulting variable beginning with an underscore will be ignored.
Thus these variables can be used as auxiliary variables in the
configuration file. For example, this configuration file will define
custom emojis for the numbered keycaps:
``````python
emojis = {}
for _x in range(10):
emojis[str(_x)] = '{}\ufe0f\u20e3'.format(_x)
``````
Finally, when a configuration file is processed, the current
configuration instance is presented as a variable named `_` (this can
be changed with `--config-variable=NAME` (_configuration variable:_ `configVariableName`)). The
following example does the same as the previous example but uses the
dedicated variable:
``````python
_.emojis.update(((str(_x), '{}\ufe0f\u20e3'.format(_x)) for _x in range(10)))
``````
:::{tip}
To make a set of configuration files automatic loaded by EmPy, use the
`EMPY_CONFIG` environment variable in your startup
shell:
% export EMPY_CONFIG=~/path/to/default.conf
To make a more general set of _options_ available, set `EMPY_OPTIONS`.
:::
#### Configuration variables
The following configuration variables exist with the given types and
their corresponding command line options and environment variables.
Default values are shown after a `=` sign. When a corresponding
command line option exists, See the [command line
options](#command-line-options) for more detailed information.
`name: str = 'default'`
: The name of this configuration. It is for organizational purposes
and is not used directly by the EmPy system.
`notes = None`
: Arbitrary data about this configuration. It can be anything from an
integer to a string to a dictionary to a class instance, or its
default, `None`. It is for organizational purposes and is not used
directly by the EmPy system.
`prefix: str = '@'` (_command line option:_ `-p/--prefix=CHAR`, _environment variable:_ `EMPY_PREFIX`)
: The prefix the interpreter is using to delimit EmPy markup. Must be
a single Unicode code point (character).
`pseudomoduleName: str = 'empy'` (_command line option:_ `-m/--pseudomodule=NAME`, _environment variable:_ `EMPY_PSEUDO`)
: The name of the pseudomodule for this interpreter.
`verbose: bool = False`
: If true, print debugging information before processing each EmPy
token.
`rawErrors: bool = False` (_command line option:_ `-r/--raw-errors`, _environment variable:_ `EMPY_RAW_ERRORS`)
: If true, print a Python traceback for every exception that is thrown.
`exitOnError: bool = True`
: If true, exit the EmPy interpreter after an error occurs. If false,
processing will continue despite the error.
`ignoreErrors: bool = False`
: If true, all errors are ignored by the EmPy interpreter. Setting
this to true also implies `exitOnError` is false.
`contextFormat: str = '%(name)s:%(line)d:%(column)d'` (_command line option:_ `--context-format`)
: The string format to use to render contexts. EmPy will
automatically determine whether or not it should use the `%`
operator or the `str.format` method with this format. See [Context
formatting](#context-formatting) for more details.
`goInteractive: bool = False` (_command line option:_ `-i/--interactive`, _environment variable:_ `EMPY_INTERACTIVE`)
: When done processing the main EmPy document (if any), go into
interactive mode by running a REPL loop with `sys.stdin`. If such
document is specified (_i.e._, EmPy is invoked with no arguments),
go into interactive mode as well.
`deleteOnError: bool = False` (_command line option:_ `-d/--delete-on-error`, _environment variable:_ `EMPY_DELETE_ON_ERROR`)
: If an output file is chosen (_e.g._, with `-o/--output=FILENAME` or one
of the other such options) and an error occurs, delete the output
file. If this is set to true with output set to `sys.stdout`, a
ConfigurationError will be raised.
`doFlatten: bool = False` (_command line option:_ `-f/--flatten`, _environment variable:_ `EMPY_FLATTEN`)
: Flatten the contents of the `empy`
pseudomodule into the globals rather than having them all under the
pseudomodule name.
`useProxy: bool = True` (_command line option:_ `-n/--no-proxy`, _environment variable:_ `EMPY_NO_PROXY`)
: If true, install a proxy object for `sys.stdout`. This should be
true if any output is being done via `print` or `sys.stdout.write`.
`relativePath: bool = False` (_command line option:_ `-l/--relative-path`)
: If true, the directory of the EmPy script's path will be prepended
to Python's `sys.path`.
`buffering: int = 16384` (_command line option:_ `-b/--buffering`, _environment variable:_ `EMPY_BUFFERING`)
: Specify the buffering for the input and output files.
`replaceNewlines: bool = False`
: If true, newlines in emoji names, Unicode character name escape
markup, and code evaluation will be changed to spaces. This can
help when writing EmPy with a word-wrapping editor.
`ignoreBangpaths: bool = True`
: If true, a bangpath (the first line of a file which starts with
`#!`) will be treated as an EmPy comment, allowing the creation of
EmPy executable scripts. If false, it will not be treated specially
and will be rendered to the output.
`noneSymbol: Optional[str] = None` (_command line option:_ `--none-symbol`)
: When an EmPy expansion evaluates to None (_e.g._, `@(None)`), this
is the string that will be rendered to the output stream. If set to
`None` (the default), no output will be rendered.
`missingConfigIsError: bool = True`
: If a configuration file is specified with `-c/--config-file=FILENAME` but
does not exist, if this variable is true an error will be raised.
`pauseAtEnd: bool = False`
: When done processing EmPy files, read a line from `sys.stdin` before
exiting the interpreter. This can be useful when testing under
consoles on Windows.
`startingLine: int = 1` (_command line option:_ `--starting-line`)
: The line to start with in contexts.
`startingColumn: int = 1` (_command line option:_ `--starting-column`)
: The column to start with in contexts.
`significatorDelimiters: tuple = ('__', '__')`
: A 2-tuple of strings representing the prefix and suffix to add to
significator names in order to determine what name to give them in
the globals.
`emptySignificator: object = None`
: The default value to use for non-stringized significators.
`autoValidateIcons: bool = True`
: When icons are used with a custom dictionary, a preprocessing phase
needs to be done to make sure that all icon starting substrings are
marked in the `icons` dictionary. If this variable is false, this
extra processing step will not be done; this is provided if the user
wants to specify their own properly-validated icons dictionary and
wishes to avoid a redundant step.
`emojiModuleNames: list[str] = ['emoji', 'emojis', 'emoji_data_python', 'unicodedata']` (_command line option:_ `--emoji-modules`)
: The list of names of supported emoji modules that the EmPy system
will attempt t use at startup.
`emojiNotFoundIsError: bool = True`
: If true, a non-existing emoji is an error.
`useBinary: bool = False` (_command line option:_ `-u/--binary/--unicode`, _environment variable:_ `EMPY_BINARY`)
: If true, open files in binary mode.
`inputEncoding: str = 'utf-8'` (_command line option:_ `--input-encoding=E`, _environment variable:_ `EMPY_INPUT_ENCODING`)
: The file input encoding to use. This needs to be set before files
are opened to take effect.
`outputEncoding: str = 'utf-8'` (_command line option:_ `--output-encoding=E`, _environment variable:_ `EMPY_OUTPUT_ENCODING`)
: The file output encoding to use. This needs to be set before files
are opened to take effect.
`inputErrors: str = 'strict'` (_command line option:_ `--input-errors=E`, _environment variable:_ `EMPY_INPUT_ERRORS`)
: the file input error handler to use. This needs to be set before files
are opened to take effect.
`outputErrors: str = 'strict'` (_command line option:_ `--output-errors=E`, _environment variable:_ `EMPY_OUTPUT_ERRORS`)
: The file output error handler to use. This needs to be set before files
are opened to take effect.
`normalizationForm: str = 'NFKC'` (_command line option:_ `-z/--normalization-form=F`)
: The normalization form to use when applying diacritic combiners.
Set to `None` or `''` in order to skip normalization.
`autoPlayDiversions: bool = True` (_command line option:_ `--no-auto-play-diversions`)
: If diversions are extant when an interpreter is ready to exist, if
this variable is true then those diversions will be undiverted to
the output stream in lexicographical order by name.
`expandUserConstructions: bool = True`
: If true, when processing configuration files, call
`os.path.expanduser` on each filename to expand `~` and `~user`
constructions.
`configVariableName: str = '_'` (_command line option:_ `--config-variable=NAME`)
: When processing configuration files, the existing configuration
object can be referenced as a variable. This indicates its name.
`successCode: int = 0` (_command line option:_ `--success-code=N`)
: The exit code to return when a processing is successful.
`failureCode: int = 1` (_command line option:_ `--failure-code=N`)
: The exit code to return when an error occurs during processing.
`unknownCode: int = 2` (_command line option:_ `--unknown-code=N`)
: The exit code to return when a configuration error is found (and
processing never starts).
`checkVariables: bool = True` (_command line option:_ `--no-check-variables`)
: If true, configuration variables will be checked to make sure they
are known variables and have the proper type on assignment.
`pathSeparator: str = ';'` (Windows) or `':'` (others) ` ` (_command line option:_ `--path-separator`)
: The path separator to use when specifying multiple filenames with
`-c/--config-file=FILENAME`. Defaults to `;` on Windows and `:` on other
platforms.
`duplicativeFirsts: list[str] = ['(', '[', '{']`
: The list of first markup characters that may be duplicated to
indicate variants. For instance, `@(...)` is expression markup,
but `@((...))` is parenthesis extension markup.
`controls: dict = {...}`
: The controls dictionary used by the [named escape
markup](#named-escape-markup).
`diacritics: dict = {...}`
: The diacritic combiners dictionary used by the [diacritic
markup](#diacritic-markup).
`icons: dict = {...}`
: The icons dictionary used by the [icon markup](#icon-markup).
`emojis: dict = {...}`
: The custom emojis dictionary which is referenced first by the [emoji
markup](#emoji-markup). Defaults to an empty dictionary.
:::{seealso}
The list of configuration variables is available in the `variables` help
topic and is summarized
[here](http://www.alcyone.com/software/empy/HELP.html#configuration-variables-summary).
:::
:::{note}
Configuration objects were introduced in EmPy version 4.0; previously
an underused options dictionary was introduced in version 2.2.2.
:::
#### Configuration methods
The following methods are supported by configuration instances:
`__init__(**kwargs)`
: The constructor. Takes a set of keyword arguments that are then set
as attributes in the configuration instance. So
```python
config = em.Configuration(prefix='$')
```
is a shorter form of
```python
config = em.Configuration()
config.prefix = '$'
```
`isInitialized() -> bool`
: Has this instance been initialized? Before initialization, no
typechecking is done even if `checkVariables` is set.
`check(inputFilename, outputFilename)`
: Check the file settings against these filenames and raise a
`ConfigurationError` is there appears to be an inconsistency.
`has(name) -> bool`
: Is this name an existing configuration variable?
`get(name, default=None) -> bool`
: Get the value of this configuration variable or return this default
if it does not exist.
`set(name, value)`
: Set the configuration variable to the given value.
`update(**kwargs)`
: Set a series of configuration variables via a set of keyword
arguments.
`clone(deep=False) -> Configuration`
: Clone this configuration and return it. If `deep` is true, make it
a deep copy.
`run(statements)`
: Execute a series of configuration commands.
`load(filename, required=None)`
: Load and execute a configuration file. If `required` is true, raise
an exception; if false, ignore; if `None`, use the default for this
configuration.
`path(path, required=None)`
: Load and execute one or more configuration files separated by the
path separator. `required` argument is the same as for `load`
above.
`hasEnvironment(name) -> bool`
: Is the given environment variable defined, regardless of its value?
`environment(name, default=None, blank=None)`
: Get the value of the environment variable. If it is not defined,
return `default`; if it is defined but is empty, return `blank`.
`hasDefaultPrefix() -> bool`
: Is the `prefix` configuration variable set to the
default?
`has{Full|No|Line|Fixed}Buffering() -> bool`
: Is buffering set to full, none, line, or some fixed value,
respectively?
`createFactory([tokens]) -> Factory`
: Create a token factory from the list of token classes and return it.
If `tokens` is not specified, use the default list.
`adjustFactory()`
: Adjust an existing factory to take into account a non-default prefix.
`getFactory([tokens], [force])`
: Get a factory, creating one if one has not yet been created, with
the given `tokens` list (if not specified, a default list will be
used). If `force` is true, then create a new one even if one
already exists.
`resetFactory()`
: Clear the current factory, if any.
`createExtensionToken(first: str, name: str, [last: str])`
: Create a new extension token class with the first character `first`
and with method name `name`. If `last` is specified, use that as
the last character; otherwise, guess for a closed form with a
default of `first`.
`hasBinary() -> bool`
: Is binary (formerly called Unicode) support enabled?
`enableBinary([major, minor])`
: Enable binary support, conditionally if `major` and `minor` (the
major and minor versions of Python) are specified and binary support
is needed for this version of Python.
`disableBinary()`
: Turn off binary/Unicode support.
`isDefaultEncodingErrors([encoding, errors, asInput]) -> bool`
: Are both the file encoding and file error handler the default?
Check for input if `asInput` is true, otherwise check for output.
`getDefaultEncoding([default]) -> str`
: Get the current default encoding, overriding with `default` if
desired.
`open(filename, mode=None, buffering=-1, encoding=None, errors=None, expand=None) -> file`
: The main wrapper around the `open`/`codecs.open` call, allowing for
seamless file opening in both binary and non-binary mode across all
supported Python versions.
`significatorReString() -> str`
: Return a regular expression string that will match significators in
EmPy code with this configuration's prefix.
:::{hint}
It can be used in Python like this:
```python
data = open('script.em', 'r').read()
for result in empy.config.significatorRe().findall(data):
string2, key2, value2, string1, key1, value1 = result
if key1:
print("Single line significator: {} = {}{}".format(
key1, value1, ' (stringized)' if string1 else ''))
else: # key2
print("Multi-line significator: {} = {}{}".format(
key2, value2, ' (stringized)' if string2 else ''))
```
:::
`significatorRe([flags]) -> re.Pattern`
: Return a compiled regular expression pattern object for this
configuration's prefix. Override the `re` `flags` if desired.
`significatorFor(key) -> str`
: Return the significator variable name for this significator key.
`setContextFormat(rawFormat)`
: Set the context format for this configuration. See [context
formatting](#context-formatting).
`renderContext(context) -> str`
: Render the given context using the existing context format string.
`calculateIconsSignature() -> tuple`
: Calculate the icons signature to try to detect any accidental
changes.
`signIcons()`
: Calculate the icons signature and update the configuration with it.
`transmogrifyIcons([icons])`
: Process the icons dictionary and make sure any keys' prefixes are
backfilled with `None` values. This is necessary for the
functioning of the [icon markup](#icon-markup). This method will be
called automatically unless `autoValidateIcons` is
false.
`validateIcons([icons])`
: Check whether the icons have possibly changed and transmogrify them
if necessary.
`initializeEmojiModules([moduleNames])`
: Scan for existing emoji modules and set up the appropriate internal
data structures. Use the list of module names in the configuration
if `moduleNames` is not specified.
`substituteEmoji(text) -> str`
: Perform emoji substitution with the detected emoji modules.
`isSuccessCode(code) -> bool`
: Is this exit code a success code?
`isExitError(error) -> bool`
: Is this exception instance an exit error rather than a real error?
`errorToExitCode(error) -> int`
: Return an appropriate exit code for this error.
`isNotAnError(error) -> bool`
: Does this exception instance not represent an actual error?
`formatError(error[, prefix, suffix]) -> str`
: Return a string representing the details of the given exception
instance, with an optional prefix and suffix.
:::{seealso}
The list of configuration methods is available in the `methods` help
topic and is summarized
[here](http://www.alcyone.com/software/empy/HELP.html#configuration-methods-summary).
:::
### Error handling
#### Error dispatchers
When an error occurs in an EmPy system, first an **error dispatcher**
is invoked. The purpose of the dispatcher is to determine at a
high-level what should be done about the error. A dispatcher is a
zero-argument callable which primarily determines whether the error
should be handled by the running interpreter, whether it should be
raise to the parent caller rather than handled by the interpreter, or
some other custom behavior.
When specified in the [`Interpreter` constructor](#constructor) or one
of the high-level interpreter methods (_e.g._, `file` or `string`), it
can take on a few special values:
| Value | Meaning | Corresponding method |
| --- | --- | --- |
| `None` | Use interpreter default | -- |
| `True` | Interpreter should handle error | `dispatch` |
| `False` | Interpreter should reraise error | `reraise` |
:::{note}
For standalone interpreters and its high-level methods, the default
dispatcher is `True` (`dispatch`); that is, the interpeter will handle
the error itself. When calling the `expand` interpreter method or the
global `expand` function, the dispatcher is `False` (`reraise`); in
other words, calls to `expand` will result in any occurring errors
being raised to the caller rather than handled by the interpteter.
:::
:::{note}
Error dispatchers were introduced in EmPy version 4.0.1.
:::
#### Error handlers
Once an error is dispatched to the interpteter, it is handled by an
**error handler**. An error handler is a callable object that will
respond to the error and take any necessary action. If no
user-specified error handler is set, the default error handler is
used, which prints a formatted EmPy error message to `sys.stderr`.
An error handler is a callable object with the following signature:
`handler(type, error, traceback) -> bool`
It takes the error type, the error instance, and the traceback object
corresponding to an exception (a tuple of which is the return value of
`sys.exc_info()`) and returns an optional Boolean. If the return
value is true, the default handler will _also_ be invoked after the
error handler is called. (Not explicitly returning anything will
implicitly return `None`, which is a false value.)
The current error that the interpreter has encountered is set in the
interpreter's `error` attribute (with `None` indicating no error).
The error handler can manually set this attribute to `None` to clear
the error if desired.
After the error handler(s) have been called, the interpreter will then
decide how to resolve the error. If the `error` attribute of the
interpreter is still non-`None` and the configuration variable
`exitOnError` is true (option: `-k/--keep-going`), the
interpreter will exit. If the `error` attribute is `None`, it will
continue running.
If the `ignoreErrors` configuration variable (option:
`-e/--ignore-errors`) is true, then no error dispatchers or error
handlers will be called.
:::{note}
Error handlers were introduced in EmPy version 4.0.
:::
#### Error classses
The following error classes are used by EmPy:
| Class | Base class | Meaning |
| --- | --- | --- |
| `Error` | `Exception` | Base error class |
| `ConsistencyError` | `Error` | An error involving inconsistent settings has occurred |
| `DiversionError` | `Error` | An error involving diversions has occurred |
| `FilterError` | `Error` | An error involving filters has occurred |
| `CoreError` | `Error` | An error involving cores has occurred |
| `ExtensionError` | `Error` | An error involving extensions has occurred |
| `StackUnderflowError` | `Error` | A stack has underflowed (internal error) |
| `UnknownEmojiError` | `Error` | An unknown emoji was requested |
| `StringError` | `Error` | An old-style string error (used internally) |
| `InvocationError` | `Error` | An error invoking the interpreter has occurred |
| `ConfigurationError` | `Error` | An error involving a bad configuration has occurred |
| `CompatibilityError` | `ConfigurationError` | An error involving backward compatibility has occurred |
| `ConfigurationFileNotFoundError` | `ConfigurationError` | A requested configuration file was missing |
| `ParseError` | `Error` | Invalid EmPy syntax was encountered |
| `TransientParseError` | `ParseError` | Invalid EmPy syntax was encountered (but may be resolved by reading further data) |
## Reference
The following reference material is available:
### Getting version and debugging information
To print the version of EmPy you have installed, run:
% em.py -V # or: --version
Welcome to EmPy version 4.1.
To print additional information including the Python implementation
and version, operating system, and machine type, run:
% em.py -W # or: --info
Welcome to EmPy version 4.1, in CPython/3.10.12, on Linux (POSIX), with x86_64.
For diagnostic details (say, to report a potential problem to the
developer), run:
% em.py -Z # or: --details
Welcome to EmPy version 4.1, in CPython/3.10.12, on Linux (POSIX), with x86_64.
Details:
- basic/implementation: CPython
- basic/machine: x86_64
- basic/os: POSIX
- basic/system: Linux
- basic/version: 3.10.12
...
### Examples and testing
For quick examples of EmPy code, check out the examples throughout
this document. For a more expansive tour of examples illustrating
EmPy features, check out tests/sample/sample.em. For a real-world
example, check out README.md.em, which is the EmPy source file from
which this documentation is generated.
EmPy has an extensive testing system. (If you have EmPy installed via
an operating system package that does not include the test system and
you wish to use it, [download the tarball](#getting-the-software).)
EmPy's testing system consists of the shell script test.sh and two
directories: tests and suites. The tests directory contains the
unit/system tests, and the suites directory contains files with lists
of tests to run. The test.sh shell script will run with any modern
Bourne-like shell.
Tests can be run changing to the directory where test.sh and both the
tests and suites directories are located, and then executing
`./test.sh` followed by the tests desired to be run following on the
command line. For example, this runs a quick test:
% ./test.sh tests/sample/sample.em
tests/sample/sample.em (python3) [PASS]
PASSES: 1/1
All tests passed (python3).
Specifying a directory will run all the tests contained in that
directory and all its subdirectories:
% ./test.sh tests/common/trivial
tests/common/trivial/empty.em (python3) [PASS]
tests/common/trivial/long.em (python3) [PASS]
tests/common/trivial/medium.em (python3) [PASS]
tests/common/trivial/short.em (python3) [PASS]
tests/common/trivial/short_no_newline.em (python3) [PASS]
PASSES: 5/5
All tests passed (python3).
:::{warning}
The tests directory contains a superset of all tests for Python
versions, so running all the tests with `./test.sh tests` will
generate test failures.
:::
Suites can be run by using the `@` character before the filename. A
suite is a list of tests, one per line, to run. Blank lines and lines
starting with `#` are ignored:
% cat suites/default
# Run tests for Python versions from 3.0 up.
tests/common
tests/modern
tests/python3
tests/sample
% ./test.sh @suites/default
tests/common/callbacks/deregister.em (python3) [PASS]
tests/common/callbacks/get_none.em (python3) [PASS]
tests/common/callbacks/get_one.em (python3) [PASS]
...
PASSES: 433/433
All tests passed (python3).
To test a version of Python other than the default (that is, other
than a Python 3._x_ interpreter named `python3`), specify it with the
`-p` option to the test script and use that version's test suite. To
test CPython 2.7, for instance:
% ./test.sh -p python2.7 @suites/python2.7
tests/common/callbacks/deregister.em (python2.7) [PASS]
tests/common/callbacks/get_none.em (python2.7) [PASS]
tests/common/callbacks/get_one.em (python2.7) [PASS]
...
Suites for all supported interpreters and versions are provided. For
example, if you have PyPy 3.10 installed:
% ./test.sh -p pypy3.10 @suites/pypy3.10
tests/common/callbacks/deregister.em (pypy3.10) [PASS]
tests/common/callbacks/get_none.em (pypy3.10) [PASS]
tests/common/callbacks/get_one.em (pypy3.10) [PASS]
...
To only report errors ("quiet mode"), use the `-q` option:
% ./test.sh -q @suites/default
PASSES: 433/433
All tests passed (python3).
For more information about the testing tool, run:
% ./test.sh -h # or: --help
Usage: ./test.sh [<option>...] [--] (<file> | <directory> | @<suite>)...
Run one or more EmPy tests, comparing the results to exemplars, and
return an exit code indicating whether all tests succeeded or whether
there were some failures. If no tests are specified, this help is
displayed. Test filenames, directory names, and suite names cannot
contain spaces.
...
:::{note}
A simple benchmark test system was introduced in EmPy version 2.1, and
was expanded to a full unit and system test suites for all supported
versions of Python in EmPy version 4.0.
:::
### Embedding EmPy
EmPy can be easily embedded into your Python programs. Simply ensure
that the em.py file is available in the `PYTHONPATH` and import `em`
as a module:
```python
import em
print(em)
```
To embed an interpreter, create an instance of the `Interpreter`
class. The interpreter constructor requires keyword arguments, all
with reasonable defaults; [see here for the list](#constructor). One
important argument to an interpreter is a
[configuration](#configuration), which, if needed, should be
constructed first and then passed into the interpreter. If no
configuration is specified, a default instance will be created and
used:
```python
import em
config = em.Configuration(...)
interp = em.Interpreter(config=config, ...)
```
Then call interpreter methods on it such as `write`, `string`,
`evaluate`, `execute`, `expand`, and so on. The full list of
interpreter methods is [here](#interpreter-methods). Exceptions that
occur during processing will be handled by the interpreter's error
handler.
:::{important}
When you create an interpreter, you must call its `shutdown` method
when you are done. This is required to remove the proxy on
`sys.stdout` that EmPy requires for proper operation and restore your
Python environment to the state it was before creating the
interpreter. This can be accomplished by creating the interpreter in
a `with` statement -- interpreters are also context managers -- or by
creating it and shutting it down in a `try`/`finally` statement.
This is not needed when calling the `expand` global function; it
creates and shuts down an ephemeral interpreter automatically.
:::
Calling the interpreter's `shutdown` can be handled with either with a
`try`/`finally` statement or a `with` statement:
```python
import em
interp = em.Interpreter(...)
try:
... do some things with the interpreter ...
finally:
interp.shutdown()
# or ...
with em.Interpreter(...) as interp:
... do other things with the interpreter ...
```
:::{warning}
If you receive a `ConsistencyError` mentioning the proxy when quitting
your program, you are likely not calling the `shutdown` method on the
interpreter. Make sure to call `shutdown` so the interpreter can
clean up after itself.
:::
There is also a global `expand` function which will expand a single
string, creating and destroying an ephemeral interpreter to do so.
You can use this function to do a one-off expansion of, say, a large
file:
```python
import em
data = open('tests/sample/sample.em').read()
print(em.expand(data))
```
If an exception occurs during `expand` processing, the exception will
be raised to the caller.
### Modules
A fully-functional EmPy system contains the following modules and files.
#### `empy` pseudomodule
The pseudomodule is not an actual module, but rather the instance of
the running EmPy interpreter exposed to the EmPy system. It is
automatically placed into the interpreter's globals and cannot be
imported explicitly. See
[Pseudomodule/interpreter](#pseudomodule-interpreter) for details.
#### `em` module
The primary EmPy module. It contains the `Configuration` and
`Interpreter` classes as well as all supporting logic. An EmPy system
can be functional with only this module present if needed.
It also includes the following global functions:
{#details}
`details(level, [prelim, postlim, file])`
: Write details about the running system to the given file, which
defaults to `sys.stdout`. The `level` parameter is an attribute of
the `em.Version` class (effectively an enum). `prelim` and
`postlim` indicate preliminary and postliminary text to output
before and after the details (and have reasonable defaults).
:::{note}
This function requires the `emlib` to be installed to function most
effectively.
:::
{#expand}
`expand(data, **kwargs) -> str`
: Create a ephemeral interpreter with the given kwargs, expand data,
shut the interpreter down, and then return the expansion. The
function takes the same keyword arguments as the [`Interpreter`
constructor](#constructor), with the following additions:
{#expand-arguments}
| Argument | Meaning | Default |
| --- | --- | --- |
| `dispatcher` | Dispatch errors or raise to caller? | `False` |
| `locals` | The locals dictionary | `{}` |
| `name` | The context filename | `""` |
If the markup that is being expanded causes an exception to be
raised, by default the exception will be let through to the caller.
:::{important}
As with the [`Interpreter` constructor](#constructor), the order of
the `expand` arguments has changed over time and is subject to
change in the future, so you must use keyword arguments to prevent
any ambiguity, _e.g._:
```python
myConfig = em.Configuration(...)
myGlobals = {...}
myOutput = open(...)
result = em.expand(source, config=myConfig, globals=myGlobals, ...)
```
Attempts have been made to make the `expand` function as backward
compatible (to 3._x_) as feasible, but some usages are ambiguous or
do not have direct mappings to configurations. A
`CompatibilityError` will be raised in these cases; if you encounter
this, redesign your use of `expand` to be compatible with [the
modern usage](#expand). In particular, in 3._x_, additional keyword
arguments were used to indicate the locals dictionary; in 4._x_,
keyword arguments are used for all arguments so the locals
dictionary must be specified as a distinct `locals` keyword
argument:
```python
myGlobals = {...}
myLocals = {...}
result = em.expand(source, globals=myGlobals, locals=myLocals)
```
:::
:::{warning}
Not all of the `Interpreter` constructor arguments are compatible
with the `expand` function. The `filters`, `handler`, `input` and
`output` arguments are immediately overridden by the inherent nature
of the ephemeral interpreter and so would not behave as expected.
Thus, if they are specified, a `ConfigurationError` will be raised.
For more detailed configuration of an interpreter, it's better to
create one yourself rather than rely on `expand`.
:::
{#invoke}
`invoke(args, **kwargs)`
: Invoke the EmPy system with the given command line arguments
(`sys.argv[1:]`, not `sys.argv`) and optional string settings. This
is the entry point used by the main EmPy function. The remaining
keyword arguments correspond to the [`Interpreter`
constructor](#constructor) arguments.
:::{warning}
Since the `invoke` function configures and manages the lifetime of
an `Interpreter`, not all of the constructor arguments are
compatible with it. Specifically, the `filespec` and `immediately`
arguments need to be managed by the function and so specifying a
starting value is nonsensical. Thus, if they are specified, a
`ConfigurationError` will be raised.
:::
#### `emlib` module
The EmPy supporting library. It contains various support classes,
including the base classes `Filter` and `Hook` to assist in creating
this supporting functionality.
#### `emhelp` module
The EmPy help system. It can be accessed from the main executable
with the `-h/--help` and `-H/--topics=TOPICS` command
line options. If the emlib module is not available to the executable,
the help system will return an error.
#### `emdoc` module
The EmPy documentation system, used to create this document.
:::{note}
Unlike the other EmPy modules, `emdoc` requires a modern Python 3._x_
interpreter.
:::
### Using EmPy with build tools
If you're using EmPy to process documents within a build system such
as GNU Make or Ninja, you'll want to use the `-o/--output=FILENAME` (or
`-a/--append=FILENAME`) and `-d/--delete-on-error` options together. This
will guarantee that a file will be output (or appended) to a file
without shell redirection, and that the file will be deleted if an
error occurs. This will prevent errors from leaving a partial file
around which subsequent invocations of the build system will mistake
as being up to date. The invocation of EmPy should look like this
(the `--` is not required if the input filename never starts with a
dash):
```shell
em.py -d -o $output -- $input
```
For GNU Make:
````make
EMPY ?= em.py
EMPY_OPTIONS ?= -d
%: %.em
$(EMPY) $(EMPY_OPTIONS) -o $@ -- $<
````
For Ninja:
```ninja
empy = em.py
empy_options = -d
rule empy
command = $empy $empy_options -o $out -- $in
```
### Context formatting
**Contexts** are objects which contain the filename, the line number,
the column number, and the character (Unicode code point) number to
record the location of an EmPy error during processing.
These are formatted into human-readable strings with a **context
format**, a string specifiable with `--context-format` (_configuration variable:_ `contextFormat`). A few different mechanisms for formatting contexts are
available:
| Mechanism | Description | Example
| --- | --- | --- |
| format | Use the `str.format` method | `{name}:{line}:{column}` |
| operator | Use the `%` operator | `%(name)s:%(line)d:%(column)d` |
| variable | Use `$` variables | `$NAME:$LINE:$COLUMN` |
The default context format is `%(name)s:%(line)d:%(column)d` and
uses the operator mechanism for backward compatibility.
When a context format is set, EmPy will attempt to detect which of the above mechanisms is needed:
| Mechanism | Criteria |
| --- | --- |
| format | string begins with `format:` or does not contain a `%` |
| operator | string begins with `operator:` or contains a `%` |
| variable | string begins with `variable:` |
### Data flow
**input ⟶ interpreter ⟶ diversions ⟶ filters ⟶ output**
Here, in summary, is how data flows through a working EmPy system:
1. Input comes from a source, such as an .em file on the command line,
`sys.stdin`, or via an `empy.include` statement.
2. The interpreter processes this material as it comes in,
processing EmPy expansions as it goes.
3. After interpretation, data is then sent through the diversion
layer, which may allow it directly through (if no diversion is
in progress) or defer it temporarily. Diversions that are
recalled initiate from this point.
4. Any filters in place are then used to filter the data and
produce filtered data as output.
5. Finally, any material surviving this far is sent to the output
stream. That stream is `sys.stdout` by default, but can be changed
with the `-o/--output=FILENAME` or `-a/--append=FILENAME` options.
6. If an error occurs, execute the error handler (which by default
prints an EmPy error) If the `-r/--raw-errors` option is
specified, then print a full Python traceback. If
`-k/--keep-going` is specified, continue processing rather than
exit; otherwise halt.
7. On unsuccessful exit, if `-d/--delete-on-error` is specified, delete any
specified output file.
### Glossary
The following terms with their definitions are used by EmPy:
*callback*
: The user-provided callback which is called when the custom markup
`@<...>` is encountered.
*command*
: A processing step which is performed before or after main document
processing. Examples are `-D/--define=DEFN`, `-F/--file=FILENAME` or
`-P/--preprocess=FILENAME`.
*configuration*
: An object encapsulating all the configurable behavior of an
interpreter which passed into interpreter on creation.
Configurations can be shared between multiple interpreters.
*context*
: An object which tracks the location of the parser in an EmPy file
for tracking and error reporting purposes.
*control markup*
: A markup used to direct high-level control flow within an EmPy
session. Control markups are expressed with the `@[...]` notation.
*core*
: An interpreter core which determines how the underlying language is
evaluated, executed, serialized, and how the `@[def ...]` control
markup works. By default, the underlying language is Python.
*custom*
: The custom markup invokes a callback which is provided by the user,
allowing any desired behavior. Custom markup is `@<...>`.
*diacritic*
: A markup which joins together a letter and one or more combining
characters from a dictionary in the configuration and outputs it.
Diacritic markup is `@^...`.
*dispatcher*
: An error dispatcher determines whether to dispatch the error to the
interpreter's error handler (`True`), to reraise the error to the
caller (`False`), or something else.
*diversion*
: A process by which output is deferred, and can be recalled later on
demand, multiple times if desired.
*document*
: An EmPy file containing EmPy markup to expand.
*embedding*
: Using an EmPy system by importing the `em` module and using the API
to create and manipulate interpreters programmatically, as opposed
to standalone.
*emoji*
: A markup which looks up a Unicode code point by name via a
customizable set of installable emoji modules, or via a dictionary
in the configuration. Emoji markup is `@:...:`.
*error*
: An exception thrown by a running EmPy system. When these occur,
they are dispatched by an error dispatcher and then (possibly)
passed to an error handler.
*escape*
: A markup designed to expand to a single (often non-printable)
character, similar to escape sequences in C or other languages.
Escape markup is `@\...`.
*expansion*
: The process of processing EmPy markups and producing output.
*expression*
: An expression markup represents a Python expression to be evaluated,
and replaced with the `str` of its value. Expression markup is
`@(...)`.
*file*
: An object which exhibits a file-like interface (methods such as
`write` and `close`).
*filter*
: A file-like object which can be chained to other filters or the
final stream, and can buffer, alter, or manipulate in any way the
data sent. Filters can be chained together in arbitrary order.
*finalizer*
: A function which is called when an interpreter exits. Multiple
finalizers can be added to each interpreter.
*globals*
: The dictionary (or dictionary-like object) which resides inside the
interpreter and holds the currently-defined variables.
*handler*
: An error handler which is called whenever an error occurs in the
EmPy system. The default error handler prints details about the
error to `sys.stderr`.
*hook*
: A callable object that can be registered in a dictionary, and which
will be invoked before, during, or after certain internal
operations, identified by name with a string. Some types of hooks
can override the behavior of the EmPy interpreter.
*icon*
: A markup which looks up a variable-length abbreviation for a string
from a lookup table in the configuration. Icon markup is `@|...`.
*interpreter*
: The application (or class instance) which processes EmPy markup.
*locals*
: Along with the globals, a locals dictionary can be passed into
individual EmPy API calls.
*markup*
: EmPy substitutions set off with a prefix (by default `@`) and
appropriate delimiters.
*named escape*
: A control character referenced by name in an escape markup,
`@\^{...}`.
*output*
: The final destination of the result of processing an EmPy file.
*prefix*
: The Unicode code point (character) used to set off an expansions.
By default, the prefix is `@`. If set to `None`, no markup will be
processed.
*processor*
: An extensible system which processes a group of EmPy files, usually
arranged in a filesystem, and scans them for significators.
*proxy*
: An object which replaces the `sys.stdout` file object and allows the
EmPy system to intercept any indirect output to `sys.stdout` (say,
by the `print` function).
*pseudomodule*
: The module-like object named `empy` (by default) which is exposed as
a global inside every EmPy system. The pseudomodule and the
interpreter are in fact the same object, an instance of the
`Interpreter` class.
*significator*
: A special form of an assignment markup in EmPy which can be easily
parsed externally, primarily designed for representing uniform
assignment across a collection of files. Significator markup is
`@%[!]... NL` and `@%%[!]...%% NL`.
*standalone*
: Using the EmPy system by running the `em.py` executable from the
command line.
*statement*
: A line of code that needs to be executed; statements do not have
return values. Statement markup is `@{...}`.
*stream*
: A file-like object which manages diversion and filtering. A stack
of these is used by the interpreter with the top one being active.
*system*
: A running EmPy environment.
*token*
: An element of EmPy parsing. Tokens are parsed and then processed
one at a time.
### Statistics
% wc emdoc.py emhelp.py emlib.py em.py test.sh LEGACY.md LICENSE.md README.md ANNOUNCE.md HELP.md README.md.em
505 1399 17316 emdoc.py
990 4408 42749 emhelp.py
1128 3354 35545 emlib.py
6324 22547 235116 em.py
773 2759 19803 test.sh
2701 15082 100991 LEGACY.md
14 230 1520 LICENSE.md
6554 29326 203221 README.md
767 4160 29317 ANNOUNCE.md
914 5106 46231 HELP.md
6686 31391 219427 README.md.em
27356 119762 951236 total
% sha1sum emdoc.py emhelp.py emlib.py em.py test.sh LEGACY.md LICENSE.md README.md ANNOUNCE.md HELP.md README.md.em
0a9d4545f1c4ef49e12811fe57e780c28166f369 emdoc.py
f3c625c51a1425e1bf84b2aefae7a74fae21deed emhelp.py
e965e967e34f58f7280c64a406f46bc7b3915ced emlib.py
9500c6e6029e2bb63396246a1d967597c06d3673 em.py
7f55200b3b67770ada2240c77074952c972ab207 test.sh
2c09a1fc1345f3dd637cd397cc674696f16fdc60 LEGACY.md
cd2b06faf3ef9188ef67c38818a0c6076f757801 LICENSE.md
595ebd968db1f64fbeb095204cc913f8fcf1e364 README.md
dcc21725b020d597ba2e4096402c511923eb242c ANNOUNCE.md
652c8fca03f510bae15afd867b59195b251b2b59 HELP.md
0afeba9e1d35c975ac5775c4e74ff17f69632c5e README.md.em
## End notes
### Author's notes
I originally conceived EmPy as a replacement for my [Web templating
system](http://www.alcyone.com/max/info/m4.html) which uses
[m4](https://www.gnu.org/software/m4/), a general macroprocessing
system for Unix.
Most of my Web sites use a variety of m4 files, some of which are
dynamically generated from databases, which are then scanned by a
cataloging tool to organize them hierarchically (so that, say, a
particular m4 file can understand where it is in the hierarchy, or
what the titles of files related to it are without duplicating
information); the results of the catalog are then written in database
form as an m4 file (which every other m4 file implicitly includes),
and then GNU Make converts each m4 to an HTML file by processing it.
As the Web sites got more complicated, the use of m4 (which I had
originally enjoyed for the challenge and abstractness) really started
to become an impediment to serious work; while I was very
knowledgeable about m4 -- having used it for so many years -- getting
even simple things done with it is awkward and often difficult. Worse
yet, as I started to use Python more and more over the years, the
cataloging programs which scanned the m4 and built m4 databases were
migrated to Python and made almost trivial, but writing out huge
awkward tables of m4 definitions simply to make them accessible in
other m4 scripts started to become almost farcical.
It occurred to me what I really wanted was an all-Python solution.
But replacing what used to be the m4 files with standalone Python
programs would result in somewhat awkward programs normally consisting
mostly of unprocessed text punctuated by small portions where
variables and small amounts of code need to be substituted. Thus the
idea was a sort of inverse of a Python interpreter: a program that
normally would just pass text through unmolested, but when it found a
special signifier would execute Python code in a normal environment.
I looked at existing Python templating systems, and didn't find
anything that appealed to me -- I wanted something where the desired
markups were simple and unobtrusive. After considering choices of
prefixes, I settled on `@` and EmPy was born.
As I developed the tool, I realized it could have general appeal, even
to those with widely varying problems to solve, provided the core tool
they needed was an interpreter that could embed Python code inside
templated text. As I continue to use the tool, I have been adding
features as unobtrusively as possible as I see areas that can be
improved.
A design goal of EmPy is that its feature set should work on several
levels; at any given level, if the user does not wish or need to use
features from another level, they are under no obligation to do so --
in fact, they wouldn't even need to know they exist. If you have no
need of diversions, for instance, you are under no obligation to use
them or even to know anything about them. If significators will not
help you organize a set of EmPy scripts globally, then you can ignore
them. New features that are being added are whenever feasible
transparently backward compatible (except for major version releases);
if you do not need them, their introduction should not affect you in
any way. Finally, the use of unknown prefix and escape sequences
results in errors, ensuring that they are reserved for future use.
### Acknowledgements
Questions, suggestions, bug reports, evangelism, and even complaints
from many people over the years have helped make EmPy what it is
today. Some, but by no means all, of these people are (in
alphabetical order by surname):
- Biswapesh Chattopadhyay
- Beni Cherniavsky
- Dr. S. Candelaria de Ram
- Eric Eide
- Dinu Gherman
- Grzegorz Adam Hankiewicz
- Robert Kroeger
- Bohdan Kushnir
- Kouichi Takahashi
- Ville Vainio
### Known issues and caveats
{#security}
- A running EmPy system is just an alternate form of a Python
interpreter; EmPy code is just as powerful as any Python code. Thus
it is vitally important that an EmPy system not expand EmPy markup
from an untrusted source; this is just as unsafe and potentially
dangerous as executing untrusted Python code.
- To function properly, EmPy must override `sys.stdout` with a proxy
file object, so that it can capture output of side effects and
support diversions for each interpreter instance. It is important
that code executed in an environment _not_ rebind `sys.stdout`,
although it is perfectly legal to reference it explicitly (_e.g._,
`@sys.stdout.write("Hello world\n")`). If one really needs to
access the "true" stdout, then use `sys.__stdout__` instead (which
should also not be rebound). EmPy uses the standard Python error
handlers when exceptions are raised in EmPy code, which print to
`sys.stderr`. `sys.stderr`, `sys.__stdout__`, and `sys.__stderr__`
are never overridden by the interpreter; only `sys.stdout` is.
- If you are using multiple interpreters with distinct output files
and are using the low-level interpreter methods (the ones not
documented here) to perform expansion and output, the `sys.stdout`
proxy will not be reliable. Only the high-level interpreter methods
(`evaluate`, `execute`, `string`, `expand` properly use the
protected stream stack on the `sys.stdout` proxy to guarantee valid
output. Either only use a single interpreter instance at a time
(creating and shutting it down with its `shutdown` method), use the
`-n/--no-proxy` option and only perform output with the
`write` method on the interpreter (_i.e._, do not use any `print`
statements in your code), or only use the high-level interpreter
methods documented here.
- The `empy` "module" exposed through the EmPy interface (_e.g._,
`@empy`) is an artificial module. It is automatically exposed in
the globals of a running interpreter and it cannot be manually
imported with the `import` statement (nor should it be -- it is an
artifact of the EmPy processing system and does not correspond
directly to any .py file).
- For an EmPy statement expansion all alone on a line, _e.g._, `@{a =
1}`, will include a blank line due to the newline following the
closing curly brace. To suppress this blank line, use the symmetric
convention `@{a = 1}@`, where the final `@` markup precedes the
newline, making it whitespace markup and thus consumed. For
instance:
````
@{a = 1}
There will be an extra newline above (following the closing brace).
Compare this to:
@{a = 1}@
There will be no extra newline above.
````
See [here](#idiom) for more details.
- Errors generated from within nested control structures (_e.g._,
`@[for ...]@[if ...]...@[end if]@[end for]` will report a context
of the start of the top-level control structure markup, not the
innermost markup, which would be much more helpful. This issue is
not new to 4.0 and will be addressed in a future release.
- Errors are very literal and could be made more useful to find the
underlying cause.
- Contexts (such as `empy.identify`) track the context of executed
_EmPy_ code, not Python code. This means, for instance, that blocks
of code delimited with `@{` and `}` will identify themselves as
appearing on the line at which the `@{` appears. If you're
tracking errors and want more information about the location of the
errors from the Python code, use the `-r/--raw-errors` option, which
will provide you with the full Python traceback.
- The `@[for ...]` variable specification supports tuples for tuple
unpacking, even recursive tuples. However, it is limited in that
the names included may only be valid Python identifiers, not
arbitrary Python "lvalues." Since this is something of an
accidental Python feature that is very unlikely to be relied on in
practice, this is not thought to be a significant limitation. As a
concrete example:
```python
a = [None]
for a[0] in range(5):
print(a)
```
is valid (but strange) Python, but the EmPy equivalent with `@[for
a[0] in range(5)]...` is invalid.
- The `:=` assignment expression syntax ("walrus operator") for
`while` loops and `if` statements, introduced in Python 3.8, is not
supported in the EmPy equivalent control markups `@[while]` and
`@[if]`. This may be supported in the future.
- As of Python 3.10, the `with` control structure supports specifying
multiple context managers separated by commas. This is not yet
supported by EmPy, but may be in a future version. For now, just
use nested `@[with]` control markups.
### For package maintainers
EmPy is available as a system package in most major Linux
distributions, though many have not updated to EmPy 4.0 yet.
EmPy can be made available as an operating system/distribution package
in several different ways. Regardless of the high-level organization,
the installed .py Python files must be made available as importable
Python modules, with the additional requirement that em.py must be
made available as an executable in the default `PATH`. If necessary,
this executable may also be named `empy`, but `em.py` is preferred --
and either way it is still important that the em.py file be available
for importing as a Python module (`em`).
:::{note}
Since EmPy 4.0 is not fully compatible with EmPy 3._x_, I suggest
making both EmPy 3._x_ and 4.0 packages available side by side until
4.0 becomes more fully adopted by the community.
:::
Here is a breakdown of the contents of a release tarball:
| File | Description |
| --- | --- |
| em.py | Main EmPy module and executable |
| emhelp.py | Help subsystem module |
| emlib.py | Supplementary EmPy facilities module |
| emdoc.py | Documentation subsystem module |
| ANNOUNCE.md | EmPy 4.0 release announcement |
| HELP.md | Help topic summaries |
| LICENSE.md | Software license |
| README.md | README (this file) |
| README.md.em | README source file |
| doc | HTML documentation directory hierarchy |
| test.sh | Test shell script |
| tests | Tests directory hierarchy |
| suites | Test suites directory hierarchy |
They can either be bundled up into a single, monolithic package, or
divided into a series of subpackages. Here's a suggestion for a
fleshed-out series of EmPy subpackages:
`empy-minimal`
: Just the em.py file, available as a Python module as well as an
executable. Note that this will not allow the use of the EmPy help
subsystem, unless the module emhelp.py is also included.
`empy-basic`
: The .md files, all the .py files (em.py,
emhelp.py, emlib.py, emdoc.py) available as Python modules, with the
em.py file also available as an executable.
`empy-doc`
: Just the contents of the README source file README.md.em and the
docs directory hierarchy.
`empy-test`
: The test script test.sh, the tests directory, and the suites
directory.
`empy`
: All of the above.
### Reporting bugs
If you find a bug in EmPy, please follow these steps:
1. Whittle a reproducible test case down to the smallest standalone
example which demonstrates the issue, the smaller the better;
2. Collect the output of `em.py -Z` (this will provide detailed
diagnostic details about your environment), or at least `em.py -W`
(which provides only basic details);
3. [Send me an email](mailto:software@alcyone.com) with _EmPy_ in the
Subject line including both files and a description of the problem.
Thank you!
### Release history
4.1 (2024 Mar 24)
: Add support for extension markup `@((...))`, `@[[...]]`,
`@{{...}}`, `@<...>`, etc., with custom callbacks retained for
backward compatibility; add `@[match]` control support; add
interpreter cores for overriding interpreter behavior; add more
command line option toggles; add notion of verbose/brief errors;
more uniform error message formatting; various documentation
updates.
{#last-minor-release}
4.0.1 (2023 Dec 24)
: Add root context argument, serializers, and idents to interpreter;
fix `setContext...` methods so they also modify the currents stack;
better backward compatibility for `expand` function and
`CompatibilityError`; fix inconsistent stack usage with `expand`
method; add error dispatchers, cleaner error handling and
`ignoreErrors`; have `expand` method/function raise
exceptions to caller; eliminate need for `FullContext` class
distinct from `Context`; support comments in "clean" controls; add
`--no-none-symbol` option; add clearer errors for removed literal
markup; add `Container` support class in `emlib`; hide non-standard
proxy attributes and methods; support string errors (why not);
update and expand tests; help subsystem and documentation updates.
4.0 (2023 Nov 29)
: A major revamp, refresh, and modernization. Major new features
include inline comments `@*...*`; backquote literals `` @`...`
``; chained if-then-else expressions; functional expressions
`@f{...}`; full support for `@[try]`, `@[while ...]` and `@[with
...]` control markup; `@[defined ...]` control markup; stringized
and multiline significators; named escapes `@\^{...}`; diacritics
`@^...`; icons `@|...`; emojis `@:...:`; configurations; full
Unicode and file buffering support; proxy now reference counted;
hooks can override behavior; many bug fixes; an extensive builtin
help system (`emhelp`); and rewritten and expanded documentation in
addition to a dedicated module (`emdoc`). Changes include
relicensing to BSD, interpreter constructor now requires keyword
arguments, `-d/--delete-on-error` instead of "fully buffered files";
cleaned up environment variables; "repr" markup replaced with emoji
markup; remove literal markups `@)`, `@]`, `@}`; context line
markup `@!...` no longer pre-adjusts line; custom markup `@<...>`
now parsed more sensibly; filter shortcuts removed; context now
track column and character count; auxiliary classes moved to `emlib`
module; use `argv` instead of `argc` for interpreter arguments. See
[Full list of changes between EmPy 3._x_ and
4.0](http://www.alcyone.com/software/empy/ANNOUNCE.html#all-changes) for a more
comprehensive list.
{#last-major-release}
3.3.4a (2021 Nov 19)
: Fix an error in setup.py in the downloadable tarball (did not affect
PIP downloads).
3.3.4 (2019 Feb 26)
: Minor fix for a Python 3._x_ compatibility issue.
3.3.3 (2017 Feb 12)
: Fix for `empy.defined` interpreter method.
3.3.2 (2014 Jan 24)
: Additional fix for source compatibility between 2._x_ and 3.0.
3.3.1 (2014 Jan 22)
: Source compatibility for 2._x_ and 3.0; 1._x_ compatibility dropped.
3.3 (2003 Oct 27)
: Custom markup `@<...>`; remove separate pseudomodule instance for
greater transparency; deprecate `Interpreter` attribute of
pseudomodule; deprecate auxiliary class name attributes associated
with pseudomodule in preparation for separate support library in
4.0; add `--no-callback-error` [defunct] and
`--no-bangpath-processing` [now
`--no-ignore-bangpaths`] command line options; add
`atToken` hook.
3.2 (2003 Oct 7)
: Reengineer hooks support to use hook instances; add `-v/--verbose`
and `-l/--relative-path` option; reversed PEP 317 style;
modify Unicode support to give less confusing errors in the case of
unknown encodings and error handlers; relicensed under LGPL.
3.1.1 (2003 Sep 20)
: Add string literal `@"..."` markup; add
`-w/--pause-at-end` option; fix improper globals collision
error via the `sys.stdout` proxy.
3.1 (2003 Aug 8)
: Unicode support (Python 2.0 and above); add Document and Processor
helper classes for processing significators [later moved to
`emlib`]; add `--no-prefix` option for suppressing all
markups.
3.0.4 (2003 Aug 7)
: Implement somewhat more robust "lvalue" parsing for `@[for]`
construct (thanks to Beni Cherniavsky for inspiration).
3.0.3 (2003 Jul 9)
: Fix bug regarding recursive tuple unpacking using `@[for]`; add
`empy.saveGlobals`, `empy.restoreGlobals`, and `empy.defined`
functions.
3.0.2 (2003 Jun 19)
: `@?` and `@!` markups for changing the current context name and
line, respectively; add `update` method to interpreter; new and
renamed context operations, `empy.setContextName`,
`empy.setContextLine`, `empy.pushContext`, `empy.popContext`.
3.0.1 (2003 Jun 9)
: Fix simple bug preventing command line preprocessing directives
(`-I/--import=MODULES`, `-D/--define=DEFN`, `-E/--execute=STATEMENT`,
`-F/--file=FILENAME`, `-P/--preprocess=FILENAME`) from executing properly;
defensive PEP 317 compliance [defunct].
3.0 (2003 Jun 1)
: Replace substitution markup with control markup `@[...]`; support
`@(...?...!...)` for conditional expressions; add acknowledgements
and glossary sections to documentation; rename buffering option back
to `-b/--buffering`; add `-m/--pseudomodule=NAME` and
`-n/--no-proxy` for suppressing `sys.stdout` proxy; rename
main error class to `Error`; add standalone `expand` function; add
`--binary` and `--chunk-size` options [defunct]; reengineer parsing
system to use tokens for easy extensibility; safeguard curly braces
in simple expressions [now used by functional expressions]; fix bug
involving custom `Interpreter` instances ignoring globals argument;
`distutils` [now `setuptools`] support.
2.3 (2003 Feb 20)
: Proper and full support for concurrent and recursive interpreters;
protection from closing the true stdout file object; detect edge
cases of interpreter globals or `sys.stdout` proxy collisions; add
globals manipulation functions `empy.getGlobals`, `empy.setGlobals`,
and `empy.updateGlobals` which properly preserve the `empy`
pseudomodule; separate usage info out into easily accessible lists
for easier presentation; have `-h` option show simple usage and `-H`
show extended usage [defunct]; add `NullFile` utility class.
2.2.6 (2003 Jan 30)
: Fix a bug in the `Filter.detach` method (which would not normally be
called anyway).
2.2.5 (2003 Jan 9)
: Strip carriage returns out of executed code blocks for DOS/Windows
compatibility.
2.2.4 (2002 Dec 23)
: Abstract Filter interface to use methods only; add `@[noop: ...]`
substitution for completeness and block commenting [defunct].
2.2.3 (2002 Dec 16)
: Support compatibility with Jython by working around a minor
difference between CPython and Jython in string splitting.
2.2.2 (2002 Dec 14)
: Include better docstrings for pseudomodule functions; segue to a
dictionary-based options system for interpreters; add
`empy.clearAllHooks` and `empy.clearGlobals`; include a short
documentation section on embedding interpreters; fix a bug in
significator regular expression.
2.2.1 (2002 Nov 30)
: Tweak test script to avoid writing unnecessary temporary file; add
`Interpreter.single` method; expose `evaluate`, `execute`,
`substitute` [defunct], and `single` methods to the pseudomodule;
add (rather obvious) `EMPY_OPTIONS` environment variable support;
add `empy.enableHooks` and `empy.disableHooks`; include optimization
to transparently disable hooks until they are actually used.
2.2 (2002 Nov 21)
: Switched to `-V/--version` option for version information;
`empy.createDiversion` for creating initially empty diversion;
direct access to diversion objects with `empy.retrieveDiversion`;
environment variable support; removed `--raw` long argument (use
`-r/--raw-errors` instead); added quaternary escape code
(well, why not).
2.1 (2002 Oct 18)
: `empy.atExit` [now `empy.appendFinalizer`] registration separate
from hooks to allow for normal interpreter support; include a
benchmark sample and test.sh verification script; expose
`empy.string` directly; `-D/--define=DEFN` option for explicit
defines on command line; remove ill-conceived support for `@else:`
separator in `@[if ...]` substitution [defunct]; handle nested
substitutions properly [defunct]; `@[macro ...]` substitution for
creating recallable expansions [defunct]; add support for finalizers
with `empy.atExit` [now `empy.appendFinalizer`].
2.0.1 (2002 Oct 8)
: Fix missing usage information; fix `after_evaluate` hook not getting
called [defunct].
2.0 (2002 Sep 30)
: Parsing system completely revamped and simplified, eliminating a
whole class of context-related bugs; builtin support for buffered
filters; support for registering hooks; support for command line
arguments; interactive mode with `-i/--interactive`; significator
value extended to be any valid Python expression.
1.5.1 (2002 Sep 24)
: Allow `@]` to represent unbalanced close brackets in `@[...]`
markups [defunct].
1.5 (2002 Sep 18)
: Escape codes (`@\...`); conditional and repeated expansion
substitutions [defunct]; replaced with control markups]; fix a few
bugs involving files which do not end in newlines.
1.4 (2002 Sep 7)
: Add in-place markup `@:...:...:` [now `@$...$...$`]; fix bug with
triple quotes; collapse conditional and protected expression
syntaxes into the single generalized `@(...)` notation;
`empy.setName` and `empy.setLine` functions [now
`empy.setContextName` and `empy.setContextLine`]; true support for
multiple concurrent interpreters with improved `sys.stdout` proxy;
proper support for `empy.expand` to return a string evaluated in a
subinterpreter as intended; reorganized parser class hierarchy.
1.3 (2002 Aug 24)
: Pseudomodule as true instance; move toward more verbose (and clear)
pseudomodule functions; fleshed out diversions model; filters;
conditional expressions; protected expressions; preprocessing with
`-P/--preprocess=FILENAME` (in preparation for possible support for command
line arguments).
1.2 (2002 Aug 16)
: Treat bangpaths as comments; `empy.quote` for the opposite process
of `empy.expand`; significators (`@%...` sequences); add
`-I/--import=MODULES` and `-f/--flatten` options; much improved
documentation.
1.1.5 (2002 Aug 15)
: Add a separate `invoke` function that can be called multiple times
with arguments to simulate multiple runs.
1.1.4 (2002 Aug 12)
: Handle strings thrown as exceptions properly; use `getopt` to
process command line arguments; cleanup file buffering with
AbstractFile; very slight documentation and code cleanup.
1.1.3 (2002 Aug 9)
: Support for changing the prefix from within the `empy` pseudomodule.
1.1.2 (2002 Aug 5)
: Renamed buffering option [defunct], added `-F/--file=FILENAME` option
for interpreting Python files from the command line, fixed improper
handling of exceptions from command line options (`-E/--execute=STATEMENT`,
`-F/--file=FILENAME`).
1.1.1 (2002 Aug 4)
: Typo bugfixes; documentation clarification.
1.1 (2002 Aug 4)
: Added option for fully buffering output (including file opens),
executing commands through the command line; some documentation
errors fixed.
{#first-major-release}
1.0 (2002 Jul 23)
: Renamed project to EmPy. Documentation and sample tweaks; added
`empy.flatten` [now `empy.flattenGlobals`]; added `-a/--append=FILENAME`
option. First official release.
0.3 (2002 Apr 14)
: Extended "simple expression" syntax, interpreter abstraction, proper
context handling, better error handling, explicit file inclusion,
extended samples.
0.2 (2002 Apr 13)
: Bugfixes, support non-expansion of `None`s, allow choice of alternate
prefix.
0.1.1 (2002 Apr 12)
: Bugfixes, support for Python 1.5._x_ [defunct], add
`-r/--raw-errors` option.
0.1 (2002 Apr 12)
: Initial early access release.
### Contact
This software was written by [Erik Max
Francis](http://www.alcyone.com/max/). If you use this software, have
suggestions for future releases, or bug reports or problems with this
documentation, [I'd love to hear about
it](mailto:software@alcyone.com).
Even if you try out EmPy for a project and find it unsuitable, I'd
like to know what stumbling blocks you ran into so they can
potentially be addressed in a future version.
I hope you enjoy using EmPy! ℰ
### About this document
This document was generated with EmPy itself using the `emdoc` module.
Both the source (README.md.em) and the resulting Markdown text
(README.md) are included in the release tarball, as is the HTML
directory hierarchy generated with Sphinx (doc).
_This documentation for EmPy version 4.1 was generated from README.md.em (SHA1 `0afeba9e1d35c975ac5775c4e74ff17f69632c5e`, 219427 bytes) at 2024-04-03 16:52:31 using EmPy version 4.1, in CPython/3.10.12, on Linux (POSIX), with x86_64._