# User's guide (version 3.3.4)


:::{important}

This documentation is for an older version of EmPy and is out of date.
There have been significant changes to EmPy starting with version 4.0.

For the latest version of this documentation, see
<http://www.alcyone.com/software/empy/README.html>.

:::

A powerful and robust templating system for Python.


## Introduction

### Overview

EmPy is a system for embedding Python expressions and statements in
template text; it takes an EmPy source file, processes it, and
produces output.  This is accomplished via expansions, which are
special signals to the EmPy system and are set off by a special prefix
(by default the at sign, `@`).  EmPy can expand arbitrary Python
expressions and statements in this way, as well as a variety of
special forms.  Textual data not explicitly delimited in this way is
sent unaffected to the output, allowing Python to be used in effect as
a markup language.  Also supported are callbacks via hooks, recording
and playback via diversions, and dynamic, chainable filters.  The
system is highly configurable via command line options and embedded
commands.

Expressions are embedded in text with the `@(...)` notation;
variations include conditional expressions with `@(...?...!...)`  and
the ability to handle thrown exceptions with `@(...$...)`.  As a
shortcut, simple variables and expressions can be abbreviated as
`@variable`, `@object.attribute`, `@function(arguments)`, `@sequence`
[index], and combinations.  Full-fledged statements are embedded with
`@{...}`.  Control flow in terms of conditional or repeated expansion
is available with `@[...]`.  A `@` followed by a whitespace character
(including a newline) expands to nothing, allowing string
concatenations and line continuations.  Comments are indicated with
`@#` and consume the rest of the line, up to and including the
trailing newline.  `@%` indicate "significators," which are special
forms of variable assignment intended to specify per-file
identification information in a format which is easy to parse
externally.  Context name and line number changes can be done with
`@?` and `@!` respectively.  `@<...>` markups are customizeable by the
user and can be used for any desired purpose.  Escape sequences
analogous to those in C can be specified with `@\...`, and finally a
`@@` sequence expands to a single literal at sign.


### Getting the software

The current version of EmPy is 3.3.4.

The official URL for this Web site is <http://www.alcyone.com/software/empy/>.

The latest version of the software is available in a tarball here:
<http://www.alcyone.com/software/empy/empy-3.3.4a.tar.gz>.

The software can be installed through PIP via this shell command:

<pre class="shell"><b><i>% python3 -m pip install 'empy==3.3.4'</i></b>
<i>...</i></pre>


For information about upgrading from 3._x_ to 4.0, see
<http://www.alcyone.com/software/empy/ANNOUNCE.html>.


### Requirements

EmPy should work with any version of Python from 2.4 onward,
including 3._x_.


### License

:::{caution}

The license for EmPy 4.0 has changed from LGPL to BSD.

:::

This code is released under the [Lesser
GPL](http://www.gnu.org/copyleft/lesser.html).


### Mailing lists

:::{error}

These mailing lists are no longer running.
:::

There are two EmPy related mailing lists available.  The first is a
receive-only, very low volume list for important announcements
(including releases).  To subscribe, send an email to
[empy-announce-list-subscribe@alcyone.com](mailto:empy-announce-list-subscribe@alcyone.com).

The second is a general discussion list for topics related to EmPy,
and is open for everyone to contribute; announcements related to EmPy
will also be made on this list.  The author of EmPy (and any future
developers) will also be on the list, so it can be used not only to
discuss EmPy features with other users, but also to ask questions of
the author(s).  To subscribe, send an email to
[empy-list-subscribe@alcyone.com](mailto:empy-list-subscribe@alcyone.com).


### Basics

EmPy is intended for embedding Python code in otherwise
unprocessed text.  Source files are processed, and the results are
written to an output file.  Normal text is sent to the output
unchanged, but markups are processed, expanded to their results,
and then written to the output file as strings (that is, with the
`str` function, not `repr`).  The act of processing EmPy source
and handling markups is called "expansion."

Code that is processed is executed exactly as if it were entered
into the Python interpreter; that is, it is executed with the
equivalent of `eval` (for expressions) and `exec` (for
statements).  EmPy is intended to be a very thin (though powerful)
layer on top of a running Python system; Python and EmPy files can
be mixed together (via command line options) without
complications.

By default the embedding prefix is the at sign (`@`), which
appears neither in valid Python code nor commonly in arbitrary
texts; it can be overridden with the `-p` option (or with the
`empy.setPrefix` function).  The prefix indicates to the EmPy
interpreter that a special sequence follows and should be
processed rather than sent to the output untouched (to indicate a
literal at sign, it can be doubled as in `@@`).

When the interpreter starts processing its target file, no modules
are imported by default, save the `empy` pseudomodule (see below),
which is placed in the globals; the `empy` pseudomodule is
associated with a particular interpreter -- in fact, they are the
same object -- and it is important that it not be removed from
that interpreter's globals, nor that it be shared with other
interpreters running concurrently (a name other than `empy` can be
specified with the `-m` option).  The globals are not cleared or
reset in any way.  It is perfectly legal to set variables or
explicitly import modules and then use them in later markups,
_e.g._, `@{import time} ... @time.time()`.  Scoping rules are as
in normal Python, although all defined variables and objects are
taken to be in the global namespace.

Indentation is significant in Python, and therefore is also
significant in EmPy.  EmPy statement markups (`@{...}`), when
spanning multiple lines, must be flush with the left margin.  This
is because (multiline) statement markups are not treated specially
in EmPy and are simply passed to the Python interpreter, where
indentation is significant.

Activities you would like to be done before any processing of the
main EmPy file can be specified with the `-I`, `-D`, `-E`, `-F`, and `-P`
options.  `-I` imports modules, `-D` executes a Python variable
assignment, `-E` executes an arbitrary Python (not EmPy) statement,
`-F` executes a Python (not EmPy) file, and `-P` processes an EmPy
(not Python) file.  These operations are done in the order they
appear on the command line; any number of each (including, of
course, zero) can be used.


## Markup

### Expansions

The following markups are supported.  For concreteness below, `@`
is taken for the sake of argument to be the prefix character,
although this can be changed.

**`@# COMMENT NEWLINE`**

: A comment.  Comments, including the trailing newline, are stripped
  out completely.  Comments should only be present outside of
  expansions.  The comment itself is not processed in any way: It is
  completely discarded.  This allows `@#` comments to be used to
  disable markups.  *Note:* As special support for "bangpaths" in
  Unix-like operating systems, if the first line of a file (or indeed
  any context) begins with `#!`, and the interpreter has a
  `processBangpaths` option set to true (default), it is treated as a
  `@#` comment.  A `#!`  sequence appearing anywhere else will be
  handled literally and unaltered in the expansion.  Example:

``````
@# This line is a comment.
@# This will NOT be expanded: @x.
``````

**`@? NAME NEWLINE`**

: Set the name of the current context to be the given string.
  Variables are not allowed here; the name is treated as a literal.
  (If you wish to use arbitrary expressions, use the
  `empy.setContextName` function instead.)  Example:

``````
@?NewName
The context name is now @empy.identify()[0] (NewName).
``````

**`@! INTEGER NEWLINE`**

: Set the line number of the current context to be the given integer
  value; this is similar to the `#line` C preprocessor directive.
  This is done in such a way that the *next* line will have the
  specified numeric value, not the current one.  Expressions are not
  allowed here; the number must be a literal integer.  (If you wish to
  use arbitrary expressions, use the `empy.setContextLine` function
  instead.)  Example:

``````
@!100
The context line is now @empy.identify()[1] (100).
``````

**`@ WHITESPACE`**

: A `@` followed by one whitespace character (a space, horizontal
  tab, vertical tab, carriage return, or newline) is expanded to
  nothing; it serves as a way to explicitly separate two elements
  which might otherwise be interpreted as being the same symbol (such
  as `@name@ s` to mean `@(name)s` -- see below).  Also, since a
  newline qualifies as whitespace here, the lone `@` at the end of a
  line represents a line continuation, similar to the backslash in
  other languages.  Coupled with statement expansion below, spurious
  newlines can be eliminated in statement expansions by use of the
  `@{...}@` construct.  Example:

``````
This will appear as one word: salt@ water.
This is a line continuation; @
this text will appear on the same line.
``````

**`@\ ESCAPE_CODE`**

: An escape code.  Escape codes in EmPy are similar to C-style escape
  codes, although they all begin with the prefix character.  Valid
  escape codes include:

| Escape code | Meaning |
| --- | --- |
| `@\0` | NUL, null |
| `@\a` | BEL, bell |
| `@\b` | BS, backspace |
| `@\d` | three-digital decimal code DDD |
| `@\e` | ESC, escape |
| `@\f` | FF, form feed |
| `@\h` | DEL, delete |
| `@\n` | LF, linefeed character, newline |
| `@\oOOO` | three-digit octal code OOO |
| `@\qQQQQ` | four-digit quaternary code QQQQ |
| `@\r` | CR, carriage return |
| `@\s` | SP, space |
| `@\t` | HT, horizontal tab |
| `@\v` | VT, vertical tab |
| `@\xHH` | two-digit hexadecimal code HH |
| `@\z` | EOT, end of transmission |
| `@^X` | the control character ^X |

  Unlike in C-style escape codes, escape codes taking some number
  of digits afterward always take the same number to prevent
  ambiguities.  Furthermore, unknown escape codes are treated as
  parse errors to discourage potential subtle mistakes.  Note
  that, while `@\0` represents the NUL character, to represent an
  octal code, one must use `@\o...`, in contrast to C.  Example:

``````
This embeds a newline.@\nThis is on the following line.
This beeps!@\a
There is a tab here:@\tSee?
This is the character with octal code 141: @\o141.
``````

**`@@`**

: A literal at sign (`@`).  To embed two adjacent at signs, use
  `@@@@`, and so on.  Any literal at sign that you wish to appear in
  your text must be written this way, so that it will not be processed
  by the system.  *Note:* If a prefix other than `@` has been chosen
  via the command line option, one expresses that literal prefix by
  doubling it, not by appending a `@`.  Example:

``````
The prefix character is @@.
To get the expansion of x you would write @@x.
``````

**`@)`, `@]`, `@}`**

: These expand to literal close parentheses, close brackets, and close
  braces, respectively; these are included for completeness and
  explicitness only.  Example:

``````
This is a close parenthesis: @).
``````

**`@'...'`, `@"..."`, `@'''...'''`, `@"""..."""`**

: These string literals expand to the literals themselves, so
  `@"test"` expands to `test`.  Since they are inherently
  no-operations, the only reason for their use is to override their
  behavior with hooks.

**`@( EXPRESSION )`**

: Evaluate an expression, and expand with the string (via a call to
  `str`) representation evaluation of that expression.  Whitespace
  immediately inside the parentheses is ignored; `@( expression )` is
  equivalent to `@(expression)`.  If the expression evaluates to
  `None`, nothing is expanded in its place; this allows function calls
  that depend on side effects (such as printing) to be called as
  expressions.  (If you really *do* want a `None` to appear in the
  output, then use the Python string `"None"`.)  *Note:* If an
  expression prints something to `sys.stdout` as a side effect, then
  that printing will be spooled to the output *before* the
  expression's return value is.  Example:

``````
2 + 2 is @(2 + 2).
4 squared is @(4**2).
The value of the variable x is @(x).
This will be blank: @(None).
``````

**`@( TEST ? THEN (! ELSE)_opt ($ EXCEPT)_opt )`**

: A special form of expression evaluation representing conditional and
  protected evaluation.  Evaluate the "test" expression; if it
  evaluates to true (in the Pythonic sense), then evaluate the "then"
  section as an expression and expand with the `str` of that result.
  If false, then the "else" section is evaluated and similarly
  expanded.  The "else" section is optional and, if omitted, is
  equivalent to `None` (that is, no expansion will take place).
  *Note*: For backward compatibility, the "else" section delimiter,
  `!`, may be expressed as a `:`.  This behavior is supported but
  deprecated.

  If the "except" section is present, then if any of the prior
  expressions raises an exception when evaluated, the expansion
  will be replaced with the evaluation of the except expression.
  (If the "except" expression itself raises, then that exception
  will be propagated normally.)  The except section is optional
  and, if omitted, is equivalent to `None` (that is, no expansion
  will take place).  An exception (cough) to this is if one of
  these first expressions raises a SyntaxError; in that case the
  protected evaluation lets the error through without evaluating
  the "except" expression.  The intent of this construct is to
  except runtime errors, and if there is actually a syntax error
  in the "try" code, that is a problem that should probably be
  diagnosed rather than hidden.  Example:

``````
What is x? x is @(x ? "true" ! "false").
Pluralization: How many words? @x word@(x != 1 ? 's').
The value of foo is @(foo $ "undefined").
Division by zero is @(x/0 $ "illegal").
``````

**`@ SIMPLE_EXPRESSION`**

: As a shortcut for the `@(...)`  notation, the parentheses can be
  omitted if it is followed by a "simple expression."  A simple
  expression consists of a name followed by a series of function
  applications, array subscriptions, or attribute resolutions, with no
  intervening whitespace.  For example:

  - a name, possibly with qualifying attributes (_e.g._, `@value`,
    `@os.environ`).

  - a straightforward function call (_e.g._, `@min(2, 3)`,
    `@time.ctime()`), with no space between the function name and the
    open parenthesis.

  - an array subscription (_e.g._, `@array[index]`,
    `@os.environ[name]`, with no space between the name and the open
    bracket.

  - any combination of the above (_e.g._,
    `@function(args).attr[sub].other[i](foo)`).

  In essence, simple expressions are expressions that can be written
  ambiguously from text, without intervening space.  Note that
  trailing dots are not considered part of the expansion (_e.g._,
  `@x.` is equivalent to `@(x).`, not `@(x.)`, which would be
  illegal anyway).  Also, whitespace is allowed within parentheses or
  brackets since it is unambiguous, but not between identifiers and
  parentheses, brackets, or dots.  Explicit `@(...)` notation can be
  used instead of the abbreviation when concatenation is what one
  really wants (_e.g._, `@(word)s` for simple pluralization of the
  contents of the variable `word`).  As above, if the expression
  evaluates to the `None` object, nothing is expanded.  Note that
  since a curly appearing where EmPy would expect an open parenthesis
  or bracket in is meaningless in Python, it is treated as a parse
  error (_e.g._, `@x{1, 2}` results in an error).  Example:

``````
The value of x is @x.
The ith value of a is @a[i].
The result of calling f with q is @f(q).
The attribute a of x is @x.a.
The current time is @time.ctime(time.time()).
The current year is @time.localtime(time.time())[0].
These are the same: @min(2,3) and @min(2, 3).
But these are not the same: @min(2, 3) vs. @min (2, 3).
The plural of @name is @(name)s, or @name@ s.
``````

**``@` EXPRESSION ` ``**

: Evaluate a expression, and expand with the `repr` (instead of the
  `str` which is the default) of the evaluation of that expression.
  This expansion is primarily intended for debugging and is unlikely
  to be useful in actual practice.  That is, a `@`...`` is identical
  to `@(repr(...))`.  Example:

``````
The repr of the value of x is @`x`.
This print the Python repr of a module: @`time`.
This actually does print None: @`None`.
``````

**`@: EXPRESSION : DUMMY :`**

: Evaluate an expression and then expand to a `@:`, the original
  expression, a `:`, the evaluation of the expression, and then a `:`.
  The current contents of the dummy area are ignored in the new
  expansion.  In this sense it is self-evaluating; the syntax is
  available for use in situations where the same text will be sent
  through the EmPy processor multiple times.  Example:

``````
This construct allows self-evaluation:
@:2 + 2:this will get replaced with 4:
``````

**`@{ STATEMENTS }`**

: Execute a (potentially compound) statement; statements have no
  return value, so the expansion is not replaced with anything.
  Multiple statements can either be separated on different lines, or
  with semicolons; indentation is significant, just as in normal
  Python code.  Statements, however, can have side effects, including
  printing; output to `sys.stdout` (explicitly or via a `print`
  statement) is collected by the interpreter and sent to the output
  (unless this behavior is suppressed with the `-n` option).  The usual
  Python indentation rules must be followed, although if the statement
  consists of only one statement, leading and trailing whitespace is
  ignored (_e.g._, `@{ print time.time() }` is equivalent to
  `@{print time.time()}`).  Example:

``````
@{x = 123}
@{a = 1; b = 2}
@{print time.time()}
@# Note that extra newlines will appear above because of the
@# newlines trailing the close braces.  To suppress them
@# use a @ before the newline:
@{
for i in range(10):
    print "i is %d" % i
}@
@{print "Welcome to EmPy."}@
``````

**`@% KEY (WHITESPACE VALUE)_opt NEWLINE`**

: Declare a significator.  Significators consume the whole line
  (including the trailing newline), and consist of a key string
  containing no whitespace, and than optional value prefixed by
  whitespace.  The key may not start with or contain internal
  whitespace, but the value may; preceding or following whitespace in
  the value is stripped.  Significators are totally optional, and are
  intended to be used for easy external (that is, outside of EmPy)
  identification when used in large scale environments with many EmPy
  files to be processed.  The purpose of significators is to provide
  identification information about each file in a special,
  easy-to-parse form so that external programs can process the
  significators and build databases, independently of EmPy.  Inside of
  EmPy, when a significator is encountered, its key, value pair is
  translated into a simple assignment of the form `__KEY__ = VALUE` ,
  where "__KEY__" is the key string with two underscores on either
  side and "VALUE" is a Python expression.  Example:

``````
@%title     "Gravitation"
@%author    "Misner", "Thorne", "Wheeler"
@%publisher "W.H. Freeman and Company"
@%pages     1279
@%keywords  'physics', 'gravity', 'Einstein', 'relativity'
@%copyright 1970, 1971
``````

**`@< CONTENTS >`**

: Invoke a custom markup.  The custom markup is a special markup
  reserved for use by the user; it has no prescribed meaning on its
  own.  If `contents` is a string representing what appears in between
  the angle brackets, then expanding this markup is equivalent to
  `empy.invokeCallback(contents)`.  See the "Custom markup" section
  for more information.


### Control

EmPy version 3 and above includes the ability to direct
conditional and repeated expansion of blocks of EmPy code with
control markups (the obsolescent "substitution" markups are
unavailable as of version 3.0).  Control markups have analogs to
control flow structures in Python such as `if/elif/else`, `for`, and
`while`.  Control markups are set off with the `@[...]` notation.

Control markups are designed to be used in precisely the same way
that their internal Python analogues are used, except that the
control markups are intended to be used where there is much more
markup than control structure.

Some control markups are considered "primary," (_e.g._, `if`,
`for`, `while`) as they begin a control markup.  Others are
considered "secondary," since they can only appear inside control
flow markups delineated by primary markups (_e.g._, `elif`,
`else`, `continue`, `break`).

Since EmPy, unlike Python, cannot use indentation to determine
where control structures begin and end, all primary control
markups *must* be followed by a corresponding terminating control
markup:

``````
@[PRIMARY ...]...@[end PRIMARY]
``````

(where `PRIMARY` represents one of the primary keywords).  The end
markup is mandatory, as is the space between the `end` and the
starting keyword.  For instance:

``````
@# If person is alive, show their age.
@person.name is @
@[if person.isAlive]@person.age@[else]dead@[end if].
``````

All primary markups must be terminated in this way, and the
keyword appearing in the appropriate `end` markup must match the
primary markup it corresponds to; if either of these conditions
are not satisfied, the result is a parse error.  Everything
between the starting control flow marker (`@[PRIMARY ...]`) and
the ending marker (`@[end PRIMARY]`) -- including other markups,
even control markups -- is considered part of the markup.  Control
markups can be nested:

``````
@# Print all non-false elements on separate lines.
@[for elem in elements]@[if elem]@elem@\n@[end if]@[end for]
``````

Three major types of primary control markups are available:
conditional (_e.g._, `if`, `try`), looping (_e.g._, `for`,
`while`), and definitional (_e.g._, `def`, discussed below).
Conditional control markups conditionally expand their contents,
whereas looping control markups repeatedly expand their contents.
The third type, definitional markups, will define new objects in
the globals relating to their contents.  Conditional and looping
markups also differ in one substantial respect: Looping constructs
support `@[continue]` and `@[break]` markups which, like their
Python equivalents, continue with the next iteration or break out
of the innermost looping construct, respectively (`@[continue]`
and `@[break]` markups have no meaning inside conditional markups
and are an error).  Also like their Python equivalents,
`@[continue]` and `@[break]` may appear inside nested markups, so
long as they ultimately are contained by at least one looping
control markup:

``````
@# Walk a long a linked list, printing each element.
@[while True]@
@node
@{node = node.next}@
@[if not node]@[break]@[end if]@
@[end while]
``````

The provided markups are designed to mimic the internal Python
control structures as closely as possible.  The supported control
markups are (the phrases in all uppercase are intended to signify
user-selectable patterns):

``````
@[if CONDITION1]...@[elif CONDITION2]...@[else]...@[end if]
@[try]...@[except ...]...@[except ...]...@[end try]
@[try]...@[finally]...@[end try]
@[for VARIABLE in SEQUENCE]...@[else]...@[end for]
@[while CONDITION]...@[else]...@[end while]
@[def SIGNATURE]...@[end def]
``````

All recognizable forms behave like their Python equivalents; `if`
can contain multiple `elif` secondary markups within it; the
`else` markups are optional (but must appear at the end), the
`try` form with the `except` clause can contain multiple ones
which are handled in sequence, the `try` form can either contain
one or more `except` clauses or one `finally` clause (but not
both), and the `for` and `while` structures can contain `continue`
or `break` clauses internally (even if contained within other
markups).

The third type of primary control markup is "definitional," in
that they create objects in the globals for later use (_e.g._,
`def`).  This allows the definition of a callable object which,
when called, will expand the contained markup (which can in turn,
of course, contain further markups).  The argument to the markup
can be any legal Python function signature:

``````
@[def f(x, y, z=2, *args, **keywords)]...@[end def]
``````

would define a function in the globals named `f` that takes the
given arguments.  A macro markup of the form `@[def
SIGNATURE]CODE@[end def]` is equivalent to the Python code:

```python
def SIGNATURE:
    r"""CODE""" # so it is a doc string
    empy.expand(r"""CODE""", locals())
```

That is, it creates a Python function with the same name and
function arguments, whose docstring is the contents of the EmPy
markup that will be expanded when called.  And, when called, it
will expand those contents, with the locals passed in.


## Features

### Unicode support

EmPy version 3.1 and above includes intrinsic Unicode support.
EmPy's Unicode support defers to Python's internal Unicode
support, available in Python 2.0 and up, in order to allow
seamless and transparent translation of different encodings to the
native Python Unicode format.

Knowledge of Python's Unicode support is expected, although not
completely required, to gain full benefit of EmPy's Unicode features.
To enable Unicode support, start EmPy with the `-u/--unicode` option.
EmPy will then transparently encode from the input stream, process
markups internally with native Unicode, and then decode transparently
to the output stream.

By default, Python sets `sys.stdin` and `sys.stdout` with a
default encoding which is accessible via
`sys.getdefaultencoding()`; encodings are represented by string
names.  These streams have encodings set by the system and
*cannot* be changed.

However, encodings for newly created files (files to be read when
specified on the command line, and/or files to be written when
used with the `-o` and `-a` arguments) can be specified for EmPy via
command line options.  The `--unicode-encoding` option
simultaneously indicates the encoding to be used for both input
and output, whereas the `--unicode-input-encoding` and
`--unicode-output-encoding` options can each be used to specify
different encodings for both input and output.  (If an encoding is
not explicitly indicated, it resorts to the system default in
`sys.getdefaultencoding()`, which is locale dependent.)

Python's Unicode implementation has the concept of error handlers,
registered with the `codecs` module, which can be specified to
determine what action should take place if input cannot be decoded
into Unicode, or Unicode cannot be encoded into output.  EmPy uses
these same "errors," as they are called, and can be specified via
command line options.  The three most common error handlers are:
`ignore`, where invalid sequences are simply ignored; `replace`,
where invalid sequences are replaced with an encoding-specific
indicator, usually a question mark; and `strict`, where invalid
sequences raise an error.  The `--unicode-errors` command line
option specifies the same error handler to be used for both input
and output, and the `--unicode-input-errors` and
`--unicode-output-errors` options can specify different error
handlers for input and output.  If an error handler is not
explicitly specified, the `strict` handler (which will raise
errors) is used.

Remember, to specify input encodings or errors that will take
effect, one cannot take input from `sys.stdin` and must explicitly
specify an EmPy file to process on the command line.  Similarly,
for output encodings or errors, `sys.stdout` cannot be used and an
explicit output file must be specified with the `-o` or `-a` options.
It is perfectly valid to enable the Unicode subsystem (`-u` option)
while using `sys.stdin` and `sys.stdout`, but the encodings and
errors of these preexisting streams cannot be changed.

Combined with the `--no-prefix` option, which disables all markup
processing, EmPy can act merely as an encoding translator, relying
on Python's Unicode facilities:

```shell
em.py --no-prefix \
    --unicode-input-encoding=utf-8 \
    --unicode-output-encoding=latin-1 \
    -o filename.Latin-1 filename.UTF-8
```

### Significators

Significators, introduced in EmPy version 1.2, are intended to
represent special assignment in a form that is easy to externally
parse.  For instance, if one has a system that contains many EmPy
files, each of which has its own title, one could use a `title`
significator in each file and use a simple regular expression to
find this significator in each file and organize a database of the
EmPy files to be built.  This is an easier proposition than, for
instance, attempting to grep for a normal Python assignment
(inside a `@{...}` expansion) of the desired variable.

Significators look like the following:

``````
@%KEY VALUE
``````

including the trailing newline, where "key" is a name and "value"
is a Python expression, and are separated by any whitespace.  This
is equivalent to the following Python code:

```python
__KEY__ = VALUE
```

That is to say, a significator key translates to a Python variable
consisting of that key surrounded by double underscores on either
side.  The value may contain spaces, but the key may not.  So:

``````
@%title "All Roads Lead to Rome"
``````

translates to the Python code:

```python
__title__ = "All Roads Lead to Rome"
```

but obviously in a way that easier to detect externally than if
this Python code were to appear somewhere in an expansion.  Since
significator keys are surrounded by double underscores,
significator keys can be any sequence of alphanumeric and
underscore characters; choosing `123` is perfectly valid for a
significator (although strange), since it maps to the name
`__123__` which is a legal Python identifier.

Note the value can be any Python expression.  The value can be
omitted; if missing, it is treated as `None`.

Significators are completely optional; it is completely legal for
a EmPy file or files to be processed without containing any
significators.  Significators can appear anywhere within a file
outside of other markups, but typically they are placed near the
top of the file to make them easy to spot and edit by humans.

A regular expression string designed to match significators (with
the default prefix) is available as `empy.SIGNIFICATOR_RE_STRING`,
and also is a toplevel definition in the `em` module itself.



### Diversions

EmPy supports an extended form of diversions, which are a
mechanism for deferring and recalling output on demand, similar to
the functionality included in m4.  Multiple "streams" of output
can be diverted (deferred) and undiverted (recalled) in this
manner.  A diversion is identified with a name, which is any
immutable object such an integer or string.  When recalled,
diverted code is *not* resent through the EmPy interpreter
(although a filter could be set up to do this).

By default, no diversions take place.  When no diversion is in
effect, processing output goes directly to the specified output
file.  This state can be explicitly requested at any time by
calling the `empy.stopDiverting` function.  It is always legal to
call this function.

When diverted, however, output goes to a deferred location which
can then be recalled later.  Output is diverted with the
`empy.startDiversion` function, which takes an argument that is
the name of the diversion.  If there is no diversion by that name,
a new diversion is created and output will be sent to that
diversion; if the diversion already exists, output will be
appended to that preexisting diversion.

Output send to diversions can be recalled in two ways.  The first
is through the `empy.playDiversion` function, which takes the
name of the diversion as an argument.  This recalls the named
diversion, sends it to the output, and then erases that
diversion.  A variant of this behavior is the
`empy.replayDiversion`, which recalls the named diversion but does
not eliminate it afterwards; `empy.replayDiversion` can be
repeatedly called with the same diversion name, and will replay
that diversion repeatedly.  `empy.createDiversion` create a
diversion without actually diverting to it, for cases where you
want to make sure a diversion exists but do not yet want to send
anything to it.

The diversion object itself can be retrieved with
`empy.retrieveDiversion`.  Diversions act as writable
file-objects, supporting the usual `write`, `writelines`, `flush`,
and `close` methods.  The data that has been diverted to them can
be retrieved in one of two ways; either through the `asString`
method, which returns the entire contents of the diversion as a
single string, or through the `asFile` method, which returns the
contents of the diversion as a readable (not writable) file-like
object.

Diversions can also be explicitly deleted without recalling them
with the `empy.purgeDiversion` function, which takes the desired
diversion name as an argument.

Additionally there are three functions which will apply the above
operations to all existing diversions: `empy.playAllDiversions`,
`empy.replayAllDiversions`, and `empy.purgeAllDiversions`.  All
three will do the equivalent of a `empy.stopDiverting` call before
they do their thing.

The name of the current diversion can be requested with the
`empy.getCurrentDiversion` function; also, the names of all
existing diversions (in sorted order) can be retrieved with
`empy.getAllDiversions`.

When all processing is finished, the equivalent of a call to
`empy.playAllDiversions` is done.


### Filters

EmPy also supports dynamic filters, introduced in version 1.3.
Filters are put in place right "before" the final output file, and
so are only invoked after all other processing has taken place
(including interpreting and diverting).  Filters take input, remap
it, and then send it to the output.

The current filter can be retrieved with the `empy.getFilter`
function.  The filter can be cleared (reset to no filter) with
`empy.resetFilter` and a special "null filter" which does not send
any output at all can be installed with `empy.nullFilter`.  A
custom filter can be set with the `empy.setFilter` function; for
convenience, specialized shortcuts for filters preexist and can be
used in lieu of actual `empy.Filter` instances for the
`empy.setFilter` or `empy.attachFilter` argument:

- `None` is a special filter meaning "no filter"; when installed,
  no filtering whatsoever will take place.  `empy.setFilter(None)`
  is equivalent to `empy.resetFilter()`.

- `0` (or any other numeric constant equal to zero) is another
  special filter that represents the null filter; when installed,
  no output will ever be sent to the filter's sink.

- A filter specified as a function (or lambda) is expected to take
  one string argument and return one string argument; this filter
  will execute the function on any input and use the return value
  as output.

- A filter that is a string is a 256-character table is
  substituted with the result of a call to `string.translate`
  using that table.

- A filter can be an instance of a subclass of `empy.Filter`.
  This is the most general form of filter.  (In actuality, it can
  be any object that exhibits a `Filter` interface, which would
  include the normal file-like `write`, `flush`, and `close`
  methods, as well as `next`, `attach`, and `detach` methods for
  filter-specific behavior.)

- Finally, the argument to `empy.setFilter` can be a Python list
  consisting of one or more of the above objects.  In that case,
  those filters are chained together in the order they appear in
  the list.  An empty list is the equivalent of `None`; all
  filters will be uninstalled.

Filters are, at their core, simply file-like objects (minimally
supporting `write`, `flush`, and `close` methods that behave in
the usual way) which, after performing whatever processing they
need to do, send their work to the next file-like object or filter
in line, called that filter's "sink."  That is to say, filters can
be "chained" together; the action of each filter takes place in
sequence, with the output of one filter being the input of the
next.  Additionally, filters support a `_flush` method (note the
leading underscore) which will always flush the filter's
underlying sink; this method should be not overridden.

Filters also support three additional methods, not part of the
traditional file interface: `attach`, which takes as an argument a
file-like object (perhaps another filter) and sets that as the
filter's "sink" -- that is, the next filter/file-like object in
line.  `detach` (which takes no arguments) is another method which
flushes the filter and removes its sink, leaving it isolated.
Finally, `next` is an accessor method which returns the filter's
sink -- or `None`, if the filter does not yet have a sink
attached.

To create your own filter, you can create an object which supports
the above described interface, or simply derive from the
`empy.Filter` class and override its `write` and possibly `flush`
methods.  You can chain filters together by passing them as
elements in a list to the `empy.setFilter` function, or you can
chain them together manually with the `attach` method:

```python
firstFilter.attach(secondFilter)
empy.setFilter(firstFilter)
```

or just let EmPy do the chaining for you:

```python
empy.setFilter([firstFilter, secondFilter])
```

In either case, EmPy will walk the filter chain and find the end
and then hook that into the appropriate interpreter stream; you
need not do this manually.  The function `empy.attachFilter` can
be used to attach a single filter (or shortcut, as above) to the
end of a currently existing chain.  Note that unlike its cousin
`empy.setFilter`, one cannot pass a sequence of filters (or filter
shortcuts) to `empy.attachFilter`.  (If there is no existing
filter chain installed, `empy.attachFilter` will behave the same
as `empy.setFilter`.)

Subclasses of `empy.Filter` are already provided with the above
null, function, and string functionality described above; they are
`NullFilter`, `FunctionFilter`, and `StringFilter`, respectively.
In addition, a filter which supports buffering, `BufferedFilter`,
is provided.  Several variants are included: `SizeBufferedFilter`,
a filter which buffers into fixed-sized chunks,
`LineBufferedFilter`, a filter which buffers by lines, and
`MaximallyBufferedFilter`, a filter which completely buffers its
input.


### Hooks

The EmPy system allows for the registry of hooks with a running
EmPy interpreter.  Originally introduced in version 2.0 and much
improved in 3.2, hooks are objects, registered with an
interpreter, whose methods represent specific callbacks.  Any
number of hook objects can be registered with an interpreter, and
when a callback is invoked, the associated method on each one of
those hook objects will be called by the interpreter in sequence.

Hooks are simply instances, nominally derived from the `empy.Hook`
class.  The `empy.Hook` class itself defines a series of methods,
with the expected arguments, which would be called by a running
EmPy interpreter.  This scenario, much improved from the prior
implementation in 2.0, allows hooks to keep state and have more
direct access to the interpreter they are running in (the
`empy.Hook` instance contains an `interpreter` attribute).

To use a hook, derive a class from `empy.Hook` and override the
desired methods (with the same signatures as they appear in the
base class).  Create an instance of that subclass, and then
register it with a running interpreter with the `empy.addHook`
function.  (This same hook instance can be removed with the
`empy.removeHook` function.)

More than one hook instance can be registered with an interpreter;
in such a case, the appropriate methods are invoked on each
instance in the order in which they were registered.  To adjust
this behavior, an optional `prepend` argument to the
`empy.addHook` function can be used dictate that the new hook
should placed at the *beginning* of the sequence of hooks, rather
than at the end (which is the default).

All hooks can be enabled and disabled entirely for a given
interpreter; this is done with the `empy.enableHooks` and
`empy.disableHooks` functions.  By default hooks are enabled, but
obviously if no hooks have been registered no hook callbacks will
be made.  Whether hooks are enabled or disabled can be determined
by calling `empy.areHooksEnabled`.  To get a (copy of) the list of
registered hooks, call `empy.getHooks`.  Finally, to invoke a hook
manually, use `empy.invokeHook`.

For a list of supported hook callbacks, see the `empy.Hook` class
definition.

As a practical example, this sample Python code would print a
pound sign followed by the name of every file that is included
with `empy.include`:

```python
class IncludeHook(empy.Hook):
    def beforeInclude(self, name, file, locals):
        print "# %s" % name

empy.addHook(IncludeHook())
```


### Custom markup

Since version 3.2.1, the markup `@<...>` is reserved for
user-defined use.  Unlike the other markups, this markup has no
specified meaning on its own, and can be provided a meaning by the
user.  This meaning is provided with the use of a "custom
callback," or just "callback," which can be set, queried, or reset
using the pseudomodule function.

The custom callback is a callable object which, when invoked, is
passed a single argument: a string representing the contents of
what was found inside the custom markup `@<...>`.

To register a callback, call `empy.registerCallback`.  To remove
one, call `empy.deregisterCallback`.  To retrieve the callback (if
any) registered with the interpreter, use `empy.getCallback`.
Finally, to invoke the callback just as if the custom markup were
encountered, call `empy.invokeCallback`.  For instance, `@<This
text>` would be equivalent to the call `empy.invokeCallback("This
text")`.

By default, to invoke a callback (either explicitly with
`empy.invokeCallback` or by processing a `@<...>` custom markup)
when no callback has been registered is an error.  This behavior
can be changed with the `CALLBACK_OPT` option, or the
`--no-callback-error` command line option.


## Customization

### Pseudomodule

The `empy` pseudomodule is available only in an operating EmPy
system.  (The name of the module, by default `empy`, can be
changed with the `-m` option or the `EMPY_PSEUDO` environment
variable).  It is called a pseudomodule because it is not actually
a module, but rather exports a module-like interface.  In fact,
the pseudmodule is actually the same internal object as the
interpreter itself.

The pseudomodule contains the following functions and objects (and
their signatures, with a suffixed `opt` indicating an optional
argument):

#### Identification-related

**`VERSION`**

: A constant variable which contains a string representation of the
  EmPy version.

**`SIGNIFICATOR_RE_STRING`**

: A constant variable representing a regular expression string (using
  the default prefix) that can be used to find significators in EmPy
  code.

**`SIGNIFICATOR_RE_SUFFIX`**

: The portion of the significator regular expression string excluding
  the prefix, so that those using non-standard prefix can build their
  own custom regular expression string with `myPrefix +
  empy.SIGNIFICATOR_RE_SUFFIX`.

**`interpreter`**

: The instance of the interpreter that is currently being used to
  perform execution.  *Note:* This is now obsolete; the pseudomodule
  is itself the interpreter.  Instead of using `empy.interpreter`,
  simply use `empy`.

**`argv`**

: A list consisting of the name of the primary EmPy script and its
  command line arguments, in analogue to the `sys.argv` list.

**`args`**

: A list of the command line arguments following the primary EmPy
  script; this is equivalent to `empy.argv[1:]`.

**`identify() -> (str, int)`**

: Retrieve identification information about the current parsing
  context.  Returns a 2-tuple consisting of a filename and a line
  number; if the file is something other than from a physical file
  (_e.g._, an explicit expansion with `empy.expand`, a file-like
  object within Python, or via the `-E` or `-F` command line options), a
  string representation is presented surrounded by angle brackets.
  Note that the context only applies to the *EmPy* context, not the
  Python context.

**`atExit(callable)`**

: Register a callable object (such as a function) taking no arguments
  which will be called at the end of a normal shutdown.  Callable
  objects registered in this way are called in the reverse order in
  which they are added, so the first callable registered with
  `empy.atExit` is the last one to be called.  Note that although the
  functionality is related to hooks, `empy.atExit` does no work via
  the hook mechanism, and you are guaranteed that the interpreter and
  stdout will be in a consistent state when the callable is invoked.

#### Context-related

**`pushContext(name_opt, line_opt)`**

: Create a new context with the given name and line and push it on the
  stack.

**`popContext()`**

: Pop the top context and dispose of it.

**`setContextName(name)`**

: Manually set the name of the current context.

**`setContextLine(line)`**

: Manually set the line number of the current context; line must be a
  numeric value.  Note that afterward the line number will increment
  by one for each newline that is encountered, as before.

Globals manipulation:

**`getGlobals()`**

: Retrieve the globals dictionary for this interpreter.  Unlike when
  calling `globals()` in Python, this dictionary *can* be manipulated
  and you *can* expect changes you make to it to be reflected in the
  interpreter that holds it.

**`setGlobals(globals)`**

: Reseat the globals dictionary associated with this interpreter to
  the provided mapping type.

**`updateGlobals(globals)`**

: Merge the given dictionary into this interpreter`s globals.

**`clearGlobals(globals_opt)`**

: Clear out the globals (restoring, of course, the `empy`
  pseudomodule).  Optionally, instead of starting with a refresh
  dictionary, use the dictionary provided.

**`saveGlobals(deep=True)`**

: Save a copy of the globals onto an internal history stack from which
  it can be restored later.  The optional `deep` argument indicates
  whether or not the copying should be a deep copy (default) or a
  shallow one.  Copying is done with `copy.deepcopy` or `copy.copy`,
  respectively.

**`restoreGlobals(destructive=True)`**

: Restore the most recently saved globals from the history stack to as
  the current globals for this instance.  The optional `destructive`
  argument indicates whether or not the restore should remove the
  restored globals from the history stack (default), or whether it
  should be left there for subsequent restores.

#### Execution-related

The following functions allow direct execution; optional `locals`
arguments, if specified, are treated as the locals dictionary in
evaluation and execution:

**`Interpreter`**

: The actual interpreter class.

**`defined(name, locals_opt)`**

: Return true if the given name is defined either in the (optional)
  locals or the interpreter globals; return false otherwise.

**`evaluate(expression, locals_opt)`**

: Evaluate the given expression.

**`serialize(expression, locals_opt)`**

: Serialize the expression, just as the interpreter would: If it is
  not None, convert it to a string with the `str` builtin function,
  and then write out the result.  If it evaluates to None, do nothing.

**`execute(statements, locals_opt)`**

: Execute the given statement(s).

**`single(source, locals_opt)`**

: Interpret the "single" source code, just as the Python interactive
  interpreter would.

**`import_(name, locals_opt)`**

: Import a module.

**`atomic(name, value, locals_opt)`**

: Perform a single, atomic assignment.  In this case name is the
  string denoating the name of the (single) variable to be assigned
  to, and value is a Python object which the name is to be bound to.

**`assign(name, value, locals_opt)`**

: Perform general assignment.  This decays to atomic assignment
  (above) in the normal case, but supports "tuple unpacking" in the
  sense that if name string contains commas, it is treated as a
  sequence of names and memberwise assignment with each member of the
  value (still a Python object, but which must be a sequence).  This
  function will raise a `TypeError` or `ValueError` just like Python
  would if tuple unpacking is not possible (that is, if the value is
  not a sequence or is of an incompatible length, respectively).  This
  only supports the assignment of Python identifiers, not arbitrary
  Python lvalues.

**`significate(key, value_opt, locals_opt)`**

: Do a manual signification.  If `value` is not specified, it is
  treated as `None`.

#### Source-related

**`include(file_or_filename, locals_opt)`**

: Include another EmPy file, by processing it in place.  The argument
  can either be a filename (which is then opened with `open` in text
  mode) or a file object, which is used as is.  Once the included file
  is processed, processing of the current file continues.  Includes
  can be nested.  The call also takes an optional locals dictionary
  which will be passed into the evaluation function.

**`expand(string, locals_opt)` -> str**

: Explicitly invoke the EmPy parsing system to process the given
  string and return its expansion.  This allows multiple levels of
  expansion, _e.g._, `@(empy.expand("@(2 + 2)"))`.  The call also
  takes an optional locals dictionary which will be passed into the
  evaluation function.  This is necessary when text is being expanded
  inside a function definition and it is desired that the function
  arguments (or just plain local variables) are available to be
  referenced within the expansion.

**`quote(string) -> str`**

: The inverse process of `empy.expand`, this will take a string and
  return a new string that, when expanded, would expand to the
  original string.  In practice, this means that appearances of the
  prefix character are doubled, except when they appear inside a
  string literal.

**`escape(string, more_opt) -> str`**

: Given a string, quote the nonprintable characters contained within
  it with EmPy escapes.  The optional `more` argument specifies
  additional characters that should be escaped.

**`flush()`**

: Do an explicit flush on the underlying stream.

**`string(string, name_opt, locals_opt)`**

: Explicitly process a string-like object.  This differs from
  `empy.expand` in that the string is directly processed into the EmPy
  system, rather than being evaluated in an isolated context and then
  returned as a string.

#### Behavior-related

**`flatten(keys_opt)`**

: Perform the equivalent of `from empy import ...` in code (which is
  not directly possible because `empy` is a pseudomodule).  If keys is
  omitted, it is taken as being everything in the `empy` pseudomodule.
  Each of the elements of this pseudomodule is flattened into the
  globals namespace; after a call to `empy.flatten`, they can be
  referred to simple as globals, _e.g._, `@divert(3)` instead of
  `@empy.divert(3)`.  If any preexisting variables are bound to these
  names, they are silently overridden.  Doing this is tantamount to
  declaring an `from ... import ...` which is often considered bad
  form in Python.

#### Prefix-related

**`getPrefix() -> str`**

: Return the current prefix.

**`setPrefix(char)`**

: Set a new prefix.  Immediately after this call finishes, the prefix
  will be changed.  Changing the prefix affects only the current
  interpreter; any other created interpreters are unaffected.  Setting
  the prefix to None or the null string means that no further markups
  will be processed, equivalent to specifying the `--no-prefix` command
  line argument.

#### Diversion-related

**`stopDiverting()`**

: Any diversions that are currently taking place are stopped;
  thereafter, output will go directly to the output file as normal.
  It is never illegal to call this function.

**`createDiversion(name)`**

: Create a diversion, but do not begin diverting to it.  This is the
  equivalent of starting a diversion and then immediately stopping
  diversion; it is used in cases where you want to make sure that a
  diversion will exist for future replaying but may be empty.

**`startDiversion(name)`**

: Start diverting to the specified diversion name.  If such a
  diversion does not already exist, it is created; if it does, then
  additional material will be appended to the preexisting diversions.

**`playDiversion(name)`**

: Recall the specified diversion and then purge it.  The provided
  diversion name must exist.

**`replayDiversion(name)`**

: Recall the specified diversion without purging it.  The provided
  diversion name must exist.

**`purgeDiversion(name)`**

: Purge the specified diversion without recalling it.  The provided
  diversion name must exist.

**`playAllDiversions()`**

: Play (and purge) all existing diversions in the sorted order of
  their names.  This call does an implicit `empy.stopDiverting` before
  executing.

**`replayAllDiversions()`**

: Replay (without purging) all existing diversions in the sorted order
  of their names.  This call does an implicit `empy.stopDiverting`
  before executing.

**`purgeAllDiversions()`**

: Purge all existing diversions without recalling them.  This call
  does an implicit `empy.stopDiverting` before executing.

**`getCurrentDiversion() -> Diversion`**

: Return the name of the current diversion.

**`getAllDiversions() -> list`**

: Return a sorted list of all existing diversions.

#### Filter-related

**`getFilter() -> Filter`**

: Retrieve the current filter.  `None` indicates no filter is
  installed.

**`resetFilter()`**

: Reset the filter so that no filtering is done.

**`nullFilter()`**

: Install a special null filter, one which consumes all text and never
  sends any text to the output.

**`setFilter(shortcut)`**

: Install a new filter.  A filter is `None` or an empty sequence
  representing no filter, or `0` for a null filter, a function for a
  function filter, a string for a string filter, or an instance of
  `empy.Filter` (or a workalike object).  If filter is a list of the
  above things, they will be chained together manually; if it is only
  one, it will be presumed to be solitary or to have already been
  manually chained together.  See the "Filters" section for more
  information.

**`attachFilter(shortcut)`**

: Attach a single filter (sequences are not allowed here) to the end
  of a currently existing filter chain, or if there is no current
  chain, install it as `empy.setFilter` would.  As with
  `empy.setFilter`, the shortcut versions of filters are also allowed
  here.

#### Hook-related

**`areHooksEnabled()`**

: Return whether or not hooks are presently enabled.

**`enableHooks()`**

: Enable invocation of hooks.  By default hooks are enabled.

**`disableHooks()`**

: Disable invocation of hooks.  Hooks can still be added, removed, and
  queried, but invocation of hooks will not occur (even explicit
  invocation with `empy.invokeHook`).

**`getHooks()`**

: Get a (copy of the) list of the hooks currently registered.

**`clearHooks()`**

: Clear all the hooks registered with this interpreter.

**`addHook(hook, prepend_opt)`**

: Add this hook to the hooks associated with this interpreter.  By
  default, the hook is appended to the end of the existing hooks, if
  any; if the optional insert argument is present and true, it will be
  prepended to the list instead.

**`removeHook(hook)`**

: Remove this hook from the hooks associated with this interpreter.

**`invokeHook(_name, ...)`**

: Manually invoke a hook method.  The remaining arguments are treated
  as keyword arguments and the resulting dictionary is passed in as
  the second argument to the hooks.

#### Custom markup-related

**`getCallback() -> callback`**

: Retrieve the current callback associated with this interpreter, or
  `None` if it does not yet have one.

**`registerCallback(callback)`**

: Register a callback to be called whenever a custom markup
  (`@<...>`) is encountered.  When encountered, `invokeCallback` is
  called.

**`deregisterCallback()`**

: Clear any callback previously registered with the interpreter for
  being called when a custom markup is encountered.

**`invokeCallback(contents)`**

: Invoke a custom callback.  This function is called whenever a custom
  markup (`@<...>`) is encountered.  It in turn calls the registered
  callback, with a single argument, `contents`, which is a string
  representing of the contents of the custom markup.


### Invocation

Basic invocation involves running the interpreter on an EmPy file
and some optional arguments.  If no file are specified, or the
file is named `-`, EmPy takes its input from stdin.  One can
suppress option evaluation (to, say, specify a file that begins
with a dash) by using the canonical `--` option.

**`-h`/`--help`**

: Print usage and exit.

**`-H`/`--extended-help`**

: Print extended usage and exit.  Extended usage includes a rundown of
  all the legal expansions, escape sequences, pseudomodule contents,
  used hooks, and supported environment variables.

**`-v`/`--verbose`**

: The EmPy system will print all manner of details about what it is
  doing and what it is processing to stderr.

**`-V`/`--version`**

: Print version and exit.

**`-a`/`--append` (filename)**

: Open the specified file for append instead of using stdout.

**`-b`/`--buffered-output`**

: Fully buffer processing output, including the file open itself.
  This is helpful when, should an error occur, you wish that no output
  file be generated at all (for instance, when using EmPy in
  conjunction with make).  When specified, either the `-o` or `-a` options
  must be specified, and the `-b` option must precede them.  This can
  also be specified through the existence of the
  `EMPY_BUFFERED_OUTPUT` environment variable.

**`-f`/`--flatten`**

: Before processing, move the contents of the `empy` pseudomodule into
  the globals, just as if `empy.flatten()` were executed immediately
  after starting the interpreter.  That is, _e.g._, `empy.include` can
  be referred to simply as `include` when this flag is specified on
  the command line.  This can also be specified through the existence
  of the `EMPY_FLATTEN` environment variable.

**`-i`/`--interactive`**

: After the main EmPy file has been processed, the state of the
  interpreter is left intact and further processing is done from
  stdin.  This is analogous to the Python interpreter's `-i` option,
  which allows interactive inspection of the state of the system after
  a main module is executed.  This behaves as expected when the main
  file is stdin itself.  This can also be specified through the
  existence of the `EMPY_INTERACTIVE` environment variable.

**`-k`/`--suppress-errors`**

: Normally when an error is encountered, information about its
  location is printed and the EmPy interpreter exits.  With this
  option, when an error is encountered (except for keyboard
  interrupts), processing stops and the interpreter enters interactive
  mode, so the state of affairs can be assessed.  This is also
  helpful, for instance, when experimenting with EmPy in an
  interactive manner.  `-k` implies `-i`.

**`-n`/`--no-override-stdout`**

: Do not override `sys.stdout` with a proxy object which the EmPy
  system interacts with.  If suppressed, this means that side effect
  printing will not be captured and routed through the EmPy system.
  However, if this option is specified, EmPy can support
  multithreading.

**`-o`/`--output` (filename)**

: Open the specified file for output instead of using stdout.  If a
  file with that name already exists it is overwritten.

**`-p`/`--prefix` (prefix)**

: Change the prefix used to detect expansions.  The argument is the
  one-character string that will be used as the prefix.  Note that
  whatever it is changed to, the way to represent the prefix literally
  is to double it, so if `$` is the prefix, a literal dollar sign is
  represented with `$$`.  Note that if the prefix is changed to one of
  the secondary characters (those that immediately follow the prefix
  to indicate the type of action EmPy should take), it will not be
  possible to represent literal prefix characters by doubling them
  (_e.g._, if the prefix were inadvisedly changed to `#` then `##`
  would already have to represent a comment, so `##` could not
  represent a literal `#`).  This can also be specified through the
  `EMPY_PREFIX` environment variable.

**`-r`/`--raw-errors`**

: Normally, EmPy catches Python exceptions and prints them alongside
  an error notation indicating the EmPy context in which it occurred.
  This option causes EmPy to display the full Python traceback; this
  is sometimes helpful for debugging.  This can also be specified
  through the existence of the `EMPY_RAW_ERRORS` environment variable.

**`-u`/`--unicode`**

: Enable the Unicode subsystem.  This option only need be present if
  you wish to enable the Unicode subsystem with the defaults; any
  other Unicode-related option (starting with `--unicode...`) will
  also enable the Unicode subsystem.

**`-D`/`--define` (assignment)**

: Execute a Python assignment of the form `variable = expression`.  If
  only a variable name is provided (*i.e.*, the statement does not
  contain an `=` sign), then it is taken as being assigned to None.
  The `-D` option is simply a specialized `-E` option that special cases
  the lack of an assignment operator.  Multiple `-D` options can be
  specified.

**`-E`/`--execute` (statement)**

: Execute the Python (not EmPy) statement before processing any files.
  Multiple `-E` options can be specified.

**`-F`/`--execute-file` (filename)**

: Execute the Python (not EmPy) file before processing any files.
  This is equivalent to `-E execfile("filename")` but provides a more
  readable context.  Multiple `-F` options can be specified.

**`-I`/`--import` (module)**

: Imports the specified module name before processing any files.
  Multiple modules can be specified by separating them by commas, or
  by specifying multiple `-I` options.

**`-P`/`--preprocess` (filename)**

: Process the EmPy file before processing the primary EmPy file on the
  command line.

**`--binary`**

: Treat the file as a binary file, and read in chunks rather than line
  by line.  In this mode, the "line" indicator represents the number
  of bytes read, not the number of lines processed.

**`--no-prefix`**

: Disable the prefixing system entirely; when specified, EmPy will not
  expand any markups.  This allows EmPy to merely act as a Unicode
  encoding translator..

**`--pause-at-end`**

: If present, then `raw_input` will be called at the end of
  processing.  Useful in systems where the output window would
  otherwise be closed by the operating system/window manager
  immediately after EmPy exited.

**`--relative-path`**

: When present, the path the EmPy script being invoked is contained in
  will be prepended to `sys.path`.  This is analogous to Python`s
  internal handling of `sys.path` and scripts.  If input is from stdin
  (`-` for a filename or no filename is specified), then nothing is
  added to the path.

**`--no-callback-error`**

: Do not consider it an error if the custom markup is invoked
  `@<...>` and there is no callback function registered for it.

**`--chunk-size` (chunk)**

: Use the specific binary chunk size rather than the default; implies
  --binary.

**`--unicode-encoding` (encoding)**

: Specify the Unicode encoding to be used for both input and output.

**`--unicode-input-encoding` (encoding)**

: Specify the Unicode encoding to be used for input.

**`--unicode-output-encoding` (encoding)**

: Specify the Unicode encoding to be used for output.

**`--unicode-input-errors` (errors)**

: Specify the Unicode error handling to be used for input.

**`--unicode-errors` (errors)**

: Specify the Unicode error handling to be used for both input and
  output.

**`--unicode-output-errors` (errors)**

: Specify the Unicode error handling to be used for output.


### Environment variables

EmPy also supports a few environment variables to predefine
certain behaviors.  The settings chosen by environment variables
can be overridden via command line arguments.  The following
environment variables have meaning to EmPy:

**`EMPY_OPTIONS`**

: If present, the contents of this environment variable will be
  treated as options, just as if they were entered on the command
  line, *before* the actual command line arguments are processed.
  Note that these arguments are *not* processed by the shell, so
  quoting, filename globbing, and the like, will not work.

**`EMPY_PREFIX`**

: If present, the value of this environment variable represents the
  prefix that will be used; this is equivalent to the `-p` command line
  option.

**`EMPY_PSEUDO`**

: If present, the value of this environment variable represents the
  name of the pseudomodule that will be incorporated into every
  running EmPy system; this is equivalent to the `-m` command line
  option.

**`EMPY_FLATTEN`**

: If defined, this is equivalent to including `-f` on the command line.

**`EMPY_RAW_ERRORS`**

: If defined, this is equivalent to including `-r` on the command line.

**`EMPY_INTERACTIVE`**

: If defined, this is equivalent to including `-i` on the command line.

**`EMPY_BUFFERED_OUTPUT`**

: If defined, this is equivalent to including `-b` on the command line.

**`EMPY_UNICODE`**

: If defined, this is equivalent to including `-u` on the command line.

**`EMPY_UNICODE_INPUT_ENCODING`**

: If present, the value of this environment variable indicates the
  name of the Unicode input encoding to be used.  This is equivalent
  to the `--unicode-input-encoding` command line option.

**`EMPY_UNICODE_OUTPUT_ENCODING`**

: If present, the value of this environment variable indicates the
  name of the Unicode output encoding to be used.  This is equivalent
  to the `--unicode-output-encoding` command line option.

**`EMPY_UNICODE_INPUT_ERRORS`**

: If present, the value of this environment variable indicates the
  name of the error handler to be used for input.  This is equivalent
  to the `--unicode-input-errors` command line option.

**`EMPY_UNICODE_OUTPUT_ERRORS`**

: If present, the value of this
  environment variable indicates the name of the error handler to
  be used for output.  This is equivalent to the
  `--unicode-output-errors` command line option.


### Examples and testing EmPy

See the sample EmPy file `sample.em` which is included with the
distribution.  Run EmPy on it by typing something like:

```shell
./em.py sample.em
```

and compare the results and the sample source file side by side.
The sample content is intended to be self-documenting, and even an
introduction to the basic features of EmPy while simultaneously
exercising them.

The file `sample.bench` is the benchmark output of the sample.
Running the EmPy interpreter on the provided `sample.em` file
should produce precisely the same results.  You can run the
provided test script to see if your EmPy environment is behaving
as expected (presuming a Unix-like operating system):

```shell
./test.sh
```

By default this will test with the first Python interpreter
available in the path; if you want to test with another
interpreter, you can provide it as the first argument on the
command line, _e.g._:

```shell
./test.sh python2.1
./test.sh /usr/bin/python1.5
./test.sh jython
```

A more comprehensive test suite and set of real-world examples is
planned for a future version.


### Embedding EmPy

For atomic applications, the `expand` function is provided (the
extra keyword arguments passed in are treated as locals):

```python
import em
print em.expand("@x + @y is @(x + y).", x=2, y=3)
```

One can specify a globals dictionary and all the other interpreter
options (below) as well.  One can specify a globals dictionary
that will be used if one wants persistence:

```python
import em
g = {}
em.expand("@{x = 10}", g)
print em.expand("x is @x.", g)
```

The standalone `expand` function, however, creates and destroys an
`Interpreter` instance each time it is called.  For repeated
expansions, this can be expensive.  Instead, you will probably
want to use the full-fledged features of embedding.  An EmPy
interpreter can be created with as code as simple as:

```python
import em
interpreter = em.Interpreter()
## The following prints the results to stdout:
interpreter.string("123\n")
## This expands to the same thing, but puts the results as a
## string in the variable result:
result = interpreter.expand("123\n")
## This just prints the value of x directly:
print interpreter.globals['x']
## Process an actual file (and output to stdout):
interpreter.file(open('/path/to/some/file'))
interpreter.shutdown() # this is important; see below
```

One can capture the output of a run in something other than stdout
by specifying the *output* parameter:

```python
import em, StringIO
output = StringIO.StringIO()
interpreter = em.Interpreter(output=output)
## Do something.
interpreter.file(open('/path/to/some/file'))
interpreter.shutdown() # again, this is important; see below
print output.getvalue() # this is the result from the session
```

When you are finished with your interpreter, it is important to
call its shutdown method:

```python
interpreter.shutdown()
```

This will ensure that the interpreter cleans up all its overhead,
entries in the `sys.stdout` proxy, and so forth.  It is usually
advisable that this be used in a try...finally clause:

```python
interpreter = em.Interpreter(...)
try:
    ...
finally:
    interpreter.shutdown()
```
 
The `em.Interpreter` constructor takes the following arguments; all
are optional.  Since options may be added in the future, it is highly
recommended that the constructor be invoked via keyword arguments,
rather than assuming their order.  The arguments are:

*output*

: The output file which the interpreter will be sending all its
 processed data to.  This need only be a file-like object; it need not
 be an actual file.  If omitted, `sys.__stdout__` is used.

*argv*

: An argument list analogous to `sys.argv`, consisting of the script
 name and zero or more arguments.  These are available to executing
 interpreters via `empy.argv` and `empy.args`.  If omitted, a
 non-descript script name is used with no arguments.

*prefix*

: The prefix (a single-character string).  Defaults to ``.  It is an
 error for this to be anything other than one character.

*pseudo*

: The name (string) of the pseudmodule.  Defaults to `empy`.

*options*

: A dictionary of options that can override the default behavior of
 the interpreter.  The names of the options are constant names ending
 in `_OPT` and their defaults are given in
 `Interpreter.DEFAULT_OPTIONS`.

*globals*

: By default, interpreters begin with a pristine dictionary of globals
 (except, of course, for the `empy` pseudomodule).  Specifying this
 argument will allow the globals to start with more.

*hooks*

: A sequence of hooks (or `None` for none) to register with the
 interpreter at startup.  Hooks can, of course, be added after the
 fact, but this allows the hooks to intercept the `atStartup` event
 (otherwise, the startup event would already have occurred by the time
 new hooks could be registered)..

Many things can be done with EmPy interpreters; for the full developer
documentation, see the generated documentation for the `em` module.


### Interpreter options

The following options (passed in as part of the options dictionary to
the Interpreter constructor) have the following meanings.  The
defaults are shown below and are also indicated in an
`Interpreter.DEFAULT_OPTIONS` dictionary.

**`BANGPATH_OPT`**

: Should a bangpath (`#!`) as the first line of an EmPy file be
  treated as if it were an EmPy comment?  Note that `#!` sequences
  starting lines or appearing anywhere else in the file are untouched
  regardless of the value of this option.  Default: true.

**`BUFFERED_OPT`**

: Should an `abort` method be called upon failure?  This relates to
  the fully-buffered option, where all output can be buffered
  including the file open; this option only relates to the
  interpreter's behavior *after* that proxy file object has been
  created.  Default: false.

**`RAW_OPT`**

: Should errors be displayed as raw Python errors (that is, the
  exception is allowed to propagate through to the toplevel so that
  the user gets a standard Python traceback)?  Default: false.

**`EXIT_OPT`**

: Upon an error, should execution continue (although the interpreter
  stacks will be purged)?  Note that even in the event this is set,
  the interpreter will halt upon receiving a `KeyboardInterrupt`.
  Default: true.

**`FLATTEN_OPT`**

: Upon initial startup, should the `empy` pseudomodule namespace be
  flattened, *i.e.*, should `empy.flatten` be called?  Note this
  option only has an effect when the interpreter is first created;
  thereafter it is ignored.  Default: false.

**`OVERRIDE_OPT`**

: Should the `sys.stdout` object be overridden with a proxy object?
  If not, side effect output cannot be captured by the EmPy system,
  but EmPy will support multithreading.  Default: true.

**`CALLBACK_OPT`**

: If a callback is invoked when none has yet been registered, should
  an error be raised or should the situation be ignored?  Default:
  true.


## Reference

### Data flow

**input ⟶ interpreter ⟶ diversions ⟶ filters ⟶ output**

Here, in summary, is how data flows through a working EmPy system:

1. Input comes from a source, such an .em file on the command line, or
   via an `empy.include` statement.

2. The interpreter processes this material as it comes in, expanding
   EmPy expansions as it goes.

3. After interpretation, data is then sent through the diversion
   layer, which may allow it directly through (if no diversion is in
   progress) or defer it temporarily.  Diversions that are recalled
   initiate from this point.

4. Any filters in place are then used to filter the data and produce
   filtered data as output.

5. Finally, any material surviving this far is sent to the output
   stream.  That stream is stdout by default, but can be changed with
   the `-o` or `-a` options, or may be fully buffered with the `-b` option
   (that is, the output file would not even be opened until the entire
   system is finished).


### Author's notes

I originally conceived EmPy as a replacement for my [Web templating
system](http://www.alcyone.com/max/info/m4.html) which uses [GNU
m4](http://www.seindal.dk/rene/gnu/), a general macroprocessing system
for Unix.

Most of my Web sites include a variety of m4 files, some of which
are dynamically generated from databases, which are then scanned
by a cataloging tool to organize them hierarchically (so that,
say, a particular m4 file can understand where it is in the
hierarchy, or what the titles of files related to it are without
duplicating information); the results of the catalog are then
written in database form as an m4 file (which every other m4 file
implicitly includes), and then GNU make converts each m4 to an
HTML file by processing it.

As the Web sites got more complicated, the use of m4 (which I had
originally enjoyed for the challenge and abstractness) really
started to become an impediment to serious work; while I am very
knowledgeable about m4 -- having used it for for so many years --
getting even simple things done with it is awkward and difficult.
Worse yet, as I started to use Python more and more over the
years, the cataloging programs which scanned the m4 and built m4
databases were migrated to Python and made almost trivial, but
writing out huge awkward tables of m4 definitions simply to make
them accessible in other m4 scripts started to become almost
farcical -- especially when coupled with the difficulty in getting
simple things done in m4.

It occurred to me what I really wanted was an all-Python solution.
But replacing what used to be the m4 files with standalone Python
programs would result in somewhat awkward programs normally
consisting mostly of unprocessed text punctuated by small portions
where variables and small amounts of code need to be substituted.
Thus the idea was a sort of inverse of a Python interpreter: a
program that normally would just pass text through unmolested, but
when it found a special signifier would execute Python code in a
normal environment.  I looked at existing Python templating
systems, and didn't find anything that appealed to me -- I wanted
something where the desired markups were simple and unobtrusive.
After considering between choices of signifiers, I settled on `@`
and EmPy was born.

As I developed the tool, I realized it could have general appeal,
even to those with widely varying problems to solve, provided the
core tool they needed was an interpreter that could embed Python
code inside templated text.  As I continue to use the tool, I have
been adding features as unintrusively as possible as I see areas
that can be improved.

A design goal of EmPy is that its feature set should work on
several levels; at each level, if the user does not wish or need
to use features from another level, they are under no obligation
to do so.  If you have no need of diversions, for instance, you
are under no obligation to use them.  If significators will not
help you organize a set of EmPy scripts globally, then you need
not use them.  New features that are being added are whenever
possible transparently backward compatible; if you do not need
them, their introduction should not affect you in any way.  The
use of unknown prefix sequences results in errors, guaranteeing
that they are reserved for future use.


### Glossary

*control*

: A control markup, used to direct high-level control flow within an
  EmPy session.  Control markups are expressed with the `@[...]`
  notation.

*diversion*

: A process by which output is deferred, and can be recalled later on
  demand, multiple times if necessary.

*document*

: The abstraction of an EmPy document as used by a processor.

*escape*

: A markup designed to expand to a single (usually non-printable)
  character, similar to escape sequences in C or other languages.

*expansion*

: The process of processing EmPy markups and producing output.

*expression*

: An expression markup represents a Python expression to be evaluated,
  and replaced with the `str` of its value.  Expression markups are
  expressed with the `@(...)`  notation.

*filter*

: A file-like object which can be chained to other objects (primarily
  the final stream) and can buffer, alter, or manipulate in any way
  the data sent.  Filters can also be chained together in arbitrary
  order.

*globals*

: The dictionary (or dictionary-like object) which resides inside the
  interpreter and holds the currently-defined variables.

*hook*

: A callable object that can be registered in a dictionary, and which
  will be invoked before, during, or after certain internal
  operations, identified by name with a string.

*interpreter*

: The application (or class instance) which processes EmPy markup.

*markup*

: EmPy substitutions set off with a prefix and appropriate delimeters.

*output*

: The final destination of the result of processing an EmPy file.

*prefix*

: The ASCII character used to set off an expansions.  By default,
  `@`.

*processor*

: An extensible system which processes a group of EmPy files, usually
  arranged in a filesystem, and scans them for significators.

*pseudomodule*

: The module-like object named `empy` which is exposed internally
  inside every EmPy system.

*shortcut*

: A special object which takes the place of an instance of the
  `Filter` class, to represent a special form of filter.  These
  include 0 for a null filter, a callable (function or lambda) to
  represent a callable filter, or a 256-character string which
  represents a translation filter.

*significator*

: A special form of an assignment markup in EmPy which can be easily
  parsed externally, primarily designed for representing uniform
  assignment across a collection of files.  Significators are
  indicated with the `@%` markup.

*statement*

: A line of code that needs to be executed; statements do not have
  return values.  In EmPy, statements are set off with `@{...}`.


### Acknowledgements

Questions, suggestions, bug reports, evangelism, and even
complaints from many people have helped make EmPy what it is
today.  Some, but by no means all, of these people are (in
alphabetical order by surname):

- Biswapesh Chattopadhyay
- Beni Cherniavsky
- Dr. S. Candelaria de Ram
- Eric Eide
- Dinu Gherman
- Grzegorz Adam Hankiewicz
- Bohdan Kushnir
- Robert Kroeger
- Kouichi Takahashi
- Ville Vainio


### Known issues and caveats

- EmPy was primarily intended for static processing of documents,
  rather than dynamic use, and hence speed of processing was not
  the primary consideration in its design.

- EmPy is not threadsafe by default.  This is because of the need
  for EmPy to override the `sys.stdout` file with a proxy object
  which can capture effects of `print` and other spooling to
  stdout.  This proxy can be suppressed with the `-n` option, which
  will result in EmPy being unable to do anything meaningful with
  this output, but will allow EmPy to be threadsafe.

- To function properly, EmPy must override `sys.stdout` with a
  proxy file object, so that it can capture output of side effects
  and support diversions for each interpreter instance.  It is
  important that code executed in an environment *not* rebind
  `sys.stdout`, although it is perfectly legal to invoke it
  explicitly (_e.g._, `@sys.stdout.write("Hello world\n")`).  If
  one really needs to access the "true" stdout, then use
  `sys.__stdout__` instead (which should also not be rebound).
  EmPy uses the standard Python error handlers when exceptions are
  raised in EmPy code, which print to `sys.stderr`.

- Due to Python's curious handling of the `print` statement --
  particularly the form with a trailing comma to suppress the
  final newline -- mixing statement expansions using prints inline
  with unexpanded text will often result in surprising behavior,
  such as extraneous (sometimes even deferred!) spaces.  This is a
  Python "feature," and occurs in non-EmPy applications as well;
  for finer control over output formatting, use `sys.stdout.write`
  or `empy.interpreter.write` directly.

- The `empy` "module" exposed through the EmPy interface (_e.g._,
  `@empy`) is an artificial module.  It cannot be imported with
  the `import` statement (and shouldn't -- it is an artifact of
  the EmPy processing system and does not correspond to any
  accessible .py file).

- For an EmPy statement expansion all alone on a line, _e.g._,
  `@{a = 1}`, note that this will expand to a blank line due to
  the newline following the closing curly brace.  To suppress this
  blank line, use the symmetric convention `@{a = 1}@`.

- When using EmPy with make, note that partial output may be
  created before an error occurs; this is a standard caveat when
  using make.  To avoid this, write to a temporary file and move
  when complete, delete the file in case of an error, use the `-b`
  option to fully buffer output (including the open), or (with GNU
  make) define a `.DELETE_ON_ERROR` target.

-  `empy.identify` tracks  the context  of executed  _EmPy_ code,  not
  Python code.   This means that  blocks of code delimited  with `@{`
  and `}` will  identify themselves as appearing on the  line at which
  the `}`  appears, and that pure  Python code executed via  the `-D`,
  `-E` and  `-F` command  line arguments  will show  up as  all taking
  place  on  line  1.   If   you're  tracking  errors  and  want  more
  information about the  location of the errors from  the Python code,
  use the  `-r` command line option,  which will provide you  with the
  full Python traceback.

- The conditional form of expression expansion `@(...?...!...)`
  allows the use of a colon instead of an exclamation point,
  _e.g._, `@(...?...:...)`.  This behavior is supported for
  backward compatibility, but is deprecated.  Due to an oversight,
  the colon was a poor choice since colons can appear legally in
  expressions (_e.g._, dictionary literals or lambda expressions).

- The `@[try]` construct only works with Python exceptions derived
  from `Exception`.  It is not able to catch string exceptions.

- The `@[for]` variable specification supports tuples for tuple
  unpacking, even recursive tuples.  However, it is limited in
  that the names included may only be valid Python identifiers,
  not arbitrary Python lvalues.  Since the internal Python
  mechanism is very rarely used for this purpose (_e.g._, `for (x,
  l[0], q.a) in sequence`), it is not thought to be a significant
  limitation.


### Wish list

Here are some random ideas for future revisions of EmPy.  If any
of these are of particular interest to you, your input would be
appreciated.

- Some real-world examples should really be included for
  demonstrating the power and expressiveness of EmPy first-hand.

- More extensive help (rather than a ridiculously long README),
  probably inherently using the EmPy system itself for building to
  HTML and other formats, thereby acting as a help facility and a
  demonstration of the working system.

- A "trivial" mode, where all the EmPy system does is scan for
  simple symbols to replace them with evaluations/executions,
  rather than having to do the contextual scanning it does now.
  This has the down side of being much less configurable and
  powerful but the upside of being extremely efficient.

- A "debug" mode, where EmPy prints the contents of everything
  it's about to evaluate (probably to stderr) before it does?

- The ability to funnel all code through a configurable `RExec`
  for user-controlled security control.  This would probably
  involve abstracting the execution functionality outside of the
  interpreter.  [This suggestion is on hold until the
  rexec/Bastion exploits are worked out.]

- Optimized handling of processing would be nice for the
  possibility of an Apache module devoted to EmPy processing.

- An EmPy emacs mode.

- An optimization of offloading diversions to files when they
  become truly huge.  (This is made possible by the abstraction of
  the `Diversion` class.)

- Support for mapping filters (specified by dictionaries).

- Support for some sort of batch processing, where several EmPy
  files can be listed at once and all of them evaluated with the
  same initial (presumably expensive) environment.
  `empy.saveGlobals` and `empy.restoreGlobals` have been
  introduced as a partial solution, but they need to be made more
  robust.

- A more elaborate interactive mode, perhaps with a prompt and
  readline support.

- A StructuredText and/or reStructuredText filter would be quite
  useful, as would SGML/HTML/XML/XHTML, s-expression, Python,
  etc. auto-indenter filters.

- An indexing filter, which can process text and pick out
  predefined keywords and thereby setup links to them.

- The ability to rerun diverted material back through the
  interpreter.  (This can be done, awkwardly, by manually creating
  a filter which itself contains an interpreter, but it might be
  helpful if this was an all-in-one operation.)

- A caching system that stores off the compilations of repeated
  evaluations and executions so that in a persistent environment
  the same code does not have to be repeatedly evaluated/executed.
  This would probably be a necessity in an Apache module-based
  solution.  Perhaps caching even to the point of generating pure
  PyWM bytecode?

- An option to change the format of the standard EmPy errors in a
  traceback.

- Support for some manner of implicitly processed /etc/empyrc
  and/or ~/.empyrc file, and of course an option to inhibit its
  processing.  This can already be accomplished (and with greater
  control) via use of EMPY_OPTIONS, though.

- More uniform handling of the preprocessing directives (`-I`, `-D`,
  `-E`, `-F`, and `-P`), probably mapping directly to methods in the
  `Interpreter` class.

- Support for integration with mod_python.

- In simple expressions, a `{...}` suffix has no meaning in Python
  (_e.g._, in Python, `@x(...)` is a call, `@x[...]` is
  subscription, but `@x{...}` is illegal).  This could be
  exploited by having a `{...}` suffix in a simple expression
  representing an encapsulation of an expanded string; _e.g._,
  `@bullet{There are @count people here}` would be equivalent to
  `@bullet(empy.expand("There are @count people here",
  locals()))}`.

- A tool to collect significator information from a hierarchy of
  .em files and put them in a database form available for
  individual scripts would be extremely useful -- this tool should
  be extensible so that users can use it to, say, build ordered
  hierarchies of their EmPy files by detecting contextual
  information like application-specific links to other EmPy
  documents.

- Extensions of the basic EmPy concepts to projects for other
  interpreted languages, such as Java, Lua, Ruby, and/or Perl.

- Ignore `SystemExit` when doing error handling, letting the
  exception progagate up?  So far no one seems to worry about
  this; deliberately exiting early in a template seems to be an
  unlikely occurrence.  (Furthermore, there are the `os.abort` and
  `os._exit` facilities for terminating without exception
  propagation.)

- The ability to obtain the filename (if relevant) and mode of the
  primary output file.

- The ability to redirect multiple streams of output; not
  diversions, but rather the ability to write to one file and then
  another.  Since output would be under the EmPy script's control,
  this would imply a useful `--no-output` option, where by default
  no output is written.  This would also suggest the usefulness of
  all the output file delegates (diversions, filters, abstract
  files, etc.) passing unrecognized method calls all the way down
  to underlying file object.

- In addition to the em.py script, an additional support library
  (non-executable) should be included which includes ancillary
  functionality for more advanced features, but which is not
  necessary to use EmPy in its basic form as a standalone
  executable.  Such features would include things like
  significator processing, metadata scanning, and advanced
  prompting systems.


### Release history

3.3.4a (2021 Nov 19)

: Fix an error in the setup.py in the downloadable tarball (did not
  affect PIP downloads).

3.3.4 (2019 Feb 26)

: Minor fix for a Python 3 compatibilty issue.

3.3.3 (2017 Feb 12)

: Fix for `defined` call.

3.3.2 (2014 Jan 24)

: Additional fix for source compatibility between 2._x_ and 3.0.

3.3.1 (2014 Jan 22)

: Source compatibility for 2._x_ and 3.0; 1._x_ and Jython
  compatibility dropped.

3.3 (2003 Oct 27)

: Custom markup `@<...>`; remove separate pseudomodule instance for
  greater transparency; deprecate `interpreter` attribute of
  pseudomodule; deprecate auxiliary class name attributes associated
  with pseudomodule in preparation for separate support library in
  4.0; add `--no-callback-error` and `--no-bangpath-processing` command
  line options; add `atToken` hook.

3.2 (2003 Oct 7)

: Reengineer hooks support to use hook instances; add `-v` option; add
  `--relative-path` option; reversed PEP 317 style; modify Unicode
  support to give less confusing errors in the case of unknown
  encodings and error handlers; relicensed under LGPL.

3.1.1 (2003 Sep 20)

: Add literal `@"..."` markup; add `--pause-at-end` command line
  option; fix improper globals collision error via the `sys.stdout`
  proxy.

3.1 (2003 Aug 8)

: Unicode support (Python 2.0 and above); add Document and Processor
  helper classes for processing significators; add `--no-prefix` option
  for suppressing all markups.

3.0.4 (2003 Aug 7)

: Implement somewhat more robust lvalue parsing for `@[for]`
  construct (thanks to Beni Cherniavsky for inspiration).

3.0.3 (2003 Jul 9)

: Fix bug regarding recursive tuple unpacking using `@[for]`; add
  `empy.saveGlobals`, `empy.restoreGlobals`, and `empy.defined`
  functions.

3.0.2 (2003 Jun 19)

: `@?` and `@!` markups for changing the current context name and
  line, respectively; add `update` method to interpreter; new and
  renamed context operations, `empy.setContextName`,
  `empy.setContextLine`, `empy.pushContext`, `empy.popContext`.

3.0.1 (2003 Jun 9)

: Fix simple bug preventing command line preprocessing directives (`-I`,
  `-D`, `-E`, `-F`, `-P`) from executing properly; defensive PEP 317
  compliance [defunct].

3.0 (2003 Jun 1)

: Control markups with `@[...]`; remove substitutions (use control
  markups instead); support `@(...?...!...)` for conditional
  expressions in addition to the now-deprecated `@(...?...:...)`
  variety; add acknowledgements and glossary sections to
  documentation; rename buffering option back to `-b`; add `-m` option
  and `EMPY_PSEUDO` environment variable for changing the pseudomodule
  name; add `-n` option and `EMPY_NO_OVERRIDE` environment variable
  for suppressing `sys.stdout` proxy; rename main error class to
  `Error`; add standalone `expand` function; add `--binary` and
  `--chunk-size` options; reengineer parsing system to use Tokens for
  easy extensibility; safeguard curly braces in simple expressions
  (meaningless in Python and thus likely a typographical error) by
  making them a parse error; fix bug involving custom Interpreter
  instances ignoring globals argument; distutils support.

2.3 (2003 Feb 20)

: Proper and full support for concurrent and recursive interpreters;
  protection from closing the true stdout file object; detect edge
  cases of interpreter globals or `sys.stdout` proxy collisions; add
  globals manipulation functions `empy.getGlobals`, `empy.setGlobals`,
  and `empy.updateGlobals` which properly preserve the `empy`
  pseudomodule; separate usage info out into easily accessible lists
  for easier presentation; have `-h` option show simple usage and `-H`
  show extened usage; add `NullFile` utility class.

2.2.6 (2003 Jan 30)

: Fix a bug in the `Filter.detach` method (which would not normally be
  called anyway).

2.2.5 (2003 Jan 9)

: Strip carriage returns out of executed code blocks for DOS/Windows
  compatibility.

2.2.4 (2002 Dec 23)

: Abstract Filter interface to use methods only; add `@[noop: ...]`
  substitution for completeness and block commenting [defunct].

2.2.3 (2002 Dec 16)

: Support compatibility with Jython by working around a minor
  difference between CPython and Jython in string splitting.

2.2.2 (2002 Dec 14)

: Include better docstrings for pseudomodule functions; segue to a
  dictionary-based options system for interpreters; add
  `empy.clearAllHooks` and `empy.clearGlobals`; include a short
  documentation section on embedding interpreters; fix a bug in
  significator regular expression.

2.2.1 (2002 Nov 30)

: Tweak test script to avoid writing unnecessary temporary file; add
  `Interpreter.single` method; expose `evaluate`, `execute`,
  `substitute` [defunct], and `single` methods to the pseudomodule;
  add (rather obvious) `EMPY_OPTIONS` environment variable support;
  add `empy.enableHooks` and `empy.disableHooks`; include optimization
  to transparently disable hooks until they are actually used.

2.2 (2002 Nov 21)

: Switched to `-V` option for version information;
  `empy.createDiversion` for creating initially empty diversion;
  direct access to diversion objects with `empy.retrieveDiversion`;
  environment variable support; removed `--raw` long argument (use
  `--raw-errors` instead); added quaternary escape code (well, why
  not).

2.1 (2002 Oct 18)

: `empy.atExit` registry separate from hooks to allow for normal
  interpreter support; include a benchmark sample and test.sh
  verification script; expose `empy.string` directly; `-D` option for
  explicit defines on command line; remove ill-conceived support for
  `@else:` separator in `@[if ...]`  substitution [defunct] ; handle
  nested substitutions properly [defunct] ; `@[macro ...]`
  substitution for creating recallable expansions [defunct].

2.0.1 (2002 Oct 8)

: Fix missing usage information; fix after_evaluate hook not getting
  called; add `empy.atExit` call to register values.

2.0 (2002 Sep 30)

: Parsing system completely revamped and simplified, eliminating a
  whole class of context-related bugs; builtin support for buffered
  filters; support for registering hooks; support for command line
  arguments; interactive mode with `-i`; significator value extended
  to be any valid Python expression.

1.5.1 (2002 Sep 24)

: Allow `@]` to represent unbalanced close brackets in `@[...]`
  markups [defunct].

1.5 (2002 Sep 18)

: Escape codes (`@\...`); conditional and repeated expansion
  substitutions [defunct] ; replaced with control
  markups]; fix a few bugs involving files which do not end in
  newlines.

1.4 (2002 Sep 7)

: Fix bug with triple quotes; collapse conditional and protected
  expression syntaxes into the single generalized `@(...)` notation;
  `empy.setName` and `empy.setLine` functions [deprecated] ; true
  support for multiple concurrent interpreters with improved
  sys.stdout proxy; proper support for `empy.expand` to return a
  string evaluated in a subinterpreter as intended; merged Context and
  Parser classes together, and separated out Scanner functionality.

1.3 (2002 Aug 24)

: Pseudomodule as true instance; move toward more verbose (and clear)
  pseudomodule functions; fleshed out diversion model; filters;
  conditional expressions; protected expressions; preprocessing with
  `-P` (in preparation for possible support for command line arguments).

1.2 (2002 Aug 16)

: Treat bangpaths as comments; `empy.quote` for the opposite process
  of `empy.expand`; significators (`@%...`  sequences); `-I` option;
  `-f` option; much improved documentation.

1.1.5 (2002 Aug 15)

: Add a separate `invoke` function that can be called multiple times
  with arguments to simulate multiple runs.

1.1.4 (2002 Aug 12)

: Handle strings thrown as exceptions properly; use getopt to process
  command line arguments; cleanup file buffering with AbstractFile;
  very slight documentation and code cleanup.

1.1.3 (2002 Aug 9)

: Support for changing the prefix from within the `empy` pseudomodule.

1.1.2 (2002 Aug 5)

: Renamed buffering option [defunct], added `-F` option for interpreting
  Python files from the command line, fixed improper handling of
  exceptions from command line options (`-E`, `-F`).

1.1.1 (2002 Aug 4)

: Typo bugfixes; documentation clarification.

1.1 (2002 Aug 4)

: Added option for fully buffering output (including file opens),
  executing commands through the command line; some documentation
  errors fixed.

1.0 (2002 Jul 23)

: Renamed project to EmPy.  Documentation and sample tweaks; added
  `empy.flatten`.  Added `-a` option.

0.3 (2002 Apr 14)

: Extended "simple expression" syntax, interpreter abstraction, proper
  context handling, better error handling, explicit file inclusion,
  extended samples.

0.2 (2002 Apr 13)

: Bugfixes, support non-expansion of Nones, allow choice of alternate
  prefix.

0.1.1 (2002 Apr 12)

: Bugfixes, support for Python 1.5._x_, add `-r` option.

0.1 (2002 Apr 12)

: Initial early access release.


## End notes

### Contact

This software was written by [Erik Max
Francis](http://www.alcyone.com/max/).  If you use this software, have
suggestions for future releases, or bug reports or problems with this
documentation, [I'd love to hear about
it](mailto:software@alcyone.com).

Even if you try out EmPy for a project and find it unsuitable, I'd
like to know what stumbling blocks you ran into so they can
potentially be addressed in a future version.

I hope you enjoy using EmPy! ℰ


### About this document

_This documentation for EmPy version 3.3.4 was generated from README.md.em (SHA1 `7cd54c8e55151243ad6c6df47fc725320a5b5b0e`, 238467 bytes) at 2024-08-25 16:18:08 using EmPy version 4.2, in CPython/3.10.12, on Linux (POSIX), with x86_64, under GCC/11.4.0._
