Next: Syntax Descriptors, Up: Syntax Tables   [Contents][Index]

36.1 Syntax Table Concepts ¶

A syntax table is a data structure which can be used to look up the syntax class and other syntactic properties of each character. Syntax tables are used by Lisp programs for scanning and moving across text.

Internally, a syntax table is a char-table (see Char-Tables). The element at index c describes the character with code c; its value is a cons cell which specifies the syntax of the character in question. See Syntax Table Internals, for details. However, instead of using aset and aref to modify and inspect syntax table contents, you should usually use the higher-level functions char-syntax and modify-syntax-entry, which are described in Syntax Table Functions.

Function: syntax-table-p object ¶

This function returns t if object is a syntax table.

Each buffer has its own major mode, and each major mode has its own idea of the syntax class of various characters. For example, in Lisp mode, the character ‘;’ begins a comment, but in C mode, it terminates a statement. To support these variations, the syntax table is local to each buffer. Typically, each major mode has its own syntax table, which it installs in all buffers that use that mode. For example, the variable emacs-lisp-mode-syntax-table holds the syntax table used by Emacs Lisp mode, and c-mode-syntax-table holds the syntax table used by C mode. Changing a major mode’s syntax table alters the syntax in all of that mode’s buffers, as well as in any buffers subsequently put in that mode. Occasionally, several similar modes share one syntax table. See Major Mode Examples, for an example of how to set up a syntax table.

A syntax table can inherit from another syntax table, which is called its parent syntax table. A syntax table can leave the syntax class of some characters unspecified, by giving them the “inherit” syntax class; such a character then acquires the syntax class specified by the parent syntax table (see Table of Syntax Classes). Emacs defines a standard syntax table, which is the default parent syntax table, and is also the syntax table used by Fundamental mode.

Function: standard-syntax-table ¶

This function returns the standard syntax table, which is the syntax table used in Fundamental mode.

Syntax tables are not used by the Emacs Lisp reader, which has its own built-in syntactic rules which cannot be changed. (Some Lisp systems provide ways to redefine the read syntax, but we decided to leave this feature out of Emacs Lisp for simplicity.)

Next: Syntax Table Functions, Previous: Syntax Table Concepts, Up: Syntax Tables   [Contents][Index]

36.2 Syntax Descriptors ¶

The syntax class of a character describes its syntactic role. Each syntax table specifies the syntax class of each character. There is no necessary relationship between the class of a character in one syntax table and its class in any other table.

Each syntax class is designated by a mnemonic character, which serves as the name of the class when you need to specify a class. Usually, this designator character is one that is often assigned that class; however, its meaning as a designator is unvarying and independent of what syntax that character currently has. Thus, ‘\’ as a designator character always stands for escape character syntax, regardless of whether the ‘\’ character actually has that syntax in the current syntax table. See Table of Syntax Classes, for a list of syntax classes and their designator characters.

A syntax descriptor is a Lisp string that describes the syntax class and other syntactic properties of a character. When you want to modify the syntax of a character, that is done by calling the function modify-syntax-entry and passing a syntax descriptor as one of its arguments (see Syntax Table Functions).

The first character in a syntax descriptor must be a syntax class designator character. The second character, if present, specifies a matching character (e.g., in Lisp, the matching character for ‘(’ is ‘)’); a space specifies that there is no matching character. Then come characters specifying additional syntax properties (see Syntax Flags).

If no matching character or flags are needed, only one character (specifying the syntax class) is sufficient.

For example, the syntax descriptor for the character ‘*’ in C mode is ". 23" (i.e., punctuation, matching character slot unused, second character of a comment-starter, first character of a comment-ender), and the entry for ‘/’ is ‘. 14’ (i.e., punctuation, matching character slot unused, first character of a comment-starter, second character of a comment-ender).

Emacs also defines raw syntax descriptors, which are used to describe syntax classes at a lower level. See Syntax Table Internals.

• Table of Syntax Classes
• Syntax Flags

Next: Syntax Properties, Previous: Syntax Descriptors, Up: Syntax Tables   [Contents][Index]

36.3 Syntax Table Functions ¶

In this section we describe functions for creating, accessing and altering syntax tables.

Function: make-syntax-table &optional table ¶

This function creates a new syntax table. If table is non-nil, the parent of the new syntax table is table; otherwise, the parent is the standard syntax table.

In the new syntax table, all characters are initially given the “inherit” (‘@’) syntax class, i.e., their syntax is inherited from the parent table (see Table of Syntax Classes).

Function: copy-syntax-table &optional table ¶

This function constructs a copy of table and returns it. If table is omitted or nil, it returns a copy of the standard syntax table. Otherwise, an error is signaled if table is not a syntax table.

Command: modify-syntax-entry char syntax-descriptor &optional table ¶

This function sets the syntax entry for char according to syntax-descriptor. char must be a character, or a cons cell of the form (min . max); in the latter case, the function sets the syntax entries for all characters in the range between min and max, inclusive.

The syntax is changed only for table, which defaults to the current buffer’s syntax table, and not in any other syntax table.

The argument syntax-descriptor is a syntax descriptor, i.e., a string whose first character is a syntax class designator and whose second and subsequent characters optionally specify a matching character and syntax flags. See Syntax Descriptors. An error is signaled if syntax-descriptor is not a valid syntax descriptor.

This function always returns nil. The old syntax information in the table for this character is discarded.

Examples:

;; Put the space character in class whitespace.
(modify-syntax-entry ?\s " ")
     ⇒ nil

;; Make ‘$’ an open parenthesis character,
;;   with ‘^’ as its matching close.
(modify-syntax-entry ?$ "(^")
     ⇒ nil

;; Make ‘^’ a close parenthesis character,
;;   with ‘$’ as its matching open.
(modify-syntax-entry ?^ ")$")
     ⇒ nil

;; Make ‘/’ a punctuation character,
;;   the first character of a start-comment sequence,
;;   and the second character of an end-comment sequence.
;;   This is used in C mode.
(modify-syntax-entry ?/ ". 14")
     ⇒ nil

Function: char-syntax character ¶

This function returns the syntax class of character, represented by its designator character (see Table of Syntax Classes). This returns only the class, not its matching character or syntax flags.

The following examples apply to C mode. (We use string to make it easier to see the character returned by char-syntax.)

;; Space characters have whitespace syntax class.
(string (char-syntax ?\s))
     ⇒ " "

;; Forward slash characters have punctuation syntax.
;; Note that this char-syntax call does not reveal
;; that it is also part of comment-start and -end sequences.
(string (char-syntax ?/))
     ⇒ "."

;; Open parenthesis characters have open parenthesis syntax.
;; Note that this char-syntax call does not reveal that
;; it has a matching character, ‘)’.
(string (char-syntax ?\())
     ⇒ "("

Function: set-syntax-table table ¶

This function makes table the syntax table for the current buffer. It returns table.

Function: syntax-table ¶

This function returns the current syntax table, which is the table for the current buffer.

Command: describe-syntax &optional buffer ¶

This command displays the contents of the syntax table of buffer (by default, the current buffer) in a help buffer.

Macro: with-syntax-table table body… ¶

This macro executes body using table as the current syntax table. It returns the value of the last form in body, after restoring the old current syntax table.

Since each buffer has its own current syntax table, we should make that more precise: with-syntax-table temporarily alters the current syntax table of whichever buffer is current at the time the macro execution starts. Other buffers are not affected.

Next: Motion and Syntax, Previous: Syntax Table Functions, Up: Syntax Tables   [Contents][Index]

36.4 Syntax Properties ¶

When the syntax table is not flexible enough to specify the syntax of a language, you can override the syntax table for specific character occurrences in the buffer, by applying a syntax-table text property. See Text Properties, for how to apply text properties.

The valid values of syntax-table text property are:

syntax-table

If the property value is a syntax table, that table is used instead of the current buffer’s syntax table to determine the syntax for the underlying text character.

(syntax-code . matching-char)

A cons cell of this format is a raw syntax descriptor (see Syntax Table Internals), which directly specifies a syntax class for the underlying text character.

nil

If the property is nil, the character’s syntax is determined from the current syntax table in the usual way.

Variable: parse-sexp-lookup-properties ¶

If this is non-nil, the syntax scanning functions, like forward-sexp, pay attention to syntax-table text properties. Otherwise they use only the current syntax table.

Variable: syntax-propertize-function ¶

This variable, if non-nil, should store a function for applying syntax-table properties to a specified stretch of text. It is intended to be used by major modes to install a function which applies syntax-table properties in some mode-appropriate way.

The function is called by syntax-ppss (see Finding the Parse State for a Position), and by Font Lock mode during syntactic fontification (see Syntactic Font Lock). It is called with two arguments, start and end, which are the starting and ending positions of the text on which it should act. It is allowed to arbitrarily move point within the region delimited by start and end; such motions don’t need to use save-excursion (see Excursions). It is also allowed to call syntax-ppss on any position before end, but if a Lisp program calls syntax-ppss on some position and later modifies the buffer at some earlier position, then it is that program’s responsibility to call syntax-ppss-flush-cache to flush the now obsolete info from the cache.

Caution: When this variable is non-nil, Emacs removes syntax-table text properties arbitrarily and relies on syntax-propertize-function to reapply them. Thus if this facility is used at all, the function must apply all syntax-table text properties used by the major mode. In particular, Modes derived from a CC Mode mode must not use this variable, since CC Mode uses other means to apply and remove these text properties.

Variable: syntax-propertize-extend-region-functions ¶

This abnormal hook is run by the syntax parsing code prior to calling syntax-propertize-function. Its role is to help locate safe starting and ending buffer positions for passing to syntax-propertize-function. For example, a major mode can add a function to this hook to identify multi-line syntactic constructs, and ensure that the boundaries do not fall in the middle of one.

Each function in this hook should accept two arguments, start and end. It should return either a cons cell of two adjusted buffer positions, (new-start . new-end), or nil if no adjustment is necessary. The hook functions are run in turn, repeatedly, until they all return nil.

Next: Parsing Expressions, Previous: Syntax Properties, Up: Syntax Tables   [Contents][Index]

36.5 Motion and Syntax ¶

This section describes functions for moving across characters that have certain syntax classes.

Function: skip-syntax-forward syntaxes &optional limit ¶

This function moves point forward across characters having syntax classes mentioned in syntaxes (a string of syntax class characters). It stops when it encounters the end of the buffer, or position limit (if specified), or a character it is not supposed to skip.

If syntaxes starts with ‘^’, then the function skips characters whose syntax is not in syntaxes.

The return value is the distance traveled, which is a nonnegative integer.

Function: skip-syntax-backward syntaxes &optional limit ¶

This function moves point backward across characters whose syntax classes are mentioned in syntaxes. It stops when it encounters the beginning of the buffer, or position limit (if specified), or a character it is not supposed to skip.

If syntaxes starts with ‘^’, then the function skips characters whose syntax is not in syntaxes.

The return value indicates the distance traveled. It is an integer that is zero or less.

Function: backward-prefix-chars ¶

This function moves point backward over any number of characters with expression prefix syntax. This includes both characters in the expression prefix syntax class, and characters with the ‘p’ flag.

Next: Syntax Table Internals, Previous: Motion and Syntax, Up: Syntax Tables   [Contents][Index]

36.6 Parsing Expressions ¶

This section describes functions for parsing and scanning balanced expressions. We will refer to such expressions as sexps, following the terminology of Lisp, even though these functions can act on languages other than Lisp. Basically, a sexp is either a balanced parenthetical grouping, a string, or a symbol (i.e., a sequence of characters whose syntax is either word constituent or symbol constituent). However, characters in the expression prefix syntax class (see Table of Syntax Classes) are treated as part of the sexp if they appear next to it.

The syntax table controls the interpretation of characters, so these functions can be used for Lisp expressions when in Lisp mode and for C expressions when in C mode. See Moving over Balanced Expressions, for convenient higher-level functions for moving over balanced expressions.

A character’s syntax controls how it changes the state of the parser, rather than describing the state itself. For example, a string delimiter character toggles the parser state between in-string and in-code, but the syntax of characters does not directly say whether they are inside a string. For example (note that 15 is the syntax code for generic string delimiters),

(put-text-property 1 9 'syntax-table '(15 . nil))

does not tell Emacs that the first eight chars of the current buffer are a string, but rather that they are all string delimiters. As a result, Emacs treats them as four consecutive empty string constants.

• Motion Commands Based on Parsing
• Finding the Parse State for a Position
• Parser State
• Low-Level Parsing
• Parameters to Control Parsing

Next: Categories, Previous: Parsing Expressions, Up: Syntax Tables   [Contents][Index]

36.7 Syntax Table Internals ¶

Syntax tables are implemented as char-tables (see Char-Tables), but most Lisp programs don’t work directly with their elements. Syntax tables do not store syntax data as syntax descriptors (see Syntax Descriptors); they use an internal format, which is documented in this section. This internal format can also be assigned as syntax properties (see Syntax Properties).

Each entry in a syntax table is a raw syntax descriptor: a cons cell of the form (syntax-code . matching-char). syntax-code is an integer which encodes the syntax class and syntax flags, according to the table below. matching-char, if non-nil, specifies a matching character (similar to the second character in a syntax descriptor).

Use aref (see Functions that Operate on Arrays) to get the raw syntax descriptor of a character, for example (aref (syntax-table) ch).

Here are the syntax codes corresponding to the various syntax classes:

For example, in the standard syntax table, the entry for ‘(’ is (4 . 41). 41 is the character code for ‘)’.

Syntax flags are encoded in higher order bits, starting 16 bits from the least significant bit. This table gives the power of two which corresponds to each syntax flag.

Function: string-to-syntax desc ¶

Given a syntax descriptor desc (a string), this function returns the corresponding raw syntax descriptor.

Function: syntax-class-to-char syntax ¶

Given a raw syntax descriptor syntax (an integer), this function returns the corresponding syntax descriptor (a character).

Function: syntax-after pos ¶

This function returns the raw syntax descriptor for the character in the buffer after position pos, taking account of syntax properties as well as the syntax table. If pos is outside the buffer’s accessible portion (see accessible portion), the return value is nil.

Function: syntax-class syntax ¶

This function returns the syntax code for the raw syntax descriptor syntax. More precisely, it takes the raw syntax descriptor’s syntax-code component, masks off the high 16 bits which record the syntax flags, and returns the resulting integer.

If syntax is nil, the return value is nil. This is so that the expression

(syntax-class (syntax-after pos))

evaluates to nil if pos is outside the buffer’s accessible portion, without throwing errors or returning an incorrect code.

Previous: Syntax Table Internals, Up: Syntax Tables   [Contents][Index]

36.8 Categories ¶

Categories provide an alternate way of classifying characters syntactically. You can define several categories as needed, then independently assign each character to one or more categories. Unlike syntax classes, categories are not mutually exclusive; it is normal for one character to belong to several categories.

Each buffer has a category table which records which categories are defined and also which characters belong to each category. Each category table defines its own categories, but normally these are initialized by copying from the standard categories table, so that the standard categories are available in all modes.

Each category has a name, which is an ASCII printing character in the range ‘ ’ to ‘~’. You specify the name of a category when you define it with define-category.

The category table is actually a char-table (see Char-Tables). The element of the category table at index c is a category set—a bool-vector—that indicates which categories character c belongs to. In this category set, if the element at index cat is t, that means category cat is a member of the set, and that character c belongs to category cat.

For the next three functions, the optional argument table defaults to the current buffer’s category table.

Function: define-category char docstring &optional table ¶

This function defines a new category, with name char and documentation docstring, for the category table table.

Here’s an example of defining a new category for characters that have strong right-to-left directionality (see Bidirectional Display) and using it in a special category table. To obtain the information about the directionality of characters, the example code uses the ‘bidi-class’ Unicode property (see bidi-class).

(defvar special-category-table-for-bidi
  ;;     Make an empty category-table.
  (let ((category-table (make-category-table))
        ;; Create a char-table which gives the 'bidi-class' Unicode
        ;; property for each character.
        (uniprop-table
         (unicode-property-table-internal 'bidi-class)))
    (define-category ?R "Characters of bidi-class R, AL, or RLO"
                     category-table)
    ;; Modify the category entry of each character whose
    ;; 'bidi-class' Unicode property is R, AL, or RLO --
    ;; these have a right-to-left directionality.
    (map-char-table
     (lambda (key val)
       (if (memq val '(R AL RLO))
           (modify-category-entry key ?R category-table)))
     uniprop-table)
    category-table))

Function: category-docstring category &optional table ¶

This function returns the documentation string of category category in category table table.

(category-docstring ?a)
     ⇒ "ASCII"
(category-docstring ?l)
     ⇒ "Latin"

Function: get-unused-category &optional table ¶

This function returns a category name (a character) which is not currently defined in table. If all possible categories are in use in table, it returns nil.

Function: category-table ¶

This function returns the current buffer’s category table.

Function: category-table-p object ¶

This function returns t if object is a category table, otherwise nil.

Function: standard-category-table ¶

This function returns the standard category table.

Function: copy-category-table &optional table ¶

This function constructs a copy of table and returns it. If table is not supplied (or is nil), it returns a copy of the standard category table. Otherwise, an error is signaled if table is not a category table.

Function: set-category-table table ¶

This function makes table the category table for the current buffer. It returns table.

Function: make-category-table ¶

This creates and returns an empty category table. In an empty category table, no categories have been allocated, and no characters belong to any categories.

Function: make-category-set categories ¶

This function returns a new category set—a bool-vector—whose initial contents are the categories listed in the string categories. The elements of categories should be category names; the new category set has t for each of those categories, and nil for all other categories.

(make-category-set "al")
     ⇒ #&128"\0\0\0\0\0\0\0\0\0\0\0\0\2\20\0\0"

Function: char-category-set char ¶

This function returns the category set for character char in the current buffer’s category table. This is the bool-vector which records which categories the character char belongs to. The function char-category-set does not allocate storage, because it returns the same bool-vector that exists in the category table.

(char-category-set ?a)
     ⇒ #&128"\0\0\0\0\0\0\0\0\0\0\0\0\2\20\0\0"

Function: category-set-mnemonics category-set ¶

This function converts the category set category-set into a string containing the characters that designate the categories that are members of the set.

(category-set-mnemonics (char-category-set ?a))
     ⇒ "al"

Function: modify-category-entry char category &optional table reset ¶

This function modifies the category set of char in category table table (which defaults to the current buffer’s category table). char can be a character, or a cons cell of the form (min . max); in the latter case, the function modifies the category sets of all characters in the range between min and max, inclusive.

Normally, it modifies a category set by adding category to it. But if reset is non-nil, then it deletes category instead.

Command: describe-categories &optional buffer-or-name ¶

This function describes the category specifications in the current category table. It inserts the descriptions in a buffer, and then displays that buffer. If buffer-or-name is non-nil, it describes the category table of that buffer instead.