Lexical structure

A Zelkova source file is read in three stages. A tokenizer turns characters into tokens; a layout pass inserts block markers by looking at where each line starts (Layout); a grammar assembles tokens into declarations. This chapter is the first stage: what the characters of a file mean, before any structure is imposed on them.

Zelkova takes its surface syntax from Elm. Every rule below is stated here in full.

Source text

A source file is Unicode text, encoded as UTF-8, in a file whose name ends in .zel.

A line ends at a line feed (U+000A). A carriage return immediately followed by a line feed (U+000D U+000A) is one line ending and means exactly the same thing, so a file written on Windows and the same file written on Unix tokenize identically. A carriage return anywhere else is not valid source text.

The space (U+0020) is the only whitespace character Zelkova recognises. It separates tokens, and in a line's leading run it is what Layout measures. A horizontal tab is invalid anywhere outside a comment — that is Layout's rule, and the reasoning is there.

Tokens are formed by maximal munch: at each point the tokenizer takes the longest run of characters that can form a token. This is why 1.5 is one float rather than 1, ., 5, and why <= is one operator rather than < followed by =. Two tokens that would otherwise run together are separated by a space.

Comments

A line comment starts at -- and runs to the end of the line.

module Example exposing ()

-- a line comment
f = 1

Because -- always starts a comment, no operator can contain two consecutive hyphens, and a--a is the single token a followed by a comment rather than anything arithmetic:

module Example exposing ()

f a =
  a--a

A block comment runs from {- to the matching -}. Block comments nest: an inner {- must be closed by its own -} before the outer comment can end. Nesting is what makes commenting out a region safe — a region that already contains a comment stays commented out as a whole.

A block comment ends exactly at its closing -}. Whatever follows on that line is ordinary source text. A block comment may appear anywhere a space may.

module Example exposing ()

f = {- a note -} 1

A comment written ahead of a top-level declaration on the same line moves the declaration's first token off column 1, just as spaces there would. The line then continues the declaration above it (Layout), and here that declaration is a complete module header, so the source is rejected with an error about the line's indentation:

module Example exposing (f)

{- a note -} f =
  1

module Example exposing ()

{- outer
   {- inner -}
   still outer
-}
f = 1

Reaching the end of a file inside a block comment is an error. A stray {- cannot delete the rest of a file without saying so.

module Example exposing (f)

f = 1
{- oops, never closed

A comment beginning {-| is an ordinary block comment. The convention that such a comment documents the declaration below it, and the @docs markup used inside the module-level one, belong to documentation tooling; the compiler discards them like any other comment.

module Example exposing ()

{-| Documentation for f. -}
f = 1

Tabs are legal inside a comment of either kind, and only there. See Layout.

Identifiers

An identifier begins with an uppercase or lowercase letter and continues with any number of letters, digits, and underscores. Precisely: the first character is in Unicode general category Lu or Ll; each subsequent character is in Lu, Ll, Lt, Lm, Lo, Nd or Nl, or is _ (U+005F).

The first letter decides what kind of thing the identifier names:

A letter that has no case cannot begin an identifier, because there would be nothing for the language to read off it. Such letters are perfectly good inside a name.

module Example exposing ()

type Форма
  = Ronde

aire côté =
  côté

Форма begins with a Cyrillic capital, so it names a type; aire and côté begin with lowercase letters, so they name a function and its parameter. A name may mix scripts freely after its first character:

module Example exposing ()

fデータ = 1

f = fデータ

A name may not begin with an uncased letter:

module Example exposing (f)

f x =
  データ

Titlecase letters (Lt — Dž, Lj, Nj, Dz) are a third case, and Zelkova reads only two. They may not begin an identifier either, for the same reason, and are ordinary continuation characters.

module Example exposing ()

Džoo = 1

f = Džoo

Known gap: that block should be rejected. Today an identifier may begin with any cased letter, titlecase included, and a titlecase-initial name is classified as lowercase — so Džoo is accepted and names a value. docs/tickets/lang-3.md tracks it.

The underscore is not a letter

_ may appear inside an identifier but may not begin one. On its own it is the wildcard pattern, and that is what it stays when it is written in front of a name: _x is two tokens, not one identifier.

In a function's parameter list, two tokens are two parameters:

module Example exposing (Flag, f)

type Flag
  = On

f : Flag -> Flag
f _x =
  On

The annotation promises one argument. f _x supplies two patterns — a wildcard and a variable named x — and the mismatch is what the compiler reports. Zelkova has no convention by which a leading underscore marks a parameter as deliberately unused; write _ for that.

Reserved words

Fifteen words are reserved and may never be used as identifiers:

as     case   else   exposing   if     import   in
infix  let    module of         then   type
class  instance

let and in are reserved although the construct that uses them is not implemented; see Layout.

Eight further words are soft keywords. Each is a keyword only in the position listed against it, and an ordinary identifier everywhere else:

These eight read as ordinary vocabulary — a tree module wants left and right, and a program modelling another language will want foreign — and each sits where one token of context says which reading is meant, so reserving the word outright would take a useful name. derived and unsafe are the two that need the token after them rather than the one before: derived alone asks for an instance to be derived, derived eq opens the derivation of a member, and derived = … or derived : … is an ordinary binding or signature; unsafe f : … marks a facade signature where unsafe : … declares a constant of that name. The other six are keywords only in the position listed against them, and ordinary identifiers everywhere else where a value is named:

module Example exposing ()

left = 1

right = 2

non = 3

foreign = 4

derived = 5

unsafe = 6

alias = 7

where = 8

Not implemented: alias is an ordinary identifier in every position today: nothing reads it after type (LANG-86).

unsafe does not get the plain-name fallback the other seven soft keywords get. left, right, non and foreign are ordinary identifiers anywhere outside their one keyword position, with no dedicated rejection for using them elsewhere — a stray one just misparses. unsafe written before a signature is read as the modifier in every module, module foreign facade or not, and a signature so marked outside a facade is rejected: unsafe f : Int in an ordinary module is a canonicalization error, not a name and not a parse error. A facade modifier only means something where there is a companion for it to be a claim about, and an ordinary module has none.

module Example exposing (twice)

unsafe twice : Int -> Int
twice x =
  x

No other word is reserved — not even ones a reader arriving from another language might expect, such as match (pattern matching's usual keyword) or await (an effect's usual one):

module Example exposing ()

match = 1

await = 2

f = match

true and false are not reserved. Zelkova has no boolean literal syntax: Bool is an ordinary union type and True and False are its constructors, resolved, imported and shadowed like any other name. Bool is also a scalar type, which is a claim about the representation a target gives it and not about its structure.

module Example exposing (Bool, not)

type Bool
  = True
  | False

not : Bool -> Bool
not b =
  case b of
    True ->
      False

    False ->
      True

So true is available as a variable name:

module Example exposing (f)

true : Int
true = 1

f : Int
f = true

Literals

Integers

An integer literal is a run of ASCII digits, or 0x followed by a run of ASCII hexadecimal digits. There are no digit separators, and no leading -: negation is an operator, described below. A pattern may still write what reads as a negative integer — -1 matches only that value — but the sign there belongs to the pattern grammar, not to this token.

module Example exposing ()

count = 42

module Example exposing (mask)

mask = 0xFF

Not implemented: hexadecimal literals are not recognised. 0xFF is currently read as the literal 0 applied to a variable named xFF, which then fails to resolve — an accident, not a diagnostic.

How large an Int may be is Evaluation semantics' rule. An integer literal outside Int's range is an error. It neither wraps nor saturates.

Int's range is not symmetric and a literal carries no sign, so the most negative Int has no literal spelling — its magnitude is one larger than the largest positive Int, and a literal of that magnitude is an error. Arithmetic reaches it by wrapping.

Floats

A float literal is a run of digits, a ., and a further run of digits — at least one digit on each side — optionally followed by an exponent: e or E, an optional + or -, and a run of digits. An exponent may also follow a bare run of digits with no . at all.

module Example exposing ()

ratio = 3.14

module Example exposing (avogadro)

avogadro = 6.022e23

Not implemented: exponents are not recognised. As with hexadecimal above, 6.022e23 currently parses as an application of 6.022 to an unresolvable e23.

Requiring a digit on each side of the point is what keeps . usable as punctuation. What a float literal denotes — the binary64 value it rounds to, and what becomes of one whose value does not fit — is Evaluation semantics's rule, not this section's.

module Example exposing ()

f = 1.

Known gap: that block should be rejected — 1. has no digit after the point. The tokenizer accepts a trailing point today. docs/tickets/lang-3.md tracks it, along with the titlecase rule above.

Characters

A character literal is a single character between single quotes. The character may be written directly, or as one of these escape sequences:

module Example exposing ()

letter = 'a'

module Example exposing (newline)

newline = '\n'

Not implemented: a character literal recognises no escape sequence. It must currently hold exactly one character between the quotes, so '\n' is rejected as an unclosed literal.

A character literal holds one Unicode scalar value. '\u{1F600}' is a single character.

Strings

A string literal is a run of characters between double quotes, using the same escapes as character literals.

module Example exposing ()

greeting = "hello"

quoted = "she said \"hi\"\n\u{1F600}"

A string may not contain an unescaped line ending: one reached before the closing quote leaves the string unclosed.

module Example exposing ()

broken = "one
two"

A multi-line string is delimited by """ and may contain line endings and unescaped double quotes. The same escapes apply.

module Example exposing (poem)

poem =
  """
  one
  two
  """

Not implemented: the multi-line form is not recognised, and the block above is rejected as an unclosed single-line string. docs/tickets/lang-79.md tracks it.

Operators

An operator is a run of one or more of these characters:

! # $ % & * + - . / < = > ? @ \ ^ | ~ :

Operators are subject to maximal munch like every other token, so a<=b is a, <=, b, and an operator only ends where a character outside that set begins. Two consecutive hyphens start a comment, so no operator may contain --.

Seven spellings are punctuation rather than operator names, and may not be declared or redefined:

Every other spelling is available. An operator has no meaning of its own: it is a name, bound by an infix declaration to an ordinary function, and its precedence and associativity are declared there rather than built in.

module Example exposing ()

infix left 6 (|+|) = combine

combine a b =
  a

A reserved spelling cannot be given one:

module Example exposing ((|), f)

infix left 6 (|) = f

f a b = a

- is not reserved: it is an ordinary operator name, and Basics binds it to subtraction like any other.

Prefix negation

A - written directly before an expression, where no left operand is available, is prefix negation: -e means the negation of e, and has the type of e, which must be a numeric one. It is not subtraction against an implied zero, and nothing about it is special-cased to literals — -x negates the variable x, and -1 negates the literal 1 rather than being a literal of its own. A pattern carries a sign differently. Where a - has a left operand and where it does not is Expressions' subject.

module Example exposing ()

infix left 6 (-) = sub

sub a b =
  a

opposite n =
  -n

Known gap: prefix negation is currently desugared to 0 - e, with 0 always an integer literal, so -3.14 is a subtraction that mixes an Int with a Float. The block above passes either way — it pins the syntax, which is unchanged — and docs/tickets/lang-4.md tracks the meaning.

Punctuation

Beyond the seven reserved operator spellings above, these characters are tokens in their own right:

module Example exposing (f)

f = [1, 2]

module Example exposing ()

f = { a = 1 }

Not implemented: lists are part of the language and are not implemented. Brackets are tokenized but no construct consumes them (LANG-44, LANG-45). Lists and Records specify their syntax; what this section fixes is that the characters are spoken for, and so unavailable as operator characters.

Numeric literals that are rejected

An integer literal outside Int's range is rejected:

module Example exposing (f)

f = 99999999999999999999

A numeric literal may contain at most one .; a second one ends the literal in an error:

module Example exposing (f)

f = 1.2.3

A numeric literal may not continue into a non-ASCII digit. 1١ looks, to a human, like two digits in a row — the second is ١, U+0661 ARABIC-INDIC DIGIT ONE — but only the ASCII digits 0-9 are recognised:

module Example exposing (f)

f = 1١

An infix declaration's precedence is a numeric literal too, and the range it may take is Declarations' rule rather than this chapter's.