Blame


1 2228b436 2024-02-10 benni .\"
2 2228b436 2024-02-10 benni .\" Copyright (c) 2023 Benjamin Stürz
3 2228b436 2024-02-10 benni .\"
4 2228b436 2024-02-10 benni .\" Permission to use, copy, modify, and distribute this software for any
5 2228b436 2024-02-10 benni .\" purpose with or without fee is hereby granted, provided that the above
6 2228b436 2024-02-10 benni .\" copyright notice and this permission notice appear in all copies.
7 2228b436 2024-02-10 benni .\"
8 2228b436 2024-02-10 benni .\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
9 2228b436 2024-02-10 benni .\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
10 2228b436 2024-02-10 benni .\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
11 2228b436 2024-02-10 benni .\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
12 2228b436 2024-02-10 benni .\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
13 2228b436 2024-02-10 benni .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
14 2228b436 2024-02-10 benni .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
15 2228b436 2024-02-10 benni .\"
16 2228b436 2024-02-10 benni .Dd $Mdocdate$
17 2228b436 2024-02-10 benni .Dt LEXICGEN 1
18 2228b436 2024-02-10 benni .Os
19 2228b436 2024-02-10 benni .Sh NAME
20 2228b436 2024-02-10 benni .Nm lexicgen
21 2228b436 2024-02-10 benni .Nd lexic generator
22 2228b436 2024-02-10 benni .Sh SYNOPSIS
23 2228b436 2024-02-10 benni .Nm
24 2228b436 2024-02-10 benni .Op Fl v Ar visibility
25 2228b436 2024-02-10 benni .Op Fl p Ar package
26 2228b436 2024-02-10 benni .Op Fl o Ar output-file
27 2228b436 2024-02-10 benni .Ar input-file
28 2228b436 2024-02-10 benni .Sh DESCRIPTION
29 2228b436 2024-02-10 benni .Nm
30 2228b436 2024-02-10 benni is a clone of lex(1) for generating lexers using Lexic.
31 2228b436 2024-02-10 benni See the FORMAT section for information about the format.
32 2228b436 2024-02-10 benni .Pp
33 2228b436 2024-02-10 benni The options are as follows:
34 2228b436 2024-02-10 benni .Bl -tag -width -Ds
35 2228b436 2024-02-10 benni .It Fl v Ar visibility
36 2228b436 2024-02-10 benni Change the visibility of the lexer-internal data structures and functions. (default: private)
37 2228b436 2024-02-10 benni .It Fl p Ar package
38 2228b436 2024-02-10 benni Set a Java package for the generated class.
39 2228b436 2024-02-10 benni .It Fl o Ar output-file
40 2228b436 2024-02-10 benni Write the output to
41 2228b436 2024-02-10 benni .Ar output-file
42 2228b436 2024-02-10 benni instead of a generated path.
43 2228b436 2024-02-10 benni If
44 2228b436 2024-02-10 benni .Ar output-file
45 2228b436 2024-02-10 benni is '-', then write to the standard output.
46 2228b436 2024-02-10 benni .El
47 2228b436 2024-02-10 benni .Sh FORMAT
48 2228b436 2024-02-10 benni A valid file consists of three sections:
49 2228b436 2024-02-10 benni .Bl -bullet -compact
50 2228b436 2024-02-10 benni .It
51 2228b436 2024-02-10 benni Declarations
52 2228b436 2024-02-10 benni .It
53 2228b436 2024-02-10 benni Definitions
54 2228b436 2024-02-10 benni .It
55 2228b436 2024-02-10 benni and Code (optional)
56 2228b436 2024-02-10 benni .El
57 2228b436 2024-02-10 benni
58 2228b436 2024-02-10 benni The declaration section can contain the following statements:
59 2228b436 2024-02-10 benni .Bl -tag -width -Ds
60 2228b436 2024-02-10 benni .It Sy %import Ar package
61 2228b436 2024-02-10 benni Import
62 2228b436 2024-02-10 benni .Ar package .
63 2228b436 2024-02-10 benni .It Sy %eof Ar name
64 2228b436 2024-02-10 benni Declare an end of file token called
65 2228b436 2024-02-10 benni .Ar name .
66 2228b436 2024-02-10 benni
67 2228b436 2024-02-10 benni Note: Should only be specified once.
68 2228b436 2024-02-10 benni .It Sy %error Ar name
69 2228b436 2024-02-10 benni Declare an error token called
70 2228b436 2024-02-10 benni .Ar name .
71 2228b436 2024-02-10 benni
72 2228b436 2024-02-10 benni Note: Should only be specified once.
73 2228b436 2024-02-10 benni .It Sy name No = Sy syntax No ;
74 2228b436 2024-02-10 benni Define a variable called
75 2228b436 2024-02-10 benni .Ar name
76 2228b436 2024-02-10 benni containing
77 2228b436 2024-02-10 benni .Ar syntax
78 2228b436 2024-02-10 benni that can be refered to from token definitions.
79 2228b436 2024-02-10 benni .El
80 2228b436 2024-02-10 benni
81 2228b436 2024-02-10 benni The definition section contains token definitions.
82 2228b436 2024-02-10 benni A token definition consists of a
83 2228b436 2024-02-10 benni .Ar name
84 2228b436 2024-02-10 benni and a
85 2228b436 2024-02-10 benni .Ar syntax expression .
86 2228b436 2024-02-10 benni The name must consist of alphabetic characters.
87 2228b436 2024-02-10 benni
88 2228b436 2024-02-10 benni The following syntax expressions are supported:
89 2228b436 2024-02-10 benni .Bl -tag -width -Ds
90 2228b436 2024-02-10 benni .It Sy ident
91 2228b436 2024-02-10 benni An identifier is a sequence of one or more alphabetic characters.
92 2228b436 2024-02-10 benni Match if the rule identified by
93 2228b436 2024-02-10 benni .Ar ident
94 2228b436 2024-02-10 benni matches.
95 2228b436 2024-02-10 benni .It Qq Sy string
96 2228b436 2024-02-10 benni Match if the input matches all characters of
97 2228b436 2024-02-10 benni .Ar string .
98 2228b436 2024-02-10 benni .It Bq Sy range
99 2228b436 2024-02-10 benni Match, if any of the fragments match.
100 2228b436 2024-02-10 benni If
101 2228b436 2024-02-10 benni .Ar range
102 2228b436 2024-02-10 benni starts with the
103 2228b436 2024-02-10 benni .Dq ^
104 2228b436 2024-02-10 benni character, then negate the matching result.
105 2228b436 2024-02-10 benni The range consists of fragments.
106 2228b436 2024-02-10 benni A fragment can be a single character, an escape sequence or a character range.
107 2228b436 2024-02-10 benni Only single-character escape sequences are supported.
108 2228b436 2024-02-10 benni A character range has the syntax a-b,
109 2228b436 2024-02-10 benni where a is the first character and b is the last.
110 2228b436 2024-02-10 benni .It Pq expr
111 2228b436 2024-02-10 benni Grouping.
112 2228b436 2024-02-10 benni .It expr?
113 2228b436 2024-02-10 benni Zero or one counts of
114 2228b436 2024-02-10 benni .Ar expr .
115 2228b436 2024-02-10 benni .It expr+
116 2228b436 2024-02-10 benni One or more counts of
117 2228b436 2024-02-10 benni .Ar expr .
118 2228b436 2024-02-10 benni .It expr*
119 2228b436 2024-02-10 benni Zero or more counts of
120 2228b436 2024-02-10 benni .Ar expr .
121 2228b436 2024-02-10 benni .It expr1 expr2
122 2228b436 2024-02-10 benni .Ar expr1
123 2228b436 2024-02-10 benni followed by
124 2228b436 2024-02-10 benni .Ar expr2 .
125 2228b436 2024-02-10 benni .It expr1 & expr2
126 2228b436 2024-02-10 benni Only match if both
127 2228b436 2024-02-10 benni .Ar expr1
128 2228b436 2024-02-10 benni and
129 2228b436 2024-02-10 benni .Ar expr2
130 2228b436 2024-02-10 benni match.
131 2228b436 2024-02-10 benni .It expr1 \e expr2
132 2228b436 2024-02-10 benni Match only if
133 2228b436 2024-02-10 benni .Ar expr1
134 2228b436 2024-02-10 benni matches, and
135 2228b436 2024-02-10 benni .Ar expr2
136 2228b436 2024-02-10 benni doesn't.
137 2228b436 2024-02-10 benni .It expr1 | expr2
138 2228b436 2024-02-10 benni Match if either or both of
139 2228b436 2024-02-10 benni .Ar expr1
140 2228b436 2024-02-10 benni or
141 2228b436 2024-02-10 benni .Ar expr2
142 2228b436 2024-02-10 benni match.
143 2228b436 2024-02-10 benni .El
144 2228b436 2024-02-10 benni
145 2228b436 2024-02-10 benni The code sections contains any arbitrary code that will be put into the generated file.
146 2228b436 2024-02-10 benni
147 2228b436 2024-02-10 benni .Sh EXAMPLES
148 2228b436 2024-02-10 benni The following code is an example of a lexicgen file:
149 2228b436 2024-02-10 benni .Bd -literal
150 2228b436 2024-02-10 benni // Declarations
151 2228b436 2024-02-10 benni %import eu.bandm.tools.lexic.*
152 2228b436 2024-02-10 benni %import java.io.IOException
153 2228b436 2024-02-10 benni %import java.io.InputStreamReader
154 2228b436 2024-02-10 benni %import java.nio.charset.StandardCharsets
155 2228b436 2024-02-10 benni
156 2228b436 2024-02-10 benni // Declare an end-of-file token called EOF.
157 2228b436 2024-02-10 benni %eof EOF
158 2228b436 2024-02-10 benni
159 2228b436 2024-02-10 benni // Declare an error token called Error.
160 2228b436 2024-02-10 benni %error Error
161 2228b436 2024-02-10 benni
162 2228b436 2024-02-10 benni Integer = "0" | [1-9][0-9]*;
163 2228b436 2024-02-10 benni Ident = [a-zA-Z_][a-zA-Z_0-9]*;
164 2228b436 2024-02-10 benni
165 2228b436 2024-02-10 benni %%
166 2228b436 2024-02-10 benni // Definitions
167 2228b436 2024-02-10 benni
168 2228b436 2024-02-10 benni Whitespace : [ \en\et\er\ef]+;
169 2228b436 2024-02-10 benni // A File can be either an integer, or an identifier.
170 2228b436 2024-02-10 benni File : Integer | Ident;
171 2228b436 2024-02-10 benni
172 2228b436 2024-02-10 benni %%
173 2228b436 2024-02-10 benni // Code
174 2228b436 2024-02-10 benni
175 2228b436 2024-02-10 benni public static void main(String[] args) {
176 2228b436 2024-02-10 benni // The construct() function is generated by lexicgen and returns a Lexer.
177 2228b436 2024-02-10 benni var lexer = construct();
178 2228b436 2024-02-10 benni
179 2228b436 2024-02-10 benni try (var rdr = new InputStreamReader(System.in, StandardCharsets.UTF_8)) {
180 2228b436 2024-02-10 benni TokenSource<String, TokenType> tokens = lexer
181 2228b436 2024-02-10 benni .lex(CodePointSource.read(rdr, e -> {}))
182 2228b436 2024-02-10 benni .removeTypes(TokenType.Whitespace);
183 2228b436 2024-02-10 benni while (true) {
184 2228b436 2024-02-10 benni var token = tokens.get();
185 2228b436 2024-02-10 benni if (token.getType() == TokenType.EOF)
186 2228b436 2024-02-10 benni break;
187 2228b436 2024-02-10 benni System.out.println(token);
188 2228b436 2024-02-10 benni }
189 2228b436 2024-02-10 benni } catch (IOException e) {}
190 2228b436 2024-02-10 benni }
191 2228b436 2024-02-10 benni .Ed
192 2228b436 2024-02-10 benni .Sh EXIT STATUS
193 2228b436 2024-02-10 benni .Ex -std
194 2228b436 2024-02-10 benni .Sh SEE ALSO
195 2228b436 2024-02-10 benni .Xr lex 1
196 2228b436 2024-02-10 benni .Sh AUTHORS
197 2228b436 2024-02-10 benni .An Benjamin Stürz Aq Mt benni@stuerz.xyz