The Flex Lexical Analyzer - Revision history

Root: Redirected page to ist:The Flex Lexical Analyzer

2018-12-05T22:14:05Z

Redirected page to ist:The Flex Lexical Analyzer

← Older revision		Revision as of 22:14, 5 December 2018
Line 1:		Line 1:
	~~{{NAVCompiladores}}~~		#REDIRECT [[ist:The Flex Lexical Analyzer]]
	~~{{TOCright}}~~

	Flex is a tool that allows the recognition of language items (elements) specified as regular expressions. These regular expressions are as specified by the corresponding formal definition (alphabet and primitive operators). In addition, Flex also supports the derived operators (see theory page) as well as a number of meta- and pseudo-characters, such as end-of-file markers or special and non-printable characters. Flex is also capable of recognizing characters by their properties (such as being a digit, and so on). All these aspects make Flex very capable in what concerns alphabet specification.

	To perform string recognition, from regular expression specifications, Flex uses a finite state machine (a finite automaton). Even though it is not its primary task, Flex is also capable of using a stack in conjunction with its state machine, making it theoretically capable of processing context-free languages (theoretically, since it is not as convenient as using a dedicated tool, such as BYACC, Bison, or others).

	~~== Basic Concepts ==~~

	Flex is a code generator that reads a specification file and generates the lexical analyzer (a scanner) as a C or C++ module (depending on the options). The description is the same in both cases (only a few details in the actions are different from one case to the other). In current versions, the code generated for the C analyzer is compatible with a C++ compiler, making it possible to program actions in this language, even without using classes. This may be a good approach, since C++ data structures are usually more programmer-friendly than C, leading to fewer bugs.

	~~A Flex specification file is composed by three parts:~~

	* Definitions
	* Rules (these are used to build the automaton), actions (for manipulating the strings that match the regular expressions)
	* Code

	~~== Structure of a Flex Specification ==~~

	~~== Basic Guidelines for a Successful Specification ==~~

	~~... Do not try to recognize "negative" patterns~~

	~~... Recognize patterns that are either of interest or that may cause problems if not recognized (example: sources of ambiguity)~~

	... if there are self-contained parts of the language being recognized, consider recognizing them in a sub-automaton, controlled by start conditions (this will make the rules simpler and the overall specification will also be simpler -- the use of sub-automata tends to make the rules less dependent On one another, in the sense that they don't interfere with each other)

	... sometimes, "bad rules" have to be written to account for badly specified language elements (e.g. octal numbers in which the digits do not belong to that base) -- these rules should be left for last, because if written too early they may complicate the overall task of rule writing (ambiguity and so on). It may be useful to activate debugging if behaviour is unexpected.

	~~== How to Debug a Flex Specification ==~~

	There are various flags and variables to activate the debug functionality in Flex, i.e., there is no fundamental need to insert code such as printfs or similar, even through, of course, that style of debugging is also possible. These other ways are often much superior to printfs, since they can easily inform the programmer about matched rules, triggered actions, and so on.

	~~Generation of debug code is requested by activating the corresponding option. In the Flex specification file, insert:~~
	~~%option debug~~

	This flag suffices to produce output information when developing in C. When using C++ scanners ('''%option c++'''), even though the above flag still generates debug code, it's still necessary to tell the scanner to actually output debug information. This can be done by calling the '''set_debug''' method with a non-zero argument (this can be conveniently done at the start of the rules section in a Flex specification file -- note how the action is indented from the left and without any rule):

	~~%option debug~~
	~~...~~

	%%
	~~{ set_debug(1); }~~

	~~... other rules...~~

	%%

	~~The YYDEBUG environment variable will activate the debug messages '''both''' in Flex and YACC (allowing simultaneous debugging of token recognition by the scanner and corresponding use by the parser).~~

	~~In the shell's command line (syntax may vary, depending on the actual environment definition circumstances):~~
	~~export YYDEBUG=1~~

	~~<!--== See Also ==~~

	* [~~http://flex.sourceforge.net/manual/ Flex manual @ sourceforge]~~
	* [~~http~~:~~//flex.sourceforge.net/ Flex site]~~
	~~-->~~
	~~== Examples ==~~

	* [[Extracting HREF attributes from HTML files]]

	~~== Exercises ==~~

	* [[The Flex Lexical Analyzer/Exercise 1 - Printing the strings present in a C/C++ program\|Exercise 1]] - Printing the strings present in a C/C++ program.
	* [[The Flex Lexical Analyzer~~/Exercise 2 - Printing the number of times access operators are used\|Exercise 2]] - Printing the number of times "." and "->" are used as operators.~~
	* [[The Flex Lexical Analyzer/Exercise 3 - Printing the comments present in a C/C++ program\|Exercise 3]] - Printing the comments present in a C/C++ program.
	* [[The Flex Lexical Analyzer/Exercise 4 - Removing the actions from a YACC specification\|Exercise 4]] - Removing the actions from a YACC specification.
	* [[The Flex Lexical Analyzer/Exercise 5 - Removing the actions from a Flex specification\|Exercise 5]] - Removing the actions from a Flex specification.
	* [[The Flex Lexical Analyzer/Exercise 6 - Playlist processing\|Exercise 6]] - Playlist processing.
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.
	* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java (Test 1, 2016/04/08).
	~~[[category:Compiladores\|Flex Lexical Analyzer]]~~
	~~[[category:Ensino~~]]

Root at 18:08, 10 February 2017

2017-02-10T18:08:28Z

← Older revision		Revision as of 18:08, 10 February 2017
Line 57:		Line 57:
	* [http://flex.sourceforge.net/ Flex site]		* [http://flex.sourceforge.net/ Flex site]
	-->		-->

	== Examples ==		== Examples ==

Root: /* See Also */

2017-02-10T18:08:06Z

Root: /* Exercises */

2016-04-10T18:17:25Z

Exercises

← Older revision		Revision as of 18:17, 10 April 2016
Line 71:		Line 71:
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.		* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.
	* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java.		* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java (Test 1, 2016/04/08).
	[[category:Compiladores\|Flex Lexical Analyzer]]		[[category:Compiladores\|Flex Lexical Analyzer]]
	[[category:Ensino]]		[[category:Ensino]]

Root: /* Exercises */

2016-04-08T18:16:21Z

Exercises

← Older revision		Revision as of 18:16, 8 April 2016
Line 71:		Line 71:
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.		* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.
			* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java.
	[[category:Compiladores\|Flex Lexical Analyzer]]		[[category:Compiladores\|Flex Lexical Analyzer]]
	[[category:Ensino]]		[[category:Ensino]]

Root at 14:51, 6 April 2015

2015-04-06T14:51:16Z

← Older revision		Revision as of 14:51, 6 April 2015
Line 72:		Line 72:
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.

	[[category:Compiladores]]		[[category:Compiladores\|Flex Lexical Analyzer]]
	[[category:Ensino]]		[[category:Ensino]]

Root at 14:33, 6 April 2015

2015-04-06T14:33:48Z

← Older revision		Revision as of 14:33, 6 April 2015
Line 72:		Line 72:
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.

	[[category:~~Compilers~~]]		[[category:Compiladores]]
	[[category:~~Teaching~~]]		[[category:Ensino]]

Root: /* How to Debug a Flex Specification */

2015-02-13T19:59:54Z

How to Debug a Flex Specification

← Older revision		Revision as of 19:59, 13 February 2015
Line 30:		Line 30:
	== How to Debug a Flex Specification ==		== How to Debug a Flex Specification ==

	There are various flags and variables to activate the debug functionality in Flex (there is no fundamental need to insert code such as printfs or similar, even through, of course, that style of debugging is also possible).		There are various flags and variables to activate the debug functionality in Flex, i.e., there is no fundamental need to insert code such as printfs or similar, even through, of course, that style of debugging is also possible. These other ways are often much superior to printfs, since they can easily inform the programmer about matched rules, triggered actions, and so on.

	In the Flex specification file:		Generation of debug code is requested by activating the corresponding option. In the Flex specification file, insert:
	%option debug		%option debug

	This flag suffices when developing in C. When using C++ scanners ('''%option c++'''), even though the above flag still generates debug code, it's still necessary to tell the scanner to actually output debug information. This can be done by calling the '''set_debug''' method with a non-zero argument (this can be done ~~conveniently~~ at the start of the rules section in a Flex specification file -- note how the action is indented from the left and without any rule):		This flag suffices to produce output information when developing in C. When using C++ scanners ('''%option c++'''), even though the above flag still generates debug code, it's still necessary to tell the scanner to actually output debug information. This can be done by calling the '''set_debug''' method with a non-zero argument (this can be conveniently done at the start of the rules section in a Flex specification file -- note how the action is indented from the left and without any rule):

	%%		%option debug
	{ set_debug(1); }		...

			%%
			{ set_debug(1); }

			... other rules...

			%%

	~~Also, the~~ YYDEBUG environment variable will activate the debug messages '''both''' in Flex and YACC (allowing simultaneous debugging of token recognition by the scanner and corresponding use by the parser). In the command line (syntax may vary, depending on the actual environment definition circumstances):		The YYDEBUG environment variable will activate the debug messages '''both''' in Flex and YACC (allowing simultaneous debugging of token recognition by the scanner and corresponding use by the parser).

			In the shell's command line (syntax may vary, depending on the actual environment definition circumstances):
	export YYDEBUG=1		export YYDEBUG=1

Root: /* Exercises */

2012-05-07T17:42:23Z

Exercises

← Older revision		Revision as of 17:42, 7 May 2012
Line 61:		Line 61:
	* [[The Flex Lexical Analyzer/Exercise 6 - Playlist processing\|Exercise 6]] - Playlist processing.		* [[The Flex Lexical Analyzer/Exercise 6 - Playlist processing\|Exercise 6]] - Playlist processing.
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.		* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
			* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.

	[[category:Compilers]]		[[category:Compilers]]
	[[category:Teaching]]		[[category:Teaching]]

Root: /* How to Debug a Flex Specification */

2012-03-20T23:58:40Z

How to Debug a Flex Specification

← Older revision		Revision as of 23:58, 20 March 2012
Line 30:		Line 30:
	== How to Debug a Flex Specification ==		== How to Debug a Flex Specification ==

	There are various flags and variables to activate the debug functionality in Flex (there is no need to insert ~~useless~~ code such as printfs or similar).		There are various flags and variables to activate the debug functionality in Flex (there is no fundamental need to insert code such as printfs or similar, even through, of course, that style of debugging is also possible).

	In the Flex specification file:		In the Flex specification file:

← Older revision		Revision as of 18:17, 10 April 2016
Line 71:		Line 71:
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.		* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.
	* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java.		* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java (Test 1, 2016/04/08).
	[[category:Compiladores\|Flex Lexical Analyzer]]		[[category:Compiladores\|Flex Lexical Analyzer]]
	[[category:Ensino]]		[[category:Ensino]]

← Older revision		Revision as of 18:16, 8 April 2016
Line 71:		Line 71:
	* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.		* [[The Flex Lexical Analyzer/Exercise 7 - Text processing\|Exercise 7]] - Text processing.
	* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.		* [[The Flex Lexical Analyzer/Exercise 8 - Message list and simple accounting\|Exercise 8]] - Message list and simple accounting.
			* [[The Flex Lexical Analyzer/Exercise 9 - Comments/code ratio in C++ and Java\|Exercise 9]] - Comments/code ratio in C++ and Java.
	[[category:Compiladores\|Flex Lexical Analyzer]]		[[category:Compiladores\|Flex Lexical Analyzer]]
	[[category:Ensino]]		[[category:Ensino]]

← Older revision		Revision as of 18:08, 10 February 2017
Line 52:		Line 52:
	export YYDEBUG=1		export YYDEBUG=1

	== See Also ==		<!--== See Also ==

	* [http://flex.sourceforge.net/manual/ Flex manual @ sourceforge]		* [http://flex.sourceforge.net/manual/ Flex manual @ sourceforge]
	* [http://flex.sourceforge.net/ Flex site]		* [http://flex.sourceforge.net/ Flex site]
			-->

	== Examples ==		== Examples ==