freeling::lexer Class Reference

Simple RegEx lexer to parse UTF8 grammar files without depending on flex (or the like) More...

#include <lexer.h>

List of all members.

Public Member Functions
	lexer (const std::vector< std::pair< freeling::regexp, int > > &)
	constructor
int	getToken (std::wistream &)
	get next token from stream (or in the buffer)
std::wstring	getText ()
	get text for last token matched
size_t	lineno ()
	get last line parsed
Private Attributes
std::vector< std::pair < freeling::regexp, int > >	rules
	to store regexps and associated tokens
std::wstring	buffer
	keep state of the parsing
std::wstring::const_iterator	beg
std::wstring::const_iterator	end
size_t	line
std::wstring	text
	resulting token
std::vector< std::wstring >	rem

---

Detailed Description

Simple RegEx lexer to parse UTF8 grammar files without depending on flex (or the like)

---

Constructor & Destructor Documentation

freeling::lexer::lexer

(

const std::vector< std::pair< freeling::regexp, int > > &

)

constructor

Constructor.

---

Member Function Documentation

wstring freeling::lexer::getText

(

)

get text for last token matched

Get text matched by last token found.

int freeling::lexer::getToken

(

std::wistream &

)

get next token from stream (or in the buffer)

Get next token from stream (or buffer)

size_t freeling::lexer::lineno

(

)

get last line parsed

Get number for last line processed.

---

Member Data Documentation

std::wstring::const_iterator freeling::lexer::beg [private]

std::wstring freeling::lexer::buffer [private]

keep state of the parsing

std::wstring::const_iterator freeling::lexer::end [private]

size_t freeling::lexer::line [private]

std::vector<std::wstring> freeling::lexer::rem [private]

std::vector<std::pair<freeling::regexp,int> > freeling::lexer::rules [private]

to store regexps and associated tokens

std::wstring freeling::lexer::text [private]

resulting token

---

The documentation for this class was generated from the following files:

• lexer.h
• lexer.cc