This entry point is used when the lexer object was allocated and initialized. The pointer p points to the LexObject.
The preprocessor at this point may alter the LexObject parameters. Here is a copy of the function lex_InitStructure from ScriptBasic v1.0build26. (other versions may slightly differ).
pLex->buffer = lexALLOC(BUFFERINCREASE*sizeof(char));
if( pLex->buffer )
pLex->cbBuffer = BUFFERINCREASE;
else
pLex->cbBuffer = 0;
CALL_PREPROCESSOR(PreprocessorLexInit,pLex);
}
void lex_InitStructure(pLexObject pLex
){
/*noverbatim
CUT*/
pLex->pfGetCharacter = NULL;
pLex->pfFileName = _MyFileName;
pLex->pfLineNumber = _MyLineNumber;
pLex->SSC = "QWERTZUIOPASDFGHJKLYXCVBNMqwertzuiopasdfghjklyxcvbnm_:$";
pLex->SCC = "QWERTZUIOPASDFGHJKLYXCVBNMqwertzuiopasdfghjklyxcvbnm_1234567890:$";
pLex->SFC = "QWERTZUIOPASDFGHJKLYXCVBNMqwertzuiopasdfghjklyxcvbnm_1234567890$";
pLex->SStC = "\"";
pLex->ESCS = "\\n\nt\tr\r\"\"\'\'";
pLex->fFlag = LEX_PROCESS_STRING_NUMBER |
LEX_PROCESS_STRING_OCTAL_NUMBER |
LEX_PROCESS_STRING_HEX_NUMBER |
0;
pLex->SKIP = " \t\r"; /* spaces to skip
\r is included to ease compilation of DOS edited
binary transfered files to run on UNIX */
pLex->pNASymbols = NULL;
pLex->pASymbols = NULL;
pLex->pCSymbols = NULL;
pLex->cbNASymbolLength = 0; /* it is to be calculated */
(Note CALL_PREPROCESSOR is a macro that call the preprocessor with appropriate arguments.)
The preprocessor may decide for example to alter the string SSC that contains all characters that may start a symbol, or SCC that contains all characters that can part a symbol or SFC that contains all characters that can be the final character of a symbol. This way for example a preprocessor may set these strings that allows Hungarian programmers to use ISO-Latin-2 accented letters in their variables. (However those characters are going to be case sensitive.)
The preprocessor may also set the pointers that point to the tables that contains the alphanumeric symbols (pASymbols), non-alpha symbols (pNASymbols) and the table used for some ScriptBasic internal debugging purpose (pCSymbols).
The preprocessor may also release and reallocate a smaller or larger buffer if wishes. (I personally see no reason.)
The function has to return zero or the error code and should set the parameter *pCmd to PreprocessorContinue, PreprocessorDone, or PreprocessorUnload.