This entry point is used when the lexer has finished the lexical analysis and the list of tokens is already in the memory. The pointer p points to the LexObject.
At this point the preprocessor may alter the tokenized form of the BASIC program. The list of tokens still contains the comment lines (also tokenized although it may make no sense), and the continuation lines are still being split containing the _ character and the new-line token.
The preprocessor may gain information from comments in case the some comments provide information for the preprocessor.
If the preprocessor uses some special symbols that drive the preprocessor processing but should be removed from the token list the preprocessor at this point may unlink the token from the list or just set the type of the token to LEX_T_SKIP or LEX_T_SKIP_SYMBOL.
If you do not or do not want to understand the difference between the two possibilities described soon then the rule of thumb is to use LEX_T_SKIP and you are safe.
The type LEX_T_SKIP should be used in case the token is handled due to ProcessLexSymbol preprocessor command and LEX_T_SKIP otherwise.
When the type is set LEX_T_SKIP_SYMBOL the lexical analyzer knows to release the string holding the symbol. If the type is LEX_T_SKIP only the token record is released.
If the symbol string is not released due to erroneously setting the type to LEX_T_SKIP instead LEX_T_SKIP_SYMBOL the memory will not be released until the interpreter finishes pre execution steps. So usually if you do not know how to set the type to skip a token LEX_T_SKIP is safe.
When processing comments the preprocessor should either use only the comments starting with the keyword rem or should carefully detect the comments starting with '.
For more information how to do it you really have to look at the function lex_RemoveComments in the file `lexer.c'.
The function has to return zero or the error code and should set the parameter *pCmd to PreprocessorContinue, PreprocessorDone, or PreprocessorUnload.