Aurel tells All: Interpreters

[Waltersmind]

@Aurel,

I think your explanations are good, but it always helps to see some code, even if it is pseudocode.

[Waltersmind]

@Bplus,

QB64 translates everything into C++, which you can see everybit of the translations in the temp folders, and then calls the C++ compiler, as well as the C++ linker, to create a full executable from the translated C++ code.

[bplus]

Hi Aurel, 

Don't worry about English, your good intentions make up for any problems there.

[eoredson]

So Erik, how did you get started with SICK? I doubt that was your first interpreter. Are you in the token selling business too?

I started with Whatis (recursive descent parser) as the engine to SICK and my token list is:

TokenList = " -+*/\^()[]{}<>=|&!%~?:#@`;,'" + Quote

and yes, 16,000 lines of code is hard to read..

Erik.

[Aurel]

Erik
It is not hard to read because you do it statement per line which increase
number of lines to 16000.
Nothing wrong with that but what you do in SIC is similar what I do in
AurelBasic - to much string operation which as result have slow interprerting.
Yes AurelBasic is very slow , some improvement I do with Ruben Interpreter
where I preparse code to string arrays( yes multiple arrays)
what produce little bit faster execution but still slow
because i dont build tokenizer and also lot of string operation is used.

Your tokenList is not token list then list of delimiters or special
signs used for direct parsing..
if you look somewhere on TJP forum i publish JimKlutho basic
originaly written in PowerBasic which use far better and simplier
method then I use in ABasic/Ruben and you in SIC.
So You and Me made similar mistakes with our code but
that is nothing strange we do that becuase we do it as we know/think that is ok.
I hope that you dont get me wrong ..i like your work it is really nice.

ps ...only thing i dont like is line numbers which are really vintage
Is that ok?

[eoredson]

Thank you,

and it is "vintage" and is inefficient parsing, but when I use a line numbered array and I

  GOTO 10

I know it gotos array element 10 without searching for the line label.

I spose it's possible to remove line number constraints but I'm not going to.

Erik.

[Pete]

Well there's remline and I wrote a similar utility 10+ years ago to remove line numbers unless they were referenced in  GOTO, GOSUB, or THEN statement. Personally, I'm glad QB languages accept both line numbers and labels but very happy line numbers are no longer mandatory. The old days sucked in that regard. At least TI was smart enough to put a line number changer as a utility in its TI BASIC suite.

Pete

[bplus]

Silly rabbit, line numbers are for kids!

But they sure do come in handy for referencing error occurrences!

[Aurel]

I know it gotos array element 10 without searching for the line label.

erik
you don't need to search for line label
label as function name is unique so what you need is position of your label
stored in ordinary array like
STRING LabelPosition[pos] = "label_name"
so when you find "name" just read index -> position
and go to next line position++ 
becuse you cannot jump on label then on next line or next statement
if you use statement delimiter as -> :

here is part from Ruben interpreter:

Code:

SUB exec_JUMP() as INT
'syn JUMP label
string mark,name : INT pos,i
mark = arg1[PC]
'print "Mark-Label" + mark
For i = 1 TO 100
'print  "LN:" + labelName[i]
    IF mark = labelName[i]
         pos = labelPos[i]
         'print "LABEL->POS:" + str(pos)
         Return pos
    END IF
Next i
print "ERROR: Label Not Found!->" + mark
perr=1 '->exit from main
Return 0
END SUB

[bplus]

Yeah, that way is probably faster than what I am doing. Already noted that at QB64.net when I saw rks(?) docs for his interpreter. He also worked in the jumps for IF blocks but he used line numbers

Well I guess everyone uses line numbers, the question is whether the user needs to punch them in the editor as necessary part of the coding process.

[phred]

As a correction(?) my interpreter uses labels but assigns the value of the line number to it during the load process so to the user it looks like a label but internally it's a line number.  I agree with bplus that this is probably not at all atypical but I haven't dissected that many interpreters to verify. 

phred (aka rks)

[bplus]

Phred that is you! 

It is a small world.

[Aurel]

finally i found ebook :
Writing Interactive Compilers and Interpreters
it looks very interesting and easy to read..
also i found in my archive SLC - stupid little compiler with very simple tokenizer
i will see how work then i post it here:

[codeguy]

Using line numbers as direct indexes to the Jump Table is far faster than using HashTable lookup. The scalar (variables that do not have a subscript, eg: ThisVariable# or the like) variables can be put in a HashTable for quick lookup. Python does this, as do other languages. Some use a pointer to its offset in memory, but this is TERRIBLE for anything other than fixed-length variables, such as string literals and numeric variables of fixed size. For these types, pointers are fine. For others, a HashTable is probably much better as it does not care about the size of the element, only its hash function ID that determines where in the HashTable it will be found.

[codeguy]

I do not understand why my code was deleted. Yes, it worked to do what it was supposed to do: find the lines that are the objects of goto, then, gosub, function calls, et cetera. If this code would have been left, that may have helped others in parsing qbxx source. The printed statements with the ascii arrows were to distinguish the lines containing these from statements that did not. I didn't know I had to be 100% serious. This is code posted exactly as it was on qb64(net) long ago and obviously this wasn't problematic. Yes, it was a beginning of a tokenizer, including recursive merging of source in $include files. There was nothing offensive in my code. It was pretty much matter-of-fact.

[codeguy]

Also included in this code was operation of the stack for jumping to subs/functions and susbsequently returning to the line/position in source code following the requested jump. I essentially created a jump table and execution stack of fully $INCLUDEd code, even nested. It also avoided infinite loops of recursive code and indicated that it may be such. which would have involved further pushes and pops onto and from the execution stack. I would have created a HashTable procedure to insert scalar variables in similar manner to what Python and other languages also do. It would have been easy to also store arrays and their data and use offsets indicated by subscripts to retrieve eg: the nth element of some array, be it string or numeric. That wouldn't have been problematic as I am sorta nimble with these data structures.

[Aurel]

Vary good explanation @codeguy
I really don't know who delete your code please post it again.

[phred]

Hey bplus, I saw my name and figured I'd say howdy!  
I've been following this thread with interest.  I've also read Writing Interactive Compilers and Interpreters, I have a copy of it in a box somewhere, (story of my life), but it is an informative book.  In my interpreter I use an index table for labels and jumps so when one is referenced it checks for name validity and retrieves the line number associated with it.  I originally used line numbers directly and that got old the first time I added a line and had to change a bunch.

[Aurel]

Hi phred
or FRED sounds more logical...
yes i agree with you ...this book offer informative frame about
"how to do that " but without real examples even author like to
tend  to BASIC code, but no examples in BASIC or any other language.

First stage of any solid state interpreter should be tokenizer
which output must be array-s of tokens ,or linked list of tokens,or
hash table of tokens or UDT.array of tokens ...
This token list must have types numbered as constant or intege

operator =  1
number =   2
variable =   3
string    =   4
etc...

is that ok?

[bplus]

You know between Aurel, Phred and yes even Bill, I have a new Interpreter boiling on my back burner called Bflat.

flat as in flat out fast as I can run and flat because no dang fat eval function!

Expressions will be mighty in their simplicity!

Well that's the gist of the dream anyway....

Hey sorry I missed Codeguys's code, what is going on with deletions????

Codeguy, have you discovered QB64.org ?

Maybe more posts will stick?