Aurel tells All: Interpreters

[bplus]

Hi Aurel, 

If you are waiting for a written invitation, here it is!

If you want us to beg for it, here it is: Please Aurel, tells us what you've learned about Interpreters

Don't let this stuff go to the grave when you do.

[Aurel]

Hi mark and everybody else!
I have learned a lot and i become confused a lot.
ok not just like this because there are few different approaches to do that.
So we must make decision which way to use that can be produced on
our members dialects.
1.
There is a way used in toy interpreter created by EdDavis
ad Mark know that ,and he also know that source code belong to Ed and
one guy called Mike pretend to credits and his modification when we work on that.
It is not perfect but is very fast because is bytecode interpreter.
Also the way how is created is not easy to understand for most our members.
So i think that such a way is not good option for all.
What you think?
2.
There is a way presented on CodeProject written in JavaScript by one young guy
PeterOlsen - similar parsing method is used but he also create AST.
AST require data structure to store parsed code which then be executed by virtual machine.
So require translation from JS to Basic functions.
Tokenizer is simple and well written .i mean in very understanding way.
3.
Next way is something like jwilla Basic written by Jerry Williams in pure C.
Whole interpreter is written in one C header file cca 300 lines of code. 
Also require translation fom C to BASIC.
4.
There is project called qb50 written in qBasic or old Visual Basic but
source code is very large and not easy to follow.

I have lot of similar things (open source files) written in other programming langs.
So we must agree about one thing first:
write just tokenizer and execute tokenized code by parser

[bplus]

Thanks Aurel, we are started.

I am interested in byte code, I have seen you write about it often and I think I want as much speed as possible for an attempt of Interpreter specially for graphics. That is my preference but others may have different ideas.

Maybe everyone can follow in their own favorite flavor or PL.

PS I'd like to see Ed Davis attend our lessons, dang I think I chased him off when I was giving mine, dang I just wanted to finish my say first. Sorry again Ed Davis.

[Aurel]

Ps Mark
I dont ..with really full respect to Ed ..but simple because of bad experience i
have in past with toy interpreter.
By the way you can still found his work on qb64.net forum.
Also his method is good for console only programs because it is  almost
impossible to implement windows message system.
Also i like to see more contribution from members of TJP forum
including Walt of course

[bplus]

YES more contributors AND students of interpreters!

[eoredson]

How come every time somebody posts interpreters they skip over mine:

http://qb64.thejoyfulprogrammer.com/show...p?tid=1095

That's just SICK!

(can U parse basic using basic??)

[bplus]

How come every time somebody posts interpreters they skip over mine:

http://qb64.thejoyfulprogrammer.com/show...p?tid=1095

That's just SICK!

(can U parse basic using basic??)

Then what would be left for you to do?

So Erik, how did you get started with SICK? I doubt that was your first interpreter. Are you in the token selling business too?

Moses parted the sea with a s(t)ick, so parting the sea of string(s) with Basic might be possible. 

[bplus]

Hi Codeguy, 

I can't tell if you are genius or madman, but this is funny!

Code:

PRINT "yeah baby --> jump to sub "; wlist(i&; "from line "; ProgramCtr&

(Dang this editor, cant delete quote and picked up foreign font from copy??? )
((OK got quote deleted but now the smilie wink??))
(((OK smilie gone in post (and 2nd time in editor), so editor is NOT WYSIWYG)))

Recursive search through a text file for keywords??? odd
Like quick sort??? what?

'* code is a LOT like quicksort, except perhaps without the quick and the sort. 

Oh ha! missed the 2nd half, too early in morning for me, LOL! 
I luv it!

Hard to follow, fun to watch!

Oh maybe genius madman:
https://www.google.com/search?q=yeah+bab...AA#imgrc=_

[Aurel]

Hi to all..

@erik
I looked at your interpreter kit and i found it nice but complete code is not
very much clear or better to say is not transparent enough.
I have build two interpreters in past but no one of them is not good enough.
So i expect  your contribution here.

@codeguy
Your code is really funny to read and let say you do it very close to my
old Aurelbasic...in fact you  wrote some sort of tokenizer.



Guys there is no need to rush ,if we wish to create solid and usefull
interpreter we must agree about few things.
1.interpreter with direct execution - very slow
2.interpreter with tokenized code but executed with parser-evaluator combination - slow
3.interpreter with tokenized code +parser which build AST and executed with tree walker- faster
4.tokenized code+ parsed to bytecode then executed by virtual machine - fast

So what to start?
It is not the same what we want to use because some dialects cannot do all of this .
And it is not same if I use o2/FreeBasic who is native compiler or SB/qb64 which are
bytecode interpreters.

[Aurel]

Still no one ?
no problems...
I have found one interesting thing which can simplify building AST tree
how? read next ...

Such trees do not necessarily need to be implemented using data structures conventionally used for trees. Instead, tokens can be stored in flat structures, such as tables, by simultaneously building a priority list which states what elements to process in which order.

This can make it more simple and easier to understand than building tree
with structures like linked list using nodes.
Above text is from parsing with precedence climbing method.

[Aurel]

Of course ...
we must first get to tokens..
tokens are parts produced by tokenizer or lexer , probably you know that?
let's look what are tokens in simple example:
"1 + 2 * 3"
our tokenizer must know how to properly exctract tokens from that string.
If we build our token list as array which is simpliest data storage then look like this:
in fact two arrays - one is tokenType and another is tokenValue
arr1-tokenType  |  arr2-tokenValue
number                    ->    1
operator                   ->    +
number                    ->    2
operator                   ->    *
number                    ->    3

so ...is that ok for start for this thematic ?

[bplus]

No one? I count me, Erik and Codeguy for starters...  3 is a major crowd here at TJP ;D

Are we suppose to agree on Basic dialect before getting started on explaining what the heck tokens are?

What byte code is? 

I suspect compilers faster than byte code interpreters so SB is out of picture but you included QB64 as byte code interpreter, why?

append: crud our posts crossed he answered before I asked

[bplus]

Yes, good start, tokens are what I call words both are symbols for some THING.

So tokens are identified by types. (Append: Here is where we diverge, for me words were typed by the position they occupy in the program line, big difference from get go!)

How do you tell a - sign for subtraction and a - sign that shows a number negative? Is it number or is it operator? (I ask because that is still a headache in BRUN 2.)

[Pete]

You guys better decide on a language pronto or what is going to take forever is going to take forever and a day... And I have golfing plans on that day.

Pete

[Aurel]

What a heck happened when i typing reply?
damn ..it looks when you Mark made reply my reply goes in the emptiness
crap !""##$#$$%%&&


Ok. i must read again your reply

[Aurel]

Well as i say there is no need to rush..
Tokens are not words,than are parts of input string or source code
splitted and extracted .
token is operator and token is number and keyword is token too.
in -3 . symbol (-) is unary operatror ... is that clear?
I am talking to much probably and my english is not the best
but i wish present it here as clear as possible because this thing is no that
easy as we may see it.
Mark .. qb64 is not native compiler which compile directly to
machine code ...then is bytecode interpreter using
C++ runtime ,so this runtime interpret C++ code which is before translated from BASIC to C++.
Simlar model use BCX which translate code to C then is compiled
with C compiler.

[bplus]

I thought when QB64 is compiled, it was then in machine language whereas, SmallBASIC is always interpreted from text file.

[Waltersmind]

@bplus,

QB64 translates to C++ and then uses the C++ compiler and linker to create the executable.

[bplus]

Yes, I knew that but the final product is binary.

Oh, don't tell me it's attached to the end of a runtime file like an SdlBasic stand alone is made.

No tell me if true, and I will have a whole new disrespect for a QB64.exe

[Aurel]

@Walter
Do you think that i explain too wide?
I have that feeling 
maybe should be better to add some code..hmmm