Aurel tells All: Interpreters

[Aurel]

what a heck is going on here

[Aurel]

what a heck is going on here
what... i must wait 33 seconds after my last post ,reply
this is rediculus SMF do that in microsecond

[bplus]

You OK Aurel?

[Aurel]

I am ok Mark
but this forum software is not
I suggest to Walt that remove all irrelevant posts or this
forum will crush...geee
I lost 3 replys on this topic..and I become pissed off..

[Pete]

I've never had a problem with it. Hey maybe it's because your posts have to go through Customs?

Pete

[bplus]

Oh I found free on-line PDF: Writing Interactive Compilers and Interpreters by PJ Brown
https://www.scribd.com/document/96516909...ters-Brown

from 1979, I guess some things never get old.


Looks like good bedtime story...

[Pete]

Bill, where the hell were you in 1979? If anyone ever needed your "Shut up and code!" rant, it was the guy who wrote that interpreter article. Either that, or I just discovered my hobby is boring, boring, boring! No wonder I don't even bother to comment my code, Geeze!

Pete [Happy balls come from California]

[Aurel]

No ...book is good and give proper direction to create interpreter/compiler
but there is not real examples

[bplus]

Curiouser and curiouser, what is a token? 

What is difference between what Aurel calls token and what I call Word?

Is it that a token is a word that is classified as some type of thing like operator, literals: number or string, or variables: number, string, array, type, object... 

What is tokenized file? like qb4.5 thingy, JB has it too I think.

Dang it, do I have to learn this from strangers in books?

You guys are condemning to read this stuff? 

What did I ever do to you to deserve this cruel punishment?

[eoredson]

What is a token?

(1) In programming languages, a single element of a programming language. For example, a token could be a keyword, an operator, or a punctuation mark. (2) In networking, a token is a special series of bits that travels around a token-ring network.

What is a word?

Basically, the gist of it is that a WORD is traditionally the size of a memory pointer, the 'most natural' size of data that can be read. DWORD is twice that size, QWORD is four times, etc.

Now, traditionally that would be 16 bits of data; a legacy to the 16 bit MSDOS days. Nowadays, though, it's generally 32 bit or 64 bit, according to your computer's architecture. So, really, a DWORD on modern computers should be 128 bit!

[bplus]

Thanks Erik,

Traditionally bplus (Mark) calls a word == that which is separated by at least one space on either side.

In the above sentence side. is a word.

The word is out now.   < 5 there

1 In the beginning was the Word, and the Word was with God, and the Word was God.




 Perhaps I am too free with my Interpreters?

x = 99 * beers / wall  ' < 7 words
x=99*beers/wall        ' < 1 word!!!

[eoredson]

Semantics/syntax/linguistics:

Definition of expression.

   String/numeric value calculated.

   Expression contains:

      [strings][numerals][constants][operators][symbols][functions][variables]

Which would you like?

[bplus]

I would like to know what you call these items:

 [strings][numerals][constants][operators][symbols][functions][variables]


generally?


What do you call the items that make up an expression?


I think you are calling them tokens and I am catching hell for calling them words when the items are separated by spaces.

[eoredson]

A term within brackets denotes a type of keyword describing the value or function:

X = X + 1

breaks down into:

[variable] [symbol] [variable] [symbol] [numeric]

which would be interpreted.

[eoredson]

1 In the beginning was the Word, and the Word was with God, and the Word was God.

So it is written, so shall it be.

(so write code)

[Aurel]

Not exactly like this and we must be more precise on this things:

      X              =                    X               +                 1
 variable   equalSym         variable      operator        number

or
      X              =                    X               +                 1
 ident       equalSym            ident       operator         number

if we wish to build tokenizer first then must be like above.
Direct interpretation does not require separate tokenization stage
than just parse to var,term,factor and evaluate it.
Do you see the difference?

[eoredson]

In a tokenized stream any indent is interpreted as a whitespace

which is Tab(9), Space(32), or CR/LF(13/10) and after that all tokens are parsed,

and whitespace is ignored. Also delimiters are parsed similar, for example , or ; could be separaters.

Of course : is specialized to separate functions.

[Aurel]

dear lord ..this forum have a such a stupid problems that i cannot
believe ..that stupid text editor not show properly my text ..damn!
i see that i must post image instead of text

[Aurel]

Erik
wait a moment i was talking about tokenizer NOT tokenized
stream ... i am still on tokenizer NOT parser...
first we need few functions(subroutines) to build tokenizer.
But yes i agree with you tokens can be also saved in
as you called token-stream if you use STEP 2 and put whole
set of tokens in one array of strings like this :
tokArray[1] = "variable"
tokArray[2] = "X"

tokArray[3] = "equalSymbol"
tokArray[4] = "="

tokArray[3] = "variable"
tokArray[5] = "X"

tokArray[6] = "operator"
tokArray[7] = "+"

tokArray[8] = "number"
tokArray[9] = "1"

[STxAxTIC]

Sorry to interrupt right in the middle, but this topic reminds me that I never explained how I cracked this particular nut in the Sxript project - and for good reason: I did zero reading, looked at zero already-completed projects before starting this, and had no idea I was stepping into a trench. Didn't know I would be calling it a "language" at all, and I still wonder if I should. What I ended up with is... hard to describe. It does all the baseline stuff a language ought to do: variables, functions, subs, recursion, supports both eager evaluation and lazy evaluation, etc etc. The downisde is it lives in the string space of other languages, so its not the fastest thing out there.

(((...And as a complete aside, that's exactly the way hobby programmers and amateurs ought to do it for their first time. I think that all of the formal discussions make WAY more sense after you spend a few months trying everything to get your toy language to perform as well as your intuition can deliver. In other words, I encourage practice first, theory later. And that's a weird statement coming from me...)))

That said, I'm trying to effectively show all of you my tokenizer, lexer, parser, and pasta maker all in one bang - and it's all mashed together in a different way, so don't look for those parts explicitly. I have prepared (After failing at a flowcahrt several times) some pseudocode. Turns out that I just needed to take my main Eval function, delete all the code and keep the comments, and suddenly I had something to show.

Here's the whole function with every detail: http://www.sxript.com/sxript/docs/flowch...wChart.txt

It probably looks bad in a code box if the text wraps, but here is the same thing without the right column for spacial efficiency:

Code:

Notes:
  (1)  The term "evaluate*" denotes recursion/looping i.e., output is used as next input.
  (2)  The (--> ^) symbol denotes (Return = LeftFragment + MidFragment + RightFragment) and return to top.

' '''''''''' '''''''''' '''''''''' '''''''''' ''''''''''                                                                
                                                                                                                      
FUNCTION EvalStep$ (TheStringIn AS STRING)                                                                            
                                                                                                                      
    ' Replace variable references with literal representations:                                                        
    '   Variables in square brackets ( [] ) are replaced outside of all occulted ( {} ) brackets.                      
    '   Variables in curvy brackets ( {} ) are replaced down to the fourth (4th) level of occulted ( {} ) brackets.  
                                                                                                                    
    ' Scan for most-embedded structure and break the input string into fragments:                                      
    '   LeftFragment = Everything to the left of the inner-most parenthesized structure or ScannedName (can be empty).  
    '   ScannedName = Word to the left of the most-embedded set of parentheses (if found).                              
    '   MidFragment = (i) Most-embedded structure contained in parens ( ), OR            
    '                 (ii) If no parens detected, the whole input string.    
    '   RightFragment = Everything to the right of the inner-most parenthesized structure (can be empty).
    '   Note: LeftFragment and RightFragment remain static. All changes occur in ScannedName and MidFragment.
                                                        
    ' If input expression contains no zero-level (unocculted) parens:  
                                                                  
        ' Pass through NumberCrunch (regardless of input type).                                    
        '   If NumberCrunch changes input string, loop until a stable result is reached. (--> ^)                  
        '   If input was already stable (emerges unchanged), pass control downward.                      
                                                                                            
        ' If handed a vector, evaluate* each component. (--> ^)          
                                                                                                        
    ' If input expression contains zero-level parens but no scanned function name:                        
                                                                                                              
        ' If the argument is singular, strip the outer parens and evaluate* content. (--> ^)                
                                                                                                          
        ' If argument is a parenthesized list, evaluate* each component and retain parens. (--> ^)        
                                                                                                                
    ' If input expression contains parens and a name was scanned:                    
                                                                                                        
        ' If ScannedName is a user-defined function:                                  
        '   Replace the function call with the literal function content.                                      
        '   Replace variables [x], [y], etc. with arguments sent to the function. (--> ^)  
                                                                                                  
        ' If ScannedName is a primitive, pass through FunctionCrunch. (--> ^)        
                                                                            
END FUNCTION                                  

' '''''''''' '''''''''' '''''''''' '''''''''' ''''''''''