SQF scripting - is it interpreted?

[Simas 12]

I was wondering about the current SQF scripting state in Alpha - does the Arma 3 scripting engine is using some sort of intermediate opcode/bytecode level or it's purely interpreted language? What for is that compile command exactly? The Wiki documentation is lacking..

You can still see the actual code when debugging the compiled "Code" type. Is this some sort of a meta output only?

If the SQF is indeed an interpreted language in the year of 2013.. I am baffled. We have issues with CPU utilization already, having the SQF use at least an intermediate opcode level (and JIT compilation in the future) would solve so many performance issues?

Pretty much all modern scripting languages use at least the opcode/bytecode level:

Lua - opcodes

LuaJIT - opcodes + JIT

v8 JavaScript - pure machine code generated on-the-fly

Python - opcodes

PyPy - Python opcodes + JIT

Did I miss something obvious about the SQF state? Are there any scripting engine performance changes planned for the final Arma 3 release (except the Java thingy)?

Edited May 15, 2013 by simast

[xendance 3]

As a computer science student, I'm intrigued about this too.

[Milyardo 10]

I was wondering about the current SQF scripting state in Alpha - does the Arma 3 scripting engine is using some sort of intermediate opcode/bytecode or it's purely interpreted language?

Neither.

What for are the compile command if nothing is actually compiled?

What makes you think nothing is compiled?

The reason I am asking this is because you can still see the actual code when debugging the compiled "Code" type. Unless it's some sort of a meta output?

Thats what debug sybols are for.

Now if the SQF is indeed an interpreted language in the year of 2013.. I am baffled. We have issues with CPU utilization already, having the SQF use at least an intermediate opcode level (or even JIT compilation in the future) would solve so many performance issues?

Bytecode VMs were introduced to address problems with application portability, not performance.

I apologise for the terse responses, in my opinion your question illustrates that you don't have enough familiarity with formal grammar, language classification, and turing machines to make this a worthwhile disucssion.

Edited May 13, 2013 by Milyardo
Grammar

[xendance 3]

Neither.

What makes you think nothing is compiled?

Thats what debug sybols are for.

Bytecode VMs were introduced to address problems with application portability, not performance.

I apologise for the terse responses, in my opinion your question illustrates that you don't have enough familiarity with formal grammar, language classification, and turing machines to make this a worthwhile disucssion.

What has formal grammar, language classification and turing machines have to do with being interested in how the engine interprets/compiles/runs the scripts? Especially since you told him "fuck you" basically.

[kremator 1064]

Gentlemen, to your corners! Seconds out, round 2!

[Simas 12]

Neither.

Which means the compile call is stripping whitespace and stuff? :) You do sound like you know something about SQF implementation.. can you please enlighten us? Why is this such a secret? The BI wiki has literary zero info on the subject.

Bytecode VMs were introduced to address problems with application portability, not performance.

I get it, MSIL for .NET, Java bytecode for JVM. But opcode layer is also orders of magnitude more efficient to "interpret" not to mention all the possible low level optimizations.

Edited May 14, 2013 by simast

[Tankbuster 1733]

in my opinion your question illustrates that you don't have enough familiarity with formal grammar, language classification, and turing machines to make this a worthwhile disucssion.

In my opinion, your reply illustrates that you are a cock.

[Milyardo 10]

In my opinion, your reply illustrates that you are a cock.

That's a little uncalled for. If I'm really being uncessarily terse then by all means elaborate on the subject and answer his question. In my opinion there isn't a question here that can't be answered meaningfully.

What has formal grammar, language classification and turing machines have to do with being interested in how the engine interprets/compiles/runs the scripts? Especially since you told him "fuck you" basically.

Those are the fundamental concepts to compilers and machine level instruction processing.

---------- Post added at 04:33 PM ---------- Previous post was at 03:54 PM ----------

Which means the compile call is stripping whitespace and stuff? :) You do sound like you know something about SQF implementation.. can you please enlighten us? Why is this such a secret? The BI wiki has literary zero info on the subject.

I won't make the claim that I know anything about the implementation. However Armed Assault config and script files can be compiled to a binarised format. This documetation(https://community.bistudio.com/wiki/raP_%28ArmA%29) is pretty outdated, but is true that there is an intermediate representation of a SQF, however this intermediate representation doesn't qualify as a bytecode because this binarised format still requires parsing by a grammar to make a set of instructions free of semantics like object type and scope.

I get it, MSIL for .NET, Java bytecode for JVM. But opcode layer is also orders of magnitude more efficient to "interpret" not to mention all the possible low level optimizations.

JIT and low level optimizations like you're infering to come from using a virtual machine to interpret a bytecode language and not an inherit property of bytecode itself. A perfect counter-example would be PHP(though PHP's interperter is becoming more virtual machine like and this is less true than it has been in the past), it has a byte-code representation, however by using bytecode format files the only benefit you gain is during program loading since you skip the step of grammar parsing.

[fred41 42]

... back to thread topic ...

I think it is very important for mod/mission developers to understand the script processing engine mechanisms.

There is a helpful wiki about optimizing script code here:

http://community.bistudio.com/wiki/Code_Optimisation

This doesnt answer the initial questions, but could help a bit to clear the fog.

The fact that compiled code is running up to 20 times faster then uncompiled, is a hint that the compiled code is something like metacode?

Maybe one of the BI-developers here, could help us a bit to understand the script processing a bit better?

Thanks in advance :)

Edited May 14, 2013 by Fred41

[Simas 12]

I won't make the claim that I know anything about the implementation. However Armed Assault config and script files can be compiled to a binarised format.

Sorry but that's apples and oranges. Config files are data, hardly a programming/scripting language. There is no logic to them at all (besides the pre-processor). Same as JSON having a BSON (binary JSON) representation.

This documetation(https://community.bistudio.com/wiki/raP_%28ArmA%29) is pretty outdated, but is true that there is an intermediate representation of a SQF, however this intermediate representation doesn't qualify as a bytecode because this binarised format still requires parsing by a grammar to make a set of instructions free of semantics like object type and scope.

And you know that based on that link? It didn't even mentioned SQF or scripting. As far as I know you cannot rapify SQF files with official BI tools. Binarized format is used for config/data files.

JIT and low level optimizations like you're infering to come from using a virtual machine to interpret a bytecode language and not an inherit property of bytecode itself.

Let's reverse this: can you have a VM without a bytecode layer? Unless you mean something even more advanced, like Google v8.

A perfect counter-example would be PHP(though PHP's interperter is becoming more virtual machine like and this is less true than it has been in the past), it has a byte-code representation, however by using bytecode format files the only benefit you gain is during program loading since you skip the step of grammar parsing.

I just happen to know A LOT about PHP as this is my primary language for 10 year now. So.. it got opcodes with Zend Engine 2 (released 9 years ago). They do optimize the opcode level during the grammar parsing process. There are also higher level optimizers in play (APC and Zend Optimzier). So what does it make it then, not a Virtual Machine?

[sbsmac 0]

>The fact that compiled code is running up to 20 times faster then uncompiled, is a hint that the compiled code is something like metacode?

The comparison being made there is compiling once vs compiling multiple times inside a loop (by using execVM). SQF is always 'compiled' (in the Arma sense) even if it's done implicitly by the engine when loading init.sqf or running execVM.

>but is true that there is an intermediate representation of a SQF, however this intermediate representation doesn't qualify as a bytecode because this binarised format still requires parsing by a grammar to make a set of instructions free of semantics like object type and scope

.rap is incapable of expressing sqf (except as an embedded string) and in any case is a storage format rather than an execution format.

*Edit* (Too long since I actually wrote any sqf!) You can get an idea of how much is determined at 'compilation' simply by checking when the engine emits errors of various kinds.

Function calls are obviously symbolic references since you can redefine the contents of a function by 'compiling' different code against the original symbol (as was done in the recent MP hack)

Type correctness (what little of it there is) is determined at runtime not 'compile' time.

IIRC operator and syntactic correctness is determined at 'compile' time which would imply that the internal representation is some kind of expression tree but perhaps I'm more disposed to think that's true because it's how squint represents code for analysis.

Edited May 14, 2013 by sbsmac

[Milyardo 10]

Sorry but that's apples and oranges. Config files are data, hardly a programming/scripting language. There is no logic to them at all (besides the pre-processor). Same as JSON having a BSON (binary JSON) representation.

And you know that based on that link? It didn't even mentioned SQF or scripting. As far as I know you cannot rapify SQF files with official BI tools. Binarized format is used for config/data files.

JSON/BSON is an excellent analogy for raPification. The docmentation states "raP encoding applies to any humanly readable text file in OFP that contains class statements.". So while it is true that raPification doesn't apply to SQF files specifically, it is not because it contains SQF contains logic or satements. Both raP and BSON are forms of serialization.

Let's reverse this: can you have a VM without a bytecode layer? Unless you mean something even more advanced, like Google v8.

I don't understand exactly what you're asking.

I just happen to know A LOT about PHP as this is my primary language for 10 year now. So.. it got opcodes with Zend Engine 2 (released 9 years ago). They do optimize the opcode level during the grammar parsing process. There are also higher level optimizers in play (APC and Zend Optimzier). So what does it make it then, not a Virtual Machine?

Yes, the Zend Engine is a virtual machine. The Zend Engine however is not the classical PHP interpreter which is not a virtual machine(more or less, like I said this has changed relatively recently). The article you linked specfically talks about features from Zend the classical php interpreter will borrow.

[Simas 12]

JSON/BSON is an excellent analogy for raPification. The docmentation states "raP encoding applies to any humanly readable text file in OFP that contains class statements.". So while it is true that raPification doesn't apply to SQF files specifically, it is not because it contains SQF contains logic or satements. Both raP and BSON are forms of serialization.

And bytecode/intermediate code (which is the question of this thread) are not serialized data. So neither raP nor BSON is relevant to SQF!

I don't understand exactly what you're asking.

You were saying - the optimizations are a property of virtual machine, not the bytecode. I am asking - can you have a virtual machine without a bytecode layer (and whats the point of your statement)?

Yes, the Zend Engine is a virtual machine.

The Zend Engine (PHP 4) is/was an interpreter.

The Zend Engine 2 (PHP 5) is a virtual machine (without JIT), just like early JVM.

The Zend Engine however is not the classical PHP interpreter which is not a virtual machine(more or less, like I said this has changed relatively recently).

I just wrote you the Zend Engine 2 (as a VM) together with PHP 5 was released 9 years ago..

The article you linked specfically talks about features from Zend the classical php interpreter will borrow.

... clearly you do not know the answer to the question. The only option left is to wait for any dev to jump in and to clear it up.

[Milyardo 10]

And bytecode/intermediate code (which is the question of this thread) are not serialized data. So neither raP nor BSON is relevant to SQF!

[sic]

... clearly you do not know the answer to the question. The only option left is to wait for any dev to jump in and to clear it up.

You asked if SQF was interpreted or if it was run from bytecode. It's neither, it's compiled. I did state there was a intermediate representation in ArmA(raP) but incorrectly assumed it applied to SQF. However I was correct in stating that intermediate representation is not a bytecode language.

The Zend Engine (PHP 4) is/was an interpreter.

The Zend Engine 2 (PHP 5) is a virtual machine (without JIT), just like early JVM.

I just wrote you the Zend Engine 2 (as a VM) together with PHP 5 was released 9 years ago..

I don't understand how this anything different from what I said, other than you seem to be implying that Zend and the classical PHP intrepreter are the same which they are not.

EDIT

You were saying - the optimizations are a property of virtual machine, not the bytecode. I am asking - can you have a virtual machine without a bytecode layer (and whats the point of your statement)?

Your question still doesn't make sense. What exactly is a 'bytecode layer'? You have machines and you have input. Bytecode is input into a virtual machine.

Edited May 14, 2013 by Milyardo

[fred41 42]

>The fact that compiled code is running up to 20 times faster then uncompiled, is a hint that the compiled code is something like metacode?

The comparison being made there is compiling once vs compiling multiple times inside a loop (by using execVM). SQF is always 'compiled' (in the Arma sense) even if it's done implicitly by the engine when loading init.sqf or running execVM.

... yes, it seems you are right here, my translation was wrong and i completly misunderstood this for weeks, thanks for enlightenment :)

BTW: your SQUINT looks very interesting (will investigate it more in the next days). Do you plan to maintain it for arma 3 too?

[GottyPlays 10]

what hte hell are you guys talking about? :O

[Simas 12]

You asked if SQF was interpreted or if it was run from bytecode. It's neither, it's compiled.

That's what the scripting command is named - "compile". C/C++ is compiled. That would be the same as Arma?

I don't understand how this anything different from what I said, other than you seem to be implying that Zend and the classical PHP intrepreter are the same which they are not.

I don't even know what you mean by "classical PHP intrepreter" - was that PHP 3? I am telling you that the first Zend Engine powered PHP 4 and it was tokenizer/interpreter implementation. But the thing was retired 9 years ago, why is this even relevant to this discussion? Current PHP 5 version is powered by Zend Engine 2.x with an opcode based virtual machine.

Your question still doesn't make sense. What exactly is a 'bytecode layer'? You have machines and you have input. Bytecode is input into a virtual machine.

Your PC is a machine (albeit not virtual). What input does it take? It's called machine code. A bytecode is the closest representation of a machine code without specific hardware dependencies (such as x86, x86_64 or ARM instruction set) - hence, portable machine code?

Edited May 15, 2013 by simast

[Insanatrix 0]

I would venture to say that it is interpreted rather than native/compiled.

Bytecode is still interpreted, it's just designed more efficiently.

As far as what everyone is talking about, Imagine that your script is written in French(SQF) and the game is compiled in English(Machine Code) and your machine only reads English. You would need something to interpret/translate the French to English so it can be processed by the Machine.

[gossamersolid 155]

SQF is interpreted, like a majority of scripting languages.

[Make Love Not War 10]

BTW: your SQUINT looks very interesting (will investigate it more in the next days). Do you plan to maintain it for arma 3 too?

I know this is a little OT, but an updated SQUINT for A3 would be awesome. Even in it's currently outdated form, it remains an absolutely invaluable tool. Many, many thanks for creating it, sbsmac!