What is the purpose of the special property of word?

[kusumotonorio]

I'm working on errors that a definition containing the primitive word mega-cache-lookup cannot be compiled.

The mega-cache-lookup is in stack-checker.known-words and the special property seems to be set to [ \ mega-cache-lookup do-not-compile ]. I suspect that my problem may have something to do with this.

{
    c-to-factor
    do-primitive
    mega-cache-lookup
    mega-cache-miss
    inline-cache-miss
    inline-cache-miss-tail
    lazy-jit-compile
    set-callstack
    set-datastack
    set-retainstack
    unwind-native-frames
} [ dup '[ _ do-not-compile ] "special" set-word-prop ] each

What is the purpose of the special property?
Why do the definitions of method not raise a compiler error when it contains this word, but my code that mimics it does?

[mrjbq7]

Hi @kusumotonorio, looking into this now... I remember vaguely how this works. Let me remind myself.

[mrjbq7]

I think that the way it works is that the word is marked no-compile, but that doesn't prevent calling it from the non-optimizing compiler...

IN: scratchpad DEFER: foo
IN: scratchpad \ foo t "no-compile" set-word-prop
IN: scratchpad : foo ( x -- y ) 1 + ;

IN: scratchpad 1 foo . 
2

IN: scratchpad 1 [ foo ] call( x -- y ) .
2

IN: scratchpad : bar ( x -- y ) [ foo ] call( x -- y ) ;
IN: scratchpad 1 bar .
2

That last definition compiles fine, and checks the quotation at runtime even though it's basically a literal, which probably introduces some overhead.

Anyway, this is a workaround to get the code working, but then we should do a pass at this to compile the mega-cache-lookup primitive if we can. There might be a reason (for example bootstrap where we build-generics and in finish-compilation-unit where we remake-generics) that it was done this way...

[mrjbq7]

You are touching some areas of Factor where assumptions were made a long time ago and we will help push on this with you. I'd love to get one of your multi-methods versions into the core when it's ready!

[kusumotonorio]

I would like to somehow bring back the results from the depths of Factor and complete the multi-method system. I would appreciate your help.

[kusumotonorio]

As a prank, when I remove the special property, instead of a do-not-compile error, it reports a different stack effect error. It is running, though.

\ mega-cache-lookup "special" remove-word-prop

resource:work/mm7/mm7-tests.factor: 117

Asset: smd-beats?

Stack effect declaration is wrong
inferred ( -- )
declared ( obj1 obj2 -- ? )

[timor]

In the commit message of 580cbe7 it says that this is because mega-cache-lookup has a non-static stack-effect. I suspect it is because it is used for all kinds of method calls, which have different stack effects.

[kusumotonorio]

So, does this mean that we need a mechanism for each generic word definition performed by the mega-cache-lookup call to have to bypass the check that it conforms to the effect declarations they have?

[mrjbq7]

I think you should call mega-cache-lookup via [ ... mega-cache-lookup ] call( -- ) syntax to get the functionality working and then let me work on making it compiled like our other primitives. I still need to figure out why the original author didn't want anyone else calling it.

[mrjbq7]

Or did I misunderstand where you are with multi-methods? Is this blocking functionality? Or is it a performance optimization?

[kusumotonorio]

The decision to use mega-cache-lookup is due to an optimization aimed at making multi-generic words run as fast as traditional generic words.

For example, in mm7-tests.factor, sd-beats? is a traditional generic word declared by GENERIC:.

IN: scratchpad \ sd-beats? def>> .
[
    {
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        M\ object sd-beats?
        sd-beats?/predicate-engine
        M\ object sd-beats?
    } 0 {
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
    } mega-cache-lookup
]
IN: scratchpad 

The stack effect of mega-cache-lookup is as follows.

PRIMITIVE: mega-cache-lookup ( methods index cache -- )

Since the defining quot contains three parameters, the effect of this quot is ( -- ), but it is actually ( x x -- x ). Because that is the effect of sd-beats?.

It is used in the same way in the multi-method I created.
smd-beats? is declared with MGENERIC:, but it has one specializer, so mega-cache-lookup is the actual behavior.

IN: scratchpad \ smd-beats? def>> .
[
    {
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        M\ object smd-beats?
        smd-beats?/predicate-engine
        M\ object smd-beats?
    } 0 {
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
        f
    } mega-cache-lookup
]

[timor]

Here are my 2 cents based on my understanding of the compiler internals, I hope that helps and is correct, please disregard otherwise.

Generally, the "special" property is used to bypass the regular behavior during the first step of SSA-Tree-IR (compiler frontend) generation. In the case of mega-cache-lookup, an error is raised because as far as I know, a quotation containing mega-cache-lookup may never make it to this stage.

For example, take the word nth:

\ nth def>>
[ ... mega-cache-lookup ]

I think this is the code which will be executed by the non-optimizing compiler when the word is executed. Since it is a primitive, it will be executed by the VM.

The definition quotation of nth will never make it to the optimizing compiler, because
if a #call to a generic word likenth is encountered in the backend optimizing compiler,
a call to one of the pic-def>> or pic-tail-def>> slots ends up in the code (Don't ask me how, though).
This seems to serve as an anchor point for the megamorphic cache in-place code modification magic in the VM (which I know next to nothing about).

Example:

: foo ( x x -- x ) nth ;
\ foo disassemble
---
00007f6774810720: 8905da68defe    mov [rip-0x1219726], eax
00007f6774810726: 8905d468defe    mov [rip-0x121972c], eax
00007f677481072c: 488d1d05000000  lea rbx, [rip+0x5]
00007f6774810733: e968f4c9ff      jmp 0x7f67744afba0 ([ \ nth ~array~ 0 ~array~ inline-cache-miss-tail ])
00007f6774810738: 0000            add [rax], al
00007f677481073a: 0000            add [rax], al
00007f677481073c: 0000            add [rax], al
00007f677481073e: 0000            add [rax], al

! If foo is actually used, the code is modified
0 V{ 42 } foo
---
42

\ foo disassemble
---
00007f6774810720: 8905da68defe    mov [rip-0x1219726], eax
00007f6774810726: 8905d468defe    mov [rip-0x121972c], eax
00007f677481072c: 488d1d05000000  lea rbx, [rip+0x5]
00007f6774810733: e948ff0200      jmp 0x7f6774840680 (nth)
00007f6774810738: 0000            add [rax], al
00007f677481073a: 0000            add [rax], al
00007f677481073c: 0000            add [rax], al
00007f677481073e: 0000            add [rax], al

It looks like that the "no-compile" property is actually provided for the purpose of preventing code reaching the compiler, which does the same as mega-cache-lookup's "special" property. In fact, from the commit history it looks like the "special" property was set on a number of primitive words before the "no-compile" behavior was implemented.

Nevertheless, it is an error if a quotation containing this word reaches the compiler, as all words which have this in their definition are handled in a special way.

[kusumotonorio]

Thanks, @timor .

Nevertheless, it is an error if a quotation containing this word reaches the compiler, as all words which have this in their definition are handled in a special way.

I have to find out why it reaches the compiler and what makes it different from traditional generic words. Am I not doing something that needs to be done, or am I going off on a tangent...?

[kusumotonorio]

I have found the solution!

I need to change optimize? in compiler.factor.

: optimize? ( word -- ? )
    {
        [ single-generic? ]
        [ multi-single-generic? ] ! <-- added
        [ primitive? ]
    } 1|| not ;

[mrjbq7]

Aha!!

…

On Wed, Jan 6, 2021 at 7:54 PM kusumotonorio ***@***.***> wrote: I have found the solution! I need to change the optimize? <https://docs.factorcode.org/content/word-optimize__que__,compiler.html> in compiler.factor <https://github.com/factor/factor/blob/master/basis/compiler/compiler.factor> . : optimize? ( word -- ? ) { [ single-generic? ] [ multi-single-generic? ] ! <-- added [ primitive? ] } 1|| not ; — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2414 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF5A53Q5KA54YADLZCAP3SYUWBFANCNFSM4VUVWQ7A> .

[kusumotonorio]

It is not directly related to the subject, but it seems to be deeply related to it indirectly, so please allow me to continue the story here.

I'm currently trying to implement call-next-multi-method, and I've gotten to the point where I'm hopeful that it might possibly work. However, I am again facing a problem.

call-next-multi-method is a syntax macro, and processing is done with the (call-next-multi-methd) word. The stack effect of this word actually depends on the generic word.

: (call-next-multi-method) ( method -- )
    dup next-multi-method-quot [ call ] [ no-next-multi-method ] ?if ;

\ (call-next-multi-method) t "no-compile" set-word-prop

SYNTAX: call-next-multi-method
   current-method get
   [ literalize suffix! \ (call-next-multi-method) suffix! ]
   [ not-in-a-multi-method-error ] if* ;

TUPLE: test-tuple1 ;
TUPLE: test-tuple2 < test-tuple1 ;
TUPLE: test-tuple3 < test-tuple2 ;

MGENERIC: next-method-test ( class -- who-am-i )

MM: next-method-test ( class: test-tuple1 -- who-am-i )
    drop "test-tuple1" ;

MM: next-method-test ( class: test-tuple2 -- who-am-i )
    call-next-multi-method ;

MM: next-method-test ( class: test-tuple3 -- who-am-i )
    drop "test-tuple3" ;


test-tuple2 new next-method-test .

I get an error that (call-next-multi-methd) cannot be compiled. So far I have not been able to find a solution to this problem as I did the other day.

[kusumotonorio]

I was able to get call-next-multi-method to work!

The key was to use define-transform to define a compiler transform for the optimizing compiler. I had noticed that (call-next-method) had been done that way, but I had assumed it was for speed optimization purposes.

: (call-next-multi-method) ( method -- )
    dup next-multi-method-quot [ call ] [ no-next-multi-method ] ?if ;

USE: stack-checker.transforms

\ (call-next-multi-method) [
    [ next-multi-method-quot ] [ '[ _ no-next-multi-method ] ] bi or
] 1 define-transform

 \ (call-next-multi-method) t "no-compile" set-word-prop

SYNTAX: call-next-multi-method
   current-method get
   [ literalize suffix! \ (call-next-multi-method) suffix! ]
   [ not-in-a-multi-method-error ] if* ;

To be honest, I'm still not sure what I meant by the code I wrote, but it seems to be working well.

IN: scratchpad 
USE: mm7

TUPLE: test-tuple1 ;
TUPLE: test-tuple2 < test-tuple1 ;
TUPLE: test-tuple3 < test-tuple2 ;

MGENERIC: next-method-test ( class -- who-i-am )

MM: next-method-test ( class: test-tuple1 -- who-i-am )
    drop "test-tuple1" ;

MM: next-method-test ( class: test-tuple2 -- who-i-am )
    "a subclass of " swap call-next-multi-method append ;

MM: next-method-test ( class: test-tuple3 -- who-i-am )
    "a subclass of " swap call-next-multi-method append ;

test-tuple1 new next-method-test .
test-tuple2 new next-method-test .
test-tuple3 new next-method-test .
"test-tuple1"
"a subclass of test-tuple1"
"a subclass of a subclass of test-tuple1"
IN: scratchpad 

[kusumotonorio]

As shown above, call-next-multi-method now works. However, the following error occurs when the vocabularies are loaded.

Asset: (call-next-multi-method)

Cannot apply “call” to a run-time computed value
macro call

If the definition containing call will actually be replaced, then it should be possible to rewrite it with some other word. I made the following change. It looks like it works.

: call-next-multi-method-quot ( quot -- )
    ! This is a word to avoid a compiler error.
    drop ;

: (call-next-multi-method) ( method -- )
    ! The content of this definition is actually replaced and never used.
    dup next-multi-method-quot [
        call-next-multi-method-quot  !  <---
    ] [ no-next-multi-method ] ?if ;

\ (call-next-multi-method) [
    [ next-multi-method-quot ] [ '[ _ no-next-multi-method ] ] bi or
] 1 define-transform

\ (call-next-multi-method) t "no-compile" set-word-prop

SYNTAX: call-next-multi-method
   current-method get
   [ literalize suffix! \ (call-next-multi-method) suffix! ]
   [ not-in-a-multi-method-error ] if* ;

But the (call-next-method) doesn't do anything like that. There must be a better way, but I don't know it at the moment.

[mrjbq7]

This is great @kusumotonorio!

When you get a moment, can you comment on your latest and greatest version and what kind of improvements it has over the current multi-methods approach?

[kusumotonorio]

Yes. There are a few things in the current code that I feel need to be fixed, and I'll introduce it again when I could fix them.

What I wanted to achieve when I started this project is working, but there are still a few things that need to be fixed.
I need to understand the forget protocol. Also, the more problematic part seems to be giving proper dependency information. This seems to be difficult to understand.